Database Support

From Count version 2.6+, the counter is able to store data in a database. It supports any of the Berkeley Db, GNU gdbm, sdbm and ndbm databases. I suggest you use Berkeley Db for robustness and reliability Note about Berkeley Db, the counter only supports version 1.x, 2.x or 3.x. Try compiling the counter with default settings first. Your system will be probed for Berkeley db 2.x/3.x/1.x first, then GNU gdbm, ndbbm and lastly If you don't have any database installed it will use the supplied sdbm database.

Databases

The counter maintains two separate databases, one is the Authentication database and the other is the Counter database. The content of the databases are key and value pairs. In the authentication database the key is the page URL and value is a ID for that URL (MD5 (RSA Data Security, Inc. MD5 Message-Digest Algorithm) of the URL). And in the counter database, the key is the ID of the URL and value is the hits:IP or just hits. The relationship between auth and counter database is shown in the figure below:

If a page can be accessed via different URL, the key for that pages in the auth database should be the same, otherwise the hits will be different if the pages are accessed differently.

Supporting databases

The counter programs supports any of the Berkeley db, GNU gdbm, sdbm and ndbm databases. At compile time, the type of databases is decided and compiled with. Please look at the following table to compare the features of the various databases. It is clear that you should use Berkeley db.

Feature	NDBM	SDBM	GDBM	Berkeyley DB
Often comes with Unix	Yes	No	No	No
Comes with Counter	No	Yes	No	No
Builds OK on Unix	N/A	Yes	Yes	Yes ^b
Code size	Varies^c	Small	Big	Big
Disk Usage	Varies^c	Small^c	Big	Ok^d
Speed	Varies	Slow^c	Ok	Fast^d
FTPable	No	Yes	Yes	Yes
Easy to build	N/A	Yes	Yes	Ok^e
Block size limits	4k	1k ^f	None	None ^e
Byte-order independent	No	No	No	Yes ^e
User-defined sort order	No	No	No	Yes ^e
Wildcard lookups	No	No	No	Yes ^e

Reference: Perl programming .., page?

The data is stored as key and value pair in the database. For authentication database, the key is the url of the page and vaue is a unique identifier ( MD5 (RSA Data Security, Inc. MD5 Message-Digest Algorithm) hash of the url) for the key. For the counter database, the key is the value from the authentication database (corresponding to the page url) and value is the hits:remote_ip.

Data storing algorithm

The Counter data storing algorithm is shown below:

Count data storing algorithm

Utilities for manipulating databases

The counter program comes with several utilities to manipulate the counter database.

dumpdb - dump the Count database. The key and values are printed separated by "=>" Please look at utils/dumpdb directory. Usage: dumpdb db_file
dumpdb.pl - Perl script to do the same.
editdb - comprehensive program for the administrators to manipulate the counter database. It's located in utils/editdb directory.
count_admin.pl- A Perl CGI program for users to manipulate their own counter values. Adminstrators can adminster the entire database. They adminstrators should configure the script allow users to manipulate their own entries only, but if needed certian users can be given more control.
rgbtxt2db - converts rgb.txt file a database for fast lookup. It is done automatically at install time. An example of rgb.txt file is:
```
255 250 250     snow
248 248 255     ghost white
248 248 255     GhostWhite
245 245 245     white smoke
245 245 245     WhiteSmoke
220 220 220     gainsboro
255 250 240     floral white
255 250 240     FloralWhite
253 245 230     old lace
253 245 230     OldLace
```
The program make the name of the color the key and the red, green, blue components the value. It also converts all the name to lower case. Please look at the Directory Structure for the location of rgbdb.

Authentication database
Authentication database is a database of URL WWW Homepage Access Counter uses. If strict_mode=yes in count.cfg file, then anyone wants to put a counter in a web page, the URL of the web page must be in this database. If the URL is not in the auth database, then the user will get a message like below:

Putting URL is authentication database is analogous to creating data files for the users in older versions of counter. It gives the administrators some control.

Authentication database can be manipulated by the program editdb or by the CGI program count_admin.pl.

Here's an example of using editdb, assuming you copied editdb somewhere in your path:



$ editdb /usr/local/etc/Counter/db/Authdb 
(Current database:Authdb)
Command> dump
<*> Locking database
 ---------------------
http://localhost/tc/=>d255dbabb12267dd9da769d529fc474e
http://localhost/=>c9db569cb388e160e4b86ca1ddff84d7
http://localhost/index.html.en=>c9db569cb388e160e4b86ca1ddff84d7
http://www.muquit.com/=>f97d3dcd5d7df510cea62bd222136aa1
http://localhost/tc/index.html=>d255dbabb12267dd9da769d529fc474e
http://127.0.0.1/tc/index.html=>d255dbabb12267dd9da769d529fc474e
http://localhost/tc/test.html=>5613f9e2ab0b292ca8a7b582f73e22bc
http://127.0.0.1/tc/test.html=>5613f9e2ab0b292ca8a7b582f73e22bc
 ---------------------
<*> Unocking database
<->Lock released.
(Current database:Authdb)
Command> edit www
<*> Locking database
<*> Unocking database
<->Lock released.

1 item matched.
   1. http://www.muquit.com/=>f97d3dcd5d7df510cea62bd222136aa1
Enter the number to edit, Q to quit: 1

Edit Key (http://www.muquit.com/): http://muquit.com/
Edit Value (f97d3dcd5d7df510cea62bd222136aa1): 
<*> Locking database
http://muquit.com/=>f97d3dcd5d7df510cea62bd222136aa1
* saved successfully!

<*> Unocking database
<->Lock released.
(Current database:Authdb)
Command> dump
<*> Locking database
 ---------------------
http://localhost/tc/=>d255dbabb12267dd9da769d529fc474e
http://localhost/=>c9db569cb388e160e4b86ca1ddff84d7
http://localhost/index.html.en=>c9db569cb388e160e4b86ca1ddff84d7
http://www.muquit.com/=>f97d3dcd5d7df510cea62bd222136aa1
http://localhost/tc/index.html=>d255dbabb12267dd9da769d529fc474e
http://127.0.0.1/tc/index.html=>d255dbabb12267dd9da769d529fc474e
http://localhost/tc/test.html=>5613f9e2ab0b292ca8a7b582f73e22bc
http://127.0.0.1/tc/test.html=>5613f9e2ab0b292ca8a7b582f73e22bc
http://muquit.com/=>f97d3dcd5d7df510cea62bd222136aa1
 ---------------------
<*> Unocking database
<->Lock released.
(Current database:Authdb)
Command> q

If you have lots of URLs to add, write them in a text file and load the file with editdb. The format of this file is:


    URL=>ID

Here, URL is the key and the ID is the value. Let put the following lines in a file called db.txt,


http://www.muquit.com/muquit/index.html=>
http://www.muquit.com/muquit/=>
http://muquit.com/muquit/=>
http://muquit.com/muquit/index.html=>
http://209.145.38.154/muquit/index.html=>
http://209.145.38.154/index.html=>

Note, the values are left empty intentionally. If the values are left empty, editdb will prompt for the value with a default. Below, we took only the first one as defualt (pressing Enter) and used this value for each of the other url, because all the urls are for the same page.
Now add the file to the auth database,


$ editdb -f db.txt foo.db
Key=http://www.muquit.com/muquit/index.html
Enter Value (3e38fe43ea71dfd2a671f245c15bbc9d): 
Key=http://www.muquit.com/muquit/
Enter Value (3f550961303f9bb899a896b5c45dc994): 3e38fe43ea71dfd2a671f245c15bbc9d
Key=http://muquit.com/muquit/
Enter Value (165931a7b0c687223484be632881eb22): 3e38fe43ea71dfd2a671f245c15bbc9d
Key=http://muquit.com/muquit/index.html
Enter Value (e925c24eef64045a921b32051f17bfed): 3e38fe43ea71dfd2a671f245c15bbc9d
Key=http://209.145.38.154/muquit/index.html
Enter Value (78509821e809b24260490776762cd05e): 3e38fe43ea71dfd2a671f245c15bbc9d
Key=http://209.145.38.154/index.html
Enter Value (32c723d328a0f49bae453169a544fcc9): 3e38fe43ea71dfd2a671f245c15bbc9d
<*> Locking database
Add:http://www.muquit.com/muquit/index.html=>3e38fe43ea71dfd2a671f245c15bbc9d
Add:http://www.muquit.com/muquit/=>3e38fe43ea71dfd2a671f245c15bbc9d
Add:http://muquit.com/muquit/=>3e38fe43ea71dfd2a671f245c15bbc9d
Add:http://muquit.com/muquit/index.html=>3e38fe43ea71dfd2a671f245c15bbc9d
Add:http://209.145.38.154/muquit/index.html=>3e38fe43ea71dfd2a671f245c15bbc9d
Add:http://209.145.38.154/index.html=>3e38fe43ea71dfd2a671f245c15bbc9d
<*> Unocking database
<->Lock released.

You can use dumpdb to dump the database.

Advantages of using databases

A single database for the whole site. No need to create a file for each page.
Very easy to call, can be called as simply as: <img src="/cgi-bin/Count.cgi"> Page will be figured out automatically.
Much less disk I/O, hence faster.
Easy to administer. Site admin can allow the users can edit their own hits using the supplied cgi program. Suitable for large sites.

Disadvantages of using databases

Solely depends on HTTP_REFERER environment variable. If the browsers stop supporting this variable, the counter will break. It's very easy to fake referer, but it's the life with www.
Counter will show different values if the host has aliases and accessed via aliases. For example, for the host www.muquit.com, the local users can access the host as www, so http://www/ will be another key in the database. However, for the practical purposes it is ok as most of the hosts in the Internet is accessed by only one name.
It's fixed now. (-mm, Sep-24-2000)