Benchmarking

There is ab and httpref e.g.

httperf --verbose --hog --timeout=25 --server=www.yourdomain.com --port=80 --uri /pagetotest.html --rate=1000 --send-buffer=2000 --recv-buffer=2000 --num-conns=500000 --num-calls=1

Rewrite Engine

Security

Turn down the Volume

On Debian we can reduce Apache's verbosity level by setting ServerTokens and ServerSignature. The reason why this makes sense is simple — when Apache delivers content or error pages, important information about the version and other details implemented on our system is broadcasted within the server header as well.

For example, the information we see when trying to access a page that does not exist (by simply changing the URL a bit) might be Apache/2.4.2 (Debian) mod_auth_passthrough/2.8 mod_log_bytes/3.2 Server at localhost Port 80.

This line within the server header exposes important version and variant information about our operating system and the Apache installation with all its modules and plugins — it indirectly exposes possible security holes which might exist to crackers, or at least make it easier for malicious attackers to gather information about our system.

To ensure that Apache does not broadcast this message to the whole world which then makes it harder for attackers to gain information, we can modify the afore mentioned directives:

sa@wks:~$ cat /etc/apache2/conf.d/security | grep -v \# | grep .
ServerTokens Prod
ServerSignature Off
TraceEnable Off
sa@wks:~$

As can be seen, we set ServerTokens Prod and ServerSignature Off which drastically brings down Apache's verbosity. What we also did is alter TraceEnable to read TraceEnable Off — this makes cross-site attacks a lot harder for attackers.

After we altered /etc/apache2/conf.d/security as can be seen above, a simple apache2ctl restart or apache2ctl graceful restarts/starts Apache and loads the new settings. apache2ctl however requires apache2-utils to be installed. If that package is not available, /etc/init.d/apache2 reload is equal to issuing apache2ctl graceful.

Analytics

Web analytics is the measurement, collection, analysis and reporting of Internet data for purposes of understanding and optimizing web products like for example this website/platform. There are two categories of web analytics software

• off-site and
• on-site web analytics

Off-site web analytics refers to web measurement and analysis irrespective of whether we own or maintain a website. It includes the measurement of a website's potential audience (opportunity), share of voice (visibility), and buzz (comments) that is happening on the Internet as a whole.

On-site web analytics measures a visitor's journey once on our website. This includes its drivers and conversions — for example, which landing pages encourage people to make a purchase.

On-site web analytics measures the performance of our website in a commercial context. This data is typically compared against key performance indicators for performance, and used to improve a web site or marketing campaign's audience response.

Historically, web analytics has referred to on-site visitor measurement. However in recent years this has blurred, mainly because vendors are producing tools that span both categories.

For my website/platform, I do on-site statistics/analytics using AWStats. How this piece of software works and how it is installed and set up is shown below.

AWStats

As mentioned, I do on-site statistics/analytics using AWStats and this is the subsection on my website/platform where we are going to take a look at it from a technical point of view — how to acquire the software, install it, and most importantly, set it up using all of its default features.

However, of course we do not stop there — after AWStats is up and running in its default setup, we are going to add a bunch of nifty plugins and do some tweaks in order to turn it into a fully blown analytics tools for this website/platform.

AWStats Intro

As there is a lot info on the Internet what AWStats is and how it work, I am not going to repeat that here. I am just providing a few pointers that might help the reader grasp things further down:

• One should know about terms used by marketing folks like for example landing pages, search engine ranking, search engine optimization, web analytics and last but not least, Internet marketing in general because that is mostly the case why AWStats (or any other analytics software for that matter) is put into place at all.
• From a more technical angle, one should at least have read the AWStats glossary and know what CGI (Common Gateway Interface) is.
• Another must-have-read document from my point of view is the AWStats FAQs page — it helps a lot if somebody is in the process of evaluating different software for doing site analytics/statistics.

Installation and Configuration of Core AWStats

We start with installing AWStats — I call it Core AWStats for the simple reason that later, we are going to install and configure additional plugins on top of the core package.

This might also be a good time to mention that the entire website/platform runs within an OpenVZ VE (Virtual Environment) i.e. an isolated container, containing nothing else but the website data and all the bits and pieces that work behind the curtains like for example AWStats.

 1  sa@wks:~$ whoami; hostname
 2  sa
 3  wks
 4  sa@wks:~$ ssh website
 5  sa@wks-ve10:~$ whoami; hostname
 6  sa
 7  wks-ve10
 8  sa@wks-ve10:~$ su
 9  Password:
10  wks-ve10:/home/sa# aptitude install awstats
11  Reading package lists... Done
12  Building dependency tree
13  Reading state information... Done
14  Reading extended state information
15  Initializing package states... Done
16  The following NEW packages will be installed:
17    awstats libnet-xwhois-perl{a}
18  0 packages upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
19  Need to get 0B/967kB of archives. After unpacking 5296kB will be used.
20  Do you want to continue? [Y/n/?] y
21  Writing extended state information... Done
22  Reading package fields... Done
23  Reading package status... Done
24  Retrieving bug reports... Done
25  Parsing Found/Fixed information... Done
26  Selecting previously deselected package awstats.
27  (Reading database... 16997 files and directories currently installed.)
28  Unpacking awstats (from.../awstats_6.9~dfsg-1_all.deb)...
29  Selecting previously deselected package libnet-xwhois-perl.
30  Unpacking libnet-xwhois-perl (from.../libnet-xwhois-perl_0.90-3_all.deb)...
31  Processing triggers for man-db...
32  Setting up awstats (6.9~dfsg-1)...
33  Setting up libnet-xwhois-perl (0.90-3)...
34
35
36  starting etckeeper post-install actions...
37  [master 28cb58e] committing changes in /etc after apt run
38   3 files changed, 1522 insertions(+), 0 deletions(-)
39   create mode 100644 awstats/awstats.conf
40   create mode 100644 awstats/awstats.conf.local
41   create mode 100644 cron.d/awstats
42
43  git repository housekeeping using git gc...
44  Counting objects: 1376, done.
45  Delta compression using 2 threads.
46  Compressing objects: 100% (783/783), done.
47  Writing objects: 100% (1376/1376), done.
48  Total 1376 (delta 254), reused 1373 (delta 253)
49  git gc finished successfully...
50
51  Reading package lists... Done
52  Building dependency tree
53  Reading state information... Done
54  Reading extended state information
55  Initializing package states... Done
56  Writing extended state information... Done
57
58  wks-ve10:/home/sa# type dpl; dpl awstat* | grep ii
59  dpl is aliased to `dpkg -l'
60  ii  awstats                         6.9~dfsg-1            powerful and featureful web server log analyzer

As I said, we start out at the HN (Hardware Node) until we use SSH (Secure Shell) to enter the VE (Virtual Environment) in line 4. There we install awstats simply by issuing line 10. Of course, by now one should have installed Apache too (I have installed apache2-mpm-worker).

What follows is the usual install procedure. However, one might not get the same output as I do from lines 36 to 49 simply because I installed and set up etckeeper.

With line 58, where I use dpl, an alias in my ~/.bashrc, we check again if the package awstats has been installed. It did install successfully as the ii indicates.

 61  wks-ve10:/home/sa# cp -a /usr/share/doc/awstats/examples/ /usr/local/awstats
 62  wks-ve10:/home/sa# cd /usr/local/awstats/
 63  wks-ve10:/usr/local/awstats# type la; la
 64  la is aliased to `ls -la'
 65  total 208
 66  drwxr-xr-x  5 root root   4096 2009-05-27 11:02 .
 67  drwxrwsr-x 11 root staff  4096 2009-05-27 11:04 ..
 68  -rw-r--r--  1 root root   1069 2009-03-03 17:21 apache.conf
 69  -rwxr-xr-x  1 root root  19256 2009-03-03 17:21 awstats_buildstaticpages.pl
 70  -rwxr-xr-x  1 root root  26216 2009-03-03 17:21 awstats_configure.pl
 71  -rwxr-xr-x  1 root root  13152 2009-03-03 17:21 awstats_exportlib.pl
 72  -rw-r--r--  1 root root  17612 2008-10-05 11:01 awstats.model.conf.gz
 73  -rwxr-xr-x  1 root root    289 2009-03-03 17:21 awstats-update
 74  -rw-r--r--  1 root root   1179 2009-03-03 17:21 awstats-update.8
 75  -rwxr-xr-x  1 root root   5504 2009-03-03 17:21 awstats_updateall.pl
 76  drwxr-xr-x  2 root root   4096 2009-05-27 11:02 css
 77  -rw-r--r--  1 root root   2562 2008-11-15 16:03 example.pm.gz
 78  drwxr-xr-x  2 root root   4096 2009-05-27 11:02 js
 79  -rwxr-xr-x  1 root root  30687 2009-03-03 17:21 logresolvemerge.pl
 80  -rwxr-xr-x  1 root root  27809 2009-03-03 17:21 maillogconvert.pl
 81  -rwxr-xr-x  1 root root    856 2009-03-03 17:21 redostats.sh
 82  -rwxr-xr-x  1 root root    594 2009-03-03 17:21 staticpages.sh
 83  -rwxr-xr-x  1 root root  10121 2008-04-27 16:47 urlaliasbuilder.pl
 84  drwxr-xr-x  2 root root   4096 2009-05-27 11:02 xslt
 85  wks-ve10:/usr/local/awstats# gunzip awstats.model.conf.gz
 86  wks-ve10:/usr/local/awstats# cp awstats_configure.pl{,_orig}
 87  wks-ve10:/usr/local/awstats# type pi; pi configure
 88  pi is aliased to `ls -la | grep'
 89  -rwxr-xr-x  1 root root  31513 2009-05-27 11:24 awstats_configure.pl
 90  -rwxr-xr-x  1 root root  31513 2009-05-27 11:30 awstats_configure.pl_orig
 91
 92
 93  [ here we use some editor to edit awstats_configure.pl... ]
 94
 95
 96  wks-ve10:/usr/local/awstats# grep -A16 Markus awstats_configure.pl
 97  # /----- Markus's changes start
 98  # $AWSTATS_PATH='';
 99  # $AWSTATS_ICON_PATH='/usr/local/awstats/wwwroot/icon';
100  # $AWSTATS_CSS_PATH='/usr/local/awstats/wwwroot/css';
101  # $AWSTATS_CLASSES_PATH='/usr/local/awstats/wwwroot/classes';
102  # $AWSTATS_CGI_PATH='/usr/local/awstats/wwwroot/cgi-bin';
103  # $AWSTATS_MODEL_CONFIG='/etc/awstats/awstats.model.conf';              # Used only when configure ran on linux
104  # $AWSTATS_DIRDATA_PATH='/var/lib/awstats';                             # Used only when configure ran on linux
105
106  $AWSTATS_PATH='/usr/share/awstats';
107  $AWSTATS_ICON_PATH='/usr/share/awstats/icon';
108  $AWSTATS_CSS_PATH='/usr/share/awstats/css';
109  $AWSTATS_CLASSES_PATH='/usr/share/awstats/lib';
110  $AWSTATS_CGI_PATH='/usr/lib/cgi-bin';
111  $AWSTATS_MODEL_CONFIG='/usr/local/share/awstats.model.conf';
112  $AWSTATS_DIRDATA_PATH='/var/lib/awstats';
113  # /----- changes end
114  wks-ve10:/usr/local/awstats# grep /wwwroot/cgi-bin awstats_configure.pl | wc -l
115  9
116  wks-ve10:/usr/local/awstats# sed -i s%/wwwroot/cgi-bin%% awstats_configure.pl
117  wks-ve10:/usr/local/awstats# grep wwwroot/cgi-bin awstats_configure.pl | wc -l
118  0

Lines 61 to 118 are all about preparing for configuring Awstats. At first we copy all the data into the right place — the one that the FHS (Filesystem Hierarchy Standard) specifies for local data, specific to a particular machine.

In line 85 we unzip awstats.model.conf.gz — a tiny fact that many overlook and thus get errors when they try running the configure script awstats_configure.pl later in the process.

In line 86 we make a backup copy of awstats_configure.pl, something which is considered good practice in general.

The awstats_configure.pl that ships with the Debian package has path settings which we need to alter. We therefore open an editor (any editor will do; I use GNU Emacs) and make the necessary changes. Those changes can be seen in lines 97 to 113.

Another thing I recommend changing is the CGI root directory which we do with line 116 — we simply delete the /wwwroot/cgi-bin portion of the path, which happens to be present 9 times as can be seen from line 115.

119  wks-ve10:/usr/local/awstats# ./awstats_configure.pl
120
121  ----- AWStats awstats_configure 1.0 (build 1.8) (c) Laurent Destailleur -----
122  This tool will help you to configure AWStats to analyze statistics for
123  one web server. You can try to use it to let it do all that is possible
124  in AWStats setup, however following the step by step manual setup
125  documentation (docs/index.html) is often a better idea. Above all if:
126  - You are not an administrator user,
127  - You want to analyze downloaded log files without web server,
128  - You want to analyze mail or ftp log files instead of web log files,
129  - You need to analyze load balanced servers log files,
130  - You want to 'understand' all possible ways to use AWStats...
131  Read the AWStats documentation (docs/index.html).
132
133  -----> Running OS detected: Linux, BSD or Unix
134
135  -----> Check for web server install
136
137  Enter full config file path of your Web server.
138  Example: /etc/httpd/httpd.conf
139  Example: /usr/local/apache2/conf/httpd.conf
140  Example: c:\Program files\apache group\apache\conf\httpd.conf
141  Config file path ('none' to skip web server setup):
142  > /etc/apache2/apache2.conf
143
144  -----> Check and complete web server config file '/etc/apache2/apache2.conf'
145    Add 'Alias /awstatsclasses "/usr/share/awstats/lib/"'
146    Add 'Alias /awstatscss "/usr/share/awstats/css/"'
147    Add 'Alias /awstatsicons "/usr/share/awstats/icon/"'
148    Add 'ScriptAlias /awstats/ "/usr/lib/cgi-bin/"'
149    Add '<Directory>' directive
150    AWStats directives added to Apache config file.
151
152  -----> Update model config file '/usr/local/awstats/awstats.model.conf'
153    File awstats.model.conf updated.
154
155  -----> Need to create a new config file ?
156  Do you want me to build a new AWStats config/profile
157  file (required if first install) [y/N] ? y
158
159  -----> Define config file name to create
160  What is the name of your web site or profile analysis ?
161  Example: www.mysite.com
162  Example: demo
163  Your web site, virtual server or profile name:
164  > www.markus-gattol.name
165
166  -----> Define config file path
167  In which directory do you plan to store your config file(s) ?
168  Default: /etc/awstats
169  Directory path to store config file(s) (Enter for default):
170  >
171
172  -----> Create config file '/etc/awstats/awstats.www.markus-gattol.name.conf'
173   Config file /etc/awstats/awstats.www.markus-gattol.name.conf created.
174
175  -----> Restart Web server with '/etc/init.d/apache restart'
176
177  -----> Add update process inside a scheduler
178  Sorry, configure.pl does not support automatic add to cron yet.
179  You can do it manually by adding the following command to your cron:
180  /usr/lib/cgi-bin/awstats.pl -update -config=www.markus-gattol.name
181  Or if you have several config files and prefer having only one command:
182  /usr/local/awstats/tools/awstats_updateall.pl now
183  Press ENTER to continue...
184
185
186  A SIMPLE config file has been created: /etc/awstats/awstats.www.markus-gattol.name.conf
187  You should have a look inside to check and change manually main parameters.
188  You can then manually update your statistics for 'www.markus-gattol.name' with command:
189  > perl awstats.pl -update -config=www.markus-gattol.name
190  You can also read your statistics for 'www.markus-gattol.name' with URL:
191  > http://localhost/awstats/awstats.pl?config=www.markus-gattol.name
192
193  Press ENTER to finish...

Lines 119 to 193 are where all the intense labor configuring AWStats happens. Thankfully to awstats_configure.pl, most of this labor gets hidden from us and we only have to answer a few questions that awstats_configure.pl asks us.

With line 142, we provide information about the location of the configuration file of our web server — which is Apache version 2 in our current case. Note lines 145 to 149, awstats_configure.pl uses the path settings we set above in lines 106 to 112. Also note line 152 and 153 and what I said above about line 85 — unzipping awstats.model.conf.gz that is.

Probably the most important step is with line 164 — we need to tell AWStats about the domain name which is, as we know, www.markus-gattol.name in our current case — note that we do not specify the protocol portion i.e. we did not specify but just the domain name. The reason why I am mentioning this explicitly is that it seems this is where many folks stumble when setting up AWStats.

Also important is what awstats_configure.pl tells us in line 173 — this is the configuration/settings file for www.markus-gattol.name called awstats.www.markus-gattol.name.conf. This file is where we change how AWStats works/behaves later on i.e. how many visitors does it show, Top 30 or maybe only Top 5, does it show the screen size of our visitors or not, what plugins do we enable/disable etc.

From line 177 to 183 awstats_configure.pl simply tells us it tried to set up a cron job but that, as of now (June 2009), it does not yet support setting up a cron job automatically. No problem, we do it ourselves further down anyway.

Last interesting thing to mention is line 189 — we are going to use this one-liner quite a bit further down in order to update AWStats statistics database (which actually is just a file on the filesystem).

Hooray!... We are done configuring AWStats! What follows are just a few more minor tweaks here and there to make things complete.

194  wks-ve10:/usr/local/awstats# chmod 644 /var/log/apache2/access.log
195
196
197  [ here we use some editor to edit /etc/awstats/awstats.www.markus-gattol.name.conf... ]
198
199
200  wks-ve10:/usr/local/awstats# grep ^LogFile /etc/awstats/awstats.www.markus-gattol.name.conf
201  LogFile="/var/log/apache2/access.log"

Next thing we need to do is set the file permissions on /var/log/apache2/access.log, the log file we tell AWStats to use (line 200). It must be readable by non-root users — as we will see later, AWStats will run as system user awstats.

202  wks-ve10:/usr/local/awstats# /usr/lib/cgi-bin/awstats.pl -update -config=www.markus-gattol.name
203  Create/Update database for config "/etc/awstats/awstats.www.markus-gattol.name.conf" by AWStats version 6.9 (build 1.925)
204  From data in log file "/var/log/apache2/access.log"...
205  Phase 1 : First bypass old records, searching new record...
206  Searching new records from beginning of log file...
207  Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
208  Jumped lines in file: 0
209  Parsed lines in file: 3164
210   Found 0 dropped records,
211   Found 0 corrupted records,
212   Found 0 old records,
213   Found 3164 new qualified records.

Remember what I said about line 180 above... now we use this command to update the AWStats database for the first time — thereby generating the database if issued for the first time.

In line 203 we can see that AWStats is using /var/log/apache2/access.log as its information source.

214  wks-ve10:/usr/local/awstats# cat /etc/cron.d/awstats
215  0,10,20,30,40,50 * * * * www-data [ -x /usr/lib/cgi-bin/awstats.pl -a -f /etc/awstats/awstats.conf -a -r /var/log/apache/access.log ] && /usr/lib/cgi-bin/awstats.pl -config=awstats -update >/dev/null
216  wks-ve10:/usr/local/awstats# [ -x /usr/lib/cgi-bin/awstats.pl -a -f /etc/awstats/awstats.conf -a -r /var/log/apache/access.log ] && echo true || echo false
217  false
218  wks-ve10:/usr/local/awstats# [ -x /usr/lib/cgi-bin/awstats.pl -a -f /etc/awstats/awstats.conf -a -r /var/log/apache2/access.log ] && echo true || echo false
219  true
220  wks-ve10:/usr/local/awstats# echo "wrong path i.e. must be apache2 instead of just apache"
221  wrong path i.e. must be apache2 instead of just apache
222  wks-ve10:/usr/local/awstats#
223
224
225  [ here we use some editor to edit /etc/cron.d/awstats... ]
226
227
228  wks-ve10:/usr/local/awstats# cat /etc/cron.d/awstats
229  */20 * * * * root [ -x /usr/lib/cgi-bin/awstats.pl -a -f /etc/awstats/awstats.conf -a -r /var/log/apache2/access.log ] && /usr/lib/cgi-bin/awstats.pl -config=www.markus-gattol.name -update >/dev/null
230  wks-ve10:/usr/local/awstats# date; tail -n3 /var/log/syslog
231  Wed May 27 12:14:51 TC 2009
232  May 27 12:14:01 wks-ve10 /USR/SBIN/CRON[10265]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
233  May 27 12:14:48 wks-ve10 /usr/sbin/cron[6369]: (*system*awstats) RELOAD (/etc/cron.d/awstats)
234  May 27 12:20:01 wks-ve10 /USR/SBIN/CRON[10300]: (root) CMD ([ -x /usr/lib/cgi-bin/awstats.pl -a -f /etc/awstats/awstats.conf -a -r /var/log/apache2/access.log ] && /usr/lib/cgi-bin/awstats.pl -config=www.markus-gattol.name -update >/dev/null)

Of course we do not want to issue line 202 manually all the time but then it would be nice to have up-to-date analytics data available at all times right?!

A solution to this problem is setting up a cron job, something we know, awstats_configure.pl is not able to do automatically so far... that it tells us above (lines 178 to 182) but at the same time it puts /etc/cron.d/awstats into place. An example of code and docu not at the same version that is ;-]

Anyway, good for us because it means less work. Line 215 shows the cron job awstats_configure.pl put into place. There are three things we are going to change about it:

• fix the bug with the path setting
• the intervall (elevate it to run every 20 minutes instead of every 10 minutes) and last but not least,
• the executing user will be root instead of www-data

Line 229 shows the cron job as we want it. Line 233 shows that crond automatically monitors changes and updates himself if changes to cron jobs (i.e. files containing them) happen (see cron article for more information on that matter).

And yes, that means we do not have to restart crond — that is one of many myths/false believes amongst novice users. Many think one would need to issue /etc/init.d/crond restart after a change to some cron job is made... which is not necessary!

The following shows a full listing of /etc/awstats/awstats.www.markus-gattol.name.conf after I walked trough it for the first time, making some minor tweaks here and there just to make AWStats work fine in its default setup — no plugins, no ExtraSections, no special tweaks etc.... just core AWStats.

This however will not be the final version — after we installed a few third party plugins (stuff not shipped with Debian's .deb for AWStats package) we will revisit /etc/awstats/awstats.www.markus-gattol.name.conf and make a second/final walk trough it, changing things as wanted/needed.

235  wks-ve10:/usr/local/awstats#
236
237
238  [ here we use some editor to edit /etc/awstats/awstats.www.markus-gattol.name.conf... enable plugins, review default settings  etc.... ]
239
240
241  wks-ve10:/usr/local/awstats# cat /etc/awstats/awstats.www.markus-gattol.name.conf | grep -v \# | grep .
242  LogFile="/var/log/apache2/access.log"
243  LogType=W
244  LogFormat=1
245  LogSeparator=" "
246  SiteDomain="www.markus-gattol.name"
247  HostAliases="www.markus-gattol.name www.www.markus-gattol.name 127.0.0.1 localhost 10.0.3.4"
248  DNSLookup=2
249  DirData="/var/lib/awstats"
250  DirCgi="/awstats"
251  DirIcons="/awstatsicons"
252  AllowToUpdateStatsFromBrowser=0
253  AllowFullYearView=2
254  EnableLockForUpdate=0
255  DNSStaticCacheFile="dnscache.txt"
256  DNSLastUpdateCacheFile="dnscachelastupdate.txt"
257  SkipDNSLookupFor=""
258  AllowAccessFromWebToAuthenticatedUsersOnly=0
259  AllowAccessFromWebToFollowingAuthenticatedUsers=""
260  AllowAccessFromWebToFollowingIPAddresses=""
261  CreateDirDataIfNotExists=0
262  BuildHistoryFormat=text
263  BuildReportFormat=html
264  SaveDatabaseFilesWithPermissionsForEveryone=0
265  PurgeLogFile=0
266  ArchiveLogRecords=0
267  KeepBackupOfHistoricFiles=1
268  DefaultFile="home.html"
269  SkipHosts=""
270  SkipUserAgents=""
271  SkipFiles=""
272  SkipReferrersBlackList=""
273  OnlyHosts=""
274  OnlyUserAgents=""
275  OnlyUsers=""
276  OnlyFiles=""
277  NotPageList="css js class gif jpg jpeg png bmp ico rss xml swf"
278  ValidHTTPCodes="200 304"
279  ValidSMTPCodes="1 250"
280  AuthenticatedUsersNotCaseSensitive=0
281  URLNotCaseSensitive=0
282  URLWithAnchor=0
283  URLQuerySeparators="?;"
284  URLWithQuery=0
285  URLWithQueryWithOnlyFollowingParameters=""
286  URLWithQueryWithoutFollowingParameters=""
287  URLReferrerWithQuery=0
288  WarningMessages=1
289  ErrorMessages=""
290  DebugMessages=0
291  NbOfLinesForCorruptedLog=50
292  WrapperScript=""
293  DecodeUA=0
294  MiscTrackerUrl="/js/awstats_misc_tracker.js"
295  UseFramesWhenCGI=1
296  DetailedReportsOnNewWindows=1
297  Expires=0
298  MaxRowsInHTMLOutput=3000
299  Lang="auto"
300  DirLang="./lang"
301  ShowMenu=1
302  ShowSummary=UVPHB
303  ShowMonthStats=UVPHB
304  ShowDaysOfMonthStats=VPHB
305  ShowDaysOfWeekStats=PHB
306  ShowHoursStats=PHB
307  ShowDomainsStats=PHB
308  ShowHostsStats=PHBL
309  ShowAuthenticatedUsers=0
310  ShowRobotsStats=HBL
311  ShowWormsStats=0
312  ShowEMailSenders=0
313  ShowEMailReceivers=0
314  ShowSessionsStats=1
315  ShowPagesStats=PBEX
316  ShowFileTypesStats=HB
317  ShowFileSizesStats=0
318  ShowOSStats=1
319  ShowBrowsersStats=1
320  ShowScreenSizeStats=0
321  ShowOriginStats=PH
322  ShowKeyphrasesStats=1
323  ShowKeywordsStats=1
324  ShowMiscStats=a
325  ShowHTTPErrorsStats=1
326  ShowSMTPErrorsStats=0
327  ShowClusterStats=0
328  AddDataArrayMonthStats=1
329  AddDataArrayShowDaysOfMonthStats=1
330  AddDataArrayShowDaysOfWeekStats=1
331  AddDataArrayShowHoursStats=1
332  IncludeInternalLinksInOriginSection=0
333  MaxNbOfDomain = 10
334  MinHitDomain  = 1
335  MaxNbOfHostsShown = 10
336  MinHitHost    = 1
337  MaxNbOfLoginShown = 10
338  MinHitLogin   = 1
339  MaxNbOfRobotShown = 10
340  MinHitRobot   = 1
341  MaxNbOfPageShown = 10
342  MinHitFile    = 1
343  MaxNbOfOsShown = 10
344  MinHitOs      = 1
345  MaxNbOfBrowsersShown = 10
346  MinHitBrowser = 1
347  MaxNbOfScreenSizesShown = 5
348  MinHitScreenSize = 1
349  MaxNbOfWindowSizesShown = 5
350  MinHitWindowSize = 1
351  MaxNbOfRefererShown = 10
352  MinHitRefer   = 1
353  MaxNbOfKeyphrasesShown = 10
354  MinHitKeyphrase = 1
355  MaxNbOfKeywordsShown = 10
356  MinHitKeyword = 1
357  MaxNbOfEMailsShown = 20
358  MinHitEMail   = 1
359  FirstDayOfWeek=1
360  ShowFlagLinks=""
361  ShowLinksOnUrl=1
362  UseHTTPSLinkForUrl=""
363  MaxLengthOfShownURL=64
364  HTMLHeadSection=""
365  HTMLEndSection=""
366  Logo="awstats_logo6.png"
367  LogoLink="http://awstats.sourceforge.net"
368  BarWidth   = 260
369  BarHeight  = 90
370  StyleSheet=""
371  LoadPlugin="tooltips"
372  LoadPlugin="hashfiles"
373  LoadPlugin="userinfo"
374  LoadPlugin="hostinfo"
375  ExtraTrackedRowsLimit=500

376  wks-ve10:/usr/local/awstats# grep -A99 AWStats /etc/apache2/apache2.conf
377  # Directives to allow use of AWStats as a CGI
378  #
379  Alias /awstatsclasses "/usr/share/awstats/lib/"
380  Alias /awstatscss "/usr/share/awstats/css/"
381  Alias /awstatsicons "/usr/share/awstats/icon/"
382  ScriptAlias /awstats/ "/usr/lib/cgi-bin/"
383
384  #
385  # This is to permit URL access to scripts/files in AWStats directory.
386  #
387  <Directory "/usr/local/awstats">
388      Options None
389      AllowOverride None
390      Order allow,deny
391      Allow from all
392  </Directory>
393

Lines 376 to 392 show the tail of Apache's config file after awstats_configure.pl applied its changes (lines 145 to 149) plus we altered line 387 to read <Directory "/usr/local/awstats"> instead of <Directory "/usr/local/awstats/wwwroot">.

394  wks-ve10:/usr/local/awstats# la /var/lib/awstats/
395  total 32
396  drwxr-x---  2 www-data www-data  4096 2009-05-27 12:00 .
397  drwxr-xr-x 24 root     root      4096 2009-05-27 11:02 ..
398  -rw-r--r--  1 root     root     10879 2009-05-27 11:52 awstats052009.www.markus-gattol.name.bak
399  -rw-r--r--  1 root     root     10873 2009-05-27 12:00 awstats052009.www.markus-gattol.name.txt
400  wks-ve10:/usr/local/awstats# type dpl; dpl apache2* | grep ^ii
401  dpl is aliased to `dpkg -l'
402  ii  apache2-mpm-worker              2.2.11-3              Apache HTTP Server - high speed threaded mod
403  ii  apache2-utils                   2.2.11-3              utility programs for web servers
404  ii  apache2.2-common                2.2.11-3              Apache HTTP Server common files
405  wks-ve10:/usr/local/awstats# type psa; psa apache
406  psa is aliased to `ps aux | grep'
407  root     10393  0.0  0.0   7264   788 pts/1    S+   03:38   0:00 grep apache
408  wks-ve10:/usr/local/awstats# apache2ctl start
409  apache2: Could not reliably determine the server's fully qualified domain name, using 10.0.3.4 for ServerName
410  wks-ve10:/usr/local/awstats# psa apache
411  root     10400  0.0  0.2  72468  2704 ?        Ss   03:38   0:00 /usr/sbin/apache2 -k start
412  www-data 10401  0.0  0.1  72200  1804 ?        S    03:38   0:00 /usr/sbin/apache2 -k start
413  www-data 10402  0.0  0.2 295916  2464 ?        Sl   03:38   0:00 /usr/sbin/apache2 -k start
414  www-data 10403  0.0  0.2 295916  2468 ?        Sl   03:38   0:00 /usr/sbin/apache2 -k start
415  root     10461  0.0  0.0   7264   784 pts/1    S+   03:39   0:00 grep apache
416
417
418  [ here we use some editor to edit /etc/logrotate.d/apache2... ]
419
420
421  wks-ve10:/usr/local/awstats# cat /etc/logrotate.d/apache2
422  /var/log/apache2/*.log {
423          weekly
424          missingok
425          rotate 52
426          compress
427          delaycompress
428          notifempty
429          create 640 root adm
430          sharedscripts
431          prerotate
432          /usr/lib/cgi-bin/awstats.pl -update -config=www.markus-gattol.name
433          endscript
434          postrotate
435                  if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
436                          /etc/init.d/apache2 reload > /dev/null
437                  fi
438          endscript
439  }
440  wks-ve10:/usr/local/awstats#

We are almost done with Awstats default configuration. What we see in lines 398 and 399 respectively is the afore mentioned database file (and its backup) where Awstats keeps all its data and where it writes the information gathered from /var/log/apache2/access.log.

Lines 401 to 404 just show the actual packages (related to Apache) which I have installed within this VE (Virtual Environment). Then, because we figure Apache is not running (line 407; I used apache2ctl stop earlier), we start it in line 408 — note that by installing Apache as .deb, Apache starts automatically when the computer (or VE for that matter) boots.

Last but not least, with adding lines 431 to 438 to /etc/logrotate.d/apache2 we avoid data loss in the event that Apache rotates its logs before every bit of it has been analyzed by Awstats and written to its database.

Plugins and Additional Tweaks

Some plugins are more or less just switches that we toggle from off to on or vice versa. Others however need a bit more attention, meaning that we might have to install additional software, edit files and do other things like that.

Tooltips

That is an easy one, just uncommenting LoadPlugin="tooltips" does the trick.

Enabling UTF8 Decoding

In order to make this one work, we need to install another package (line 441), and also uncomment it (line 485).

441  wks-ve10:/usr/local/awstats# aptitude install liburi-perl
442  Reading package lists... Done
443  Building dependency tree
444  Reading state information... Done
445  Reading extended state information
446  Initializing package states... Done
447  The following NEW packages will be installed:
448    liburi-perl
449  0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
450  Need to get 88.3kB of archives. After unpacking 381kB will be used.
451  Writing extended state information... Done
452  Get:1 ftp://ftp2.de.debian.org testing/main liburi-perl 1.37+dfsg-1 [88.3kB]
453  Fetched 88.3kB in 0s (170kB/s)
454
455
456  starting etckeeper pre-install actions...
457  [master abea2f3] saving uncommitted changes in /etc prior to apt run
458   1 files changed, 1 insertions(+), 1 deletions(-)
459  Reading package fields... Done
460  Reading package status... Done
461  Retrieving bug reports... Done
462  Parsing Found/Fixed information... Done
463  Selecting previously deselected package liburi-perl.
464  (Reading database... 20129 files and directories currently installed.)
465  Unpacking liburi-perl (from.../liburi-perl_1.37+dfsg-1_all.deb)...
466  Processing triggers for man-db...
467  Setting up liburi-perl (1.37+dfsg-1)...
468
469  git repository housekeeping using git gc...
470  Counting objects: 1433, done.
471  Delta compression using up to 2 threads.
472  Compressing objects: 100% (797/797), done.
473  Writing objects: 100% (1433/1433), done.
474  Total 1433 (delta 292), reused 1427 (delta 289)
475  git gc finished successfully...
476
477  Reading package lists... Done
478  Building dependency tree
479  Reading state information... Done
480  Reading extended state information
481  Initializing package states... Done
482  Writing extended state information... Done
483
484  wks-ve10:/usr/local/awstats# grep decodeutfkeys /etc/awstats/awstats.www.markus-gattol.name.conf
485  LoadPlugin="decodeutfkeys"

GeoIP

Those are plugins that enable AWStats to map IP addresses to locations i.e. it tells us the country a visitor came from, what city and sometimes, if it can be computed, the company he makes the visit from.

There is one base plugin called GeoIP Country which provides us with location information at country level. If we want more fine-grained location information i.e. maybe not just the country like for example that the visitor came from Austria, but maybe even the city within this country, like for example Vienna, then we need to install and configure the GeoIP City plugin on top of the GeoIP Country plugin.

The fun does not stop there. If we want even more detail i.e. if we not just want the country and city information regarding a visitor but also the organization the visitor came from, then we go ahead and install the GeoIP Org/AS plugin.

Below I am going to show how to install and configure all three of this plugins, starting with the base plugin GeoIP Country.

GeoIP Country

As mentioned above, we need to setup the GeoIP Country plugin at first.

486  wks-ve10:/usr/local/awstats# aptitude install build-essential libgeoip1 libgeoip-dev
487  Reading package lists... Done
488  Building dependency tree
489  Reading state information... Done
490  Reading extended state information
491  Initializing package states... Done
492  The following NEW packages will be installed:
493    binutils{a} build-essential bzip2{a} cpp{a} cpp-4.3{a} dpkg-dev{a} g++{a} g++-4.3{a} gcc{a} gcc-4.3{a} geoip-database{a} libc6-dev{a} libgeoip-dev
494    libgeoip1 libgmp3c2{a} libmpfr1ldbl{a} libstdc++6-4.3-dev{a} libtimedate-perl{a} linux-libc-dev{a}
495  0 packages upgraded, 19 newly installed, 0 to remove and 0 not upgraded.
496  Need to get 20.8MB/20.8MB of archives. After unpacking 64.4MB will be used.
497  Do you want to continue? [Y/n/?] y
498  Writing extended state information... Done
499  Reading package fields... Done
500  Reading package status... Done
501  Retrieving bug reports... Done
502  Parsing Found/Fixed information... Done
503  Selecting previously deselected package binutils.
504  (Reading database... 17743 files and directories currently installed.)
505  Unpacking binutils (from.../binutils_2.19.1-1_amd64.deb)...
506
507
508  [skipping a lot of lines...]
509
510
511  Initializing package states... Done
512  Writing extended state information... Done
513
514  wks-ve10:/usr/local/awstats# cd /tmp
515  wks-ve10:/tmp# wget http://geolite.maxmind.com/download/geoip/api/perl/Geo-IP-1.38.tar.gz
516  --2009-05-27 16:59:52--  http://geolite.maxmind.com/download/geoip/api/perl/Geo-IP-1.38.tar.gz
517  Resolving geolite.maxmind.com... 64.246.48.99
518  Connecting to geolite.maxmind.com|64.246.48.99|:80... connected.
519  HTTP request sent, awaiting response... 200 OK
520  Length: 63418 (62K) [application/x-gzip]
521  Saving to: `Geo-IP-1.38.tar.gz'
522
523  100%[==================================================================================================================>] 63,418       111K/s   in 0.6s
524
525  2009-05-27 16:59:53 (111 KB/s) - `Geo-IP-1.38.tar.gz' saved [63418/63418]
526
527  wks-ve10:/tmp# tar -xzf Geo-IP-1.38.tar.gz

Because we need to do a bit of compiling, we install a few additional packages as can be seen in line 486. Then, in lines 515 to 526, we actually download the plugin and extract the tarball as can be seen in line 527.

528  wks-ve10:/tmp# cd Geo-IP-1.38/
529  wks-ve10:/tmp/Geo-IP-1.38# perl Makefile.PL
530  Checking if your kit is complete...
531  Looks good
532  Writing Makefile for Geo::IP
533  wks-ve10:/tmp/Geo-IP-1.38# make
534  cp lib/Geo/Mirror.pm blib/lib/Geo/Mirror.pm
535  cp lib/Geo/IP/Record.pod blib/lib/Geo/IP/Record.pod
536  cp lib/Geo/IP/Record.pm blib/lib/Geo/IP/Record.pm
537  cp lib/Geo/IP.pm blib/lib/Geo/IP.pm
538  /usr/bin/perl /usr/share/perl/5.10/ExtUtils/xsubpp  -typemap /usr/share/perl/5.10/ExtUtils/typemap -typemap typemap  IP.xs > IP.xsc && mv IP.xsc IP.c
539  cc -c   -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g   -DVERSION=\"1.38\" -DXS_VERSION=\"1.38\" -fPIC "-I/usr/lib/perl/5.10/CORE"  -DPERL_EXT IP.c
540  Running Mkbootstrap for Geo::IP ()
541  chmod 644 IP.bs
542  rm -f blib/arch/auto/Geo/IP/IP.so
543  cc  -shared -O2 -g -L/usr/local/lib IP.o  -o blib/arch/auto/Geo/IP/IP.so        \
544             -lGeoIP      \
545

The compilation is done in a few steps — lines 529, 533, 552 and 560 show it step by step.

546  chmod 755 blib/arch/auto/Geo/IP/IP.so
547  cp IP.bs blib/arch/auto/Geo/IP/IP.bs
548  chmod 644 blib/arch/auto/Geo/IP/IP.bs
549  Manifying blib/man3/Geo::Mirror.3pm
550  Manifying blib/man3/Geo::IP::Record.3pm
551  Manifying blib/man3/Geo::IP.3pm
552  wks-ve10:/tmp/Geo-IP-1.38# make test
553  PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
554  t/0_base..........ok
555  t/1_lookup........ok
556  t/2_namelookup....ok
557  t/3_mirror........ok
558  All tests successful.
559  Files=4, Tests=33,  0 wallclock secs ( 0.10 cusr +  0.04 csys =  0.14 CPU)
560  wks-ve10:/tmp/Geo-IP-1.38# make install
561  Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
562  Installing /usr/local/lib/perl/5.10.0/auto/Geo/IP/IP.bs
563  Installing /usr/local/lib/perl/5.10.0/auto/Geo/IP/IP.so
564  Installing /usr/local/lib/perl/5.10.0/Geo/IP.pm
565  Installing /usr/local/lib/perl/5.10.0/Geo/Mirror.pm
566  Installing /usr/local/lib/perl/5.10.0/Geo/IP/Record.pod
567  Installing /usr/local/lib/perl/5.10.0/Geo/IP/Record.pm
568  Installing /usr/local/man/man3/Geo::Mirror.3pm
569  Installing /usr/local/man/man3/Geo::IP.3pm
570  Installing /usr/local/man/man3/Geo::IP::Record.3pm
571  Writing /usr/local/lib/perl/5.10.0/auto/Geo/IP/.packlist
572  Appending installation info to /usr/local/lib/perl/5.10.0/perllocal.pod
573  wks-ve10:/tmp/Geo-IP-1.38# la
574  total 400
575  drwxr-xr-x 6  501 root   4096 2009-05-27 17:00 .
576  drwxrwxrwt 5 root root   4096 2009-05-27 17:00 ..
577  drwxr-xr-x 8 root root   4096 2009-05-27 17:00 blib
578  -rw-r--r-- 1  501 root   8775 2009-05-17 22:20 Changes
579  drwxr-xr-x 2  501 root   4096 2009-05-17 22:24 example
580  -rw-r--r-- 1  501 root    556 2009-05-17 14:05 INSTALL
581  -rw-r--r-- 1 root root      0 2009-05-27 17:00 IP.bs
582  -rw-r--r-- 1 root root  36446 2009-05-27 17:00 IP.c
583  -rw-r--r-- 1 root root 268064 2009-05-27 17:00 IP.o
584  -rw-r--r-- 1  501 root   6412 2009-05-17 14:05 IP.xs
585  drwxr-xr-x 3  501 root   4096 2009-05-17 22:24 lib
586  -rw-r--r-- 1 root root  26844 2009-05-27 17:00 Makefile
587  -rw-r--r-- 1  501 root   3297 2009-05-17 14:05 Makefile.PL
588  -rw-r--r-- 1  501 root    353 2009-05-17 14:05 MANIFEST
589  -rw-r--r-- 1  501 root    464 2009-05-17 22:24 META.yml
590  -rw-r--r-- 1 root root      0 2009-05-27 17:00 pm_to_blib
591  -rw-r--r-- 1  501 root   1069 2009-05-17 14:05 README
592  drwxr-xr-x 2  501 root   4096 2009-05-17 22:24 t
593  -rw-r--r-- 1  501 root    965 2009-05-17 14:05 typemap
594  wks-ve10:/tmp/Geo-IP-1.38# locate GeoIP.dat
595  /usr/share/GeoIP/GeoIP.dat
596  wks-ve10:/tmp/Geo-IP-1.38# cd /usr/local/awstats/
597
598
599  [ here we use some editor to edit /etc/awstats/awstats.www.markus-gattol.name.conf... ]
600
601
602  wks-ve10:/usr/local/awstats# grep GeoIP.dat /etc/awstats/awstats.www.markus-gattol.name.conf
603  LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat"

The result of all the compilation steps can be seen in line 595. Now we need to edit /etc/awstats/awstats.www.markus-gattol.name.conf again and make it so that we end up with line 603.

Things are working immediately i.e. assuming we have Apache up and running and followed this article step by step until now, we may only need to issue /usr/lib/cgi-bin/awstats.pl -config=www.markus-gattol.name -update in case the cron job did not run in the meantime and the results would show when we visit the URL /site_statistics/awstats.pl?config=www.markus-gattol.name.

GeoIP City

This one depends on GeoIP Country. Installing and configuring it is already a lot simpler as taking all the necessary hurdles in case of the GeoIP Country plugin.

604  wks-ve10:/usr/local/awstats# cd /usr/share/
605  wks-ve10:/usr/share# pi Geo
606  drwxr-xr-x   2 root root  4096 2009-05-27 16:48 GeoIP
607  wks-ve10:/usr/share# mkdir GeoIP_city
608  wks-ve10:/usr/share# cd GeoIP_city/
609  wks-ve10:/usr/share/GeoIP_city# wget -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
610  wks-ve10:/usr/share/GeoIP_city# gunzip GeoLiteCity.dat.gz
611  wks-ve10:/usr/share/GeoIP_city# la
612  total 42164
613  drwxr-xr-x  2 root root     4096 2009-05-27 17:49 .
614  drwxr-xr-x 91 root root     4096 2009-05-27 17:48 ..
615  -rw-r--r--  1 root root 43115074 2009-05-04 00:10 GeoLiteCity.dat
616  wks-ve10:/usr/share/GeoIP_city# grep GeoLiteCity.dat /etc/awstats/awstats.www.markus-gattol.name.conf
617  LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /usr/share/GeoIP_city/GeoLiteCity.dat"
618  wks-ve10:/usr/share/GeoIP_city# cd ..

There are basically just two things we do — downloading the GeoIP City plugin in line 609 and once again altering /etc/awstats/awstats.www.markus-gattol.name.conf to end up with line 617. As for the GeoIP base plugin, this works immediately.

GeoIP Org/AS

The last GeoIP plugin is set up in the same way as the former one. The screendump speaks for itself so no need for me to explain.

619  wks-ve10:/usr/share# mkdir GeoIP_asnum
620  wks-ve10:/usr/share# cd GeoIP_asnum/
621  wks-ve10:/usr/share/GeoIP_asnum# wget -q http://geolite.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
622  wks-ve10:/usr/share/GeoIP_asnum# gunzip GeoIPASNum.dat.gz
623  wks-ve10:/usr/share/GeoIP_asnum# ll
624  total 2.2M
625  -rw-r--r-- 1 root root 2.2M 2009-03-22 00:48 GeoIPASNum.dat
626  wks-ve10:/usr/share/GeoIP_asnum# grep GeoIPASNum.dat /etc/awstats/awstats.www.markus-gattol.name.conf
627  LoadPlugin="geoip_org_maxmind GEOIP_STANDARD /usr/share/GeoIP_city/GeoIPASNum.dat"
628  wks-ve10:/usr/share/GeoIP_asnum#

Finally, we have enabled a bunch of additional features on top of core AWStats (doing so with the help of plugins) which suit my needs perfectly for additional location information with my site statistics/analytics perfectly fine.

629  wks-ve10:/usr/share/GeoIP_asnum# cd /usr/local/awstats/
630  wks-ve10:/usr/local/awstats# grep ^LoadPlugin /etc/awstats/awstats.www.markus-gattol.name.conf
631  LoadPlugin="tooltips"
632  LoadPlugin="decodeutfkeys"
633  LoadPlugin="hashfiles"
634  LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat"
635  LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /usr/share/GeoIP_city/GeoLiteCity.dat"
636  LoadPlugin="geoip_org_maxmind GEOIP_STANDARD /usr/share/GeoIP_city/GeoIPASNum.dat"
637  LoadPlugin="userinfo"
638  LoadPlugin="hostinfo"

What can be done further is tweaking the configuration a bit further here and there as I will show below.

Show Domains/Country Chart

Per default the country table does not show unique visitors and number of visits per country. Of course we want to enable that nifty feature:

639  wks-ve10:/usr/local/awstats# grep ^ShowDomainsStats /etc/awstats/awstats.www.markus-gattol.name.conf
640  ShowDomainsStats=UVPHB

Screen Size and other Goodies

One of the coolest things about AWStats is the leap forward we can make by bringing a bit of javascript magic to the table.

Imagine how cool it would be to know and visualize things like for example the screen size of our visitors and sort it based on how often a particular screen size is seen.

Or, imagine how cool it would be to know and visualize if somebody has javascript enabled, java support enabled, what color depth he has, if the macromedia director plugin is installed, if the macromedia shockwave plugin is installed, if he has the realplayer g2 plugin installed, if the quicktime plugin is installed, if the mediaplayer plugin is installed or last but not least, if he had the acrobat PDF plugin installed. Guess what, that we can do! Here is how:

641  wks-ve10:/usr/local/awstats# locate awstats_misc_tracker
642  /usr/local/awstats/js/awstats_misc_tracker.js.gz
643  /usr/share/doc/awstats/examples/js/awstats_misc_tracker.js.gz
644  wks-ve10:/usr/local/awstats# cd js/
645  wks-ve10:/usr/local/awstats/js# la
646  total 8
647  drwxr-xr-x 2 root root   39 2009-05-27 11:02 .
648  drwxr-xr-x 5 root root 4096 2009-05-27 11:41 ..
649  -rw-r--r-- 1 root root 2692 2008-04-27 16:47 awstats_misc_tracker.js.gz
650  wks-ve10:/usr/local/awstats/js# gunzip awstats_misc_tracker.js.gz
651  wks-ve10:/usr/local/awstats/js# grep DocumentRoot /etc/apache2/sites-available/default
652          DocumentRoot /home/sa/0/0/
653  wks-ve10:/usr/local/awstats# cp -a js/ /home/sa/0/0/
654  wks-ve10:/usr/local/awstats# chown -R awstats\: /home/sa/0/0/js/
655  wks-ve10:/usr/local/awstats# grep ^MiscTrackerUrl /etc/awstats/awstats.www.markus-gattol.name.conf
656  MiscTrackerUrl="/js/awstats_misc_tracker.js"
657  wks-ve10:/usr/local/awstats# grep ^ShowMiscStats /etc/awstats/awstats.www.markus-gattol.name.conf
658  ShowMiscStats=anjdfrqwp
659  wks-ve10:/usr/local/awstats# grep ^ShowScreenSizeStats /etc/awstats/awstats.www.markus-gattol.name.conf
660  ShowScreenSizeStats=1
661  wks-ve10:/usr/local/awstats#

There is not much to say about it except that we need to place (see line 653) ../js/awstats_misc_tracker.js into Apaches document-root (line 652) and see that /etc/awstats/awstats.www.markus-gattol.name.conf has the correct settings (lines 656, 658 and 660).

It is also a good idea to change its owner as we do in line 654 — I realize, if one has followed the article in a linear manner so far, he would not have a system user awstats just yet. I show how to create one in the security section further down. The reader can simply skip the step in line 654 for now and do it after he created the system user awstats as shown below.

Last but not least, to make it work, we need to copy a javascript snippet (one only needs lines 664 and 665 as the rest is specific to my setup i.e. 662 is elisp comment and 663 and 666 are tags to not let Muse interpret the javascript code in lines 664 and 665)

662  ; needed for AWStats
663  <literal>
664  <script type="text/javascript" src="/js/awstats_misc_tracker.js"></script>
665  <noscript><img src="/js/awstats_misc_tracker.js?nojs=y" height=0 width=0 border=0 style="display: none"></noscript>
666  </literal>

into the source code — preferable at the bottom of the index page e.g. index.html, index.php or whatever one uses for his main page. In my case it is home.html as can be seen

667  wks-ve10:/usr/local/awstats# grep home.html /etc/awstats/awstats.www.markus-gattol.name.conf
668  DefaultFile="home.html"

and also visited if we just use the URL .

Speed

There are benchmarks available in order to give people an idea what might be true for their particular use case.

Personally, I do not care about speed much simply because I always use up-to-date hardware and scale ahead of demand which becomes pretty easy by using OpenVZ — if a VE (Virtual Environment) hits a barrier/limit, I simply elevate it and in case the HN (Hardware Node) runs out of juce, I move the entire VE (e.g. this one which contains this website, AWStats, Apache, etc.) to another, more powerful HN. I do this on-line i.e. totally transparent for users because there is no downtime.

However, why not making a few settings that help with making AWStats more snappy. Actually there is only one setting that I find worth setting:

669  wks-ve10:/usr/local/awstats# grep ^DNSLoo /etc/awstats/awstats.www.markus-gattol.name.conf
670  DNSLookup=0
671  wks-ve10:/usr/local/awstats#

Appearance

AWStats appearance can be changed by changing its CSS (Cascading Style Sheets). The Debian package ships a CSS directory which we need to place consistently with where AWStats looks for its CSS files. We made this setting above in /etc/apache2/apache2.conf (line 380).

669  wks-ve10:/usr/local/awstats# cp -a css/ /usr/share/awstats/
670  wks-ve10:/usr/local/awstats# grep ^StyleSheet /etc/awstats/awstats.www.markus-gattol.name.conf
671  StyleSheet="/awstatscss/awstats_bw.css"

In line 669 we copy the CSS directory containing AWStats's CSS files into the place according to the settings in line 380. We also specify what specific CSS file to use (there are several available) as can be seen in line 671.

In addition to the CSS file, we can set a few CSS settings (a collection of the most important ones) directly within /etc/awstats/awstats.www.markus-gattol.name.conf:

672  wks-ve10:/usr/local/awstats# grep ^color /etc/awstats/awstats.www.markus-gattol.name.conf
673  color_Background="FFFFFF"               # Background color for main page (Default = "FFFFFF")
674  color_TableBGTitle="CCCCDD"             # Background color for table title (Default = "CCCCDD")
675  color_TableTitle="000000"               # Table title font color (Default = "000000")
676  color_TableBG="CCCCDD"                  # Background color for table (Default = "CCCCDD")
677  color_TableRowTitle="FFFFFF"            # Table row title font color (Default = "FFFFFF")
678  color_TableBGRowTitle="ECECEC"          # Background color for row title (Default = "ECECEC")
679  color_TableBorder="ECECEC"              # Table border color (Default = "ECECEC")
680  color_text="000000"                     # Color of text (Default = "000000")
681  color_textpercent="606060"              # Color of text for percent values (Default = "606060")
682  color_titletext="000000"                # Color of text title within colored Title Rows (Default = "000000")
683  color_weekend="EAEAEA"                  # Color for week-end days (Default = "EAEAEA")
684  color_link="0011BB"                     # Color of HTML links (Default = "0011BB")
685  color_hover="605040"                    # Color of HTML on-mouseover links (Default = "605040")
686  color_u="f7f828"                        # Background color for number of unique visitors (Default = "FFAA66")
687  color_v="f7f828"                        # Background color for number of visites (Default = "F4F090")
688  color_p="5afc24"                        # Background color for number of pages (Default = "4477DD")
689  color_h="66DDEE"                        # Background color for number of hits (Default = "66DDEE")
690  color_k="f84628"                        # Background color for number of bytes (Default = "2EA495")
691  color_s="5afc24"                        # Background color for number of search (Default = "8888DD")
692  color_e="5afc24"                        # Background color for number of entry pages (Default = "CEC2E8")
693  color_x="f84628"                        # Background color for number of exit pages (Default = "C1B2E2")

Security

Security with AWStats in essence is about potential risks when using AWStats in a dynamic CGI (Common Gateway Interface) setup rather than creating static HTML reports.

Personally, I think, as with any other technology, if used with some safety precautions in place, an AWStats CGI setup is secure enough to not be worried at all.

For the benefit of having dynamic analytics data available at all times we are going to use AWStats in a CGI setup. We are however going to tighten security in order to make it a secure setup.

Change default Path

694  wks-ve10:/usr/local/awstats# grep ScriptAlias /etc/apache2/apache2.conf
695  ScriptAlias /awstats/ "/usr/lib/cgi-bin/"
696
697
698  [ here we use some editor to edit /etc/apache2/apache2.conf... ]
699
700
701  wks-ve10:/usr/local/awstats# grep ScriptAlias /etc/apache2/apache2.conf
702  ScriptAlias /site_statistics "/usr/lib/cgi-bin/"
703  wks-ve10:/usr/local/awstats# apache2ctl restart
704  apache2: Could not reliably determine the server's fully qualified domain name, using 10.0.3.4 for ServerName

First thing we do is render any standard attack on the default path useless — this is pretty much the same principle as with moving the sshd port away from port 22 so the random script kiddie or worm can do no harm. This alone will make more than 90% percent of attacks go away before they even occur.

What we want to achieve is to make the site statistics available under /site_statistics/awstats.pl?config=www.markus-gattol.name rather than /awstats/awstats.pl?config=www.markus-gattol.name.

In order to do so, we make a change to /etc/apache2/apache2.conf — compare line 695 with line 702. That is all... was not that hard, was it? ;-]

Secure AWStats Script

Another, way more powerful thing we can do is to make the CGI script (awstats.pl) driving this whole shebang more secure plus create a dedicated system user account that we use to run AWStats related things. However, before we do so, I would make a few things clear:

• We are only going to make the CGI (Common Gateway Interface) security concern go away — we are not addressing the who can see what subject simply because I want everybody to be able to see the entire statistics for my website/platform. Those who need/want to restrict access to their site statistics can do so of course. The AWStats manuals have information on how to do so.
• In order to harden awstats.pl, we are going to use a utility called chattr which happens to only for work for ext3/4 filesystems as of now (June 2009). In particular, we are going to set the i (immutable) flag on awstats.pl which not only makes it impossible to change the file contents but also to apply chmod and chown settings.

705  wks-ve10:/usr/local/awstats# la /usr/lib/cgi-bin/
706  total 640
707  drwxr-xr-x  2 root root   4096 2009-05-27 11:02 .
708  drwxr-xr-x 42 root root  20480 2009-05-27 16:48 ..
709  -rwxr-xr-x  1 root root 625732 2009-03-03 17:21 awstats.pl
710  wks-ve10:/usr/local/awstats# cd /usr/lib/cgi-bin/
711  wks-ve10:/usr/lib/cgi-bin# adduser --system awstats
712  Adding system user `awstats' (UID 104)...
713  Adding new user `awstats' (UID 104) with group `nogroup'...
714  Creating home directory `/home/awstats'...
715  wks-ve10:/usr/lib/cgi-bin# chown awstats\: awstats.pl
716  wks-ve10:/usr/lib/cgi-bin# la
717  total 640
718  drwxr-xr-x  2 root    root      4096 2009-05-27 11:02 .
719  drwxr-xr-x 42 root    root     20480 2009-05-27 16:48 ..
720  -rwxr-xr-x  1 awstats nogroup 625732 2009-03-03 17:21 awstats.pl
721  wks-ve10:/usr/lib/cgi-bin# lsattr awstats.pl
722  ------------------- awstats.pl
723  wks-ve10:/usr/lib/cgi-bin# chattr +i awstats.pl
724  wks-ve10:/usr/lib/cgi-bin# lsattr awstats.pl
725  ----i-------------- awstats.pl
726  wks-ve10:/usr/lib/cgi-bin# chown sa\: awstats.pl
727  chown: changing ownership of `awstats.pl': Operation not permitted
728  wks-ve10:/usr/lib/cgi-bin# chmod 777 awstats.pl
729  chmod: changing permissions of `awstats.pl': Operation not permitted
730  wks-ve10:/usr/lib/cgi-bin# echo "I own this server now" > awstats.pl
731  bash: awstats.pl: Permission denied
732  wks-ve10:/usr/lib/cgi-bin# la
733  total 640
734  drwxr-xr-x  2 root    root      4096 2009-05-27 11:02 .
735  drwxr-xr-x 42 root    root     20480 2009-05-27 16:48 ..
736  -rwxr-xr-x  1 awstats nogroup 625732 2009-03-03 17:21 awstats.pl
737  wks-ve10:/usr/lib/cgi-bin#

At first we create the new system user in lines 711 to 714 — we use the --system switch i.e. this is not a fully fledged user account simply because it does not need to be. Actually it should not be a fully fledged user account!

The main benefit with regards to security when creating a system user account rather than a fully fledged user account is that the new system user will have the shell /bin/false and has login capabilities disabled. On top of that, the user awstats does not belong to any group (well, its nogroup but...) per default which is good because it limits his radius of action from the very beginning.

Line 715 is where we change the owner of awstats.pl which works as we see from comparing line 709 and line 720. Next thing to do is to set the immutable flag in line 723. Once set, we can see that neither changing awstats.pl owner nor its permissions, not even altering its contents work anymore. Great! Now, let the attacks begin ;-]

Extra Sections

The AWStats ExtraSection features are powerful setup options which allow us to add our own report in case it is not already provides by default with AWStats.

We can use it to build special reports, like for example the number of sales for a particular product, marketing reports, counting for a particular user or user agent, get a list of referring sites (also known as backlinks; those sites that link to ours), have a chart that shows what path through the website visitors took (clickstream analysis), etc.

The only concern we might have with regards of using ExtraSections is their speed impact — each ExtraSection reduces AWStats speed by around 8%, which, depending on the size of our logfile (/var/log/apache2/access.log) and hardware resources might not matter at all or, it might be quite the opposite where an already slow system is brought to its knees.

What I am trying to say is simply that one should evaluate his individual situation and then act accordingly. For example, I started out with AWStats core and the plugins I set up, tested, figured speed does not concern me at all i.e. I have more resources than this website/platform could possible ever need and thus I enabled a few extra sections.

Below are the ExtraSections I am using/toying around with right now — /etc/awstats/awstats.www.markus-gattol.name.conf has a lot of documentation so there is no need for me to mention it here again.

wks-ve10:/usr/local/awstats# exit
exit
sa@wks-ve10:~$ grep -A11 ^ExtraSectionName /etc/awstats/awstats.www.markus-gattol.name.conf
ExtraSectionName1="Google Searches"
ExtraSectionCodeFilter1="200 304"
ExtraSectionCondition1="REFERER,(.*www\.google.*)"
ExtraSectionFirstColumnTitle1="Search"
ExtraSectionFirstColumnValues1="REFERER,p=([^&]+)||REFERER,q=([^&]+)||REFERER,as_p=([^&]+)||REFERER,as_q=([^&]+)"
ExtraSectionFirstColumnFormat1="<a href='http://www.google.com/search?q=%s' title='Click to execute search'>%s</a>"
ExtraSectionStatTypes1=PHBL
ExtraSectionAddAverageRow1=0
ExtraSectionAddSumRow1=1
MaxNbOfExtra1=50
MinHitExtra1=1

--
ExtraSectionName2="Referring Sites by domain"
ExtraSectionCodeFilter2="200 304"
ExtraSectionCondition2="REFERER,^(?!http:\/\/sunoano\.name)"
ExtraSectionFirstColumnTitle2="Site"
ExtraSectionFirstColumnValues2="REFERER,^[hH][tT][tT][pP]:\/\/([^\/]+)\/"
ExtraSectionFirstColumnFormat2="<a href='http://%s/' rel='nofollow' title='http://%s/ [new window]'>%s</a>"
ExtraSectionStatTypes2=PHL
ExtraSectionAddAverageRow2=1
ExtraSectionAddSumRow2=1
MaxNbOfExtra2=50
MinHitExtra2=1

--
ExtraSectionName3="Navigation from home.html to"
ExtraSectionCodeFilter3="200 304"
ExtraSectionCondition3="REFERER,http:\/\/sunoano\.name\/"
ExtraSectionFirstColumnTitle3="URL"
ExtraSectionFirstColumnValues3="URL,(.*html$)"
ExtraSectionFirstColumnFormat3="%s"
ExtraSectionStatTypes3=PHBL
ExtraSectionAddAverageRow3=0
ExtraSectionAddSumRow3=1
MaxNbOfExtra3=25
MinHitExtra3=1

--
ExtraSectionName4="Hotlinking pages"
ExtraSectionCodeFilter4="200 304"
ExtraSectionCondition4="URL,\.mpeg$||URL,\.mpg$||URL,\.avi$||URL,\.jpg$||URL,\.gif$"
ExtraSectionFirstColumnTitle4="Referrer"
ExtraSectionFirstColumnValues4="REFERER,^(?!http:\/\/sunoano\.name)http:\/\/(.*)$"
ExtraSectionFirstColumnFormat4="<a href='http://%s' target='_blank'>%s</a>"
ExtraSectionStatTypes4=HBL
ExtraSectionAddSumRow4=1
MaxNbOfExtra4=25
MinHitExtra4=1


ExtraSectionName5="Pages with debian in name"
ExtraSectionCodeFilter5="200 304"
ExtraSectionCondition5="URL,(.*\/debian.*)"
ExtraSectionFirstColumnTitle5="URL"
ExtraSectionFirstColumnValues5="URL,(.*)"
ExtraSectionStatTypes5=PHBL
ExtraSectionAddAverageRow5=0
ExtraSectionAddSumRow5=0
MaxNbOfExtra5=5
MinHitExtra5=1


ExtraSectionName6="Pages with business in name"
ExtraSectionCodeFilter6="200 304"
ExtraSectionCondition6="URL,(.*\/business.*)"
ExtraSectionFirstColumnTitle6="URL"
ExtraSectionFirstColumnValues6="URL,(.*)"
ExtraSectionStatTypes6=PHBL
ExtraSectionAddAverageRow6=0
ExtraSectionAddSumRow6=0
MaxNbOfExtra6=5
MinHitExtra6=1


ExtraSectionName7="Pages with pim in name"
ExtraSectionCodeFilter7="200 304"
ExtraSectionCondition7="URL,(.*\/pim.*)"
ExtraSectionFirstColumnTitle7="URL"
ExtraSectionFirstColumnValues7="URL,(.*)"
ExtraSectionStatTypes7=PHBL
ExtraSectionAddAverageRow7=0
ExtraSectionAddSumRow7=0
MaxNbOfExtra7=5
MinHitExtra7=1


sa@wks-ve10:~$

AWBot

• http://awbot.sourceforge.net

WRITEME