High-Performance Logging System

July 1997.

Conventional Web-servers log (archive) a variety of information about each hit or request. However, if a request invokes a program (using CGI, for example), the Web-server has no access to the operation of that program, and can not log any data thereof. This is a major limitation; for example, it cannot log the number of results that matched a search query.

Logging such information must intrinsically be the responsibility of the program. However, due care must be taken not to adversely impact performance. The logging system should not reduce throughput significantly. Furthermore, failures in the logging mechanism should not affect in any way the proper functioning of the program.

I designed a logging system to satisfy these constraints. It had several components:

All these components were implemented in Perl, with the system-level operations (UDP, semaphores, and shared memory) being implemented using the system-level calls. A bug in the Perl interpreter that handled semaphores proved an obstacle, whereupon I identified and fixed the bug in the Perl source-code and recompiled Perl.

The system ran close to the native speed of the machines, and could handle upto 1000 logging requests a second. In practice, the speed of the archival system to move the information into Oracle (about 100/s) proved to be the limiting factor; this was outside my control. However, even this lower speed was quite adequate for the purposes of Industry.Net.

Rujith de Silva 1997-05-13