Annvix - system logging with socklog
Welcome to the tutorial guide. The tutorial will provide a user with advise and guidance on system logging with socklog.
As of Annvix 1.1-RELEASE the socklog program has been provided as an alternative to syslogd and klogd. As of 2.0-RELEASE, socklog is the default logger installed and the sysklogd package has been removed.
A user should note that socklog is an alternative secure system logger with a lot of features. It uses the default svlogd program to handle the actual logging; socklog itself just listens to sockets, be they UNIX sockets, TCP ports, or UDP ports. Because of this, socklog is very extensible and offers features that syslogd can’t, including:
remote logging via TCP
receiving remote logs via TCP
automatic log rotation
Let’s have a look at log directories versus log files.
Log directories vs. log files
The socklog package in Annvix comes with three default services: socklog-unix, socklog-klog, and socklog-rklog. The first service listens to the standard /dev/log socket that is used by applications to send messages to the syslog. The second receives messages from /proc/kmsg (normally klogd would receive these messages and relay them to the syslog); the socklog-klog service does not relay these messages to the system logger but manages the logs itself. Likewise, socklog-rklog receives messages from /proc/rsbac-info/rmsg which are RSBAC-related kernel messages (NOTE, as of RSBAC 1.2.5.1 it does not look like the configuration option CONFIG_RSBAC_RMSG_EXCL works so messages from RSBAC still end up being sent via /proc/kmsg in addition to /proc/rsbac-info/rkmsg).
As a result, instead of kernel logs ending up in /var/log/kernel (and in /var/log/messages, the kernel logs end up in the /var/log/system/kmsg directory, with the current file being the most recent logfile. The RSBAC kernel logs end up in /var/log/system/rkmsg.
A user should note that normal logfiles that would normally populate /var/log (such as auth.log, boot.log, messages, and so forth are no longer used and can be safely removed from the system once socklog is installed. Instead, the default socklog configuration is to log the following facilities to the following log directories:
all logs: /var/log/system/all
auth.* and authpriv.*: /var/log/system/auth
local7.*: /var/log/system/boot
cron.*: /var/log/system/cron
daemon.*: /var/log/system/daemon
*.debug: /var/log/system/debug
ftp.*: /var/log/system/ftp
kern.*: /var/log/system/kern
local.*: /var/log/system/local
mail.*: /var/log/system/mail
*.info (excluding mail.info, news.info, authpriv.info and kern.info): /var/log/system/messages
news.*: /var/log/system/news
syslog.*: /var/log/system/syslog
user.*: /var/log/system/user
Within each directory is the current file which is the latest log and other files using a TAI64 timestamp that indicate the date of rotation (i.e. @4000000043ed4a8b03f03b6c.s). Also, within each file is a config file which tells svlogd what to log; for instance the /var/log/system/messages/config file looks like:
-*.*
+*.info*
-mail.*
-news.*
-authpriv.*
-kern.*
Order is important; here everything is deselected first, then all *.info priorities are added and finally, selectively, various facilities are removed that are not desired to be logged there.
Customizing socklog
The default socklog configuration is adequate for most people with a single-log system (i.e. not receiving remote logs nor sending them). However, it is desirable to be able to send (and receive) logs to other systems. Even adding new facilities or manipulating logging on a single system may be desirable.
Configuring svlogd
svlogd is told what directories to read config files from via the logging runscript for a service, so for socklog-unix this would be in the file /var/service/socklog-unix/log/run. This file contains, by default:
#!/bin/execlineb
/bin/cd /var/log/system
/sbin/chpst -u syslogd /sbin/svlogd
/var/log/system/all
/var/log/system/auth
/var/log/system/boot
/var/log/system/cron
/var/log/system/daemon
/var/log/system/debug
/var/log/system/ftp
/var/log/system/kern
/var/log/system/local
/var/log/system/mail
/var/log/system/messages
/var/log/system/news
/var/log/system/syslog
/var/log/system/user
As can be seen here, arguments to svlogd are the full paths to directories containing config files (which will also in turn contain log files). To, for instance, log openldap messages (which, when the openldap-server package is installed modifies /etc/syslog.conf to send local0.* logs to /var/log/ldap/ldap.log), a user would add /var/log/system/ldap to the end of the directory list in the run script.
A user can now create the /var/log/system/ldap directory:
# mkdir /var/log/system/ldap
# chmod 770 /var/log/system/ldap
# chown root:syslogd /var/log/system/ldap
Then create the /var/log/system/ldap/config file, which would contain:
-*
+local0.*
If user wants to restart svlogd, then he/she needs to send it a HUP using runsvctrl: (/sbin/runsvctrl h /service/socklog-unix/log). Now svlogd has been restarted and will begin to log the newly added ldap configuration.
Logging to a remote server using UDP
It should be noted that while socklog supports receiving logs via UDP, using UDP as a transmission medium should be avoided if at all possible because UDP is both unreliable and insecure. If at all possible, use TCP to transmit log data (if the sending/receiving servers support it, i.e. using socklog or syslog-ng or another system logger than can send/receive logs via TCP).
If a user wnats to instruct svlogd to log to a remote server over UDP (for instance, to a remote syslogd server), then he/she has to create a new configuration directory and file as indicated above but use the directory /var/log/system/remote with the contens of /var/log/system/remote/config being:
+*.*
s9999
n2
U192.168.1.2:514
Then a user has to restart svlogd . This will instruct svlogd to send what it receives to the remote system (192.168.1.2) on port 514 using the UDP protocol. This also tells svlogd to rotate the log every 10KB (triggering the sending of the log to the remote host) and tells svlogd to maintain a maximum number of 2 old log files (in the case of transmission failure).
Please note that using “U” here tells svlogd not to maintain a local copy of the logs (although any errors are logged to current); instead it will send the log files as soon as it receives them. A user can send UDP logs onto another logging directory (say all), by using “u” instead of “U” (read the svlogd(8) manpage for more info). For ease of configuration, having a separate logging directory designated as the “to-remote” log may be a better idea than modifying other configurations.
Receiving logs from a remote server using UDP
Install the socklog-remote package; this package comes with default run scripts to receive TCP- and UDP-based log messages. Before starting socklog-udp, edit the /var/service/socklog-unix/log/run as by default it receives all logs to /var/log/system/remote/all-udp and it is sometimes a good idea to further split the logs based on the sender’s IP address. The resulting run script may look like:
#!/bin/execlineb
/bin/cd /var/log/system/remote
/sbin/chpst -u syslogd /sbin/svlogd
/var/log/system/remote/all-udp
/var/log/system/remote/192.168.1.12
The first run script tells socklog to listen to port 514 on all hosts with the inet protocol UDP.
By default, socklog-udp listens on UDP port 514 (the default syslog port) which can be modified by changing /var/service/socklog-udp/env/PORT. A user can also specify what IP address to bind by creating /var/service/socklog-udp/env/IP with the IP address a user wants the service to bind to (the default is all).
The received logs are printed to STDOUT (prefixed with the sender’s IP address), so the log/run script handles the actual logging. In this case, it will log all received UDP syslog messages to /var/log/system/remote/all and will log messages from the sender IP 192.168.1.12 to /var/log/system/remote/192.168.1.12 via the following config files:
The /var/log/system/remote/all-udp/config file, by default, looks like:
+*.*
and the /var/log/system/remote/192.168.1.12/config file should look like:
-*
+192.168.1.12:*
A user can add subsequent IP-based directories (i.e. perhaps a /var/log/system/remote/192.168.1.13 directory) to handle logging for more than one sending IP. If separating logs per host is not a concern, do away with the IP-based logging directories. The rationale behind an “all” directory and IP-based directories is to ease separation of logs per host (something syslog cannot easily do) and also provide a “master” log that can be used by tools such as swatch.
If a user wants to add this new service, then he/she can use srv –add socklog-udp (if a user can make the appropriate changes in /var/service/socklog-udp before adding the service, a user can save himself/herself a restart). A user should remember to send the socklog-udp/log service a KILL signal if configuration changes are made; this can be accomplished with either runsvctrl k /service/socklog-udp/log or srv -f –restart socklog-udp/log.
Logging to a remote server using TCP
When logging to a remote server, a user can choose to send the logs to the remote server instead of keeping them on the system, or a user can log both to the system and to the remote server. Note that this will not pick up kernel logs (which the socklog-klog service monitors), although a user can do a similar type of configuration for socklog-klog or a duplicate service to handle it.
If a user wants to log to a remote system via TCP instead of logging locally, change /service/socklog-unix/log/run to look like:
#!/bin/execlineb
/bin/cd /var/log/system
/sbin/chpst -u syslogd /sbin/svlogd
/var/log/system/all
Next, edit the /var/log/system/all/config file which should look similar to:
+*.*
s2048
n20
!tryto -pv nc -q 1 192.168.1.2 12000
This instructs svlog to set the log size to 2KB and to keep a maximum of 20 log files. The “!tryto” indicates using the tryto program as a pre-processor; svlogd will run tryto with the specified arguments before rotating the log. The “-pv” option tells tryto that it is being used as a svlogd pre-processor and to be verbose about what it is doing. The next option tells tryto to execute nc (netcat); what it does is feed it’s STDIN to nc as STDIN. In this case, nc is told to send it’s STDIN to the IP address 192.168.1.2 on port 12000. Change the IP address and port number to suit a user configuration. A user will need to install the nc package. The “-q 1″ option to nc tells it to wait 1 second after an EOF is detected; if a user doesn’t include this option nc will continue to hang around long after it should.
Please note that depending on the system, 2KB chunks may be too fast or too slow so a user should determine how often they’re sent to the remote system before settling on a particular size. What happens here is that on every rotation, nc is being used to send the contents of current; in other words if it takes three weeks to fill up 4KB of content, it will be three weeks before the log is transmitted.
If a user wants to mantain system logs and log remotely, change /service/socklog-unix/log/run and add another directory to the end of the directory list, perhaps /var/log/system/toremote, then create the directory (making sure it is owned root:syslogd and mode 0770) with a config file with contents identical to what was previously noted:
+*.*
s2048
n20
!tryto -pv nc 192.168.1.2 12000
This acts the same as the previous configuration, however here we are adding another logging directory who’s sole job is to feed received logs to the remote system, without interfering with what is locally logged.
Receiving logs from a remote server using TCP
As with receiving logs over UDP, install the socklog-remote package to get the default run scripts, for this service we’re interested in socklog-tcp. By default the service uses tcpsvd to listen to the local TCP port (5140 by default but this can be changed by modifying /var/service/socklog-tcp/env/PORT). When connections are received on this port, the connection is passed off to socklog which is using the ucspi connection method. A user can also specify what IP address to bind to (if a user wants to be explicit) by creating /var/service/socklog-tcp/env/IP with the IP address a user wants the service to bind to (the default is all).
Handling the logs is very similar to handling incoming UDP logs. Modify the /var/service/socklog-tcp/log/run script to look like the following (to again allow for per-IP directory logging; the default simply logs all TCP-based logs to /var/log/system/remote/all-tcp):
#!/bin/execlineb
/bin/cd /var/log/system/remote
/sbin/chpst -u syslogd /sbin/svlogd
/var/log/system/remote/all-tcp
/var/log/system/remote/192.168.1.12
The received logs are printed to STDOUT (via socklog and prefixed with the sender’s IP address), so the log/run script handles the actual logging. In this case, it will log all received TCP syslog messages to /var/log/system/remote/all-tcp and will log messages from the sender IP 192.168.1.12 to /var/log/system/remote/192.168.1.12 via the following config files:
The /var/log/system/remote/all-tcp/config file looks like:
+*.*
and the /var/log/system/remote/192.168.1.12/config file should look like:
-*
+192.168.1.12:*
Add subsequent IP-based directories (i.e. perhaps a /var/log/system/remote/192.168.1.13 directory) to handle logging for more than one sending IP. If separating logs per host is not a concern, do away with the IP-based logging directories. The rationale behind an “all” directory and IP-based directories is to ease separation of logs per host (something syslog cannot easily do) and also provide a “master” log that can be used by tools such as swatch.
If a user wants to add this new service, then he/she should use srv –add socklog-tcp (if a user can make the appropriate changes in /var/service/socklog-tcp before adding the service, a user can save himself/herself a restart).
A user should remember to send the socklog-tcp/log service a KILL signal if configuration changes are made; this can be accomplished with either runsvctrl k /service/socklog-tcp/log or srv -f –restart socklog-tcp/log.
If a user has followed this tutorial guide then he/she would have learnt about system logging with socklog.













