Annvix - Managing Supervised Services
Welcome to the tutorial guide. The tutorial will provide a user with advise and guidance on how to manage supervised services. Please note that Annvix uses a different means of managing daemons and services. Initscripts is a means of starting and stopping of system services. They are called as:
# /etc/init.d/script start
Or:
# service script start
There are some weaknesses associated with these initscripts. The initscripts are big and bulky and, more often than not, unnecessarily bloated. This makes them slow to start services. The problems can occur if the administrators restarts services. This will inherit variables from the current user’s shell environment. If a service stops then the service stays down. This may be good if a particular service is being configured, but can be very problematic on systems that require high availability.
Please note that a wrapper was written to handle talking to services. The wrapper script is called srv.
How to use srv?
It is good to know more about srv. srv controls services in a same way as service. If a user installs a daemon service on Annvix, it does not automatically start for a user. A user needs to explicitly enable the service before it will ever start. By default, openssh-server is installed on Annvix but, as with all daemon services, it is not started by default or even made available by default. If a user wants to obtain a list of available services, then he/she should execute following:
# srv –list
service status pid started
————————————————————-
crond up 3397 02/02/2007 12:03:45 AM
crond/log up 3366 02/02/2007 12:03:45 AM
mingetty-tty1 up 5785 02/02/2007 12:10:41 AM
mingetty-tty2 up 3302 02/02/2007 12:03:44 AM
mingetty-tty3 up 3399 02/02/2007 12:03:45 AM
mingetty-tty4 up 3332 02/02/2007 12:03:44 AM
mingetty-tty5 up 3331 02/02/2007 12:03:44 AM
mingetty-tty6 up 3362 02/02/2007 12:03:46 AM
socklog-klog up 3409 02/02/2007 12:03:45 AM
socklog-klog/log up 3408 02/02/2007 12:03:45 AM
socklog-rklog - - -
socklog-rklog/log - - -
socklog-unix up 3444 02/02/2007 12:03:45 AM
socklog-unix/log up 3443 02/02/2007 12:03:45 AM
sshd - - -
sshd/log - - -
Please note that only crond, socklog-klog, socklog-unix, and the mingetty services are available and running. These services are the only ones to start “out of the box”. A user will also be able to view that sshd is installed, but is not available. If a user wants to enable sshd, then he/she should execute following code:
# srv –add sshd
Adding service ’sshd’: …….done
# srv –list
service status pid started
————————————————————-
crond up 3397 02/02/2007 12:03:45 AM
crond/log up 3366 02/02/2007 12:03:45 AM
mingetty-tty1 up 5785 02/02/2007 12:10:41 AM
mingetty-tty2 up 3302 02/02/2007 12:03:44 AM
mingetty-tty3 up 3399 02/02/2007 12:03:45 AM
mingetty-tty4 up 3332 02/02/2007 12:03:44 AM
mingetty-tty5 up 3331 02/02/2007 12:03:44 AM
mingetty-tty6 up 3362 02/02/2007 12:03:46 AM
socklog-klog up 3409 02/02/2007 12:03:45 AM
socklog-klog/log up 3408 02/02/2007 12:03:45 AM
socklog-rklog - - -
socklog-rklog/log - - -
socklog-unix up 3444 02/02/2007 12:03:45 AM
socklog-unix/log up 3443 02/02/2007 12:03:45 AM
sshd up 5644 02/10/2007 10:25:02 AM
sshd/log up 9177 02/10/2007 10:25:55 AM
A user can note that sshd is available and running.
Retaining Status Across Reboots
If a user wants to turn down by using following code:
# srv –down sshd
then on the next reboot, sshd will stay down.
srv Options
The best source of information on srv is the manpage. Following options will summarise all the command that a user can pass to srv:
–help - lists all available commands
–list | -l ([service]) / - lists all available services or just the named service
–info | -i [service] - displays verbose information on a single service
–add | -a [service] - adds a service to /service and starts it
–del | -z [service] - removes a service from /service, stopping it first
–up | -u [service] - starts the named service
–down | -d [service] - stops the named service; on a reboot it will remain down
–restart | -r [service] - stops then starts the named service
–reload | -h [service] - restarts a service by sending it a HUP signal
The –add and –del commands are similar to those from chkconfig. The –list command is similar to chkconfig –list.
The –up and –down commands work similar to the service commands “start” and “stop” for initscripts.
The –list command will show what services are installed, which are available to be started, and what their status is by showing a “-” (available but not in /service), “up” (running), or “down” (not running).
The –info command gives extended status information on a particular service. If a user wants to check the status of sshd, then a user should use:
# srv –info sshd
Service: sshd
————————————————
Installed: Thu 01 Feb 2007 11:19:45 PM MST
RPM: openssh-server-4.5p1-6584avx
RPM Desc: OpenSSH Secure Shell protocol server (sshd)
————————————————
Managed: Yes
Status: Up
Pid: 3376
Processes:
tcpsvd(3376)-+-sshd(24424)—sshd(24426)—sh(24427)—screen(24428)—zsh(24429)
`-sshd(8785)—sshd(8787)—zsh(8788)—su(8803)—bash(8813)
Started: 729012 seconds ago
When: 02/02/2007 12:03:46 AM
Is Logged: Yes
Log Dir: /var/log/service/sshd
Log Status: Up
Log Pid: 3375
Log Started: 729013 seconds ago
Log When: 02/02/2007 12:03:45 AM
Dependencies: sshd
————————————————
Requires:
Needed By:
By using runit, the srv wrapper script makes it very easy to manage system daemons. Please note that every daemon shipped with Annvix uses srv.
Peers
srv also handles a concept called “peers”, which is an access control mechanism similar to using /etc/hosts.allow and /etc/hosts.deny. Services that use tcpsvd make use of peers to allow or deny access to those services. tcpsvd is another “super-server” similar to xinetd. It answers any connection requests, determines if the connecting system is permitted to use the service, and if so, hands it off to the appropriate program.
For example the sshd service uses tcpsvd in exactly this way. It’s run script looks like as following:
#!/bin/execlineb
/bin/fdmove -c 2 1
/bin/export PATH “/sbin:/bin:/usr/sbin:/usr/bin”
/sbin/chpst -e /etc/sysconfig/env/tcpsvd/
/sbin/chpst -e ./env/
/bin/multisubstitute {
import -D “localhost” HOSTNAME
import -D 0 IP
import -D 22 PORT
import -D 20 MAX_CONN
import -D 5 MAX_PER_HOST
import -D 20 MAX_BACKLOG
import OPTIONS
}
/sbin/tcpsvd -v -l ${HOSTNAME} -x peers.cdb -c ${MAX_CONN} -C ${MAX_PER_HOST} -b ${MAX_BACKLOG} ${IP} ${PORT}
/usr/sbin/sshd -i ${OPTIONS}
Essentially what happens here is that a number of variables are imported first from the global /etc/sysconfig/env/tcpsvd environment directory, then the service’s own ./env environment directory. The important thing to note here is the -x peers.cdb argument to tcpsvd.
This argument instructs tcpsvd to use peers information for ACLs as found in the peers.cdb file for this service. This file is a CDB copy of the contents of the ./peers directory for this service. This file is constructed every time srv is used to manage peers information for the service, by first writing the peers information to ./peers and then creating the CDB file by using the ipsvd-cdb tool.
Structure of ./peers
The peers directory is a regular directory that contains IP addresses, partial IP addresses, or hostnames with certain permissions that tell tcpsvd whether or not that IP (or hostname) is allowed to connect to the service. The catch-all file is ./peers/0, and it’s default permissions determine the default access type. This directory is essentially an ipsvd instructions directory; as such you can get more details from the ipsvd-instruct(5) manpage.
The file’s permissions determines the access to the service for that IP or hostname:
If neither the user’s read permission, nor the user’s execute permission is set for the file, the connection is closed immediately
If the file has the user’s execute permission set, ipsvd reads the contents of the file and runs /bin/sh -c ” instead of the default program given att he command line for this connection (i.e. the service itself)
If the file has the user’s read permission set, ipsvd reads the contents of the file and interprets each line as an instruction for this connection; thus allowing the connection
Essentially, to deny access to a connection, chmod 0 the file. To allow access to a connection, chmod 0644 the file. If you want to get fancy, you can have the file contain an alternate program to execute as it’s contents and chmod 0755 the file.
For instance, to disallow all connections to the sshd service, but allow connections from the 10.0.5.0 network and a single remote IP address (say 12.34.56.78), the ./peers directory would look thus:
# ls -l
total 4
———- 1 root admin 0 Dec 29 16:09 0
-rw-r–r– 1 root root 0 Feb 4 17:43 10.0.5
-rw-r–r– 1 root root 2 Nov 18 07:28 12.34.56.78
How srv handles peers
Manually manipulating ./peers is fine, but srv makes it simple to do basic ACLs. It doesn’t do things such as executing alternate programs based on the connection, but it will manipulate the peers information to the point that you can allow and deny access based on IP or hostname or network, and you can also set per-peer environment pair values, as well as the number of concurrent connections to the service.
The related srv commands are:
–peers | -p [service] - set peer information for the service (see below), or rebuild the peer instructions for the service if no peer control options are given
–env | -e [service] - set global environment pair values for the service
Along with the –peers command, a user can use the following peer control options:
–grant | -g - grant access to this service to the specified hosts, IPs, or networks
–exclude | -x - deny access to this service to the specified hosts, IPs, or networks
–remove | -y - remove peer information for the specified hosts, IPs, or networks
–env | -e - set per-peer environment pair values
–max | -m - set the number of concurrent connections for the specifies hosts, IPs, or networks to this service
An example will help towards gaining more understanding:
# srv –peers rsync -g 192.168.0.12 -e TRUSTED=yes -m 5
This command sets up peer information for the rsync service. It grants access to the IP 192.168.0.12, sets the environment variable $TRUSTED to “yes” (this is made available to the service), and sets the maximum concurrent connections to 5.
This command also explains that a user can have default “accept” to a service, a user can also override certain configuration options based on connecting IP or hostname. In this case, if access to the rsync service was globally permitted, the maximum number of concurrent connections per IP might be 2 for every IP other than that specified, which would permit 5. The environment variable $TRUSTED is set when this IP connects where it is not set for all others
By looking at this, it can be said that peers management is simple and powerful. This because a user can completely change how a service responds based on the connecting system.
Let’s have a look at another example:
# chmod 0 ./peers/0
# srv –env sshd MAX_PER_HOST=1
# srv –peers sshd -g secure.host.com -e OPTIONS=’-o “PermitRootLogin yes”‘-m 10
# srv –peers sshd -g host.com -m 3
A user will note that this series of commands will carry out following:
- disable all connections to sshd by default.
- set the maximum concurrency for all hosts to one connection per IP.
- grant access to sshd to secure.host.com with the environment variable $OPTIONS set to ‘-o “PermitRootLogin yes”‘ and a concurrency of 10 connections.
- raise the concurrency to three connections per IP to any *.host.com connections.
The third command is interesting. This is because the sshd service passes the value of $OPTIONS directly to sshd itself, a user can directly manipulate the server based on the incoming connection. A user can note that “PermitRootLogin” is set to “No” in sshd_config. All connections this will be the case, however, if a connection comes from secure.host.com, all of a sudden sshd will allow root logins because that option is passed to it.
The resulting ./peers/secure.host.com file would look like:
C10
+ OPTIONS=-o “PermitRootLogin yes”
The use of host names is fine, but it is recommended that a user should look up via DNS first. This means that if the IP 12.34.56.78 does not have a reverse DNS entry to secure.host.com, access will be granted based on the IP address and not the hostname.
Please note that in order to make use of this, a user will need to add -h to the tcpsvd command in the ./run script. By default, no services will attempt to do DNS lookups on incoming connections as it can get expensive in terms of connection speeds. If a user wants to enable this then he/she can change ./run so that instead of looking like:
/sbin/tcpsvd -v -l ${HOSTNAME} -x peers.cdb -c ${MAX_CONN} -C ${MAX_PER_HOST} -b ${MAX_BACKLOG} ${IP} ${PORT}
/usr/sbin/sshd -i ${OPTIONS}
it looks like:
/sbin/tcpsvd -v -h -l ${HOSTNAME} -x peers.cdb -c ${MAX_CONN} -C ${MAX_PER_HOST} -b ${MAX_BACKLOG} ${IP} ${PORT}
/usr/sbin/sshd -i ${OPTIONS}
If a user can change a run script, then he/she should keep in mind that upgrades to that package will never overwrite the changes. However, the new script will be named run.rpmnew and a user will examine the file for changes that may have been made and merge them in manually.
The way it works
At boot, Annvix uses runit rather than init to start the system. In runit’s stage 2 process, runsvdir is started to manage everything under /service. All that is in the /service directory are symlinks to directories in /var/service which is where rpm packages place directories and information in order to run supervised services. These directories are named after the service they start. Unlike initscripts that can (and often do) start more than one daemon, each service starts one and only one service. So, for instance, when running LDAP you would use service ldap start on most Linux distributions, but on Annvix you would use srv –up slapd; srv –up slurpd. In this case there are two daemons, thus two services: slapd and slurpd.
Inside each directory is a single file called run. run is the equivalent of an initscript’s start() function; all it does is start the service. The run script of spamd, for instance looks like this:
#!/bin/sh
PATH=”/sbin:/usr/sbin:/bin:/usr/bin”
# this runs spamd supervised
OPTIONS=”-c -m5 -H”
if [ -f ./env/OPTIONS ]; then
OPTIONS=`head -1 ./env/OPTIONS`
fi
exec /usr/bin/spamd $OPTIONS 2>&1
Like an initscript, it sources certain configuration files to get options for the daemon (in this case, the /var/service/spamd/env/OPTIONS file). You also see that the daemon is started with exec and that STDERR is redirected to STDOUT. This is not always desirable, however, and needs to be dealt with on a case-by-case basis. In the case of spamd, a sub-service is also started that writes all of spamd’s output to a logfile.
The logging service is another service that supervise handles, tied specifically to that particular service. This is done by using a log/ subdirectory of the service; so spamd’s logging service run script is /service/spamd/log/run. The contents of this file are:
#!/bin/execlineb
# logging for the spamd service
/bin/foreground { /usr/bin/install -m 0700 -d -o logger -g logger /var/log/service/spamd }
/bin/cd /var/log/service
/sbin/chpst -u logger /sbin/svlogd -tt /var/log/service/spamd
This script basically takes STDOUT from spamd and writes it, through the svlogd tool (which handles the naming and automatic rotation of log files) to the directory /var/log/service/spamd. In this directory you will find a few files, some with odd names that are rotated logfiles, and one of interest, particularly current. The current logfile is the latest output of the service.
If a user wants to debug a service or figure out why it isn’t starting. Often errors from a service will be written to STDERR, which, if you remember, we are redirecting to STDOUT, which is then being written to the logfiles that svlogd is handling.
If a user browses through the /var/log/service directory, then he/she will view a number of subdirectories. These subdirectories are log directories for other services, although not all services uses logging services (socklog is a notable exceptions). More often than not, the log files will be 0 bytes unless the program (like snort) is fairly verbose. As well, in many cases, services need to be started with debugging enabled in order to prevent them from forking into the background. In order for runsvdir to be able to control a service, it must not fork into the background.
If a user follows this tutorial guide then he/she would have learnt about managing supervised services.













