In this article a user will learn about securing Linux production systems. It will cover areas such as physical Security, verifying security action items, retiring linux servers with sensitive data, backups, removing unnecessary software packages, patching linux systems and closing network ports and disabling runlevel system services and Xinetd services.
Now, let’s look at following areas of cover:
Physical Security
A user should consider physical security as of high importance and for that Linux production servers should be in locked datacenters where only people with passed security checks have access. Another useful thing regarding security is to take boot loader passwords.
Verifying Security Action Items
A user should also look into scripts that can verify that all the security action items have been executed. Please note that even the best sysadmins can make mistakes and miss steps. If a user has a larger Linux environment, it would be a good investment to write scripts for checking Linux security action items.
Retiring Linux Servers with Sensitive Data
If a user is interested in retiring Linux servers with sensitive data then he/she should understand that data cannot be recovered from the hard disks. In order to verify that all traces of data are removed, the DiskSanitizer tool can be used. This tool can be operated from a floppy disk and it removes data.
Backups
If a users system gets compromised, then the backups become invaluable. Backups are of high importance as in cases like bugs, accidents etc. backups can be used to compare the current system against the backed-up system. Backups hold importance for production systems and the backups should be taken offsite for cases like disasters.
Disk Partitions
A user should be aware that servers should have separate partitions for at least /, /boot, /usr, /var, /tmp, and /home. A user shouldn’t have logging and temporary space under /var and /tmp fill up the root partition. Third party applications should be on separate filesystems as well, e.g. under /opt.
Firewall (iptables)
A user should note that firewalls can be used to protect the servers in a production network.
SELinux
Please note that SELinux is an advanced technology for securing Linux systems. Hardening Linux using SELinux technology, on its own, warrants it’s own security. The SELinux technology is recommended for securing Linux systems and this is why a user should consider using SELinux.
FTP, telnet, and rlogin (rsh)
FTP, telnet, and rlogin (rsh) are vulnerable to eavesdropping, which is one of the reasons why SSH/SCP/SFTP should be used instead. It is highly recommended not to run these services. Due to the high risk, this guide does not cover these services. It would also be a good idea not to have FTP and Telnet server RPMs installed on the system.
How to remove unecessary Software Packages (RPMs)?
It is a good idea to establish the software packages that are not necessary. This is because some of the software packages that are not necessary might not comply with the security policy. After removing the unnecessary software packages, it will be easy for a user to update nad maintain when security alerts and patches are released. A good example is of packages such as Apache or Samba installed on the system which a user doesn’t use them. Another useful advise is not to have development packages and desktop software packages (such as X Server) etc. installed on production servers. Also, packages such as FTP and Telnet daemons should not be installed as well unless there is a justified business reason for it (SSH/SCP/SFTP should be used instead).
The first step is to create a Linux image that only contains RPMs needed by the applications, and needed for maintenance and troubleshooting purposes. A good approach is to start with a minimum list of RPMs and then add packages as needed. It may be time-consuming but worth the efforts.
Useful commands
A list of useful commands that a user can note are provided below:
- If a user wants to get a list of all installed RPMs then he/she can use the following command:
rpm –qa
- If a user wants to know more about a particular RPM, then he/she can run:
rpm -qi
- If a user wants to check for and report potential conflicts and dependencies for deleting a RPM, then he/she can run:
rpm -e –test
Patching Linux Systems
A user should also note that in order to secure Linux production environments, it is a good idea to build an infrastructure for patch management. It is recommended to have a written security policy and procedure to handle Linux security updates and issues.
For example, a security policy should detail the timeframe for assessment, testing, and rollout of patches. Network related security vulnerabilities should get the highest priority and should be addressed immediately within a short timeframe.
For example, a security procedure should detail the process for assesment, testing, and rollout of patches. The assessment phase should occur within a testing lab, and initial rollout should occur on development systems first.
A separate security log should detail what Linux security notices have been received, when patches have been researched and assessed, when patches have been applied etc. For Red Hat systems, it is recommended to use Red Hat Network (RHN) for patch management. In fact, for secure environments a user can also take into account the Red Hat’s Satellite solution.
Detecting Listening Network Ports
One of the most important tasks is to detect and close network ports that are not needed.
In order to obtain a list of listening network ports (TCP and UDP sockets), a user can run the following command:
# netstat -tulp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 *:auth *:* LISTEN 2328/xinetd
tcp 0 0 localhost.localdomain:smtp *:* LISTEN 2360/sendmail: acce
tcp 0 0 *:ssh *:* LISTEN 2317/sshd
A user can easily see from the output that xinetd, sendmail, and sshd are listening.
On all newer Red Hat Linux distributions sendmail is configured to listen for local connections only. Sendmail should not listen for incoming network connections unless the server is a mail or relay server. If a user wants to confirm that then he/she should run a port scan from another server. In order to run a port scan from another server a user can run following command:
# nmap -sTU
Starting nmap 3.70 ( http://www.insecure.org/nmap/ ) at 2004-12-10 22:51 CST
Interesting ports on jupitor (172.16.0.1):
(The 3131 ports scanned but not shown below are in state: closed)
PORT STATE SERVICE
22/tcp open ssh
113/tcp open auth
Nmap run completed — 1 IP address (1 host up) scanned in 221.669 seconds
#
Please note that the mentioned nmap command can take a while. If a user wants to remove the UDP port scan (without the option “-U”), then nmap will finish the port scan immediately. If a user runs it on the local machine it will also complete very fast. A user should also note that nmap might not show all listening network sockets if a firewall is being used to block ports.
From the output above a user can see that the xinetd daemon is listening on port auth (port 113) for IDENT. A user can also see that sendmail is not listening for remote incoming network connections.
There is another method to list all of the TCP and UDP sockets to which programs are listening is lsof:
# lsof -i -n | egrep ‘COMMAND|LISTEN|UDP’
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 2317 root 3u IPv6 6579 TCP *:ssh (LISTEN)
xinetd 2328 root 5u IPv4 6698 TCP *:auth (LISTEN)
sendmail 2360 root 3u IPv4 6729 TCP 127.0.0.1:smtp (LISTEN)
#
Closing Network Ports and Disabling Runlevel System Services
One of the most important tasks is to remove any network services from the system startup process that are not required.
On Red Hat systems if a user wants a list of all services that are started at bootup then he/she can run following command:
chkconfig –list |grep on
A user will also notice that there are quite a lot of services enabled on the system. But many runlevel services (Stand-Alone Services) are not network related services like kudzu which is responsible for detecting and configuring new and/or changed hardware on your system. This service is only run during the boot process. It is important to know that a user should not disable runlevel services that are required by the system to run smoothly.
Red Hat Runlevel System services
There are a number of Red Hat Runlevel Systerm servies that a user can enable if he/she wants to depending on his needs and requirements:
gpm needed if you want to use the mouse at the console
kudzu important for detecting new hardware
syslog important for syslog services
netfs needed only if there are NFS shares that should be mounted at boot time
network important for starting network interfaces (e.g. eth0, eth1, bonding,…)
random used for the system entropy pool
atd needed if the at(1) service is used instead of cron
apmd Advanced Power Management (APM) daemon is used for laptops and some desktops
isdn needed if ISDN is being used
iptables needed if Netfilter (iptables) Firewall is being used
ip6tables needed if ip6tables Firewall is being used
pcmcia not needed on servers - needed for laptops
irqbalance important for distributing interrupts across all CPUs
sendmail needed if Sendmail is used - Procmail should be used which is more secure
autofs needed if automounter is used - production applications should not be dependent on automounter
sshd important for logins via SSH
portmap needed if e.g. NFS is being used
nfslock needed if NFS shares are mounted
nfs needed if server runs the NFS server
mdmonitor needed only if software RAID is being used
crond important for running cron jobs
xinetd needed if xinetd services are being used, see /etc/xinetd.d/ for list of services
cups needed if CUPS is used for the printing system
rhnsd needed if server should connect to RHN to check for software updates etc.
sysstat needed to reset system statistics logs
audit needed only if Linux Audit Subsystem (LAuS) should run for collecting system call audit records
psacct needed only if kernel process accounting information is needed
smartd important for monitoring disk problems if hard disks support SMART technology
netdump important if kernel oops data and memory dumps should be sent to a Netdump server for server crashes
Please note that the start/stop scripts of all runlevel services can be found in the /etc/init.d directory. For example, if a user doesn’t know what the atd service does, then he/she can go to /etc/init.d and open the file atd. And in the script look for lines that start programs. In the atd script the “daemon /usr/sbin/atd” line starts the binary atd. Now having the name of the program that is started by this service, a user can check the online pages of atd by running man atd. This will help a user to find out more about a system service.
If a user wants to disable a service (such as the runlevel service nfs) on a permanent basis then he/she can run following command:
chkconfig nfs off
If a user wants to disable the runlevel service nfs immediately then he/she can run following command:
/etc/init.d/nfs stop
Closing Network Ports and Disabling Xinetd Services
The xinetd daemon is a replacement for inetd, the internet services daemon. It monitors the ports for all network services configured in /etc/xinetd.d, and starts the services in response to incoming connections.
If a user want to check if xinetd is enabled and running, then he/she can execute following command:
# chkconfig –list xinetd
xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
# /etc/init.d/xinetd status
xinetd (pid 2619) is running…
#
If xinetd is active, it is important to check which Unix services are active and controlled by xinetd. A command is provided below that shows all the services configured in /etc/xinetd.d and wheter xinetd monitors the ports for these services:
# chkconfig –list | awk ‘/xinetd based services/,/”"/’
xinetd based services:
krb5-telnet: off
rsync: off
eklogin: off
gssftp: off
klogin: off
chargen-udp: off
kshell: off
auth: on
chargen: off
daytime-udp: off
daytime: off
echo-udp: off
echo: off
services: off
time: off
time-udp: off
cups-lpd: off
#
If a user wants to get a list of only active services for which xinetd monitors the ports, then he/she should run following command:
# chkconfig –list | awk ‘/xinetd based services/,/”"/’ | grep -v off
xinetd based services:
auth: on
#
This shows that the telnet-server RPM is not installed on the system. If the Telnet Server package telnet-server would be installed, it would show up on the list whether it’s active or not.
If a user wants to disable a service such as telnet and also wants to view how the telnet service entries are being update then a user can run following command
# chkconfig –list telnet
telnet on
# cat /etc/xinetd.d/telnet | grep disable
disable = no
# chkconfig telnet off
# chkconfig –list telnet
telnet off
# cat /etc/xinetd.d/telnet | grep disable
disable = yes
#
Please note that for the telnet service it would be better to remove the package from the system since SSH should be used instead:
# rpm -e telnet-server
It is important to investigate all active xinetd services and to disable them if they are not needed.
Here is an example how to find out what a service does. This example is based on the assumption that a user doesn’t know what the auth service does?. This can be done by running following command:
# grep ” server” /etc/xinetd.d/auth
server = /usr/sbin/in.authd
server_args = -t60 –xerror –os -E
# man in.auth
No manual entry for in.auth
# rpm -qf /usr/sbin/in.authd
authd-1.4.1-1.rhel3
# rpm -qi authd-1.4.1-1.rhel3 | awk ‘/Description/,/”"/’
Description :
authd is a small and fast RFC 1413 ident protocol daemon
with both xinetd server and interactive modes that
supports IPv6 and IPv4 as well as the more popular features
of pidentd.
# rpm -ql authd-1.4.1-1.rhel3
/etc/ident.key
/etc/xinetd.d/auth
/usr/sbin/in.authd
/usr/share/doc/authd-1.4.1
/usr/share/doc/authd-1.4.1/COPYING
/usr/share/doc/authd-1.4.1/README.html
/usr/share/doc/authd-1.4.1/rfc1413.txt
/usr/share/locale/ja/LC_MESSAGES/authd.mo
#
This example shows what can be done if there exists no online manuals for the binary in.authd that is started by xinetd. The steps above should be helpful for finding out more about services.
The auth service (aka IDENT, see RFC 1413) allows remote daemons to query information about users establishing TCP connections on the local server. In a trusted environment it helps a server to identify who is trying to use it. For example, it can provide vital information for troubleshooting and who has done what. IDENT requests are needed by some applications like IRC. A user should note that IDENT can be a security risk.
If a user wants to disable the auth service, then he/she should run the following command:
# chkconfig auth off
The xinetd daemon is quite flexible and has many features. Here are just a few functionalities of Xinetd:
- Acces control for TCP, UDP, and RPC services
- Acess limitations based on time
- Provides mechanisms to prevent DoS attacks
If a user has read this article then he/she would have learnt about physical Security, verifying security action items, retiring linux servers with sensitive data, backups, removing unnecessary software packages, patching linux systems and closing network ports and disabling runlevel system services and Xinetd services.