Jonathan's blog

This blog, and my other blogs, used to run on a rather old server: two 1GHz Pentium III processors, 1GB memory and 2 x 18GB SCSI hard drives. I host with Ridgeon Network, which is owned by my friend Chris. I help him out with some networking stuff from time to time so he loaned me this spare server for personal use.

A few weeks back one of the hard drives failed. Service continued as normal but it was an acute reminder that the server was getting old.

Recently Chris bought a powerful server for use as a VMWare ESXi hypervisor, along with a large iSCSI SAN to host all the disk images, and powerful shared MySQL database server. As he was moving lots of his servers from physical boxes to virtual machines, I decided to do likewise.

So this website, and my other sites, are now hosted on a CentOS virtual machine, with their databases on a separate CentOS database server. Given that the load average on the old P3 wasn’t very high I wasn’t expecting a noticeable improvement in performance. But how wrong I was! The site is noticeably faster to load and navigate, and in particular the WordPress management interface is miles faster.

All in all, I’m happy with the new platform. To anyone else considering replacing old servers with a virtualised infrastructure, I say go for it. You’ll save tons of electricity, take up less rack space, pave the way for later expansion (by adding more hypervisors or more disks to the SAN) and have better manageability and backupabilitiy.

I’ve had a server at home for years now, and I’ve also been a professional sysadmin for at least three years. I know my way around Linux pretty well and for some time I’ve run my own web server and also other services.

But one thing I’ve steered clear of until now is running my own email server.

I’ve always thought it would be fairly easy to set up, but much harder to make secure. I don’t want to receive tonnes of spam and I don’t want spammers using my SMTP server as an open relay. In the past I’ve read about building SMTP servers with sendmail, postfix and exim but there was all sorts of conflicting information when it came to integrating milters and so on. Different guides all seemed to give contradictory advice and require all sorts of strange configuration steps that I couldn’t understand.

But all that changed when I heard about ClearOS. In short, it’s a spin of CentOS which uses a custom web interface to configure various software “modules”, including things like web server, email server, firewall gateway, database server, and so on.

I installed it on a virtual machine and after only a few clicks I was running a mail server: an MX for receiving mail for my domains, an authenticated SMTP server for personal outgoing mail, and a secure IMAP server for storing and accessing my mail. The frontend sets up postfix and cyrus to do its dirty work.

For ultimate ease, users (just me, in this case) are authenticated using a local LDAP directory, rather than by using system accounts. All SSL certificates for IMAPS and HTTPS were added automatically. Email antivirus scanning is done by Amavis and spam filtering is done by Spamassassin.

I had a little bit of trouble setting up Horde to access webmail and a web interface for configuring sieve rules. By “trouble” I mean the default Apache virtual host declarations needed some changing around and some aliases adding. If you’re familiar with Apache this won’t be a problem.

There are some aspects of ClearOS I don’t like so much, and I would prefer to use CentOS. But now ClearOS has written out all my configs it should be trivial to move my new mail setup to a plain old CentOS installation, where I already run my websites from. I have definitely learnt a lot about how email works by simply reading and understanding the config files written by the frontend.

So if you want to build an email server but don’t know where to start – try ClearOS. It’s a great introduction to the “scary” parts of setting up an email server, like milters and certificates.

Today a friend (from a Windows background – still a friend?! ) asked me how to go about setting up a LAMP (Linux, Apache, MySQL & PHP) server. I wrote him a few notes, not only on how to configure the LAMP stack, but also on how to configure a Linux system properly from scratch, and how to do so securely. There are millions of guides out there that explain how to serve web pages with Apache, but not many of them explain the basics of setting up a secure system too.

I’ve edited these notes slightly to make them suitable for a wider audience, but in essence it’s the same stuff. Hope it’s useful!

OS installation

I recommend using CentOS. It doesn’t really matter whether you choose 32-bit (i386) or 64-bit (x86_64) but use ideally use 64-bit unless there’s a reason not to.

Boot from the CD or DVD of your choice. It doesn’t matter whether you use the full DVD, or the network install CD.

Choose the text-based installer from the boot prompt by typing linux text. The text installer doesn’t install as much extra rubbish as the GUI installer.

In most cases the default options are good enough. One option you should change is to use an NTP time server. This is especially important with virtual machines, since they suffer badly from clock drift.

Choose a strong root password. You will only need it once again. After that, you won’t even even need it for logging on, so there is no need to pick anything memorable. In fact, you are best off choosing a long, random string of mixed-case letters and numbers.

When it comes to choosing packages, deselect as many of the groups as possible. We will add the packages we need individually later on.

Let the installer run its course, and reboot.

Users and passwords

Upon first boot, log in as root using the password you picked before. Now create new user accounts and set passwords:

useradd yourusername
passwd yourusername

Now for setting sudo access. This is like “run as admin” on Windows. Type visudo. In the text file that opens, read down to the line that says

root    ALL=(ALL)       ALL

Duplicate it twice by pressing yyp. Go into insert mode by pressing i and change the username root to your username. When you are done, hit Esc and type :wq to save and exit. Gotta love vi commands

To disable remote root login via ssh, edit the file /etc/ssh/sshd_config using your favourite editor. If you don’t already have a favourite editor, use vi.

Find the line:

#PermitRootLogin yes

and uncomment it and change the value to no:

PermitRootLogin no

Restart the ssh daemon by doing

sudo /sbin/service sshd restart

From now on you can gain root access by using the sudo command, and you won’t need to log in as root again. Log out now by typing exit and re-login as your own user. Forget the root password forever.

Installing packages

First we add a couple of third-party software repositories that have useful stuff.

sudo rpm -Uvh http://download1.rpmfusion.org/free/el/updates/testing/5/i386/rpmfusion-free-release-5-0.1.noarch.rpm http://download1.rpmfusion.org/nonfree/el/updates/testing/5/i386/rpmfusion-nonfree-release-5-0.1.noarch.rpm

Let’s get rid of the stuff we don’t want or need. There are no doubt more than things that can be removed than I’ve listed here, but they can be removed later.

sudo yum remove bluez* pcsc*

Update the system so you’re sure that that latest versions of all software are installed.

sudo yum update

Now we can install the stuff we want for LAMP!

sudo yum install httpd mysql-server php php-mysql

If you are wanting to use any PHP modules/libraries they can be installed here too, such as the commonly-used graphics library gd.

Services

Let’s start the two daemons for Apache and MySQL, and tell them to start on boot.

sudo /sbin/service httpd start
sudo /sbin/service mysqld start
sudo /sbin/chkconfig httpd on
sudo /sbin/chkconfig mysqld on

Apache in its default state will run out of the box. MySQL just needs a root password setting.

mysqladmin -u root password NEWPASSWORD

From now on it’s advisable to GRANT access to specific users on specific databases/tables. Go read about MySQL users.

Firewall

Let’s assume you want HTTP on port 80 open to the world. Open /etc/sysconfig/iptables for editing, and add this line.

-A RH-INPUT -p tcp -m tcp --dport 80 -j ACCEPT

Save and close, and run this to make the changes live.

sudo /sbin/service iptables restart

Editing configs

The main config file for Apache is at /etc/httpd/conf/httpd.conf. It doesn’t need any changes for basic operation, but if you edit it you need to restart the httpd service to pick up the changes.

If you get serious with web publishing from a LAMP platform, you will probably want to read about name-based virtual hosts.

Adding content

In its basic configuration, you should add PHP scripts, HTML pages and other content like images and stylesheets to /var/www/html/. You do not need to restart the daemon for it to pick up new content.

When debugging pages, you will probably find it handy to refer to the error log, at /var/log/httpd/error_log.

Tip: Open two SSH windows to the server – one for editing stuff, and the other for watching the log scroll by as events occur. Use Ctrl-C to break out of it. Do this:

sudo tail -f /var/log/httpd/error_log

For those who have been following the issues around Samba suddenly breaking upon upgrade, I’ve now got to the bottom of it.

I’ve updated the original post with details, so if you wish to leave comments, please leave them on that post.

Some time ago I bought a cheap USB thermometer called TEMPer. I was disappointed to find that it didn’t work on Fedora. It would only work on Windows using a poor piece of proprietary software.

I eventually found the blog of Tollef Fog Heen, who had managed to get his TEMPer to work. Unfortunately his solution involved patching and compiling a kernel.

However since then, it seems his patch has been integrated into the stock Fedora kernel and it is now possible to read the temperature from it.

The TEMPer device appears to be a USB-serial adapter, with a serial I2C device at the end of it. It’s not straightforward to extract the temperature from it, but Tollef Fog Heen has written a simple C program to return the temperature.

His program polls the TEMPer every second and prints the temperature to the command line. It doesn’t stop until you kill the program. I made a couple of tweaks to the code so it prints the temperature once, formatted as a raw number with no extra text, and then quits. You can find my modified source here.

Disclaimer: I don’t know C. I haven’t changed any of the logic of the code, only the way it prints the output. If the code is buggy, it wasn’t me!

Now I have an executable that returns the temperature from the TEMPer, I can think about building some application that could use this. How about a Nagios plugin?

I have a samba fileserver which has been happily running for a couple of years with identical config but on different versions of samba, and on Fedora and CentOS.

The latest incarnation was running samba 3.2.11.

But the other day, my samba package was upgraded to 3.4.1

Samba shares on the server immediately stopped working and access is immediately denied to all users.

I looked at the config and nothing has changed during the upgrade.

I looked at the logs and no access attempts are recorded; no errors are logged.

I noticed that the new template config file is a little different from previous versions, so I made the necessary changes and migrated my config to the new file. No change to samba’s behaviour at all.

For a while I wondered if I had a rogue samba server on my network, but stopping my samba service causes requests to time out rather than be denied. So it’s definitely my samba daemon that’s responding, but goodness knows why it behaving like this.

Restarting samba puts the following in the log file:

[2009/10/02 10:33:54,  0] smbd/server.c:1065(main)
smbd version 3.4.1-0.41.fc11 started.
Copyright Andrew Tridgell and the Samba Team 1992-2009
[2009/10/02 10:33:54,  0] smbd/server.c:457(smbd_open_one_socket)
smbd_open_once_socket: open_socket_in: Address already in use
[2009/10/02 10:33:54,  0] smbd/server.c:457(smbd_open_one_socket)
smbd_open_once_socket: open_socket_in: Address already in use

I will keep hunting until I find what’s caused this. Unfortunately I can’t watch any of my recorded TV programmes until then!

I’ll post back here when I’ve tracked it down.

Update: I managed to get Samba working again. Sort of. This is a snippet from my now-working smb.conf:

# ----------------------- Standalone Server Options ------------------------
#
# Scurity can be set to user, share(deprecated) or server(deprecated)
#
# Backend to store user information in. New installations should
# use either tdbsam or ldapsam. smbpasswd is available for backwards
# compatibility. tdbsam requires no further configuration.

security = user
#       passdb backend = tdbsam
passdb backend = smbpasswd

As you can see, I simply reverted to the older smbpasswd authentication after yum upgraded Samba and switched to tdbsam and my shares magically sprung back to life. It’s a shame, because I don’t like going backwards. I like going forwards – hence I run Fedora.

I can confirm that this “fix” works with the latest version of Samba at the time of writing – version 3.4.2.

So despite the claim that tdbsam requires no extra configuration, clearly there’s more to it than that. I will once again post back here when I’ve found a way to enable tdbsam without breaking everything.

This is a +1 for Easy Firewall Generator for iptables.

Of course any self-respecting sysadmin should be able to set up iptables, but sometimes starting off can be tricky. So I use the this website, which lets you define the basics using a handful of checkboxes, and it generates a script that configures your computer’s iptables firewall.

It works for single hosts and servers that do NAT, and includes protection against a great many nasties.

Once you have this, it’s then an easy task to hand-configure the result to your heart’s content.

I decided that I need to sort out the way I do my personal calendaring.

Currently I only use my phone’s built-in calendar. I nearly always have my phone with me, but it’s a bit of a pain to enter stuff on when I’m sat at a computer anyway, and carrying all that information solely on my phone presents a huge risk of loss, theft or breakage.

I need some kind of centralised store of information that is able to sync with all the devices and programs I want to use, namely:

• Some sort of cross-platform calendar client – mainly for use on Linux but also nice to be able to use similar software if I’m on Windows or OS X.
• Sony-Ericsson P1i (Symbian) built-in calendar
• iPhone, for when I get one
• Web interface, for those times when I’m borrowing a computer and can’t install a client.

Google Calendar seems to be a good choice. It’s flexible and can sync with lots of things.

Linux

So I installed Lightning on all my Fedora and Ubuntu machines. It’s a calendar extension for Thunderbird, which I already use. To install it yourself:

On Fedora:
yum install thunderbird-lightning
On Ubuntu:
apt-get thunderbird-lightning

It’s easy to set up, too. Suppose your Google account is joebloggs@gmail.com, then you would…

• Add a new calendar to Lightning by right-clicking in the Calendar area
• Choose On the Network
• Select CalDAV
• Enter your location as https://www.google.com/calendar/dav/joebloggs@gmail.com/events
• Give the calendar a name

OS X and Windows

It’s a little more work to install Lightning on OS X. You have to download the add-on from Mozilla, and install it in Thunderbird. Same story for Windows.

It’s quite straightforward and there are instructions on the website.

When you’re done, follow the same instructions as for Linux to subscribe to your Google calendar in Lightning.

Sony Ericsson UIQ

Setting up Google Calendar on my Sony Ericsson P1i was a bit of a pain, too. The P1i can’t interact with Google natively, I had to set up an account with Goosync to enable this. Goosync talks to Google, and your phone talks to Goosync using SyncML.

But once you have a Goosync account, you can synchronise a lot of handsets with Google calendar.

So first, you will need to set up an account with Goosync. It’s free and very easy. Goosync will ask you to tie your Goosync account to your Google account.

There’s also an option to have the settings for your phone sent automatically to your handset. However this didn’t work for me so I had to enter the settings manually.

Assuming the sync task on your phone has been set up properly, do a  test run to make sure it all works.

• If possible, connect to a wireless network first. If not, 3G will have to do.
• Go to the Main Menu
• Go to Tools
• Go to Remote Sync
• Find the profile that syncs with Goosync
• Find the sync task called Calendar. Make sure it is ticked, and then tap Sync to start off the first synchronisation.

If that worked, you can now run the sync task whenever you like from within the calendar itself.

• Open your phone calendar
• Tap More
• Tap Calendar manager
• Tap Synchronise

That’s all there is to it! Unfortunately there’s no way of making your calendar synchronise automatically at set intervals, but that’s probably a good thing, because you can’t get stung for 3G charges!

iPhone and iPod touch

Coming soon…

This a short and simple guide, explaining how to set up remote monitoring of Linux hosts using NRPE in Nagios. The procedure is simple, but having searched for information on this earlier today I didn’t find a straightforward all-inclusive guide, so I’ve written my own.

These instructions were written with Nagios 3.0.6, and they assume that you already have a working Nagios monitoring server. They assume that the monitoring server was installed from RPM, not from source (some paths will vary).

Configuring the remote server

First, we install the NRPE on the remote server to be monitored. This comes as standard in the Fedora repositories, but on CentOS you’ll need to add the EPEL repository first.

yum install nrpe

We’ll need to make one or two changes to get it working. First open up /etc/nagios/nrpe.cfg and find the allowed_hosts directive. Replace it with the IP address of your Nagios monitoring server.

allowed_hosts=123.123.123.123

Edit your /etc/sysconfig/iptables and add a line to allow port 5666/TCP from the monitoring server’s IP address.

-A INPUT -m tcp -p tcp -s 123.123.123.123--dport 5666 -j ACCEPT

Finally, restart iptables and start NRPE to get it working. We also tell NRPE to start on boot.

service iptables restart
service nrpe start
chkconfig nrpe on

Configuring the Nagios server

Edit your commands.cfg (usually in /etc/nagios/objects/ if you installed from RPM) and add the following command definition:

define command{
        command_name    check_nrpe
        command_line    $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
        }

If this is your first remote Linux host to monitor, create a new host definition file in the same directory as commands.cfg, e.g. linux.cfg. Make a host definition for your new server:

define host{
        use                     linux-server
        host_name               myserver
        alias                   My Server
        address                 234.234.234.234
        }

Add the following to it as a test to show it works:

define service{
        use                         generic-service
        host_name                   myserver
        service_description         PING
        check_command               check_ping!100.0,20%!500.0,60%
        }

define service{
        use                         generic-service
        host_name                   yourserver
        service_description         Load
        check_command               check_nrpe!check_load
        }

Restart Nagios and ensure that both tests work OK. If so, we can move on to some custom test.

Custom checks

The default NRPE client comes with a handful of built-in tests. You can see these near the bottom of nrpe.cfg on your remote machine. But they’re not very exciting, and you’ll probably want to use some of the other checks. If you want to see a list of the available checks in your yum repo, try this:

yum list available nagios-plugins-*

Install any that take your fancy. You’ll need to set up a definition for them in your nrpe.cfg. Use the examples in the file, and try running the Nagios plugin yourself to see if it gives you any clues about the arguments it wants.

Please note, in the default config of NRPE, you cannot use placeholders like $ARG1$, for security reasons. Either hardcode the values in, like

command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1

or enable dont_blame_nrpe=1 further up in the file. There is a security risk associated with doing this. Your funeral!

Restart NRPE again, and let’s move on to setting up your Nagios server. There is no need to create a new command definition, since we are using NRPE again. So open up linux.cfg and let’s add a service definition for the check_hda1 that exists in nrpe.cfg.

define service{
        use                             generic-service
        host_name                       myserver
        service_description             Disk status
        check_command                   check_nrpe!check_hda1
        }

Restart Nagios again and your new checks should appear. Go ahead and install any useful plugins from your yum repository, or have a look at Monitoring Exchange, a great source of free Nagios plugins.

I wrote my own plugins for monitoring your account balance with AQL and checking for the latest installed kernel. One day I will probably get round to uploading them to Monitoring Exchange.

I’ve just written a module for Nagios that will determine if the currently running kernel is the latest kernel available on the system. It will not tell you if there is a newer kernel in a yum repository or similar.

The main gotcha is that you need an RPM-based system for my script to work, e.g. RHEL, CentOS, Fedora and many others. It is most certainly not bulletproof, but it works on my systems.

All feedback welcome.

N.B. I’ve now published this module on Monitoring Exchange. Please download the plugin from there, as I will keep that copy up to date if there are changes in the future (and the copy on this page is likely to go out of date).

check_kernel

#!/usr/bin/perl -w

# Usage:   check_kernel

use strict;
use lib "/usr/local/nagios/libexec";
use utils qw(%ERRORS);

my $running_kernel=`uname -r`;
my $installed_kernel=`rpm -q kernel | tail -n 1`;
my $rpm = `which rpm`;

chomp $running_kernel;
chomp $installed_kernel;

if ($rpm =~ m/no rpm in/i) {
   print "UNKNOWN - You must be running an RPM-based system\n";
   exit $ERRORS{'UNKNOWN'};
}

if (!defined $running_kernel || !defined $installed_kernel) {
   print "UNKNOWN - Test failed\n";
   exit $ERRORS{'UNKNOWN'};
}

# Strip off the "kernel-" prefix so the strings will match
$installed_kernel =~ s/kernel-//gi;

# Do the test
if ($running_kernel eq $installed_kernel) {
   print "OK - running latest installed kernel ($running_kernel)\n";
   exit $ERRORS{'OK'};
} else {
   print "WARNING - reboot to run latest installed kernel ($installed_kernel)\n";
   exit $ERRORS{'WARNING'};
}