Small Daemons
My home network consists of two Linksys WRT54G wireless routers, a Linksys NSLU2 network storage device, a Mac Mini running MythTV, and laptops for each member of the house. All but one of these devices use 802.11 wireless networking.
One of the WRT54G routers lives in my bedroom, connected to the DSL modem there. This router runs the wonderful OpenWRT firmware, providing me a good old fashioned Linux command line with which to administer the router. I use iptables rules to enact network address translation, allowing all the computers in my house to have access to the internet.
The NSLU2 is the only device in my network that lacks built-in wireless networking. I could purchase a USB wireless adapter, but these aren't as cheap as I'd like, and the NSLU2 only has two USB ports, both of which I would prefer to use to connect to USB hard drives (though currently only one is in use). The NSLU2 instead connects to my other WRT54G, which in turn is running the dd-wrt firmware. This router acts solely as a wireless bridge, allowing wired-only devices to access my network and the internet, and provides no other services. I run Debian on the NSLU2, giving me a fully featured operating system on an itty bitty platform.
All of these Linksys products run Linux, and can be classified as "embedded systems" because they have very specific limitations with respect to storage and memory. As such, certain tradeoffs need to be made. The WRT54G routers have 16 and 8 megabytes of RAM, respectively, and they run the entire system in memory so there's no room left for extraneous services or wasted space. The routers, for example, all use Busybox to provide basic UNIX utilities without using a lot of space. The default installation of OpenWRT provides the Dropbear SSH daemon, which is a substantially smaller package than the more traditional OpenSSH. Dropbear works great, though, and provides everything I need to connect to and administer the routers.
The NSLU2 isn't as limited when it comes to storage, since I can (and do) use a USB hard drive to store an entire Debian installation; and it has (by comparison) a whopping 32 megabytes of RAM. Still, I need to be careful about memory usage so as not to cause the system to thrash swap space or have the kernel OOM killer forcibly terminate services. Taking a cue from the OpenWRT installation, I replaced the standard OpenSSH server on the NSLU2 with Dropbear, in order to free up more RAM. I also installed dash instead of bash as my primary shell, in order to eek out a little extra memory. dash lacks all of the handy features I like from bash, but I use the NSLU2 for pretty specific backup purposes so I don't miss the features all that much. I disabled all the unnecessary (for my use) services, like nfs-common, portmap, openbsd-inetd, etc. I replaced Exim4 with SSMTP. Finally, I installed thttpd. My intent with this is to use the NSLU2 to store large media files I might like to link from my blog without having to worry about bandwidth usage on my blog's server: serving the files from my home DSL line has no bandwidth cap, whereas the VPS hosting my blog does have a bandwidth cap.
For the past little while I've been contemplating OpenVPN on one or more device in my network. I find myself growing more and more uncomfortable using unencrypted wireless networks at coffee shops and hotels, and would like to be able to tunnel my connections through a secured channel to a host under my control. I've had tremendous success with OpenVPN at my previous job, and I advocate it all the time to people looking for VPN solutions. Its set up, while not hard, is non-trivial, and truthfully I worried about the resource drain the OpenVPN daemon might put on my limited resource Linksys devices.
During a conversation with Matt in #habari, I provided the link to using SSH as a SOCKS proxy. Duh! There was the answer I was looking for! So last night, I installed the full-blown OpenSSH daemon onto one of my WRT54G routers. I configured it to listen on TCP port 443, so that it wouldn't interfere with the currently running dropbear (should something have gone wrong, I still wanted to be able to access the system!) and because the proxied traffic will largely look like requests to an HTTPS website to the casual eye (and most firewalls). The two applications I most frequently use (Firefox and XChat) both support SOCKS proxies, so I was quickly in business. I defined the SOCKS proxy in XChat, established an ssh connection to the router, and watched as XChat reported
* Looking up irc.freenode.net
* Looking up localhost
* Connecting to localhost (127.0.0.1) port 2600...
* Connected. Now logging in...
I installed the FoxyProxy Firefox extension, and added my localhost proxy to its configuration. It was hard to test with certainty from home, but this morning at work there was no doubt: after selecting the localhost proxy from the FoxyProxy menu, I visited WhatIsMyIP and saw my home IP address displayed! Success!
Not every application I'll ever use supports SOCKS, so this might not be as robust a long-term solution as OpenVPN would have been, but it allows me to minimize the number of running services while re-using familiar applications.
Offsite Backup
Rick contacted me recently to propose a mutually beneficially technical configuration: use one another's NSLU2 network storage devices as an offsite backup solution. That is, Rick will send his backups to me, and I'll send my backups to him. I've been exceedingly lax in my backup strategy of late, so this seemed like an excellent opportunity to remedy that situation.
I'm running Debian on my slug, and Rick is running Unslung. We created accounts for one another on our respective slugs, configured port forwarding on our firewalls, and yesterday I sat down with Rick to help him get ssh keys working for passwordless logins. Rick sent over a few dozen photos for storage on my slug, and it all worked perfectly.
When I got home, I set to configuring my backups. My goal was a little more complex, because I want to backup my web / mail server, in addition to my desktop data. I created a daily cron job on my server to dump all the MySQL databases, gzip compress them, and then encrypt them using GnuPG. The script first empties out my backup directory, removing any previous backups, and then dumps all of the above into the backup directory. I created a weekly cron job to archive all my users' mail spools, tar and gzip them, and finally encrypt them with GnuPG. Since the weekly cron task executes after the daily, the mail backups will coexist with the database backups for one day, and will be purged the following morning.
On my slug, I created a cron job to run at 8:15 AM every day (after both the normal daily and weekly tasks execute), in order to make sure that the web server had finished all of its archive tasks. The slug uses scp to grab *.gpg from the backup directory on the server, and stores them in the local backup directory. It then uses the find command to identify files in the backup directory that have not been modified in more than 30 days. These are deleted. Finally, the script invokes rsync to synchronize my slug's backup directory with the backup directory on Rick's slug. Files that were deleted on my slug (due to age) should be deleted from Rick's slug, and new files should be transferred over.
In this way, I have a month's worth of daily database backups, and weekly snapshots of mail, stored in my house, and also stored offsite at Rick's house. Everything is GPG encrypted, so if anyone robs my house (or Rick's house), I won't have to worry too much about any sensitive data that might be contained in the database dumps or in email messages.
While preparing these backup scripts, I was reminded of OfflineIMAP, a utility used to synchronize IMAP folders between multiple computers. It's a pretty interesting package, and it might be a great way to keep more frequent backups of all the mail on my server. I don't know if it offers anything unique over just a plain ol' rsync of my server's maildir, but it's another tool in the toolbox. One of the neat tricks you can do with OfflineIMAP is to use ssh to prepare a "preauth tunnel", so that you don't need to use your IMAP password at all (relying instead on key-based ssh logins) -- that's pretty clever! I am, however, a little leery about keeping an unencrypted copy of all our mail on the slug locally, but I can't adequately articulate why: if it's acceptable to keep the mail on the VPS, on hardware outside my physical control, why should I be worried about a duplicate copy stored on a hard drive in my house? If you're using OfflineIMAP, do please let me know what you think in the comments.
What backup strategies are you using? How have they worked for you so far?
