Mindstab.net

So after being alterted to the existence of DKIM by this article posted on HackerNews I wanted to implement it immediatly on my server. DKIM is Domain Keys for Identified Mail, a crypo signing protocol where a pub key sits in your DNS and your mail servers sign your mail as it passes through your server. Seems a little stronger than SPF from a few years ago for authenticating mail’s origin so I was keen to adopt it.

So I found the freshest instructions on the Gentoo wiki and followed them. They were a bit spartan so I went looking for a bit more material and found this Ubuntu tutorial which had some helpful suggestions like the testing section.

After giving the OpenDKIM instructions a first run through I gave the testing a try.

First using dkimcore.org/tools/ I found that the Gentoo OpenDKIM config tool had spat out invalid TXT. It had spat out

v=DKIM1;=rsa; p=MIGfM......

And after some quick internet consultation I found out I needed to fix it to

v=DKIM1; k=rsa; p=MIGfM.....

The second test from the Ubuntu docs was an auto-respond test email system that along with wikipedia I learned about ADSP from. So I added

_adsp._domainkey.mindstab.net. IN TXT "dkim=discardable"

to my Bind config as well. (I’m still not 100% about the final ‘.’). Also it seems the autoresponder email tool doesn’t update its DNS too often so I may have to wait a bit to retest.

So now it seems I should have DKIM signed/valid email! :) Just another step to make sure my email is valid, slightly less spoofable and liked/accepted by the big email providers.

Also, seeing results like this from Gmail after receiving my email seems good:

Received-SPF: pass (google.com: domain of dan@mindstab.net designates 69.164.214.81 as permitted sender) client-ip=69.164.214.81;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan@mindstab.net designates 69.164.214.81 as permitted sender) 
  smtp.mail=dan@mindstab.net; dkim=pass header.i=@mindstab.net

Just a note: I’ve now been running mindstab.net for 10 years. It’s been a long ride since I started on a 486 in my bedroom, but there’s lots more to come! Just thought I’d take a moment to mark the occasion.

A while ago I tried with mixed success to get Lisp onto my Gentoo Hardened server. I had to go a binary only route and kind of stopped there not taking it any farther. Now, 2 years later, I need the full meal deal, lisp + emacs + slime, on my server, which is now a Xen VPS with as much hardening as I could get (much less kernel based hardening since it’s the VPS’s kernel). It was still too much for SBCL to compile in portage so here’s what I did to get it all working.

So you need an out of tree binary copy of SBCL. Live with it. It works. The problem with going with out of tree software, especially for a language, is that what ever binary you get isn’t supported and hasn’t been tested against all the software in-tree. For instance I initially tried the newest version of SBCL (1.0.42) but ran into problems with portage’s stable slime.
Ultimately I went with the closest I could get to portage’s stable version. Portage has 1.0.19 marked as the most recent stable version so I went out and downloaded the binary of that version

$ wget  http://sourceforge.net/projects/sbcl/files/sbcl/1.0.19/sbcl-1.0.19-x86-linux-binary.tar.bz2/download
$ tar -xjf sbcl-1.0.19-x86-linux-binary.tar.bz2

So change into the directory and check out INSTALL. Basically installation is easy. Binary SBCL is configured around installing into /usr/local but that can be gotten around. So we’ll go with a more traditional install into /usr

Note: My test box is a VPS with a Xen kernel not a hardened kernel so I didn’t have any PaX problems, but my notes for the last time I tired this on a full hardened install mention that you need do disable some PaX features before SBCL will work:

$ paxctl -p -e -m -r -x -s " on src/runtime/sbcl

Install to /usr

# INSTALL_ROOT=/usr sh install.sh

Now SBCL is installed but it won’t work because the binary is preconfigured to look for the core in /usr/local. So we’ll borrow the gentoo SBCL config files to get that setup properly.

/etc/env.d/50sbcl

SBCL_HOME=/usr/lib/sbcl
SBCL_SOURCE_ROOT=/usr/lib/sbcl/src

# env-update

The above file and command set up the system environment variables to tell SBCL where it’s really installed. Now is as good a time as and to ‘source /etc/profile‘ to get those changes.

Now SBCL is installed and working, we need to let portage know that. There used to be a ‘emerge –inject’ method, but that’s been deprecated in place of a new provides file

/etc/portage/profile/package.provided

dev-lisp/sbcl-1.0.19

Now portage knows about our SBCL so we can start installing things that depend on it like the rest of our tool chain

# emerge cl-asdf emacs slime -va

So now we have all the pieces, all they need is some gluing together. Again we’ll borrow from the Gentoo SBCL files.

/etc/sbclrc

;;; The following is required if you want source location functions to
;;; work in SLIME, for example.
 
(setf (logical-pathname-translations "SYS")
    '(("SYS:SRC;**;*.*.*" #p"/usr/$(get_libdir)/sbcl/src/**/*.*")
          ("SYS:CONTRIB;**;*.*.*" #p"/usr/$(get_libdir)/sbcl/**/*.*")))
 
;;; Setup ASDF
(load "/etc/gentoo-init.lisp")

/etc/gentoo-init.lisp

(in-package #:cl-user)
#+(or sbcl ecl) (require :asdf)
#-(or sbcl ecl) (load #p"/usr/share/common-lisp/source/asdf/asdf.lisp")
(push #p"/usr/share/common-lisp/systems/" asdf:*central-registry*)
(asdf:oos 'asdf:load-op :asdf-binary-locations)
(setf asdf:*centralize-lisp-binaries* t)
(setf asdf:*source-to-target-mappings* '((#p"/usr/lib/sbcl/" nil) (#p"/usr/lib64/sbcl/" nil)))

Now everything should work. You just need to set up your emacs and slime

~/.emacs

; your SLIME directory
(add-to-list 'load-path "/usr/share/emacs/site-lisp/slime/")
; your Lisp system
(setq inferior-lisp-program "/usr/bin/sbcl")
(require 'slime)
(slime-setup)
 
(global-set-key (kbd "C-c C-q") 'slime-close-all-parens-in-sexp)

Now It’s all glued together, give it a go

$ emacs
M-x slime

If you don’t get any compilation errors you should be in emacs + slime.

And there you have it, SBCL emacs and slime on Gentoo Hardened.

Cavets

1) For some reason this approach adds some annoying extra text to vanilla SBCL start up that I can’t seem to get rid of

$ sbcl
This is SBCL 1.0.19, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http: //www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
; loading system definition from
; /usr/share/common-lisp/systems/asdf-binary-locations.asd into
; #<package "ASDF0">
; registering #<system ASDF-BINARY-LOCATIONS {AAF8F51}> as ASDF-BINARY-LOCATIONS
* 

2) The system I tested this on is a VPS so the kernel is a Xen kernel, not a hardened kernel, so there may be additional complications on a full hardened install. Please let me know if you have any, and especially any working solutions.

So, what have I got on the go?

• School: Last semester, just two classes, but they are looking like they’ll be delicious and meaty
  • CS 411: Compiler design: We build a java compiler
  • CS 415: We build an operating system, fun times with C!
  • …actually, I’m also taking spanish!
• “Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp” by Peter Norvig. I got this for christmas and have started to work through it. I’m really excited about getting further into it. In the middle of it I’ll be implementing prolog in Lisp and the in the latter half I’ll be working on a natural language processor! Not to mention tons of other stuff, this book is huge and dense.
• BattleCode 2010 has just started so my team and I are just about to start digging into that. Lots of AI coding to be done there.
• Peter Michaux’s Scheme from Scratch. I stumbled upon this from Hacker News. This fellow wants to write his own scheme to scratch a mental itch, and he’s blogging each step and posting the code as well. I think it looks like a great amount of fun and that I too have that mental itch, so I’m following along, using his blog as a guide and looking at his code as well when I get stuck, but doing my best to do it myself.
• The great mindstab.net migration to the cloud! Yes, setting up an entirely new server and migrating years of site history and email etc can take a lot of work.

So yeah, I have an insane amount of work on my plate, but I couldn’t be more excited! All of it is thrilling and amazing!

Also, if I haven’t mentioned it before, the dynamic duo of Jono Bacon and Stuart Langridge of Lug Radio fame are back with a new podcast Shot of Jaq! It’s fun. Really, those two Brits have been the source of the only podcast’s I’ve ever listened to. They are a great source of both Linux and British in my weekly diet.

But now I’m stoked to just find out that Ximian/Linux rockstar coder Nat Friedman and Tomboy creator and a rock star in his own right Alex Graveley have started a brand new podcast Hacker Medley that is the first new podcast that I’m actually quite excited to try out.

Finally, I’m reading “Pattern Recognition” by William Gibson in my spare time (read: on the bus) and finding it pleasant.

… and we’re back! Just in time for our 8th birthday.

Mindstab.net’s been around. It started back in January of 2002 running on a 486 from my bedroom. A few years later when I had some money, I bought a trusty 1U rackmount server, kvasir, and migrated mindstab there. Kvasir started co-located but when money dried up some, I brought kvasir home and mindstab has been living in my room for the last several years.

That all ended yesterday morning at 3:30am when I was awoken by kvasir’s PSU fan screaming to death. The writing had been on the wall for a bit that things were going to have to change. I knew the fan was on its last legs, and hosting from home had been getting ever more precarious as both the local ISPs seem unable to provide a stable and reliable net connection. To that end I’d been playing with a VPS at linode.com (recommended in a blog post by Brian Carper) but I hadn’t gotten very far. Actually, after a month I was still poking at the kernel trying to get my general paranoid level of security with grsecurity and PaX to work with the Xen virtual host at Linode. Anyways, at 3:30am yesterday I went into emergency mode and acceptable that their stock kernel “might just suffice” and started madly installing infrastructure software. Since then I’ve been copying over data and starting to bring services back up. So far we have a firewall, DNS, Jabber and I’m just starting to get the websites back up, this being the first.

Lots of work ahead. I need to get the rest of the sites up. Then I have to dive into the mess that is e-mail. I’d previously been using a qmail/vpopmail/courier system but I think I’m chucking that for a postfix/dovecot system, but I’m sure it’ll take some kicking to get it up and get all my mail into it. Hopefully not too much. Then onto less crucial systems, then go back and re-vet alot of things.

So yeah, as of now, mindstab.net is coming to you from my new server Loki, in the cloud… or New Jersey more specifically :P

I decided I really needed to work on a server backup system, so here are my notes on the system I have now.

First things first, the files I want to backup are owned by all different users, so the only user who can run the backup process is root. Therefore I can’t just run rsync from my local machine and grab the files from the server, the backup process has to be run on the server and backup to the backup machine. Now in my case this was a bit of a trick because the backup machine was behind a firewall, so the server had no direct line of communication to it. So I wrote a script to turn on a reverse ssh port forward.

recv-serv-backup.sh

#!/bin/sh
 
ssh -R 8000:127.0.0.1:22 -N user@kvasir.mindstab.net

When run on the backup machine behind a firewall, it connects to the server (kvasir) and listens on port 8000. When ssh on kvasir connects to port 8000 it redirects that traffic to local port 22, the ssh port of the backup machine. This is how the firewall is gotten around. Reverse port mapping is a cool trick to master.

Next as root on kvasir I generated a public ssh key and put it on the backup machine so root could automatically log on repeatedly to the back up machine (think lots of rsync calls) once the key was loaded once. Then I hooked up the key to keychain. That is all better outlined at:
Gentoo Linux Documentation: OpenSSH key management, Part 1 and
Gentoo Linux Documentation: OpenSSH key management, Part 2.

The all I needed to do was poke the rsync syntax to use the nonstandard ssh port for backup. The best method I found was

rsync -e "ssh -p 8000" -av 

rsync for those of you who don’t know is a great little backup tools. It’s like a smart (in that it only copies files that have been modified since the last backup) network aware (since it can use ssh) copy tool. Simple but really useful.

So with that I wrote a backup script on the server

server-backup.sh

#!/bin/sh
 
backup () {
        echo "$1..."
        DST=`echo "$1" | awk '{split($0,a,"/");  result = "/"; for (i=2 ; i < length(a)  ; i++)   result = result "/" a[i];  print result;  };'`
        rsync -e "ssh -p 8000" -acv $1 dan@127.0.0.1:~/kvasir$DST ;
 
}
 
backup "/svn"
backup "/git"
...

There is one caveat, rsync won’t create subdirectories on the other side specified in the path so you need to create the basic directory structure.

 rsync -e "ssh -p 8000" -av /git user@127.0.0.1:~/kvasir/git 

is fine because it will create /git just fine, but

 rsync -e "ssh -p 8000" -av /home/user user@127.0.0.1:~/kvasir/home/user 

will fail if ~/kvasir/home doesn't exist. So you'll need to create the basic directory structure or enhance the backup function to strip out extra directories in the target path.

Finally, I didn't want anyone and everyone to potentially be able to gain access to private data on the backup machine, so the target directory needed to be encrypted. There are a lot of options, but I opted for the easy encFS route and just installed "cryptkeeper" and had it setup the directory.

Now all I have to do is mount the encrypted backup directory, run the script to turn on the reverse ssh tunnel, and run the backup script, and I have an encrypted backup solution for my server that gets around firewalls.

Not bad.

References

• What Goes Out Can Come Back In -- SSH tricks
• Google

Mindstab.net returns! Let me explain what happened:

I hosted my server on a small business line with Telus. A little over 2 weeks ago my house fell off the internet. I called them and they said my modem must be old and broken so they’d send me a new one in 5 business days tops.

I waited the 5 days and called on the last after not receiving a new modem and they said it’d only be a day or two more tops. I called a few days later after still receiving no modem and now having been offline for a week and a half and they double checked and it turns out my new modem order hadn’t even yet been processed! Even after a reminder call. Unbelievable. So I canceled my (lack of) service with them and registered with the other head of our local two headed ISP monopoly, Shaw. They had me online with in 4 days, and I just got my static IPs today. So mindstab.net is back online, as am I. Lets see how this goes.

The internet connection for mindstab.net has become extremely unstable. We’re pretty sure it’s a crap ADSL modem, and a replacement is on the way.

I’ve finally gotten around to some long overdue server maintenance on Kvasir (mindstab.net et all). This is where running a Gentoo server can kind of be fun. I can mostly ignore it for ages, just poking at it when a GLSA (Gentoo security advisement) comes out for software it’s running, and then when I feel like sitting down to it, I can upgrade all the software it’s running to the latest stable versions.

The difference in comparison to other distros is of course that you are always current with their stable, but their stable can only stay current so long before they have to release a new version or else everything will break on some upgrades. In Gentoo they give you the tools to deal with it and pass the breakage onto you. So I upgrade slowly and cautiously and then fix a few things when config files change or libraries change and more things need to be recompiled, but it works. I installed Gentoo on this server 4 or 5 years ago and look, it’s still going and running all new software. I think that’s pretty cool.

And to top off the changes, I finally got around to installing a “new” 512MB ram stick in the server as well, now doubling its ram to 1GB. Which is cool, and just in time because the new clamav is eating RAM like candy. I’m actually wondering if something is wrong with it because it’s really eating ram…

Ah well, anyways, the server got some well needed love and attention and is feeling much better.

So, I’m leaving tomorrow morning for a month to Mexico. I’m excited and aprehensive as to be expected, but I think I’m in good shape (ie. packed etc).

Anyways, because I’m such a public fellow, it should be pretty easy to follow my progress. You can get up to the minute updates when I can make them directly from my twitter account at twitter.com/dan_ballard. Usually within half an hour twitter updates will then show up here on the site, plus I’ll probably make the occasional update directly to the site when internet is available. So if you want all my travel stories etc, you can subscribe to the site RSS feed. If you’re on LJ, of course all of this will still be pushed there, so fear not, you get the full Dan vacation experience.

For those of you who could care less about my travel exploits and are just interested in my Linux exploits or what not, I’ve first requested Planet Larry change my feed to a Linux only one for the duration of the trip and you can do the same. Try subscribing to mindstab.net/wordpress/archives/tag/linux/feed to get only Linux stuff.

Ok, wish me luck, my flight is at 7am Wednesday morning.