• School: Last semester, just two classes, but they are looking like they'll be delicious and meaty
  • CS 411: Compiler design: We build a java compiler
  • CS 415: We build an operating system, fun times with C!
  • ...actually, I'm also taking spanish!
• "Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp" by Peter Norvig. I got this for christmas and have started to work through it. I'm really excited about getting further into it. In the middle of it I'll be implementing prolog in Lisp and the in the latter half I'll be working on a natural language processor! Not to mention tons of other stuff, this book is huge and dense.
• BattleCode 2010 has just started so my team and I are just about to start digging into that. Lots of AI coding to be done there.
• Peter Michaux's Scheme from Scratch. I stumbled upon this from Hacker News. This fellow wants to write his own scheme to scratch a mental itch, and he's blogging each step and posting the code as well. I think it looks like a great amount of fun and that I too have that mental itch, so I'm following along, using his blog as a guide and looking at his code as well when I get stuck, but doing my best to do it myself.
• The great mindstab.net migration to the cloud! Yes, setting up an entirely new server and migrating years of site history and email etc can take a lot of work.

So yeah, I have an insane amount of work on my plate, but I couldn't be more excited! All of it is thrilling and amazing!

Also, if I haven't mentioned it before, the dynamic duo of Jono Bacon and Stuart Langridge of Lug Radio fame are back with a new podcast Shot of Jaq! It's fun. Really, those two Brits have been the source of the only podcast's I've ever listened to. They are a great source of both Linux and British in my weekly diet.

But now I'm stoked to just find out that Ximian/Linux rockstar coder Nat Friedman and Tomboy creator and a rock star in his own right Alex Graveley have started a brand new podcast Hacker Medley that is the first new podcast that I'm actually quite excited to try out.

Finally, I'm reading "Pattern Recognition" by William Gibson in my spare time (read: on the bus) and finding it pleasant.

Mindstab.net's been around. It started back in January of 2002 running on a 486 from my bedroom. A few years later when I had some money, I bought a trusty 1U rackmount server, kvasir, and migrated mindstab there. Kvasir started co-located but when money dried up some, I brought kvasir home and mindstab has been living in my room for the last several years.

That all ended yesterday morning at 3:30am when I was awoken by kvasir's PSU fan screaming to death. The writing had been on the wall for a bit that things were going to have to change. I knew the fan was on its last legs, and hosting from home had been getting ever more precarious as both the local ISPs seem unable to provide a stable and reliable net connection. To that end I'd been playing with a VPS at linode.com (recommended in a blog post by Brian Carper) but I hadn't gotten very far. Actually, after a month I was still poking at the kernel trying to get my general paranoid level of security with grsecurity and PaX to work with the Xen virtual host at Linode. Anyways, at 3:30am yesterday I went into emergency mode and acceptable that their stock kernel "might just suffice" and started madly installing infrastructure software. Since then I've been copying over data and starting to bring services back up. So far we have a firewall, DNS, Jabber and I'm just starting to get the websites back up, this being the first.

Lots of work ahead. I need to get the rest of the sites up. Then I have to dive into the mess that is e-mail. I'd previously been using a qmail/vpopmail/courier system but I think I'm chucking that for a postfix/dovecot system, but I'm sure it'll take some kicking to get it up and get all my mail into it. Hopefully not too much. Then onto less crucial systems, then go back and re-vet alot of things.

So yeah, as of now, mindstab.net is coming to you from my new server Loki, in the cloud... or New Jersey more specifically :P

First things first, the files I want to backup are owned by all different users, so the only user who can run the backup process is root. Therefore I can't just run rsync from my local machine and grab the files from the server, the backup process has to be run on the server and backup to the backup machine. Now in my case this was a bit of a trick because the backup machine was behind a firewall, so the server had no direct line of communication to it. So I wrote a script to turn on a reverse ssh port forward.

recv-serv-backup.sh

 
#!/bin/sh
 
ssh -R 8000:127.0.0.1:22 -N user@kvasir.mindstab.net
 

When run on the backup machine behind a firewall, it connects to the server (kvasir) and listens on port 8000. When ssh on kvasir connects to port 8000 it redirects that traffic to local port 22, the ssh port of the backup machine. This is how the firewall is gotten around. Reverse port mapping is a cool trick to master.

Next as root on kvasir I generated a public ssh key and put it on the backup machine so root could automatically log on repeatedly to the back up machine (think lots of rsync calls) once the key was loaded once. Then I hooked up the key to keychain. That is all better outlined at:
Gentoo Linux Documentation: OpenSSH key management, Part 1 and
Gentoo Linux Documentation: OpenSSH key management, Part 2.

The all I needed to do was poke the rsync syntax to use the nonstandard ssh port for backup. The best method I found was

rsync -e "ssh -p 8000" -av

rsync for those of you who don't know is a great little backup tools. It's like a smart (in that it only copies files that have been modified since the last backup) network aware (since it can use ssh) copy tool. Simple but really useful.

So with that I wrote a backup script on the server

server-backup.sh

 
#!/bin/sh
 
backup () {
        echo "$1..."
        DST=`echo "$1" | awk '{split($0,a,"/");  result = "/"; for (i=2 ; i < length(a)  ; i++)   result = result "/" a[i];  print result;  };'`
        rsync -e "ssh -p 8000" -acv $1 dan@127.0.0.1:~/kvasir$DST ;
 
}
 
backup "/svn"
backup "/git"
...
 

There is one caveat, rsync won't create subdirectories on the other side specified in the path so you need to create the basic directory structure.

 rsync -e "ssh -p 8000" -av /git user@127.0.0.1:~/kvasir/git

is fine because it will create /git just fine, but

 rsync -e "ssh -p 8000" -av /home/user user@127.0.0.1:~/kvasir/home/user

will fail if ~/kvasir/home doesn't exist. So you'll need to create the basic directory structure or enhance the backup function to strip out extra directories in the target path.

Finally, I didn't want anyone and everyone to potentially be able to gain access to private data on the backup machine, so the target directory needed to be encrypted. There are a lot of options, but I opted for the easy encFS route and just installed "cryptkeeper" and had it setup the directory.

Now all I have to do is mount the encrypted backup directory, run the script to turn on the reverse ssh tunnel, and run the backup script, and I have an encrypted backup solution for my server that gets around firewalls.

Not bad.

References

• What Goes Out Can Come Back In -- SSH tricks
• Google

I hosted my server on a small business line with Telus. A little over 2 weeks ago my house fell off the internet. I called them and they said my modem must be old and broken so they'd send me a new one in 5 business days tops.

I waited the 5 days and called on the last after not receiving a new modem and they said it'd only be a day or two more tops. I called a few days later after still receiving no modem and now having been offline for a week and a half and they double checked and it turns out my new modem order hadn't even yet been processed! Even after a reminder call. Unbelievable. So I canceled my (lack of) service with them and registered with the other head of our local two headed ISP monopoly, Shaw. They had me online with in 4 days, and I just got my static IPs today. So mindstab.net is back online, as am I. Lets see how this goes.

The difference in comparison to other distros is of course that you are always current with their stable, but their stable can only stay current so long before they have to release a new version or else everything will break on some upgrades. In Gentoo they give you the tools to deal with it and pass the breakage onto you. So I upgrade slowly and cautiously and then fix a few things when config files change or libraries change and more things need to be recompiled, but it works. I installed Gentoo on this server 4 or 5 years ago and look, it's still going and running all new software. I think that's pretty cool.

And to top off the changes, I finally got around to installing a "new" 512MB ram stick in the server as well, now doubling its ram to 1GB. Which is cool, and just in time because the new clamav is eating RAM like candy. I'm actually wondering if something is wrong with it because it's really eating ram...

Ah well, anyways, the server got some well needed love and attention and is feeling much better.

Anyways, because I'm such a public fellow, it should be pretty easy to follow my progress. You can get up to the minute updates when I can make them directly from my twitter account at twitter.com/dan_ballard. Usually within half an hour twitter updates will then show up here on the site, plus I'll probably make the occasional update directly to the site when internet is available. So if you want all my travel stories etc, you can subscribe to the site RSS feed. If you're on LJ, of course all of this will still be pushed there, so fear not, you get the full Dan vacation experience.

For those of you who could care less about my travel exploits and are just interested in my Linux exploits or what not, I've first requested Planet Larry change my feed to a Linux only one for the duration of the trip and you can do the same. Try subscribing to mindstab.net/wordpress/archives/tag/linux/feed to get only Linux stuff.

Ok, wish me luck, my flight is at 7am Wednesday morning.

I've found documentation on the setup of git servers and public repositories kind of lacking, so here is my best attempt at documenting what works for me. Feel free to comment with bugs or enhancements please.

Contents

• 1. Setting Up A Local Repository
  • 1.1 From Scratch
  • 1.2 From An Existing Project
• 2. Setting Up A Remote Repository
  • 2.1 Remote Repository For Developer Only (ssh)
  • 2.2 Remote Repository For Public Access (git://)
  • 2.3 Shared Multi-Developer Public Repository
• 3. Managing Multiple Developers, Repositories and Branches
• References

1. Setting Up A Local Repository

Alice is going to start developing a project and she wants to add source control to it. There are a couple of reasons to set up a local repository that Alice likes including branch control, so that she can revert her code to previous releases, fix, patch or merge a bug fix, roll a release, and then pop back to the current development branch.

1.1 From Scratch

To set up a git repository for her project, Alice does the following:

alice@home $ mkdir proj
alice@home $ cd proj
alice@home $ git init

The project directory is now an empty git repository. As she creates files, she can add them to the respository with

alice@home $ git add newfile.src

And when she's done work or at least reached some break point, she can commit the new files, and all changes with

alice@home $ git commit -a

1.2 From An Existing Project

Also, occasionally Alice gets excited and starts coding before creating a repository. To create a repository from an already started project is as simple as

alice@home $ cd ~/proj
alice@home $ git init

and either

alice@home $ git add .

to add all the files, or

alice@home $ git add file1 file2 file3

to add just some of the files, both followed by

alice@home $ git commit -a

for the initial commit of the code to the new repository.

2. Setting Up A Remote Repository

Some times Alice needs her repositories to be remote and internet accessible. Sometimes she needs to work on them from several locations, and sometimes she wants her project's code to always be accessible to the public.

There are two primary methods for making remote git repositories accessible online. The first method is over ssh, which developers can use to both read and write to the repository and the second is through a dedicated git server which the public can use for read only access.

2.1 Remote Repository For Developer Only (ssh)

If Alice's project is personal and she just needs a central repository to access from a few locations like both work and home, she can set up a repository on any unix machine she has access to as follows.

Alice needs to create a bare repository clone of her working code and then transfer it to the server she will be using as the repository host

alice@home $ git clone --bare ~/proj proj.git
alice@home $ tar -czf proj.git.tar.gz proj.git
alice@home $ scp proj.git.tar.gz alice@server.com:~

Then, on the server

alice@server $ tar -xzf proj.git.tar.gz
alice@server $ mv proj.git proj

Now Alice can create working copies of the repository from anywhere, like work, and work on the code as normal as follows

alice@work $ git clone ssh://alice@server.com/home/alice/proj
alice@work $ cd proj
...
alice@work $ commit -a

However all this does is create a local clone of the repository and commit the changes to the new local clone. To push changes to the local repository back to the central repository, Alice does

alice@work $ git push

(As a note, Alice will also need to perform this clone of the remote repository at home so that her repository is aware of the remote repository, or she can use 'git remote add' to make her current original repository aware of the remote one)

When Alice gets home she can check out the latest changes with a simple

alice@home $ git pull

which pulls all the latest changes from the remote repository. Then she can develop, commit and push her changes and then the next day at work she can pull all those changes.

2.2 Remote Repository For Public Access (git://)

Now, to allow public read only access of the repository over the git:// protocol the steps of setting up a remote repository are all the same, however there are additional steps that need to be taken.

At a minimum, Alice needs to setup the git daemon on the server and tell each git repository that she wants to be publically accessible that it is so.

Setting up the a basic git daemon is up to Alice and her server's distribution, but once it is installed and running, it will try to export any directory on the server filesystem that is a) a git repository, and is b) flagged to be publically accessible.

To make her repositories accessible, Alice does the following

alice@server $ touch ~/proj/git-daemon-export-ok

Now when Bob hears about Alice's project, he can check out a copy of the repository himself as follows

bob@home $ git clone git://server.com/home/alice/proj

Bob actually ends up with the a full clone of the repository and can work with the code, and if he wants he can make changes and commit them to his local clone of the repository as normal. However, the one thing Bob cannot do is 'push' his changes back to the central repository.

He can, however, even stay up to date with the repository with git pull

bob@home $ git pull

and he'll always get the latest changes.

2.3 Shared Multi-Developer Public Repository

(Note: This is for those more used to CVS and Subversion style source control. Defacto and "proper" git style is outlined in section 3. Managing Multiple Developers, Repositories and Branches.)

Alice happens to have root access to her server and wants to set up a multiple developer git repository.

First she creates a git user group and makes a root git directory.

root@server # groupadd git
root@server # mkdir /git

Then Alice configures the git daemon to only export repositories in /git in the git-daemon's config file

GITDAEMON_OPTS="--syslog --verbose /git"

Now Alice creates a shared repository. She untars the git repository like normal, but sets its group to git and makes sure it'll stick by setting the stick bit, and then she makes it "shared" which means all the files are writeable by the group git.

root@server # cd /git
root@server # tar -xzf proj.git.tar.gz
root@server # mv proj.git proj
root@server # chgrp -R git proj
root@server # chmod g+ws proj -R
root@server # cd proj
root@server # git config core.sharedRepository true

And of course if Alice wants it to be publically viewable

root@server # touch git-daemon-export-ok

Now Alice has a git repository that several developers on the server can all use. Anyone in the git group can commit to the repository.

Alice's friend Charlie wants to develop for the project so Alice gives him an account on the server. Charlie can then start developing just like normal

charlie@home $ git clone ssh://charlie@server.com/git/proj
charlie@home $ cd proj
...
charlie@home $ git commit -a
charlie@home $ git push

Alice can get these changes at home, and any she's made from work with a simple

alice@home $ git pull

And if the repository was made public and exportable then Bob can checkout the code and keep up to date too

bob@home $ git clone git://server.com:/git/proj
bob@home $ cd proj
...
bob@home $ git pull

3. Managing Multiple Developers, Repositories and Branches

The proper way to use git with multiple developers is for each developer to have their own repository and branches and have a central manager who pulls from all the other branches and merges the code together before release. This is how Linux works (git was created by Linux's creator Linus).

Note: I know this is the proper way but I haven't really had any experience with it, so until I get time to play with it unfortunately this part of the document will be empty. Check out the official git manual for a good idea of how this should be managed, especially chapter 4 Sharing Development.

References

• Git User's Manual (for version 1.5.3 or newer)
• Setting up your Git repositories for open source projects at GitHub
• Google

It's as simple as this

rsync -a -e ssh Documents haplo@kvasir.mindstab.net:~/backup

And then the reverse to get new stuff.

The full article is at www.enterprisenetworkingplanet.com/netos/article.php/10951_1573881_2. I'm doing this with a bunch of my data now! I'm excited because I'd been thinking about setting up some backup solution for a bit but this is so trivially easy and more than good enough it's delightful.