So following common advice to “fix a problem two ways to prevent it in the future” I fixed the DNS, but I also wanted an automated way to track my computers when and if their IPs changed.

So the first thing I needed was shared place to store the IP information. Thinking about it I realized that Dropbox would work well for that. So all I needed was a simple script.

So the solution was to put a script that determined the IP of the computer in Dropbox and have cron on all the computers run it. Each user can call cron with

$ crontab -e

And I created a crontab directory that I could add more scripts to later if need be with and run them hourly with the following entry

0 * * * * cd /home/dan && run-parts Dropbox/cron

The script itself was a file called getip and it used whatsmyip.com’s automation detection script.

getip

#!/bin/sh 
 
wget -O /tmp/`hostname`.ip http://automation.whatismyip.com/n09230945.asp
tmp_file=/tmp/`hostname`.ip
dst_file=Dropbox/var/log/`hostname`.ip
if ! diff -q ${tmp_file} ${dst_file} > /dev/null  ; then
  `cp ${tmp_file}  ${dst_file}`
fi

Then I just created Dropbox/var/log and installed the crontab on all my computers, and volia, homebrew IP tracking for all my compters accessible to me from anywhere.

After about 2 hours debugging we figured it all out. When the module is initialized, it pulls out all the config data, puts it in a hash, and hands it off to Drupal in the menu hook. Drupal stores it forever, and hands it off to the search function whenever it is called and so what ever language is active when the module is initialized determines what colleciton is used forever for all languages. Not cunning. The fix however was straight forward and one line: in the search function, just disregard the passed in settings value for the collection and re-pull it.

For what it’s worth I filed a bug and patch (against 6.x-2.0-beta1, though it should work fine for all versions) with the module at http://drupal.org/node/1208892 but it seems it’s suffering from low maintainership.

Here it is as well, though it’s probably just as easy to copy/paste the line in directly.

Index: google_appliance.module
===================================================================
--- google_appliance.module	(revision 14982)
+++ google_appliance.module	(working copy)
@@ -1055,6 +1055,9 @@
  *   Themed search results.
  */
 function google_appliance_search_view($search_base, $client, $collection, $keys = NULL, $title = NULL) {
+  
+  $collection =  trim(variable_get('google_appliance_default_collection', 'default_collection'));
+  
   $form = drupal_get_form('google_appliance_search_form', NULL, $search_base, $client, $collection, $keys);
   // When POSTing back to an existing search-results page, the original
   // URL is accessed (which re-runs that search) and then the redirect for

Now it works just fine and all our different language sites have searches and results in just their language. Splendid!

It turns out there are two ways flash players server videos these days. The first and easiest is that a simple flash player loads in your browser, and uses your browser to make a GET request to the server to load a .flv file (FLash Video). This is relatively easy to intercept, there are lots of tools and plugins for Firefox that do this automatically for you. Even better, on Linux for example, these videos are usually stored in /tmp so your browser does the whole job and gives them to you. No work required.

The other more complicated but more secure option is that the flash player connects to a dedicated rtmp server that streams flash video. The flash plugin does the networking and there is no file to save, it’s a stream.

If you are lucky enough to have a player using the first option, you are done. Assuming you have the second option, then your fun has just begun.

First we need to try and figure out where the server that your flash video is.

My first approach was to use wireshark to sniff the traffic. Through this I discovered the basics, like the address the server and the port, 1935.

Next I installed rtmpdump. RTMP is the Real Time Messaging Protocol and rtmpdump is a program that can connect to an RTMP server, get a stream and save it to a file. Sadly the data I got from wireshark didn’t have all the parameters I needed to get the file. Or I couldn’t read it properly. So while I knew where the server was and could now connect to it, I still didn’t know how to ask for the video I wanted.

Thankfully rtmpdump comes with several other utilities. After reading its README I went the rtmpsuck route. I set local redirecting of all port 1935 requests to localhost with iptables and ran the rtmpsuck proxy server. In theory it was supposed to intercept all calls from the flash player to the rtmp server, decode and spit them out, and then forward them along. Even better, it would try to save the stream on the way back as it passed through it.

# iptables -t nat -A OUTPUT -p tcp --dport 1935 -m owner --uid-owner OWNER_UID  -j REDIRECT
$ ./rtmpsuck

Where OWNER_UID is the uid of the user running rtmpsuck. With this running I just reloaded the page with the player (twice, it’s a bit glitchy) and then tried to skip to where my part was so it would save the stream from there.

It was partially successful. It spit out on the console all the pertinent path parameters about the video on the server, but it kept chocking on bad packets of data and stopped recording. Also for some reason the video it did store was very large, space-consuming wise.

Armed with the right parameters though I was able to use rtmpdump to suck down the whole video from the server surprisingly quickly and in a reasonably sized format.

$ ./rtmpdump  -r rtmp://server.net/app_name/blah/event/date/date-1 -o video.flv

Now the video was liberated from its flash interface and in my possession, I just had to cut out my small part and then convert it to a more common format.

$ mencoder -ss 1:15:50  -endpos 0:05:57  -ovc copy -oac copy video.flv -o result.flv
$ ffmpeg -i result.flv result.avi

And volia. I now have just my part of the video and in a common format. I mean you hypothetically do! Yes…

Completely unrelatedly, you can expect to see my presentation on my project Cortex from the BCNet Broadband Innovation Challenge (where I got second place) online soon.

However, this is Android! And so heroically the CyanogenMod people have been following the Android code base and releaseing an free up to date version that works on some more common phones. The good folks of Open Etna have taken that work and customized it and made sure it works on the LG Eve specifically. So following the instructions on their and other sites, I have been able to upgrade my phone to Android 2.2, and it’s awesome. There is a lot more software available, voice commands, live wallpapers, and a JIT compiler so software should run faster, and more that I’m just discovering (ships with a good terminal app). And just generally newer default apps. So thank you very much to those communities of volunteers for doing vastly more than what my Phone company was incapable of.

And now briefly what I did:

I followed the Open Etna Installation Guide. It’s pretty straight forward and simple. However there were a few other resources. I can’t stress enough how important good backups are, so make sure your contacts are synced to your google account and make a list of your favorite apps because you are about to wipe everything clean. Better instructions on how to run the backup procedure are at www.zacpod.com/p/71. Bellow the videos on the Open Etna guide is a important reminded to install the Google apps. Considering how important this is I’m surprised they placed it a bit out of the way, and it’s important because it allows you to then resync all your contacts and access all the other Google goodies that help make these phones great (and who wants to reenter all their contacts when all you have to do is hit a button to restore them).

So that’s about it. Now my phone is to date and has a whole lot more life to it. Thanks Android community!

So you need an out of tree binary copy of SBCL. Live with it. It works. The problem with going with out of tree software, especially for a language, is that what ever binary you get isn’t supported and hasn’t been tested against all the software in-tree. For instance I initially tried the newest version of SBCL (1.0.42) but ran into problems with portage’s stable slime.
Ultimately I went with the closest I could get to portage’s stable version. Portage has 1.0.19 marked as the most recent stable version so I went out and downloaded the binary of that version

$ wget  http://sourceforge.net/projects/sbcl/files/sbcl/1.0.19/sbcl-1.0.19-x86-linux-binary.tar.bz2/download
$ tar -xjf sbcl-1.0.19-x86-linux-binary.tar.bz2

So change into the directory and check out INSTALL. Basically installation is easy. Binary SBCL is configured around installing into /usr/local but that can be gotten around. So we’ll go with a more traditional install into /usr

Note: My test box is a VPS with a Xen kernel not a hardened kernel so I didn’t have any PaX problems, but my notes for the last time I tired this on a full hardened install mention that you need do disable some PaX features before SBCL will work:

$ paxctl -p -e -m -r -x -s " on src/runtime/sbcl

Install to /usr

# INSTALL_ROOT=/usr sh install.sh

Now SBCL is installed but it won’t work because the binary is preconfigured to look for the core in /usr/local. So we’ll borrow the gentoo SBCL config files to get that setup properly.

/etc/env.d/50sbcl

SBCL_HOME=/usr/lib/sbcl
SBCL_SOURCE_ROOT=/usr/lib/sbcl/src

# env-update

The above file and command set up the system environment variables to tell SBCL where it’s really installed. Now is as good a time as and to ‘source /etc/profile‘ to get those changes.

Now SBCL is installed and working, we need to let portage know that. There used to be a ‘emerge –inject’ method, but that’s been deprecated in place of a new provides file

/etc/portage/profile/package.provided

dev-lisp/sbcl-1.0.19

Now portage knows about our SBCL so we can start installing things that depend on it like the rest of our tool chain

# emerge cl-asdf emacs slime -va

So now we have all the pieces, all they need is some gluing together. Again we’ll borrow from the Gentoo SBCL files.

/etc/sbclrc

;;; The following is required if you want source location functions to
;;; work in SLIME, for example.
 
(setf (logical-pathname-translations "SYS")
    '(("SYS:SRC;**;*.*.*" #p"/usr/$(get_libdir)/sbcl/src/**/*.*")
          ("SYS:CONTRIB;**;*.*.*" #p"/usr/$(get_libdir)/sbcl/**/*.*")))
 
;;; Setup ASDF
(load "/etc/gentoo-init.lisp")

/etc/gentoo-init.lisp

(in-package #:cl-user)
#+(or sbcl ecl) (require :asdf)
#-(or sbcl ecl) (load #p"/usr/share/common-lisp/source/asdf/asdf.lisp")
(push #p"/usr/share/common-lisp/systems/" asdf:*central-registry*)
(asdf:oos 'asdf:load-op :asdf-binary-locations)
(setf asdf:*centralize-lisp-binaries* t)
(setf asdf:*source-to-target-mappings* '((#p"/usr/lib/sbcl/" nil) (#p"/usr/lib64/sbcl/" nil)))

Now everything should work. You just need to set up your emacs and slime

~/.emacs

; your SLIME directory
(add-to-list 'load-path "/usr/share/emacs/site-lisp/slime/")
; your Lisp system
(setq inferior-lisp-program "/usr/bin/sbcl")
(require 'slime)
(slime-setup)
 
(global-set-key (kbd "C-c C-q") 'slime-close-all-parens-in-sexp)

Now It’s all glued together, give it a go

$ emacs
M-x slime

If you don’t get any compilation errors you should be in emacs + slime.

And there you have it, SBCL emacs and slime on Gentoo Hardened.

Cavets

1) For some reason this approach adds some annoying extra text to vanilla SBCL start up that I can’t seem to get rid of

$ sbcl
This is SBCL 1.0.19, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http: //www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
; loading system definition from
; /usr/share/common-lisp/systems/asdf-binary-locations.asd into
; #<package "ASDF0">
; registering #<system ASDF-BINARY-LOCATIONS {AAF8F51}> as ASDF-BINARY-LOCATIONS
* 

2) The system I tested this on is a VPS so the kernel is a Xen kernel, not a hardened kernel, so there may be additional complications on a full hardened install. Please let me know if you have any, and especially any working solutions.

clc-register-user-package to the rescue! Create ~/.clc/source and put your ASDF package there, then simply run

$ clc-register-user-package ~/.clc/source/package/package.asd

and volia, you can (require :package) from any of your Lisp systems there after.

It’s pretty awesome. :)

First things first, the files I want to backup are owned by all different users, so the only user who can run the backup process is root. Therefore I can’t just run rsync from my local machine and grab the files from the server, the backup process has to be run on the server and backup to the backup machine. Now in my case this was a bit of a trick because the backup machine was behind a firewall, so the server had no direct line of communication to it. So I wrote a script to turn on a reverse ssh port forward.

recv-serv-backup.sh

#!/bin/sh
 
ssh -R 8000:127.0.0.1:22 -N user@kvasir.mindstab.net

When run on the backup machine behind a firewall, it connects to the server (kvasir) and listens on port 8000. When ssh on kvasir connects to port 8000 it redirects that traffic to local port 22, the ssh port of the backup machine. This is how the firewall is gotten around. Reverse port mapping is a cool trick to master.

Next as root on kvasir I generated a public ssh key and put it on the backup machine so root could automatically log on repeatedly to the back up machine (think lots of rsync calls) once the key was loaded once. Then I hooked up the key to keychain. That is all better outlined at:
Gentoo Linux Documentation: OpenSSH key management, Part 1 and
Gentoo Linux Documentation: OpenSSH key management, Part 2.

The all I needed to do was poke the rsync syntax to use the nonstandard ssh port for backup. The best method I found was

rsync -e "ssh -p 8000" -av

rsync for those of you who don’t know is a great little backup tools. It’s like a smart (in that it only copies files that have been modified since the last backup) network aware (since it can use ssh) copy tool. Simple but really useful.

So with that I wrote a backup script on the server

server-backup.sh

#!/bin/sh
 
backup () {
        echo "$1..."
        DST=`echo "$1" | awk '{split($0,a,"/");  result = "/"; for (i=2 ; i < length(a)  ; i++)   result = result "/" a[i];  print result;  };'`
        rsync -e "ssh -p 8000" -acv $1 dan@127.0.0.1:~/kvasir$DST ;
 
}
 
backup "/svn"
backup "/git"
...

There is one caveat, rsync won’t create subdirectories on the other side specified in the path so you need to create the basic directory structure.

 rsync -e "ssh -p 8000" -av /git user@127.0.0.1:~/kvasir/git

is fine because it will create /git just fine, but

 rsync -e "ssh -p 8000" -av /home/user user@127.0.0.1:~/kvasir/home/user

will fail if ~/kvasir/home doesn't exist. So you'll need to create the basic directory structure or enhance the backup function to strip out extra directories in the target path.

Finally, I didn't want anyone and everyone to potentially be able to gain access to private data on the backup machine, so the target directory needed to be encrypted. There are a lot of options, but I opted for the easy encFS route and just installed "cryptkeeper" and had it setup the directory.

Now all I have to do is mount the encrypted backup directory, run the script to turn on the reverse ssh tunnel, and run the backup script, and I have an encrypted backup solution for my server that gets around firewalls.

Not bad.

References

• What Goes Out Can Come Back In -- SSH tricks
• Google

In Firefox, goto “about:config” and put a filter of “vendor” in and hit enter. Change general.useragent.vendor from “Ubuntu” to “Firefox”.

Then it works fine. It’s annoying MS is still doing such browser specific tweeking and it breaks on something so small, but then again, it does seem that they at least support vanilla Firefox. I don’t know how important the vendor string is but if it’s breaking compatibility maybe Canoncial should leave it alone?

My server, mindstab.net, runs Hardened Gentoo. I like it. It provides nice features from grsecurity and PaX like memory randomization, non executable writable memory, etc. However, it really doesn’t get along so well with Lisp. Lisp in general seems to like executable and writable memory, and SBCL at least also doesn’t like randomized memory. So it took a bit of work to get Lisp onto my server.

Approach 1: Failure
I spent a bunch of time trying to patch the build process in portage to coax SBCL into building. First, of course, I used gcc-config to disable the hardened gcc profile, and just use the vanilla one. Then I created a suid root shell script to call “paxctl -m -p -r -e $1” so that the sandboxed build process could disable PaX features on the SBCL binaries. I added the command to the ebuild, and created a patch to insert the command into SBCL’s build process. The process goes like this, portage download’s the SBCL source and a pre-compiled SBCL binary. The patched ebuild then calls my suid root script which disables PaX on the pre-compiled binary so it actually runs (as opposed to crashing under PaX) and then a new SBCL binary is built from the source and the pre-compiled binary builds a core file from the SBCL lisp source. The patched SBCL make.sh then again calls the suid root script on the new binary, so it will run. Then it should load the new core and recompile the system for itself. Sadly, while it runs at least, it chokes on the core file and hangs while using 100% cpu. I couldn’t get past this so I eventually gave up. If anyone has any suggestions that’d be great.

Approach 2: Success
So the actually solution was as follows: Download the most recent precompiled SBCL binary from the website (1.0.15 for x86), run “paxctl -p -e -m -r -x -s ” on src/runtime/sbcl (to cover all the bases). Then run “sh install.sh” to install SBCL to /usr/local. That’s it.

The problem with this is you can’t emerge lisp packages in portage, you have to install them by hand (unless maybe you want to fake inject the package into the portage database).

I downloaded a copy of slime, untarred it and popped it in my .emacs and I had a full lisp environment ready to go, and on my hardened machine no less. Not so bad.

It’s really easy, especially with Ubuntu.

To start with, you need to be a router, so you need firewall software that can do NAT (network address translation). This is part kernel side (NAT and iptables options enabled and modules loaded) and part user space side, in the form of the program ‘iptables’, so make sure it is installed, which it is by default on Ubuntu (I think).

Now all you really need to do is add two routing rules, one says anything coming on the ethernet port should go through the NAT procedure, which basically means it’s IP headers are tweaked to make it look like they originated from your computer and then you send them along to the internet. The second rule helps facilitate this (I’m a little less sure what it does, but it’s needed).

In this case we are assuming the interface eth0 is the wired network and eth1 is the wireless. Change as required.

iptables --table nat --append POSTROUTING --out-interface eth1 -j MASQUERADE
iptables --append FORWARD --in-interface eth0 -j ACCEPT

Next you just need to tell the kernel port forwarding should be turned on, and you can do this through the wonderful /proc filesystem.

echo 1  > /proc/sys/net/ipv4/ip_forward

Now you’re pretty much ready to go. Connect to the wireless, presumably though NetworkManager or your wireless toll of choice. Then enable the wired network manually.

ifconfig eth0 up 192.168.1.1

This turns on eth0 with a local network address of 192.168.1.1. Now plug your friend’s computer into the ehternet port with a crossover ethernet cable or into a hub and then the hub into you with regular ethernet cable and have them manually pick an address on 192.168.1.* (or whatever local network you chose, it doesn’t matter) and set you (192.168.1.1) as the gateway router.

If this is a little much for them or they are running an OS that makes this non trivial, than its really another easy step for you to set up a DHCP server and do all the configuration for them :).

So install dhcpd, on Ubuntu it’s ‘dhcp3-server’, on Gentoo I think it’s just ‘dhcpd’.

Now we have to configure it. We have to tell it the gateway router, which is us (192.168.1.1), the nameservers (the servers in /etc/resolv.conf) and the pool of IPs to use and what interface/network to listen on.

Open the config file, on Ubuntu ‘/etc/dhcp3dhcpd.conf‘.

The relevant parts are as follows

...
# servers in /etc/resolv.conf
option domain-name-servers 192.168.0.1;

...

# the local network you created
subnet 192.168.1.0 netmask 255.255.255.0 {
        #IPs free to assign
        range 192.168.1.100 192.168.1.200;
        #your computer, the router
        option routers 192.168.1.1;
}

And that’s it. (Re)Start the server

/etc/init.d/dhcp3-server restart

And you are now serving all the information your friend’s computer will need to automatically connect properly.

They should now be online once they restart their internet connection.

One annoying thing about Ubuntu vs Gentoo is that on Ubuntu, the init system is a bit more kludgy and old fashioned. Any server software installed is automatically configured to start at boot time, forever, which in this case isn’t what you want. You only want the dhcpd server to run very rarely, at parties, the rest of the time it’s a waste. So we need to turn it’s auto starting off. Apparently the /ubuntu init system barely supports this, we have to force it.

update-rc.d -f dhcp3-server remove

Now just turn it on when you need with its init.d file.