Mindstab.net

So after being alterted to the existence of DKIM by this article posted on HackerNews I wanted to implement it immediatly on my server. DKIM is Domain Keys for Identified Mail, a crypo signing protocol where a pub key sits in your DNS and your mail servers sign your mail as it passes through your server. Seems a little stronger than SPF from a few years ago for authenticating mail’s origin so I was keen to adopt it.

So I found the freshest instructions on the Gentoo wiki and followed them. They were a bit spartan so I went looking for a bit more material and found this Ubuntu tutorial which had some helpful suggestions like the testing section.

After giving the OpenDKIM instructions a first run through I gave the testing a try.

First using dkimcore.org/tools/ I found that the Gentoo OpenDKIM config tool had spat out invalid TXT. It had spat out

v=DKIM1;=rsa; p=MIGfM......

And after some quick internet consultation I found out I needed to fix it to

v=DKIM1; k=rsa; p=MIGfM.....

The second test from the Ubuntu docs was an auto-respond test email system that along with wikipedia I learned about ADSP from. So I added

_adsp._domainkey.mindstab.net. IN TXT "dkim=discardable"

to my Bind config as well. (I’m still not 100% about the final ‘.’). Also it seems the autoresponder email tool doesn’t update its DNS too often so I may have to wait a bit to retest.

So now it seems I should have DKIM signed/valid email! :) Just another step to make sure my email is valid, slightly less spoofable and liked/accepted by the big email providers.

Also, seeing results like this from Gmail after receiving my email seems good:

Received-SPF: pass (google.com: domain of dan@mindstab.net designates 69.164.214.81 as permitted sender) client-ip=69.164.214.81;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of dan@mindstab.net designates 69.164.214.81 as permitted sender) 
  smtp.mail=dan@mindstab.net; dkim=pass header.i=@mindstab.net

Family got a new computer and Mom wanted Ubuntu on it. A good sign. They’ve won her over. The battle should be done right?

• Time spent waiting while the installer is unresponsive and gives no indication it is working or if it is dead: 30 minutes
• First install booted up with a thin stripe of graphics about 1cm tall at the top of the screen and blank black for the rest. Completely unuseable
• Second install, I chrooted it from the liveCD, updated all the software and installed fglrx
• Second install booted, started using Unity (the new Ubuntu interface):
  • Only a few visual artifacts :/
  • Software discover-ability is about zero compared the the old well structured Gnome menu :(
  • Having to type the names of the apps I want or sort through a giant unstructured alphabetical list of all software installed is not the way I want to launch my programs and is slow and cumbersome
    • Also, after typing there is a noticable pause before the search list populates. Live AJAX web services like google search which are remote are/feel more responsive :(
  • New sidebar buttons are huge, greatly limiting the number of shortcuts I can pin to it
  • No smooth and wobbly windows anymore.
  • No applets
• After about 30-45 minutes use the screen blanked. I changed to a shell (alt-f1) and rebooted. It booted up and the monitor got no signal. Repeated with same result. It completely died. (remounting from the liveCD Xorg.0.log reports some problem with the fglrx driver not loading (but it worked from the liveCD and for the first while of the fresh install?). What happened to the good old indestructible bullet proof X Ubuntu touted a few years ago? Do modern graphics cards no longer support fall back graphics modes like VESA?)
• Mom is stuck with Windows 7 because it works

:(

So I’m setting up a new website for a client (an artist) so the easiest what I’ve always done is use Gallery. I’ve used 2.* for years and now 3.0 is out, and has been for a few months. So I figured why not give it a try.

I’ve never knows a website to have intermittent bugs, but Gallery3 has a good couple of them. Sometimes the spacing around photo/album items is just way too big, and after a mouse over, they jump position. A good half the time trying to delete an item takes you to a blank white page with a single option, “delete” and then that takes you to a dead end page with clearly an AJAX reply. But only sometimes. How do you track down bugs that only happen half the time? The default theme, wind, seems to lean heavily on jquery and I think, but am not sure, this is where the instability is coming from, but having not boned up to the level of jquery master, I certainly can’t dive in. Also apparently some of these issues aren’t even unknown, but still haven’t been fixed so we could assume better minds than mine have looked.

So that’s a bit disappointing and a waste of a days work that I get to eat. Gallery3 is not stable and usable. Back to Gallery2.

(Also, I’m not even really a fan of hiding things like item names by default and only showing on mouse over, it’s bad for the kind of galleries I’m putting up, but there isn’t even an option about that, and again, not recoding a ton of jquery code.)

Frustrated.

So 2010′s been a year.

Nearer it’s start some friends and I competed for the second time in MIT’s BattleCode, this time getting second non-MIT spot, or 18th overall. These competitions have been good for us: they are fun, it’s a good group project, and we work on our group project skills like planning, coordinating, and so forth. We spend so much time on our own, or in school even, working solo it’s good to work these skills as they will be needed later. Also it’s fun to learn about and catch up on low level AI stuff, like swarming and flocking movement/coordination techniques etc.

I also entered a school project into BCNet’s Broadband Innovation Challenge and got awarded second place. My project was “Cortex” a P2P processing app that runs with no software install entirely in your web browser. It was comprised of a small Java Applet webserver used as a backbone for communication and then a JavaScript front end, with all the control logic of the P2P network also written in JavaScript. I pretty much wrote a P2P app in JavaScript just using Java only to get around the AJAX/Server of Origin security policy issue. It was an interesting and challenging project and I’m pleased with how it did in the competition.

Over the summer I was in China which was amazing.

Then in the fall while finishing off my degree in CS once and for all I also competed in the Google sponsored University of Waterloo AI Contest. This, while being a simpler solo competition, was notable for me as it was my third major project undertaken in Lisp. I thoroughly enjoyed the challenge and again learned lots more about Lisp and again improved my Lisp style. Lisp and the emacs environment just take longer to learn and wrap my head around. And since I don’t get to work in them constantly, between work and school, it takes time. I’m by no means a master, but after convincing a friend to take a stab at the same competition in Lisp for his first try with Lisp, I at least see how far I’ve come. I’m getting more used to thinking functionally, especially with respect to using Lisp mapping functions instead of loops to modify, filter, or build on data. I placed disappointingly poorly due to lack of time, but I’m satisfied with what I learned (and also proud by association that the winner was a Lisp entry!). It was a good experience. I look forward to being able to undertake some more Lisp projects in the new year.

I also boned up on my Python this fall for a small work project, a multi threaded web crawler for a client. Played successfully with Python’s threading, so that was fun.

And that brings us to now. I’m in Colombia for the holidays, and in my vacation spare time I’ve finally gotten around to looking at the codebase to my school project “Cortex”. As school projects are, it worked, and well, but the codebase was a bit of a mess due to strong time constraints. Now that I have some time I’m doing some massive cleanups and adding a few features I’d wanted to but didn’t have time to. Hopefully early in the new year it’ll be in shape that I can release it. That would be nice.

So 2010 was a great year. I got to write a lot of cool code in several different language. I got to travel more than I ever have before, and I read a lot more than 2009 (traveling facilitates a lot of reading :)). It’s been a good year.

For 2011 though, now that I’m done with school, I’d like to start by releasing more code, starting with Cortex; getting more paying work; and looking at maybe starting a startup. I’d like to spend more time working on both AI (if you hadn’t noticed, obviously a hobby of mine) and in Lisp, starting with getting back into my signed copy of Peter Norvig’s “Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp” (yes getting it signed was awesome and a ridiculously geeky moment) and moving on from there. I’d like to at least keep up with the reading. I have high hopes for it to be an interesting year.

So here’s to 2010, you’ve been great, lets see if I can’t build on that for a more amazing 2011.

So now Yahoo is shutting down Delicious. I really can’t think of a single service that has been made better under Yahoo, and plenty that have died horrible slow deaths after taking off so quickly and getting bought for decent prices. Yahoo is where you go to die, even if you are healthy.

Anyways, a quick and simple backup command to get an XML file of your bookmarks from Delicious care of Ixiaus 9 on Hacker News

curl https://{your username}:{your password}@api.del.icio.us/v1/posts/all > bookmarks.xml

Ha, coupled with massive lay offs just before xmas I think Yahoo’s holiday spirit can be summed up in “Bah Humbug!”

A while ago I tried with mixed success to get Lisp onto my Gentoo Hardened server. I had to go a binary only route and kind of stopped there not taking it any farther. Now, 2 years later, I need the full meal deal, lisp + emacs + slime, on my server, which is now a Xen VPS with as much hardening as I could get (much less kernel based hardening since it’s the VPS’s kernel). It was still too much for SBCL to compile in portage so here’s what I did to get it all working.

So you need an out of tree binary copy of SBCL. Live with it. It works. The problem with going with out of tree software, especially for a language, is that what ever binary you get isn’t supported and hasn’t been tested against all the software in-tree. For instance I initially tried the newest version of SBCL (1.0.42) but ran into problems with portage’s stable slime.
Ultimately I went with the closest I could get to portage’s stable version. Portage has 1.0.19 marked as the most recent stable version so I went out and downloaded the binary of that version

$ wget  http://sourceforge.net/projects/sbcl/files/sbcl/1.0.19/sbcl-1.0.19-x86-linux-binary.tar.bz2/download
$ tar -xjf sbcl-1.0.19-x86-linux-binary.tar.bz2

So change into the directory and check out INSTALL. Basically installation is easy. Binary SBCL is configured around installing into /usr/local but that can be gotten around. So we’ll go with a more traditional install into /usr

Note: My test box is a VPS with a Xen kernel not a hardened kernel so I didn’t have any PaX problems, but my notes for the last time I tired this on a full hardened install mention that you need do disable some PaX features before SBCL will work:

$ paxctl -p -e -m -r -x -s " on src/runtime/sbcl

Install to /usr

# INSTALL_ROOT=/usr sh install.sh

Now SBCL is installed but it won’t work because the binary is preconfigured to look for the core in /usr/local. So we’ll borrow the gentoo SBCL config files to get that setup properly.

/etc/env.d/50sbcl

SBCL_HOME=/usr/lib/sbcl
SBCL_SOURCE_ROOT=/usr/lib/sbcl/src

# env-update

The above file and command set up the system environment variables to tell SBCL where it’s really installed. Now is as good a time as and to ‘source /etc/profile‘ to get those changes.

Now SBCL is installed and working, we need to let portage know that. There used to be a ‘emerge –inject’ method, but that’s been deprecated in place of a new provides file

/etc/portage/profile/package.provided

dev-lisp/sbcl-1.0.19

Now portage knows about our SBCL so we can start installing things that depend on it like the rest of our tool chain

# emerge cl-asdf emacs slime -va

So now we have all the pieces, all they need is some gluing together. Again we’ll borrow from the Gentoo SBCL files.

/etc/sbclrc

;;; The following is required if you want source location functions to
;;; work in SLIME, for example.
 
(setf (logical-pathname-translations "SYS")
    '(("SYS:SRC;**;*.*.*" #p"/usr/$(get_libdir)/sbcl/src/**/*.*")
          ("SYS:CONTRIB;**;*.*.*" #p"/usr/$(get_libdir)/sbcl/**/*.*")))
 
;;; Setup ASDF
(load "/etc/gentoo-init.lisp")

/etc/gentoo-init.lisp

(in-package #:cl-user)
#+(or sbcl ecl) (require :asdf)
#-(or sbcl ecl) (load #p"/usr/share/common-lisp/source/asdf/asdf.lisp")
(push #p"/usr/share/common-lisp/systems/" asdf:*central-registry*)
(asdf:oos 'asdf:load-op :asdf-binary-locations)
(setf asdf:*centralize-lisp-binaries* t)
(setf asdf:*source-to-target-mappings* '((#p"/usr/lib/sbcl/" nil) (#p"/usr/lib64/sbcl/" nil)))

Now everything should work. You just need to set up your emacs and slime

~/.emacs

; your SLIME directory
(add-to-list 'load-path "/usr/share/emacs/site-lisp/slime/")
; your Lisp system
(setq inferior-lisp-program "/usr/bin/sbcl")
(require 'slime)
(slime-setup)
 
(global-set-key (kbd "C-c C-q") 'slime-close-all-parens-in-sexp)

Now It’s all glued together, give it a go

$ emacs
M-x slime

If you don’t get any compilation errors you should be in emacs + slime.

And there you have it, SBCL emacs and slime on Gentoo Hardened.

Cavets

1) For some reason this approach adds some annoying extra text to vanilla SBCL start up that I can’t seem to get rid of

$ sbcl
This is SBCL 1.0.19, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http: //www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
; loading system definition from
; /usr/share/common-lisp/systems/asdf-binary-locations.asd into
; #<package "ASDF0">
; registering #<system ASDF-BINARY-LOCATIONS {AAF8F51}> as ASDF-BINARY-LOCATIONS
* 

2) The system I tested this on is a VPS so the kernel is a Xen kernel, not a hardened kernel, so there may be additional complications on a full hardened install. Please let me know if you have any, and especially any working solutions.

Wow, I think my addiction to the bleeding edge is dying due to risk aversion. I don’t have time to deal with breaks anymore, work is taking priority over time to fool around with buggy betas. Part of the reason my bleeding egde addiction has lasted this long (long out living my general move away from Gentoo to Ubuntu on the desktop) is because generally Ubuntu late alphas and betas have all been surprisingly stable and usable. Until now.

I just upgraded to Ubuntu 10.10 on my netbook. Wow it’s unusable. The new default interface appears to rely on acceleration or something but that isn’t working on my Gateway netbook with an Intel n10 graphics card. So it’s unusably slow. I can’t even tell how the side bar is really supposed to work, it doesn’t really scroll properly or quickly and apps don’t really seem to launch in any kind of timely manor. On an interface designed for small vertical real estate I question preloading the sidebar with so many apps that I don’t want to use, and not supplying an obvious editing mechanism requiring me to rely on the slow and broken scrolling mechanism. Also why use such large icons, it’s just wasteful and ugly. Then they have a new desktop launcher that seems to lack some the normal free desktop menu categories, with unusually small icons. Who is picking sizes in this interface, these icons have the room to be larger, unlike the sidebar. Additionally it’s more sluggish than non native apps like Google Suggest. I type in a search and in more time the a webapp takes it finally starts popping up apps that match, then seconds later loads their icons. Searching my “menu” for my apps should not take seconds. Unacceptable.

Additionally the fallback desktop options do not include the 10.04 interface, just bad remixes of it like the normal gnome desktop with or without the 10.04 launcher running as the wallpaper. That is massively suboptimal.

Finally external monitors don’t work with my netbook under 10.10 which is a show stopper. I can’t really work without an external monitor.

I’ve loved Ubuntu netbook remix 10.04 because it makes my netbook feel fast and really usable. 10.10 is unusably slow and buggy. It’s also surprising since normally I don’t wait this long before updating and things normally work just fine. I know it’s a beta, but in the past the late alphas and betas have always been workable and acceptable to pretty much perfect. This is the first time it’s been unusable, and it’s only a month till release. I’m really worried that all of the bugs I have will not be fixed and the some of these problems are design decisions that I just disagree with.

Now I get to reinstall 10.04 so I can get back to work. My addiction to updating before release seems to have been killed dead and I’m still quite worried about upgrading to 10.10 later. The beta has left a horrible taste in my mouth.

I love tomboy. I drop tons of my ideas into it. And plans. Lists. Anything on my brain. The problem is as soon as I have more than one computer, like now with my netbook and laptop, all my thoughts are split in half. Ever since tomboy hit the scene years ago I’ve loved it but also known it needed synchronization to be the best and truly useful to me. I’ve only had sporadic luck with the sshfs sync. It worked last year. Now it doesn’t.
Anyways, I finally couldn’t stand it any longer and decided to sign up for an Ubuntu One account that goes with the Ubuntu Web Sync plugin shipped with Tomboy on Ubuntu. I’m not normally one for web services, I like to roll my own where I can (email, jabber, blog), but here since sshfs sync wasn’t working and snowy (tomboy online) is pre alpha, I didn’t have a choice.
(Actually, in hindsight I realize I could have tried and been using sync to local folder and placed that in my DropBox, whoops)
Anyways, it was painless and now my notes are synced to the cloud and my brain is healed. So thank you Ubuntu. I’m not a huge fan of proprietary solutions but no one else has a solution at all, so you win by virtue of just shipping. (Of course this isn’t exactly a multi platform approach, I just thankfully only run Ubuntu on my laptops right now. A few years ago when I ran Gentoo on desktops/laptops this would have been a no go, and Dropbox would have probably floated to the top of my mind sooner)
Let that be a lesson to the rest of you, ideals are good, but solid options are better.

I need to get back into blogging, but like so many others, twitter has been eroding at that. We shall see.

… and we’re back! Just in time for our 8th birthday.

Mindstab.net’s been around. It started back in January of 2002 running on a 486 from my bedroom. A few years later when I had some money, I bought a trusty 1U rackmount server, kvasir, and migrated mindstab there. Kvasir started co-located but when money dried up some, I brought kvasir home and mindstab has been living in my room for the last several years.

That all ended yesterday morning at 3:30am when I was awoken by kvasir’s PSU fan screaming to death. The writing had been on the wall for a bit that things were going to have to change. I knew the fan was on its last legs, and hosting from home had been getting ever more precarious as both the local ISPs seem unable to provide a stable and reliable net connection. To that end I’d been playing with a VPS at linode.com (recommended in a blog post by Brian Carper) but I hadn’t gotten very far. Actually, after a month I was still poking at the kernel trying to get my general paranoid level of security with grsecurity and PaX to work with the Xen virtual host at Linode. Anyways, at 3:30am yesterday I went into emergency mode and acceptable that their stock kernel “might just suffice” and started madly installing infrastructure software. Since then I’ve been copying over data and starting to bring services back up. So far we have a firewall, DNS, Jabber and I’m just starting to get the websites back up, this being the first.

Lots of work ahead. I need to get the rest of the sites up. Then I have to dive into the mess that is e-mail. I’d previously been using a qmail/vpopmail/courier system but I think I’m chucking that for a postfix/dovecot system, but I’m sure it’ll take some kicking to get it up and get all my mail into it. Hopefully not too much. Then onto less crucial systems, then go back and re-vet alot of things.

So yeah, as of now, mindstab.net is coming to you from my new server Loki, in the cloud… or New Jersey more specifically :P

So you may have noticed a few days ago a link to an article on teaching to program in a newish language called Shoes. Its a cute language on top of Ruby for whipping up fun cute little GUI apps, event oriented and good for introductions to programming. So I wanted to play with it but Ubuntu and Debian ship the old version 1 and version 2 has been out since December 2008. So I checked Ubuntu’s bugzilla and sure enough, there was a bug from April asking for a version bump with no response. So I figured it might be time to step up to the plate. So I brought up the Ubuntu Packaging Guide and gave it a read. Turns out Shoes wasn’t trivial to package but with the old version 1 Deb package as a starting point I was able to get version 2 packaged! It’s now sitting in Ubuntu’s bugzilla at https://bugs.launchpad.net/ubuntu/+source/shoes/+bug/359031 and if you just want the Shoes 2 i386 deb, its at http://launchpadlibrarian.net/30491206/shoes_0.r1134-1_i386.deb. So yeah, check it out, give it a whirl, have fun.

As a side note, I’ve found Ubuntu’s bugzilla to be sporadically responsive which sucks a bit, but does encourage one to step up some… But looking at Debian, where this package actually originated from is even worse. They have no web interface for entering bugs, they only accept them via email or a command line tool. It does seems like a epic usability fail. So here’s hoping that now that Shoes 2 has been packaged as a .deb we’ll see it in Ubuntu sooner rather than later. Maybe I should just make a new bug for it?

(I hate to say it, but I still have found the Gentoo bugzilla to be blazingly responsive and have fond memories of it. I wish other communities could learn from it, what ever it is they are doing right.)