Mindstab.net

So 3 years ago as I was learning Lisp and looking for things to do I noticed it seemed to lack perl style pack/unpack encoding functions that many other languages shared (like PHP, Ruby, Python) so I wrote cl-pack (github) to fill that void. I released it back at the end of the summer of 2009 and moved on. Then this week on github I got a pull request. It seems cl-pack has found some use by people and a github user Taksatou had noticed a bug in the BER encoding and fixed it! So awesome, and thank you! The fix is merged and cl-pack continues to fill a small gap in the Lisp ecosystem :)

So summer seems to have come to a close both by calendar reckoning and by checking with the weather. So what have I been up to?

Well probably primarily just plain old work. Aside from that? Well, in the spring my overly ambitious plan to take 5 Coursera classes at the same time as working full time collapsed a bit. I did manager to pull off completing Software as a Service, Model Thinking and Game Theory.

After that I took some time off of “school”. I poked my old Lisp OpenGL flight sim but not to much noticeable effect. Reworked some of the model and rendering code and did manage to add an animated sidejet that pushes the ship sideways (all that work is in the rotate branch). Possibly in a year or two more at this slow pace I’ll have a fully movable ship. Anyways, it did make a good break.

I also ramped up my reading and discovered Goodreads into which I important my current to-read list and then explored its recomendation and lists to substantially expand it including most of the top reddit and other sci-fi lists. I’ve started cracking away at it so that should keep me occupied for a year or two.

But the break from school wasn’t to last. My friend Rob caught the first run through of the crypto class and broguht me back to coursera. So right now I’m just finishing off the Quantum Computing course (and doing rather poorly in it as its very much math and very little programming), and just starting the second run of Cryptography. And next month as crypto winds down I want to do the Scala class (taught by the author of the language! :D) and possibly the Human-Computer Interaction course if I can cram it in. We shall see.

On top of that I’ve finally gotten around to another goal I had for this year which was to start playing with Django a web framework in Python. I’ve just finished the 4 part intro tutorial and am ready to try a simple test site I’ve had in mind for a bit.

I also added one more new money and time consuming hobby this year: WarMachine. So that’s also a thing taking some of my time. For anyone curious, I play Cygnar.

Goals for the last quarter of the year? Well, actually finish my coursera courses, do the previously mentioned django test site, blog more than once every few months, cram/learn some spanish before my new years trip to Colombia to visit family, and read very many books. So we’ll see how that goes.

Hope you all had a good, fun and or productive summer!

Well, aside from being pretty busy with work, I spent some more time playing with OpenGL and Lisp. I took my demo test of a spinning diamond up to the beginnings of a space flight sim. Now we have a ship that has a thruster that applies forward thrust, there is the beginnings of physics. The next logical step would be to add turning, but I wanted to do it with physics, which meant writing torque code and that’s about where I got side tracked and have stopped for the moment. Not super interesting yet, but not bad for someone who has never done 3D programming before. The code is at https://github.com/dballard/flight-sim.

I then got distracted by the idea of a checkers AI by a few articles I read in August. I decided to dust off my Ruby and give that a go but it didn’t get too far before I realized that I was going to have to get side tracked from search in to aggressive pruning and heuristics. I was satisfied with the project even though it doesn’t really play because at it renewed me Ruby which had been shelved since some time last year and reminded me that search of this kind is bloody hard. The code is at https://github.com/dballard/checkers. It can simulate a full game to any depth if you have the time. If you have less than days or years per turn, it’s pretty much no good.

Now I’m getting further distracted by a few ideas in different directions yet again and I’m starting the Stanford AI class and Machine Learning class that they are offering online for free on monday. That should keep me pretty much tied up until years end and I’m looking forward to it all!

Also thanks to my Kobo Touch eReader I’ve been reading a ton, which is nice. I may have to give up my pace of a book every week or two for the next month or two to keep up with the new classes, we shall see.

• Zach's Journal – Making a small Lisp project with quickproject and Quicklisp
  Zach outlines how to start projects using quicklisp and quickproject
  (tags: lisp quicklisp project depends)

I woke up today to a message from a physics PhD student in the US asking about my old Lisp Go AI bot code (from the Mindstab Go AI competition of 2008) and if I was ever planning on releasing it or if he could get a look at it since Go and programming were two of his hobbies.

“fink”, as my Go AI bot was called, was my frist real Lisp project, and in hindsight probably not that good, but I learned a lot from it and it was fun, which is all that really matters. Possibly it’s most notable feature now is that it’s a Lisp program that speaks GTP and may provide a base for someone to write a better bot with. So if someone else can learn something from it or use it, that would be pretty cool. So here it is

https://github.com/dballard/fink

There’s some notes about it in it’s README. I don’t recall if I documented it much but hopefully it can be made sense of :)

So as you may know I’m trying to keep up with programming exploratory side projects in my spare time, somewhat spurred on by among many things, The Cult of Done. After learning a bit about HTML5 and Canvas and writing the The Game of Life for Canvas I wanted to move on to something more meaty. That’s about as far as I’d ever gone in graphics programming so I wanted to keep pushing on in that vein for a bit. So I figured why not check out 3D land. I briefly looked at WebGL but it’s not entirely ready just yet. Some of the tech demos look decent but for one thing, there’s almost no interactivity in the more complicated ones like flight sims, I assume because a good IO system isn’t in place, so I figured why not go for straight up OpenGL.

The defacto language for OpenGL programming is C/C++ however I’ve done my share of C programming and I’m enjoying higher level languages for many things these days. I have an ok relationship with Python but it doesn’t get me excited or anything, and I’m still working at getting better at Lisp which I am fond of so I thought why not use this as another excuess to work in Lisp? And so against some people’s advice I’ve been exploring OpenGL programmign in Lisp.

I Googled around and found that the basic package is cl-opengl and using Quicklisp got that installed and found a very basic tutorial for getting a window to show with a tutorial at nklein.com from NeHe. It worked okish but not really. Then I found 3bb.cc’s cl-opengl tutorials which were better. They used SDL instead of GLUT to make a window and the first one worked and I had a raindow triangle. Instead of going on, I started looking at the opengl.com tutorial’s in C and tried translating them. I worked through the cube.c one, but it didn’t quite work, so after a bit I gave up.

I went back to the 3bb code and instead of continuing on with the tutorials, I just started working on it myself and googling for parts I needed and getting some help from a friend who’d done some heavier OpenGL programming in C. I made the color change over time (like I learned for the game of life). Then I made the triangle spin. Then I added a second back triangle and made them spin. Then I fleshed it out and made pyramid, then a diamond that spun. Then I finally made the spinning color changing diamond that also moved in a circle, refactoring the code a few times in the process.

Now I have a nice simple tech demo and some code and understandings of the basics of OpenGL and 3D programming. For about 8 days work in spare time I am very pleased. I have more plans to move on to from here, but I’ll probably start a new project for it. The code for all my demos is on github at https://github.com/dballard/cl-opengl-tests. The only one that really works is 3bb.lisp and you need cl-opengl and lispbuilder-SDL.

Something I discovered is that the linear algebra scene for Lisp is confusing and a mess (IMHO). There are a bunch of projects for it, all kind of a mess with no apparent blessed/defacto standard library. I looked at L-math, but the version in quicklisp seemed a bit broken :/. This whole Lisp philosophy of always using the latest dev code instead of stable releases seems… against better sense. And seemed to fail me here. I looked at Femlisp/matlisp but their own FAQ talked about how their linear algebra implementation was slow and that’s all I wanted it for. So in the end I just rolled my own. It was fun and I had to remember some basic math from a few years ago.

Here’s the final product:

So 2010′s been a year.

Nearer it’s start some friends and I competed for the second time in MIT’s BattleCode, this time getting second non-MIT spot, or 18th overall. These competitions have been good for us: they are fun, it’s a good group project, and we work on our group project skills like planning, coordinating, and so forth. We spend so much time on our own, or in school even, working solo it’s good to work these skills as they will be needed later. Also it’s fun to learn about and catch up on low level AI stuff, like swarming and flocking movement/coordination techniques etc.

I also entered a school project into BCNet’s Broadband Innovation Challenge and got awarded second place. My project was “Cortex” a P2P processing app that runs with no software install entirely in your web browser. It was comprised of a small Java Applet webserver used as a backbone for communication and then a JavaScript front end, with all the control logic of the P2P network also written in JavaScript. I pretty much wrote a P2P app in JavaScript just using Java only to get around the AJAX/Server of Origin security policy issue. It was an interesting and challenging project and I’m pleased with how it did in the competition.

Over the summer I was in China which was amazing.

Then in the fall while finishing off my degree in CS once and for all I also competed in the Google sponsored University of Waterloo AI Contest. This, while being a simpler solo competition, was notable for me as it was my third major project undertaken in Lisp. I thoroughly enjoyed the challenge and again learned lots more about Lisp and again improved my Lisp style. Lisp and the emacs environment just take longer to learn and wrap my head around. And since I don’t get to work in them constantly, between work and school, it takes time. I’m by no means a master, but after convincing a friend to take a stab at the same competition in Lisp for his first try with Lisp, I at least see how far I’ve come. I’m getting more used to thinking functionally, especially with respect to using Lisp mapping functions instead of loops to modify, filter, or build on data. I placed disappointingly poorly due to lack of time, but I’m satisfied with what I learned (and also proud by association that the winner was a Lisp entry!). It was a good experience. I look forward to being able to undertake some more Lisp projects in the new year.

I also boned up on my Python this fall for a small work project, a multi threaded web crawler for a client. Played successfully with Python’s threading, so that was fun.

And that brings us to now. I’m in Colombia for the holidays, and in my vacation spare time I’ve finally gotten around to looking at the codebase to my school project “Cortex”. As school projects are, it worked, and well, but the codebase was a bit of a mess due to strong time constraints. Now that I have some time I’m doing some massive cleanups and adding a few features I’d wanted to but didn’t have time to. Hopefully early in the new year it’ll be in shape that I can release it. That would be nice.

So 2010 was a great year. I got to write a lot of cool code in several different language. I got to travel more than I ever have before, and I read a lot more than 2009 (traveling facilitates a lot of reading :)). It’s been a good year.

For 2011 though, now that I’m done with school, I’d like to start by releasing more code, starting with Cortex; getting more paying work; and looking at maybe starting a startup. I’d like to spend more time working on both AI (if you hadn’t noticed, obviously a hobby of mine) and in Lisp, starting with getting back into my signed copy of Peter Norvig’s “Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp” (yes getting it signed was awesome and a ridiculously geeky moment) and moving on from there. I’d like to at least keep up with the reading. I have high hopes for it to be an interesting year.

So here’s to 2010, you’ve been great, lets see if I can’t build on that for a more amazing 2011.

A while ago I tried with mixed success to get Lisp onto my Gentoo Hardened server. I had to go a binary only route and kind of stopped there not taking it any farther. Now, 2 years later, I need the full meal deal, lisp + emacs + slime, on my server, which is now a Xen VPS with as much hardening as I could get (much less kernel based hardening since it’s the VPS’s kernel). It was still too much for SBCL to compile in portage so here’s what I did to get it all working.

So you need an out of tree binary copy of SBCL. Live with it. It works. The problem with going with out of tree software, especially for a language, is that what ever binary you get isn’t supported and hasn’t been tested against all the software in-tree. For instance I initially tried the newest version of SBCL (1.0.42) but ran into problems with portage’s stable slime.
Ultimately I went with the closest I could get to portage’s stable version. Portage has 1.0.19 marked as the most recent stable version so I went out and downloaded the binary of that version

$ wget  http://sourceforge.net/projects/sbcl/files/sbcl/1.0.19/sbcl-1.0.19-x86-linux-binary.tar.bz2/download
$ tar -xjf sbcl-1.0.19-x86-linux-binary.tar.bz2

So change into the directory and check out INSTALL. Basically installation is easy. Binary SBCL is configured around installing into /usr/local but that can be gotten around. So we’ll go with a more traditional install into /usr

Note: My test box is a VPS with a Xen kernel not a hardened kernel so I didn’t have any PaX problems, but my notes for the last time I tired this on a full hardened install mention that you need do disable some PaX features before SBCL will work:

$ paxctl -p -e -m -r -x -s " on src/runtime/sbcl

Install to /usr

# INSTALL_ROOT=/usr sh install.sh

Now SBCL is installed but it won’t work because the binary is preconfigured to look for the core in /usr/local. So we’ll borrow the gentoo SBCL config files to get that setup properly.

/etc/env.d/50sbcl

SBCL_HOME=/usr/lib/sbcl
SBCL_SOURCE_ROOT=/usr/lib/sbcl/src

# env-update

The above file and command set up the system environment variables to tell SBCL where it’s really installed. Now is as good a time as and to ‘source /etc/profile‘ to get those changes.

Now SBCL is installed and working, we need to let portage know that. There used to be a ‘emerge –inject’ method, but that’s been deprecated in place of a new provides file

/etc/portage/profile/package.provided

dev-lisp/sbcl-1.0.19

Now portage knows about our SBCL so we can start installing things that depend on it like the rest of our tool chain

# emerge cl-asdf emacs slime -va

So now we have all the pieces, all they need is some gluing together. Again we’ll borrow from the Gentoo SBCL files.

/etc/sbclrc

;;; The following is required if you want source location functions to
;;; work in SLIME, for example.
 
(setf (logical-pathname-translations "SYS")
    '(("SYS:SRC;**;*.*.*" #p"/usr/$(get_libdir)/sbcl/src/**/*.*")
          ("SYS:CONTRIB;**;*.*.*" #p"/usr/$(get_libdir)/sbcl/**/*.*")))
 
;;; Setup ASDF
(load "/etc/gentoo-init.lisp")

/etc/gentoo-init.lisp

(in-package #:cl-user)
#+(or sbcl ecl) (require :asdf)
#-(or sbcl ecl) (load #p"/usr/share/common-lisp/source/asdf/asdf.lisp")
(push #p"/usr/share/common-lisp/systems/" asdf:*central-registry*)
(asdf:oos 'asdf:load-op :asdf-binary-locations)
(setf asdf:*centralize-lisp-binaries* t)
(setf asdf:*source-to-target-mappings* '((#p"/usr/lib/sbcl/" nil) (#p"/usr/lib64/sbcl/" nil)))

Now everything should work. You just need to set up your emacs and slime

~/.emacs

; your SLIME directory
(add-to-list 'load-path "/usr/share/emacs/site-lisp/slime/")
; your Lisp system
(setq inferior-lisp-program "/usr/bin/sbcl")
(require 'slime)
(slime-setup)
 
(global-set-key (kbd "C-c C-q") 'slime-close-all-parens-in-sexp)

Now It’s all glued together, give it a go

$ emacs
M-x slime

If you don’t get any compilation errors you should be in emacs + slime.

And there you have it, SBCL emacs and slime on Gentoo Hardened.

Cavets

1) For some reason this approach adds some annoying extra text to vanilla SBCL start up that I can’t seem to get rid of

$ sbcl
This is SBCL 1.0.19, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http: //www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
; loading system definition from
; /usr/share/common-lisp/systems/asdf-binary-locations.asd into
; #<package "ASDF0">
; registering #<system ASDF-BINARY-LOCATIONS {AAF8F51}> as ASDF-BINARY-LOCATIONS
* 

2) The system I tested this on is a VPS so the kernel is a Xen kernel, not a hardened kernel, so there may be additional complications on a full hardened install. Please let me know if you have any, and especially any working solutions.

From UBC CSSS News letter:

1. Distinguished Lecture Series*
Speaker: *Peter Norvig, Director of Research, Google*
Date: Thursday, September 23
Time: 3:30 – 4:50 pm
Location: DMP 110

For those you you who forget, Peter Norvig is Google’s head of R&D and wrote “Paradigms of Artificial Intelligence Programming: Case Studies In Common Lisp” which I am slowly working through. I’m excited!

So, what have I got on the go?

• School: Last semester, just two classes, but they are looking like they’ll be delicious and meaty
  • CS 411: Compiler design: We build a java compiler
  • CS 415: We build an operating system, fun times with C!
  • …actually, I’m also taking spanish!
• “Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp” by Peter Norvig. I got this for christmas and have started to work through it. I’m really excited about getting further into it. In the middle of it I’ll be implementing prolog in Lisp and the in the latter half I’ll be working on a natural language processor! Not to mention tons of other stuff, this book is huge and dense.
• BattleCode 2010 has just started so my team and I are just about to start digging into that. Lots of AI coding to be done there.
• Peter Michaux’s Scheme from Scratch. I stumbled upon this from Hacker News. This fellow wants to write his own scheme to scratch a mental itch, and he’s blogging each step and posting the code as well. I think it looks like a great amount of fun and that I too have that mental itch, so I’m following along, using his blog as a guide and looking at his code as well when I get stuck, but doing my best to do it myself.
• The great mindstab.net migration to the cloud! Yes, setting up an entirely new server and migrating years of site history and email etc can take a lot of work.

So yeah, I have an insane amount of work on my plate, but I couldn’t be more excited! All of it is thrilling and amazing!

Also, if I haven’t mentioned it before, the dynamic duo of Jono Bacon and Stuart Langridge of Lug Radio fame are back with a new podcast Shot of Jaq! It’s fun. Really, those two Brits have been the source of the only podcast’s I’ve ever listened to. They are a great source of both Linux and British in my weekly diet.

But now I’m stoked to just find out that Ximian/Linux rockstar coder Nat Friedman and Tomboy creator and a rock star in his own right Alex Graveley have started a brand new podcast Hacker Medley that is the first new podcast that I’m actually quite excited to try out.

Finally, I’m reading “Pattern Recognition” by William Gibson in my spare time (read: on the bus) and finding it pleasant.