Mindstab.net

So summer seems to have come to a close both by calendar reckoning and by checking with the weather. So what have I been up to?

Well probably primarily just plain old work. Aside from that? Well, in the spring my overly ambitious plan to take 5 Coursera classes at the same time as working full time collapsed a bit. I did manager to pull off completing Software as a Service, Model Thinking and Game Theory.

After that I took some time off of “school”. I poked my old Lisp OpenGL flight sim but not to much noticeable effect. Reworked some of the model and rendering code and did manage to add an animated sidejet that pushes the ship sideways (all that work is in the rotate branch). Possibly in a year or two more at this slow pace I’ll have a fully movable ship. Anyways, it did make a good break.

I also ramped up my reading and discovered Goodreads into which I important my current to-read list and then explored its recomendation and lists to substantially expand it including most of the top reddit and other sci-fi lists. I’ve started cracking away at it so that should keep me occupied for a year or two.

But the break from school wasn’t to last. My friend Rob caught the first run through of the crypto class and broguht me back to coursera. So right now I’m just finishing off the Quantum Computing course (and doing rather poorly in it as its very much math and very little programming), and just starting the second run of Cryptography. And next month as crypto winds down I want to do the Scala class (taught by the author of the language! :D) and possibly the Human-Computer Interaction course if I can cram it in. We shall see.

On top of that I’ve finally gotten around to another goal I had for this year which was to start playing with Django a web framework in Python. I’ve just finished the 4 part intro tutorial and am ready to try a simple test site I’ve had in mind for a bit.

I also added one more new money and time consuming hobby this year: WarMachine. So that’s also a thing taking some of my time. For anyone curious, I play Cygnar.

Goals for the last quarter of the year? Well, actually finish my coursera courses, do the previously mentioned django test site, blog more than once every few months, cram/learn some spanish before my new years trip to Colombia to visit family, and read very many books. So we’ll see how that goes.

Hope you all had a good, fun and or productive summer!

So 2010′s been a year.

Nearer it’s start some friends and I competed for the second time in MIT’s BattleCode, this time getting second non-MIT spot, or 18th overall. These competitions have been good for us: they are fun, it’s a good group project, and we work on our group project skills like planning, coordinating, and so forth. We spend so much time on our own, or in school even, working solo it’s good to work these skills as they will be needed later. Also it’s fun to learn about and catch up on low level AI stuff, like swarming and flocking movement/coordination techniques etc.

I also entered a school project into BCNet’s Broadband Innovation Challenge and got awarded second place. My project was “Cortex” a P2P processing app that runs with no software install entirely in your web browser. It was comprised of a small Java Applet webserver used as a backbone for communication and then a JavaScript front end, with all the control logic of the P2P network also written in JavaScript. I pretty much wrote a P2P app in JavaScript just using Java only to get around the AJAX/Server of Origin security policy issue. It was an interesting and challenging project and I’m pleased with how it did in the competition.

Over the summer I was in China which was amazing.

Then in the fall while finishing off my degree in CS once and for all I also competed in the Google sponsored University of Waterloo AI Contest. This, while being a simpler solo competition, was notable for me as it was my third major project undertaken in Lisp. I thoroughly enjoyed the challenge and again learned lots more about Lisp and again improved my Lisp style. Lisp and the emacs environment just take longer to learn and wrap my head around. And since I don’t get to work in them constantly, between work and school, it takes time. I’m by no means a master, but after convincing a friend to take a stab at the same competition in Lisp for his first try with Lisp, I at least see how far I’ve come. I’m getting more used to thinking functionally, especially with respect to using Lisp mapping functions instead of loops to modify, filter, or build on data. I placed disappointingly poorly due to lack of time, but I’m satisfied with what I learned (and also proud by association that the winner was a Lisp entry!). It was a good experience. I look forward to being able to undertake some more Lisp projects in the new year.

I also boned up on my Python this fall for a small work project, a multi threaded web crawler for a client. Played successfully with Python’s threading, so that was fun.

And that brings us to now. I’m in Colombia for the holidays, and in my vacation spare time I’ve finally gotten around to looking at the codebase to my school project “Cortex”. As school projects are, it worked, and well, but the codebase was a bit of a mess due to strong time constraints. Now that I have some time I’m doing some massive cleanups and adding a few features I’d wanted to but didn’t have time to. Hopefully early in the new year it’ll be in shape that I can release it. That would be nice.

So 2010 was a great year. I got to write a lot of cool code in several different language. I got to travel more than I ever have before, and I read a lot more than 2009 (traveling facilitates a lot of reading :)). It’s been a good year.

For 2011 though, now that I’m done with school, I’d like to start by releasing more code, starting with Cortex; getting more paying work; and looking at maybe starting a startup. I’d like to spend more time working on both AI (if you hadn’t noticed, obviously a hobby of mine) and in Lisp, starting with getting back into my signed copy of Peter Norvig’s “Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp” (yes getting it signed was awesome and a ridiculously geeky moment) and moving on from there. I’d like to at least keep up with the reading. I have high hopes for it to be an interesting year.

So here’s to 2010, you’ve been great, lets see if I can’t build on that for a more amazing 2011.

Update (2011-05-06): Reader and generally helpful person Brian Visel has posted a solution in the comments. Thanks!

So I’m writing an app for a customer. The language used is python. The part I’m working on now is a bit like a web spider. It needs to be multi-threaded and handle Unicode and be able to rotate through multiple proxies. I’ve been using urllib2 to do the data fetching and it’s served me well so far and getting it to work with HTTP proxies wasn’t too problematical, just create a ProxyHandler, and then build_opener() with that and call opener.open() and volia, a proxied thread-safe Unicode solution.

However things took a turn for the more complicated when I tried to add SOCKS proxy support. It’s apparently not natively handled by urllib2 because it’s implemented at a lower level. A bit of googleing turned me onto my first not-a-solution: SocksiPy whos solution is to create a new socket that routes through the SOCKs proxy and replace the global socket.socket. Then urllib2 will of course have it’s traffic routed through the SOCKS proxy. The only problem is this is massively not thread safe if other threads are using other SOCKS or HTTP proxies.

So I did some more googeling and found that I could bypass urllib2 entierly and try pycURL a thin python wrapper for cURL. This actually was thread safe, however I can’t seem to get unicode support out of it for urls or returned data, which is a no go because the sites this code will be crawling use Unicode.

So I’m basically stuck not being able to handle SOCKS thread-safely with unicode in python :/ I have other parts of the project to work on now before my deadline which I’m moving onto. If I get done early I guess I’ll come back to this and see if I can find something better.

So this has been in the works for a bit, and now that I have the basic server software up and running I can let the cat out of the bag! We (some friends and I) are having ourselves an AI competition writing Go bots (Go the ancient Japanese board game). All the details of the competition are on the wiki at ai.mindstab.net and the client (and server) software can be gotten from ai.mindstab.net/wiki/index.php/Goserver.

Anyone who is interested is more than welcome to join our merry little band of novice AI hackers. Just sign up on the wiki and make your presence known.

As for writing the Goserver software, that was an interesting little adventure. We built it around GnuGo’s twogtp.py initially. This was my first time really working in Python, and I’ve got to say it was a pleasure. The language does a really credible job of getting out of your way and just letting you do stuff. There are a few odd little things about the language but once you get up to speed coding in it they hardly seem a problem. I still think I like Ruby overall for elegance, but until Ruby 2.0 lands with it’s proper VM, execution speed isn’t even comparable. Not that for this server that was an issue, more that we had a bunch of already written python code in gtptwo.py, which is is why Python was chosen. Still defiantly pleased, it’s a fine language. So yeah, fun times.

As for a little more history for the 0 people who are interested, we’re using GTP (Go Text Protocol) for the communication between engines and server. It’s a protocol penned by the folks at GNU for GNU Go but being adopted else where. So there’s plenty of software that can speak GTP as a client and several engine vs engine pieces of code using GTP out on the web GPLed for public use. However, running an actually Go server means you need to facilitate lots and lots of matches and also keep track of scores. Code for all that for the different internet Go servers is all closed source, which is why we had to write our own before we could start the competition. That done, we’re all ready to Go!