Mindstab.net

Just a note: I’ve now been running mindstab.net for 10 years. It’s been a long ride since I started on a 486 in my bedroom, but there’s lots more to come! Just thought I’d take a moment to mark the occasion.

I just stumpled upon the Cult of Done Manifesto on Hacker News yesterday and I like it. It seems like some good advice, and I sure haven’t been too productive as of late with my new job going on, so in the first act I’m releasing what I do have done for Cortex. I set up a github repo for cortex and pushed the latest code. The master branch is a slightly improved version of Cortex from where it was last year for the BCNet presentation and the modular branch is a work in progress to chop it up and make it more modular for better future extensibility. I’ve been stalled on it but as the guide suggests that may be mental and either way I should get what I have out there. Now I can more happily comfortably move on (if it’s been stalling me, which i think a little it has) or even work on it more, now with less “pressure” of some looming release ahead of me :). So there it is, Cortex.

Also, things like Febrice Bellard’s Java Script CPU emulator that can run Linux are amazing, imressive and inspiring, and make me realize I really need to to more hacking.

So here’s to getting stuff out there! I want in on the Cult of Done.

• Reddit Q&A: IBM Watson Research Team Answers Your Questions
  IBM answer's questions about Watson's makeup from hardware, software to processes
  (tags: IBM AI Watson)
• IBM Watson – How to build your own "Watson Jr." in your basement
  Instructions on using off the shelf hardware and software to build a Watson Jr at home
  (tags: AI IBM watson homebrew)

I’ve finally finished school. It’s been a bit of a journey, about 6 and 1/2 years, but I did it. In a few months a Bachelors of Science, majoring in Computer Science will be given to me by UBC. I’ve had a good go. I took a lot of neat class on a lot of interesting subjects. I’m happy with my education and record. I had time to write some really cool code on my own, like the mindstab Go AI competition, learning Lisp, and Cortex (partly for school) to name a few. And in between all that I also have had the time to travel, to go to some really cool places: Mexico, Guatemala, China, South Korea, Hong Kong, and Colombia. Life has been lucky and good.

Now time for something new, a new phase. And so to kick that off, I’ve landed (luckily) a one month trial contract at a web company downtown, as a PHP (and other assorted opensource technologies) developer. This is an amazing opportunity and I hope it goes well. From the two days I’ve had with them so far I really like it and would be very happy there permanently. Either way, I’m now much busier than I’ve been in a while.

Let’s say you did a presentation that was recorded and you’d like to post it to your website. Sadly, let’s now say there are some problems, like that your 5 minute presentation is part of a nearly 2 hour video only available in a flash player that doesn’t even have a time display so you couldn’t even point people to the video and say jump to 1 hour and 15 minutes to see me. It sucks. Technically your presentation is available online, but it’s not really accessible. So here is how you might rescue it!

It turns out there are two ways flash players server videos these days. The first and easiest is that a simple flash player loads in your browser, and uses your browser to make a GET request to the server to load a .flv file (FLash Video). This is relatively easy to intercept, there are lots of tools and plugins for Firefox that do this automatically for you. Even better, on Linux for example, these videos are usually stored in /tmp so your browser does the whole job and gives them to you. No work required.

The other more complicated but more secure option is that the flash player connects to a dedicated rtmp server that streams flash video. The flash plugin does the networking and there is no file to save, it’s a stream.

If you are lucky enough to have a player using the first option, you are done. Assuming you have the second option, then your fun has just begun.

First we need to try and figure out where the server that your flash video is.

My first approach was to use wireshark to sniff the traffic. Through this I discovered the basics, like the address the server and the port, 1935.

Next I installed rtmpdump. RTMP is the Real Time Messaging Protocol and rtmpdump is a program that can connect to an RTMP server, get a stream and save it to a file. Sadly the data I got from wireshark didn’t have all the parameters I needed to get the file. Or I couldn’t read it properly. So while I knew where the server was and could now connect to it, I still didn’t know how to ask for the video I wanted.

Thankfully rtmpdump comes with several other utilities. After reading its README I went the rtmpsuck route. I set local redirecting of all port 1935 requests to localhost with iptables and ran the rtmpsuck proxy server. In theory it was supposed to intercept all calls from the flash player to the rtmp server, decode and spit them out, and then forward them along. Even better, it would try to save the stream on the way back as it passed through it.

# iptables -t nat -A OUTPUT -p tcp --dport 1935 -m owner --uid-owner OWNER_UID  -j REDIRECT
$ ./rtmpsuck

Where OWNER_UID is the uid of the user running rtmpsuck. With this running I just reloaded the page with the player (twice, it’s a bit glitchy) and then tried to skip to where my part was so it would save the stream from there.

It was partially successful. It spit out on the console all the pertinent path parameters about the video on the server, but it kept chocking on bad packets of data and stopped recording. Also for some reason the video it did store was very large, space-consuming wise.

Armed with the right parameters though I was able to use rtmpdump to suck down the whole video from the server surprisingly quickly and in a reasonably sized format.

$ ./rtmpdump  -r rtmp://server.net/app_name/blah/event/date/date-1 -o video.flv

Now the video was liberated from its flash interface and in my possession, I just had to cut out my small part and then convert it to a more common format.

$ mencoder -ss 1:15:50  -endpos 0:05:57  -ovc copy -oac copy video.flv -o result.flv
$ ffmpeg -i result.flv result.avi

And volia. I now have just my part of the video and in a common format. I mean you hypothetically do! Yes…

Completely unrelatedly, you can expect to see my presentation on my project Cortex from the BCNet Broadband Innovation Challenge (where I got second place) online soon.

So 2010′s been a year.

Nearer it’s start some friends and I competed for the second time in MIT’s BattleCode, this time getting second non-MIT spot, or 18th overall. These competitions have been good for us: they are fun, it’s a good group project, and we work on our group project skills like planning, coordinating, and so forth. We spend so much time on our own, or in school even, working solo it’s good to work these skills as they will be needed later. Also it’s fun to learn about and catch up on low level AI stuff, like swarming and flocking movement/coordination techniques etc.

I also entered a school project into BCNet’s Broadband Innovation Challenge and got awarded second place. My project was “Cortex” a P2P processing app that runs with no software install entirely in your web browser. It was comprised of a small Java Applet webserver used as a backbone for communication and then a JavaScript front end, with all the control logic of the P2P network also written in JavaScript. I pretty much wrote a P2P app in JavaScript just using Java only to get around the AJAX/Server of Origin security policy issue. It was an interesting and challenging project and I’m pleased with how it did in the competition.

Over the summer I was in China which was amazing.

Then in the fall while finishing off my degree in CS once and for all I also competed in the Google sponsored University of Waterloo AI Contest. This, while being a simpler solo competition, was notable for me as it was my third major project undertaken in Lisp. I thoroughly enjoyed the challenge and again learned lots more about Lisp and again improved my Lisp style. Lisp and the emacs environment just take longer to learn and wrap my head around. And since I don’t get to work in them constantly, between work and school, it takes time. I’m by no means a master, but after convincing a friend to take a stab at the same competition in Lisp for his first try with Lisp, I at least see how far I’ve come. I’m getting more used to thinking functionally, especially with respect to using Lisp mapping functions instead of loops to modify, filter, or build on data. I placed disappointingly poorly due to lack of time, but I’m satisfied with what I learned (and also proud by association that the winner was a Lisp entry!). It was a good experience. I look forward to being able to undertake some more Lisp projects in the new year.

I also boned up on my Python this fall for a small work project, a multi threaded web crawler for a client. Played successfully with Python’s threading, so that was fun.

And that brings us to now. I’m in Colombia for the holidays, and in my vacation spare time I’ve finally gotten around to looking at the codebase to my school project “Cortex”. As school projects are, it worked, and well, but the codebase was a bit of a mess due to strong time constraints. Now that I have some time I’m doing some massive cleanups and adding a few features I’d wanted to but didn’t have time to. Hopefully early in the new year it’ll be in shape that I can release it. That would be nice.

So 2010 was a great year. I got to write a lot of cool code in several different language. I got to travel more than I ever have before, and I read a lot more than 2009 (traveling facilitates a lot of reading :)). It’s been a good year.

For 2011 though, now that I’m done with school, I’d like to start by releasing more code, starting with Cortex; getting more paying work; and looking at maybe starting a startup. I’d like to spend more time working on both AI (if you hadn’t noticed, obviously a hobby of mine) and in Lisp, starting with getting back into my signed copy of Peter Norvig’s “Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp” (yes getting it signed was awesome and a ridiculously geeky moment) and moving on from there. I’d like to at least keep up with the reading. I have high hopes for it to be an interesting year.

So here’s to 2010, you’ve been great, lets see if I can’t build on that for a more amazing 2011.

From UBC CSSS News letter:

1. Distinguished Lecture Series*
Speaker: *Peter Norvig, Director of Research, Google*
Date: Thursday, September 23
Time: 3:30 – 4:50 pm
Location: DMP 110

For those you you who forget, Peter Norvig is Google’s head of R&D and wrote “Paradigms of Artificial Intelligence Programming: Case Studies In Common Lisp” which I am slowly working through. I’m excited!

So, what have I got on the go?

• School: Last semester, just two classes, but they are looking like they’ll be delicious and meaty
  • CS 411: Compiler design: We build a java compiler
  • CS 415: We build an operating system, fun times with C!
  • …actually, I’m also taking spanish!
• “Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp” by Peter Norvig. I got this for christmas and have started to work through it. I’m really excited about getting further into it. In the middle of it I’ll be implementing prolog in Lisp and the in the latter half I’ll be working on a natural language processor! Not to mention tons of other stuff, this book is huge and dense.
• BattleCode 2010 has just started so my team and I are just about to start digging into that. Lots of AI coding to be done there.
• Peter Michaux’s Scheme from Scratch. I stumbled upon this from Hacker News. This fellow wants to write his own scheme to scratch a mental itch, and he’s blogging each step and posting the code as well. I think it looks like a great amount of fun and that I too have that mental itch, so I’m following along, using his blog as a guide and looking at his code as well when I get stuck, but doing my best to do it myself.
• The great mindstab.net migration to the cloud! Yes, setting up an entirely new server and migrating years of site history and email etc can take a lot of work.

So yeah, I have an insane amount of work on my plate, but I couldn’t be more excited! All of it is thrilling and amazing!

Also, if I haven’t mentioned it before, the dynamic duo of Jono Bacon and Stuart Langridge of Lug Radio fame are back with a new podcast Shot of Jaq! It’s fun. Really, those two Brits have been the source of the only podcast’s I’ve ever listened to. They are a great source of both Linux and British in my weekly diet.

But now I’m stoked to just find out that Ximian/Linux rockstar coder Nat Friedman and Tomboy creator and a rock star in his own right Alex Graveley have started a brand new podcast Hacker Medley that is the first new podcast that I’m actually quite excited to try out.

Finally, I’m reading “Pattern Recognition” by William Gibson in my spare time (read: on the bus) and finding it pleasant.

If you’re wondering what’s been eating my time, the simple answer is school. It’s my last year, there are things to be done. I’m helping a lot more, which takes time, for one thing.

The other thing is my CS 416 (distributed computing) term project. It’s kind of neat, Douglas Hofstadter would approve. It’s a P2P website: a website with a java applet that is a webserver, that can server the whole web site to other browsers, and it also servers it self to itself, and by using AJAX, the website can talk to “itself”, or more specifically, the server, which can also talk via AJAX to other servers. So not only does the website server itself and talk to itself, it forms a network of all the websites. I’ll probably put it online when it’s a bit more finished.

So yeah. I’m doing ok with keeping on top of my classes, but it pretty much takes all my free time that went to coding this summer. I’ll be back eventually ;)

Mindstab.net returns! Let me explain what happened:

I hosted my server on a small business line with Telus. A little over 2 weeks ago my house fell off the internet. I called them and they said my modem must be old and broken so they’d send me a new one in 5 business days tops.

I waited the 5 days and called on the last after not receiving a new modem and they said it’d only be a day or two more tops. I called a few days later after still receiving no modem and now having been offline for a week and a half and they double checked and it turns out my new modem order hadn’t even yet been processed! Even after a reminder call. Unbelievable. So I canceled my (lack of) service with them and registered with the other head of our local two headed ISP monopoly, Shaw. They had me online with in 4 days, and I just got my static IPs today. So mindstab.net is back online, as am I. Lets see how this goes.