Mindstab.net

So today is the 11th year anniversary of the site. It’s really starting to get some age on it :) Last year was pretty light on posts and side project work progress as work and coursera classes ate most of my time but I got some stuff done. Also a lot of reading.

As for this year? Who knows. I always intend to blog more than I do but at least I also haven’t given up and that’s the most important thing. The site still pulls in some little traffic from google so google analytics tells me so that’s good too.

This year has started off on vacation in Colombia with my girlfriend’s family. We spent new years in Santa Marta and it was beautiful. Now we’re relaxing back home in Bogota. I head back to Vancouver and work at the end of next week. The break has been good and 17 hour car trips and chilling at the beach have gotten the year off to a good start reading wise too.

This year I really would like to spend more time on projects than learning I think and continue to poke more at python, I’m still pretty green at it compared to say my PHP lately. We shall see.

In a lot of ways outside of work last year was pretty chill, I’d like to shake it up a bit more this year, again, we shall see.

Ah well, Happy New years to every one and here’s to doing the things we want to!

So summer seems to have come to a close both by calendar reckoning and by checking with the weather. So what have I been up to?

Well probably primarily just plain old work. Aside from that? Well, in the spring my overly ambitious plan to take 5 Coursera classes at the same time as working full time collapsed a bit. I did manager to pull off completing Software as a Service, Model Thinking and Game Theory.

After that I took some time off of “school”. I poked my old Lisp OpenGL flight sim but not to much noticeable effect. Reworked some of the model and rendering code and did manage to add an animated sidejet that pushes the ship sideways (all that work is in the rotate branch). Possibly in a year or two more at this slow pace I’ll have a fully movable ship. Anyways, it did make a good break.

I also ramped up my reading and discovered Goodreads into which I important my current to-read list and then explored its recomendation and lists to substantially expand it including most of the top reddit and other sci-fi lists. I’ve started cracking away at it so that should keep me occupied for a year or two.

But the break from school wasn’t to last. My friend Rob caught the first run through of the crypto class and broguht me back to coursera. So right now I’m just finishing off the Quantum Computing course (and doing rather poorly in it as its very much math and very little programming), and just starting the second run of Cryptography. And next month as crypto winds down I want to do the Scala class (taught by the author of the language! :D) and possibly the Human-Computer Interaction course if I can cram it in. We shall see.

On top of that I’ve finally gotten around to another goal I had for this year which was to start playing with Django a web framework in Python. I’ve just finished the 4 part intro tutorial and am ready to try a simple test site I’ve had in mind for a bit.

I also added one more new money and time consuming hobby this year: WarMachine. So that’s also a thing taking some of my time. For anyone curious, I play Cygnar.

Goals for the last quarter of the year? Well, actually finish my coursera courses, do the previously mentioned django test site, blog more than once every few months, cram/learn some spanish before my new years trip to Colombia to visit family, and read very many books. So we’ll see how that goes.

Hope you all had a good, fun and or productive summer!

So this has been a bleak year for internet freedom. ACTA, SOPA, PIPA, and in Canada Bill C30. Everyone’s angry, but what are we doing about it? Sharing links on facebook? Signing petitions. It is clearly not enough. It’s barely anything. Meanwhile, the oposition is pouring tanks of money onto the battle. They can afford to do so because they are rich, rich off us no less. But we have the advantage of vastly out numbering them. So let’s do something to actually be heard. We have agents of our own, and if we can help them, we should. So to that end I’ve donated to the following:

• The Electronic Frontier Foundation: To help fight against SOPA and PIPA and what comes next in the States, because it’s effects will be felt up here too.
• Open Media .ca: A Canadian organization supporting a free internet. Today they are even having a donate and be matched by Acanac drive, so what ever you donate will count double!
• Canadian Internet Policy and Public Interest Clinic: On advice from a lawyer friend who interned there I also sent some money their way.

All in all I may not have lots, but I have some to spare, and you may have even less, but if you can spare even $10, spend a bit to help fight for our free internet.

Just a note: I’ve now been running mindstab.net for 10 years. It’s been a long ride since I started on a 486 in my bedroom, but there’s lots more to come! Just thought I’d take a moment to mark the occasion.

I just stumpled upon the Cult of Done Manifesto on Hacker News yesterday and I like it. It seems like some good advice, and I sure haven’t been too productive as of late with my new job going on, so in the first act I’m releasing what I do have done for Cortex. I set up a github repo for cortex and pushed the latest code. The master branch is a slightly improved version of Cortex from where it was last year for the BCNet presentation and the modular branch is a work in progress to chop it up and make it more modular for better future extensibility. I’ve been stalled on it but as the guide suggests that may be mental and either way I should get what I have out there. Now I can more happily comfortably move on (if it’s been stalling me, which i think a little it has) or even work on it more, now with less “pressure” of some looming release ahead of me :). So there it is, Cortex.

Also, things like Febrice Bellard’s Java Script CPU emulator that can run Linux are amazing, imressive and inspiring, and make me realize I really need to to more hacking.

So here’s to getting stuff out there! I want in on the Cult of Done.

• Reddit Q&A: IBM Watson Research Team Answers Your Questions
  IBM answer's questions about Watson's makeup from hardware, software to processes
  (tags: IBM AI Watson)
• IBM Watson – How to build your own "Watson Jr." in your basement
  Instructions on using off the shelf hardware and software to build a Watson Jr at home
  (tags: AI IBM watson homebrew)

I’ve finally finished school. It’s been a bit of a journey, about 6 and 1/2 years, but I did it. In a few months a Bachelors of Science, majoring in Computer Science will be given to me by UBC. I’ve had a good go. I took a lot of neat class on a lot of interesting subjects. I’m happy with my education and record. I had time to write some really cool code on my own, like the mindstab Go AI competition, learning Lisp, and Cortex (partly for school) to name a few. And in between all that I also have had the time to travel, to go to some really cool places: Mexico, Guatemala, China, South Korea, Hong Kong, and Colombia. Life has been lucky and good.

Now time for something new, a new phase. And so to kick that off, I’ve landed (luckily) a one month trial contract at a web company downtown, as a PHP (and other assorted opensource technologies) developer. This is an amazing opportunity and I hope it goes well. From the two days I’ve had with them so far I really like it and would be very happy there permanently. Either way, I’m now much busier than I’ve been in a while.

Let’s say you did a presentation that was recorded and you’d like to post it to your website. Sadly, let’s now say there are some problems, like that your 5 minute presentation is part of a nearly 2 hour video only available in a flash player that doesn’t even have a time display so you couldn’t even point people to the video and say jump to 1 hour and 15 minutes to see me. It sucks. Technically your presentation is available online, but it’s not really accessible. So here is how you might rescue it!

It turns out there are two ways flash players server videos these days. The first and easiest is that a simple flash player loads in your browser, and uses your browser to make a GET request to the server to load a .flv file (FLash Video). This is relatively easy to intercept, there are lots of tools and plugins for Firefox that do this automatically for you. Even better, on Linux for example, these videos are usually stored in /tmp so your browser does the whole job and gives them to you. No work required.

The other more complicated but more secure option is that the flash player connects to a dedicated rtmp server that streams flash video. The flash plugin does the networking and there is no file to save, it’s a stream.

If you are lucky enough to have a player using the first option, you are done. Assuming you have the second option, then your fun has just begun.

First we need to try and figure out where the server that your flash video is.

My first approach was to use wireshark to sniff the traffic. Through this I discovered the basics, like the address the server and the port, 1935.

Next I installed rtmpdump. RTMP is the Real Time Messaging Protocol and rtmpdump is a program that can connect to an RTMP server, get a stream and save it to a file. Sadly the data I got from wireshark didn’t have all the parameters I needed to get the file. Or I couldn’t read it properly. So while I knew where the server was and could now connect to it, I still didn’t know how to ask for the video I wanted.

Thankfully rtmpdump comes with several other utilities. After reading its README I went the rtmpsuck route. I set local redirecting of all port 1935 requests to localhost with iptables and ran the rtmpsuck proxy server. In theory it was supposed to intercept all calls from the flash player to the rtmp server, decode and spit them out, and then forward them along. Even better, it would try to save the stream on the way back as it passed through it.

# iptables -t nat -A OUTPUT -p tcp --dport 1935 -m owner --uid-owner OWNER_UID  -j REDIRECT
$ ./rtmpsuck

Where OWNER_UID is the uid of the user running rtmpsuck. With this running I just reloaded the page with the player (twice, it’s a bit glitchy) and then tried to skip to where my part was so it would save the stream from there.

It was partially successful. It spit out on the console all the pertinent path parameters about the video on the server, but it kept chocking on bad packets of data and stopped recording. Also for some reason the video it did store was very large, space-consuming wise.

Armed with the right parameters though I was able to use rtmpdump to suck down the whole video from the server surprisingly quickly and in a reasonably sized format.

$ ./rtmpdump  -r rtmp://server.net/app_name/blah/event/date/date-1 -o video.flv

Now the video was liberated from its flash interface and in my possession, I just had to cut out my small part and then convert it to a more common format.

$ mencoder -ss 1:15:50  -endpos 0:05:57  -ovc copy -oac copy video.flv -o result.flv
$ ffmpeg -i result.flv result.avi

And volia. I now have just my part of the video and in a common format. I mean you hypothetically do! Yes…

Completely unrelatedly, you can expect to see my presentation on my project Cortex from the BCNet Broadband Innovation Challenge (where I got second place) online soon.

So 2010′s been a year.

Nearer it’s start some friends and I competed for the second time in MIT’s BattleCode, this time getting second non-MIT spot, or 18th overall. These competitions have been good for us: they are fun, it’s a good group project, and we work on our group project skills like planning, coordinating, and so forth. We spend so much time on our own, or in school even, working solo it’s good to work these skills as they will be needed later. Also it’s fun to learn about and catch up on low level AI stuff, like swarming and flocking movement/coordination techniques etc.

I also entered a school project into BCNet’s Broadband Innovation Challenge and got awarded second place. My project was “Cortex” a P2P processing app that runs with no software install entirely in your web browser. It was comprised of a small Java Applet webserver used as a backbone for communication and then a JavaScript front end, with all the control logic of the P2P network also written in JavaScript. I pretty much wrote a P2P app in JavaScript just using Java only to get around the AJAX/Server of Origin security policy issue. It was an interesting and challenging project and I’m pleased with how it did in the competition.

Over the summer I was in China which was amazing.

Then in the fall while finishing off my degree in CS once and for all I also competed in the Google sponsored University of Waterloo AI Contest. This, while being a simpler solo competition, was notable for me as it was my third major project undertaken in Lisp. I thoroughly enjoyed the challenge and again learned lots more about Lisp and again improved my Lisp style. Lisp and the emacs environment just take longer to learn and wrap my head around. And since I don’t get to work in them constantly, between work and school, it takes time. I’m by no means a master, but after convincing a friend to take a stab at the same competition in Lisp for his first try with Lisp, I at least see how far I’ve come. I’m getting more used to thinking functionally, especially with respect to using Lisp mapping functions instead of loops to modify, filter, or build on data. I placed disappointingly poorly due to lack of time, but I’m satisfied with what I learned (and also proud by association that the winner was a Lisp entry!). It was a good experience. I look forward to being able to undertake some more Lisp projects in the new year.

I also boned up on my Python this fall for a small work project, a multi threaded web crawler for a client. Played successfully with Python’s threading, so that was fun.

And that brings us to now. I’m in Colombia for the holidays, and in my vacation spare time I’ve finally gotten around to looking at the codebase to my school project “Cortex”. As school projects are, it worked, and well, but the codebase was a bit of a mess due to strong time constraints. Now that I have some time I’m doing some massive cleanups and adding a few features I’d wanted to but didn’t have time to. Hopefully early in the new year it’ll be in shape that I can release it. That would be nice.

So 2010 was a great year. I got to write a lot of cool code in several different language. I got to travel more than I ever have before, and I read a lot more than 2009 (traveling facilitates a lot of reading :)). It’s been a good year.

For 2011 though, now that I’m done with school, I’d like to start by releasing more code, starting with Cortex; getting more paying work; and looking at maybe starting a startup. I’d like to spend more time working on both AI (if you hadn’t noticed, obviously a hobby of mine) and in Lisp, starting with getting back into my signed copy of Peter Norvig’s “Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp” (yes getting it signed was awesome and a ridiculously geeky moment) and moving on from there. I’d like to at least keep up with the reading. I have high hopes for it to be an interesting year.

So here’s to 2010, you’ve been great, lets see if I can’t build on that for a more amazing 2011.

From UBC CSSS News letter:

1. Distinguished Lecture Series*
Speaker: *Peter Norvig, Director of Research, Google*
Date: Thursday, September 23
Time: 3:30 – 4:50 pm
Location: DMP 110

For those you you who forget, Peter Norvig is Google’s head of R&D and wrote “Paradigms of Artificial Intelligence Programming: Case Studies In Common Lisp” which I am slowly working through. I’m excited!