Mindstab.net

Well, since, I got Debian installed on my GP2X, the first thing I did was run some benchmarks. As per usual, I fell back on my "raw and simple math computation" benchmark suite Primes, a collection of prime number finders written in many languages. For the rest of the article we will thus be evaluating languages purely on math computational speed and nothing else.

I had already gathered data from some of my computers from when I bought Bast, my G3, so I just ran the benchmarks on the GP2X and added them in for comparison.

Machines

Results

Time in seconds to find all prime numbers between 1 and one million

	Inferno	Nika    Kvasir  Bast    GP2X
C	1.19	0.79	0.45	2.83	35.1
ObjC	1.19	0.8		2.83	43.4
C++	1.93	1.06	1.1	4.76	50.1
Java	3.59	1.63	2.14	40.3
C#	3.69	1.87		10.5	140
Awk	32.1	27.1	30	199	2065
Perl	38.2	21	23.3	145	1280
PHP	15.1	8.89	13.4	64.9	758
Python	54	38	43.8	211	1526
Lisp	10.4	5.19		36.3	2674

As you can see, of course, the 200MHz ARM GP2X was destroyed by everyone else. This really doesn't tell us anything at all actually.

Clearly scaling needed to be introduced so that we could see how the languages faired on the different architectures. All being equal I assumed that GCC would give the best optimization per platform and that the C results would be the least skewed, so I choose to represent all the results as multiples of the C time results, or divide all the times by the time that the C prime number finder took on that computer. It yielded the following results:

Time in multiples of C time to find all prime numbers between 1 and one million

	Inferno	Nika    Kvasir  Bast    GP2X
C	1.00	1.00	1.00	1.00	1.00
ObjC	1.00	1.01		1.00	1.24
C++	1.62	1.34	2.44	1.68	1.43
Java	3.02	2.06	4.76	14.24
C#	3.10	2.37		3.71	3.99
Awk	26.97	34.30	66.67	70.32	58.83
Perl	32.10	26.58	51.78	51.24	36.47
PHP	12.69	11.25	29.78	22.93	21.60
Python	45.38	48.10	97.33	74.56	43.48
Lisp	8.74	6.57		12.83	76.18

Analysis

x86 machines

You would think we could use these 3 machines to establish a baseline but we find a fairly large variance in results here. First of all, inexplicably, Inferno is the only machine to have a worse result for Perl than Awk. Also, we can see that when it comes to the interpreted languages, Kvasir is near the worst in all cases. I would attribute this to Hardened Gentoo adding very viable overhead to the interpreters. It seems to show there is a definite cost for added security in terms of efficiency.

PPC

The most notable result here is the Java result which is clearly markedly bad. Java for the PPC could use some optimization. C# care of mono on the other hand is competitive. The rest of the interpreted languages also lag behind on the PPC, although Lisp (SBCL) comes pretty close the the x86 scaled results.

ARM

And now what you've been waiting for, the ARM results for the GP2X. Well, Sadly, I just couldn't really get Java for the GP2X. Sun Java appeared to be unavailable. I could have used something like Kaffe or Jikes, but it probably wouldn't have been to fair, and also, on Debian, they insisted in pulling in some parts of Xorg as dependencies, and I didn't have a lot of space to play around with on the GP2X. Kind of lame, oh well. Again, as with the PPC, C# care of mono is very respectable. Other than that, the Lisp result was way off by a staggering amount, but that can be explained by SBCL not being available for ARM. I had to resort to the clearly slower CLisp compiler/'interpreter.'

Languages

As much hype as there is around Python these days, it appears to be fairly slow across the board compared to other interpreted languages. PHP on fact is the fastest of PERL, PHP, and Python on all tested architectures, so if you need speed and yet still the flexibility of an interpreted language, you might seriously want to use the CLI version of PHP. It seems the interpreter is very optimized across the board.

If you happened to be on an architectures supported by a good Lisp compiler like SBCL it is also a very attractive and viable option, but unfortunately for ARM, it looks like CLisp isn't, speed wise anyway.

As for Java, hopefully the GPLing of it will allow it to a) be further optimized for alternate architectures like PPC, and b) for it to be fully ported to 'new' architectures like ARM. If you're looking for speed but the middle ground flexibility of a VM language than in the mean time C# in the form of Mono is a fantastic looking choice.

Note: Ruby has again been omitted because until 2.0 is released, which includes a real VM, it is sadly not even remotely competitive in this area (math computation). It is about an order of magnitude slower than any of the other interpreted languages. Ruby 1.9 CVS last I checked (half a year ago) was competitive with Python :)

And that's it from me on the Primes front (at east until I acquire a Sparc box :P ...)

Well, now that Bast was up and running I had to start testing it. I mean that's why I bought a Mac. So I tossed on some of my old code, specifically, VM-Proto, Fast-Lang, and Primes.

VM-proto was a tiny virtual machine and compiler I wrote several years ago to learn about basic compilation to byte code, and execution of said byte code. I also then made it very portable in order to learn about low level portability issues like endian safe code. I was pleased when vm-proto compiled, was able to compile source (a prime number finder) to object code and execute that object code. Better yet, object code vm-proto compiled on Bast (PPC) was executable by vm-proto on Inferno (x86), and object code compiled by vm-proto on Inferno, was executable on Bast. A complete success.

Next up was fast-lang, a simpler interpreter that I wrote this summer just to see how fast I could make a simple language. It compiled and was able to execute a simple stack test program I wrote for it, but failed on the more complex prime number finder with a seg fault. Clearly not so portable, although that was never a goal with it it, I just wanted it to be fast on what I had, which was x86.

Finally came my Primes suite, which is a collection of prime number finders I wrote in about 37 languages and a simple perl program to run them all and store benchmarks on how fast they ran. It was a fun way to get introduced to a lot of languages and also fun to race them against each other.

I've often used Primes as a crude benchmark of languages' computational power and now I was going to use it to see how well languages worked on a Mac as compared to x86.

I ran a subset of tests in C, Object C, C++, Java, C# (mono), Awk, Perl, PHP, Python, and Common Lisp (sbcl). I ran the tests on four of my computers for comparison: Inferno, a 1.5GHz Athlon-XP; Kvasir, a 2.8GHz Pentium 4; Nika, a 1.5GHz Pentium M, and of course Bast, my new 350MHz PPC G3. The results are below:

	Inferno	Nika	Kvasir	Bast
C	1.19	0.79	0.45	2.83
ObjC	1.19	0.8		2.83
C++	1.93	1.06	1.1	4.76
Java	3.59	1.63	2.14	40.3
C#	3.69	1.87		10.5
Awk	32.1	27.1	30	199
Perl	38.2	21	23.3	145
PHP	15.1	8.89	13.4	64.9
Python	54	38	43.8	211
Lisp	10.4	5.19		36.3

There were some surprising results in there, but let's start with Bast, the G3. C, Object C, and C++ were about par for the course, but then look at the spike in time for Java. For the record, Nika and Inferno are both using Sun's JDK 1.5, and Bast is using IBM's JDK 1.5. There really shouldn't be that difference. Clearly Java is a lot less optimized for the PPC. (Hopefully now that it's GPLed that can get some attention?). Next up was C# care of Mono, which looks good comparatively. Then a huge spike with Awk, which looks like it could be missing some PPC love. It's a bit hard to tell with perl, but it looks reasonable when compared with the PHP and Python results. Comparatively speaking PHP is looking pretty good on the PPC. Python is slowest, but it's slowest on x86 too so no surprise. And Finally Common Lisp care of SBCL, looking good, and beating Java on the PPC. Impressive. So what have we learned? Java and Awk need some PPC love. (I didn't bother to include Ruby results, because until 1.9 becomes the stable 2.0, Ruby doesn't officially have a VM implementation and is incomparably slow. Ruby 1.9/2.0 with YARV is competitive, but we'll leave that for another day).

Now to my surprise, aside from C, Kvasir lost to Nika on all tests, even though Nika is half the speed. Nika also beat out Inferno and they should be much more comparable. Also, on Inferno you can see that Perl actually performs worse than awk. So I'll try to guess and explain that. Inferno and Kvasir both use GCC 3.4. and compiled all the languages with GCC 3.4, while Nika uses GCC 4.1. Inferno and Nika both use -O3 optimization while Kvasir uses -O2 optimization, and Kvasir is using the hardened tool chain with a hardened kernel. So we can see that GCC 4.1 seems to provide a solid edge over 3.4 (as seen from Nika beating Inferno), and GCC 3.4 + the hardened tool chain + a hardened kernel gives a massive performance hit compared to vanilla gcc 4.1 as seen from Nika beating all languages but C on the nearly twice as powerful Kvasir. I'm a bit shocked to find that my laptop which I would have guessed would be the clear looser with respect to interpreted languages number crunching won out. Neat to know.

So now I've learned a few things.

You can grab Primes from my git server

git clone git://git.mindstab.net/git/primes

But be warned, the autoTest.pl code is old and beyond hideous. But it kind of gets the job done.

(And please no complaints that these tests aren't fair. I'm not claiming they are anything remotely definitive. They are what they are: prime number finders in a bunch of languages)

Edit: I forgot to mention two things:

1. Getting Java for PPC is still a pain. Sun-SDK in portage isn't available for PPC. All that is is the IBM version, which is still fetch restricted. You have to sign up with IBM to get it which involved you telling them all about your business, weather you have one or not, and all about yourself, and full addresses for both, plus a phone number and email address. Extremely lame and I hope the new GPL Java fixes this stat.

2. I didn't include fortran results because GCC 4.1 seems to have a new and improved fortran compiler and my terrible old code doesn't compile anymore and I wasn't in the mood to go brush up on fortran to figure out why. Would have been neat though.

Well, I now have Bast, my new Mac G3, booting under its own power. It took a bit more work than normal because I ran into a few problems. But its all up and running now, so let's go over the install.

Bast came with MacOS 9 installed. I booted into it, and it was pretty worthless. Admittedly, it did auto detect the net and I got to look at mindstab.net in Mac IE 5. That was about it. None the less, I had intended to try and save the MacOS partition since it was only using a few hundred megs out of a 20GB harddrive.

So put the Gentoo Minimal PPC 2006.1 disk in and rebooted. Unlike x86 where you can jump into the BIOS and change boot order, on a Mac you just hold down the C key when it boots for a while until you get to the cd's boot prompt. You can probably just push it, but it's hard to tell when, since there is no visual feed back, just a black screen, then a grey screen.

My first problems popup up here, or technically a little earlier. Even back in the 90s when this box was made, Apple had apparently moved to all USB peripherals, and cockily enough, not even bothered to include PS/2 ports on their towers. Kind of lame since I have a 4 port PS/2 KVM switch which this box was supposed to use. Thankfully I had picked up a cheap old Apple USB keyboard when I bough t the box. Reportedly though, it had the shortest keyboard cable I've ever seen. I quickly borrowed my usb hub from another box and got reasonable length from them combined. So all is good? Not quite. The Mac boot loader refused to acknowledge the USB keyboard through the USB hub, even with the hub plugged into external power. So I had to go stand over in the corner by the box, and plug the keyboard in and hit C to boot from cdrom and then ENTER to boot the cdrom, and then unplug the keyboard and plug it into the hub before I could sit down, every time I rebooted the mac. Pretty lame.

Anyways, the LiveCD booted up and after the boot loaders, no one had any problem with the Mac keyboard through the hub. Except of course the keyboard was glitchy and old and kept disconnecting/reconnecting so I lost keys when I typed. Had to watch what I typed very carefully. So I immediately got sshd up and running and that was that for the keyboard.

Next I brought up parted and took a look at the partition table. I tried to resize and shrink MacOS 9's HFS+ file system and partition but parted failed on me every time. Everything on the net assured me it would be doable, but they seemed wrong.

From there things went more smoothly. I followed the Gentoo PPC install guide and everything worked pretty well. At times though, I did feel like I was on a second class architecture because they'd give an x86 dependent example, and then below it as though slapped on as after after thought were some PPC instructions. Of course if the instructions didn't really differ, some times they only showed the x86 related version.

I was pretty sure I had the kernel configured properly, but it would have payed off a bit to read dmesg a little bit closer. More about that shortly. The only recommended/stable bootloader for PPC is yaboot. Yaboot doesn't really seem as advanced as grub, but it's not bad. But the docs probably could have elaborated on some points a little more, like what is OpenBoot? and all the options related to it and do I need to configure them? (apparently not)

Anyways, I thought I had everything and rebooted. Apparently not. I got a kernel panic because it couldn't find the harddrive. It took me several say to figure that one out. First of all, it seems to have 2 IDE controllers, one for the Zip drive and CDRom, and another for the harddrive. I had to look closer at dmesg to find the second, a CMD646. The other problem didn't become apparent until more reading.

On the liveCD the Apple IDE controller is compiled in and so those drives get hda and hdb. The CDM646 driver is a module and gets loaded later so on the liveCD the harddrive is hdc. However, once I was compiling the CDM646 driver into my kernel it still wasn't working because now it was detecting the harddrive first and calling it hda, and oddly the cdrom and zip drive because hde and hdf. But I couldn't see that unless I watched the system boot because once it kernel panicked I couldn't scroll back up.

This was a frustrating bug to diagnose and fix, but I finally got it and now the box booted just fine. I was very close but had one more thing to look into. Both on the LiveCD and the new system, /proc/cpuinfo was reporting 50 bogomips, for what should have been more like 700. This bothered me because I wasn't sure if I'd misconfigured something and was missing out on a lot of performance. Finally though, again, after a lot of searching, I found out that in recent kernels, the timing/scheduling mechanism has been altered to be more sane of PPC with the result that bogomips have become a useless measurement tool for performance. The Internet assures me I'm getting my full performance.

And so I now have Gentoo on a PPC computer, my alternate arch test box. Also, I've since picked up a Dynex PS/2 to USB converter and have the G3 hooked up to my KVM switch as it should be.

I've been meaning to pick up a PPC powered computer for a bit, because I'm a nutter like that. Anyway, I finally found a reasonable deal at TheMacMarket.com and picked up a 350MHz G3 with a 20GB hard drive and 256MB ram for $139. This box is now officially my alternate arch test box (big endian yeah!). I've wiped Mac OS 9 off of it and am installing Gentoo Linux. Details about the install later when it's done.

One question though, a G3 350MHz should be rating about 700ish bogomips, but I'm getting 50. Anyone know what might be up with that? Could it just be the LiveCD?