Mindstab.net

Well, now that Bast was up and running I had to start testing it. I mean that's why I bought a Mac. So I tossed on some of my old code, specifically, VM-Proto, Fast-Lang, and Primes.

VM-proto was a tiny virtual machine and compiler I wrote several years ago to learn about basic compilation to byte code, and execution of said byte code. I also then made it very portable in order to learn about low level portability issues like endian safe code. I was pleased when vm-proto compiled, was able to compile source (a prime number finder) to object code and execute that object code. Better yet, object code vm-proto compiled on Bast (PPC) was executable by vm-proto on Inferno (x86), and object code compiled by vm-proto on Inferno, was executable on Bast. A complete success.

Next up was fast-lang, a simpler interpreter that I wrote this summer just to see how fast I could make a simple language. It compiled and was able to execute a simple stack test program I wrote for it, but failed on the more complex prime number finder with a seg fault. Clearly not so portable, although that was never a goal with it it, I just wanted it to be fast on what I had, which was x86.

Finally came my Primes suite, which is a collection of prime number finders I wrote in about 37 languages and a simple perl program to run them all and store benchmarks on how fast they ran. It was a fun way to get introduced to a lot of languages and also fun to race them against each other.

I've often used Primes as a crude benchmark of languages' computational power and now I was going to use it to see how well languages worked on a Mac as compared to x86.

I ran a subset of tests in C, Object C, C++, Java, C# (mono), Awk, Perl, PHP, Python, and Common Lisp (sbcl). I ran the tests on four of my computers for comparison: Inferno, a 1.5GHz Athlon-XP; Kvasir, a 2.8GHz Pentium 4; Nika, a 1.5GHz Pentium M, and of course Bast, my new 350MHz PPC G3. The results are below:

	Inferno	Nika	Kvasir	Bast
C	1.19	0.79	0.45	2.83
ObjC	1.19	0.8		2.83
C++	1.93	1.06	1.1	4.76
Java	3.59	1.63	2.14	40.3
C#	3.69	1.87		10.5
Awk	32.1	27.1	30	199
Perl	38.2	21	23.3	145
PHP	15.1	8.89	13.4	64.9
Python	54	38	43.8	211
Lisp	10.4	5.19		36.3

There were some surprising results in there, but let's start with Bast, the G3. C, Object C, and C++ were about par for the course, but then look at the spike in time for Java. For the record, Nika and Inferno are both using Sun's JDK 1.5, and Bast is using IBM's JDK 1.5. There really shouldn't be that difference. Clearly Java is a lot less optimized for the PPC. (Hopefully now that it's GPLed that can get some attention?). Next up was C# care of Mono, which looks good comparatively. Then a huge spike with Awk, which looks like it could be missing some PPC love. It's a bit hard to tell with perl, but it looks reasonable when compared with the PHP and Python results. Comparatively speaking PHP is looking pretty good on the PPC. Python is slowest, but it's slowest on x86 too so no surprise. And Finally Common Lisp care of SBCL, looking good, and beating Java on the PPC. Impressive. So what have we learned? Java and Awk need some PPC love. (I didn't bother to include Ruby results, because until 1.9 becomes the stable 2.0, Ruby doesn't officially have a VM implementation and is incomparably slow. Ruby 1.9/2.0 with YARV is competitive, but we'll leave that for another day).

Now to my surprise, aside from C, Kvasir lost to Nika on all tests, even though Nika is half the speed. Nika also beat out Inferno and they should be much more comparable. Also, on Inferno you can see that Perl actually performs worse than awk. So I'll try to guess and explain that. Inferno and Kvasir both use GCC 3.4. and compiled all the languages with GCC 3.4, while Nika uses GCC 4.1. Inferno and Nika both use -O3 optimization while Kvasir uses -O2 optimization, and Kvasir is using the hardened tool chain with a hardened kernel. So we can see that GCC 4.1 seems to provide a solid edge over 3.4 (as seen from Nika beating Inferno), and GCC 3.4 + the hardened tool chain + a hardened kernel gives a massive performance hit compared to vanilla gcc 4.1 as seen from Nika beating all languages but C on the nearly twice as powerful Kvasir. I'm a bit shocked to find that my laptop which I would have guessed would be the clear looser with respect to interpreted languages number crunching won out. Neat to know.

So now I've learned a few things.

You can grab Primes from my git server

git clone git://git.mindstab.net/git/primes

But be warned, the autoTest.pl code is old and beyond hideous. But it kind of gets the job done.

(And please no complaints that these tests aren't fair. I'm not claiming they are anything remotely definitive. They are what they are: prime number finders in a bunch of languages)

Edit: I forgot to mention two things:

1. Getting Java for PPC is still a pain. Sun-SDK in portage isn't available for PPC. All that is is the IBM version, which is still fetch restricted. You have to sign up with IBM to get it which involved you telling them all about your business, weather you have one or not, and all about yourself, and full addresses for both, plus a phone number and email address. Extremely lame and I hope the new GPL Java fixes this stat.

2. I didn't include fortran results because GCC 4.1 seems to have a new and improved fortran compiler and my terrible old code doesn't compile anymore and I wasn't in the mood to go brush up on fortran to figure out why. Would have been neat though.