hubFS: THE place for F#

Hi,

In examining the viability of converting an existing ocaml codebase to F# I recently ran some performance benchmarks and was pleasantly surprised by the results (reproduced below), The code used to benchmark is not my own - it comes from the ocaml submissions to the Computer Language Benchmarks Game. I think everybody realises how little you can read into benchmarks. They are afterall particular implementations of particular algorithms (often not written in the dominant idiom of the language), compiled/interpreted with a particular implementation, run on particular hardware etc. So with that said, I hope nobody takes this post as an attempt to 'prove' something definite about the performance of F# code relative to other languages. I've posted this purely because I thought one or two people might be interested.

Some notes on the methodology (in no particular order):

1. Programs were taken from the language shootout ocaml submissions where the ocaml source either needed no changes to compile or the changes were extremely trivial changes in syntax (i.e removal of ocaml named arguments, addition of spaces between brackets).

2. All tests were run three times and the mean taken as the result. The time was measured using the *nix 'time' utility. On windows the tests were run from a cygwin shell in order to have access to this utility.

3. Ocaml programs were built with the native code compiler (ocamlopt) using the flags suggested by the shootout.

4. F# programs were built using VS 2005 release mode on Windows XP. Mono version was 1.2.2.1 with flag --optimize=all. The ahead of time compiler was not used because fsc produces .NET 2.0 assemblies for which aot doesn't work (apparently, I'm no Mono expert).

5. gcc programs were built with the flags suggested by the shootout except with march changed to athlon-xp and -mfpmath=sse -msse2 added in 2 cases where they are omitted by the shootout but produce a performance increase on my machine.

6. Mono F# and gcc were both built and run on Linux (Gentoo). I realise it would have been better to run all benchmarks under one operating system but things didn't work out that way and as stated above - this exercise was always intended to be taken with a pinch of salt.

Some notes on the results:

1. F# performance with the Microsoft runtime is generally very good. In two cases actually beating c code compiled specifically for the testing hardware (see n-body and spectral norm). I'm guessing that this is to do with some very clever floating point arithmetic optimisations in the .NET compiler.

2. F# does poorly on two benchmarks (pidigits and startup). I think its safe to ignore the startup benchmark since most programs actually do some work. I've no idea why F# does badly on pidigits, although one thing I did notice is that ocaml buffers output whereas F# seems to flush the output stream after every call to printf. C blows the competition out of the water on pidigits probably because of the use of gmp.

3. Mono F# is definitely the poorest performer on the benchmark, although most of the times are respectable. This may in part be due to my lack of knowledge about Mono. Personally, I expect Mono performance to increase with future releases.

Results:

	F# .NET	F# Mono (Linux)	ocaml(XP)	ocaml (Linux)	gcc (Linux)
Binary Trees (n = 16)	2.563	7.499	2.281	2.269	2.408
Fannkuch (n = 11)	6.188	9.337	6.610	6.303	4.130
Nsieve (n = 9)	0.906	1.805	0.844	0.712	0.539
N-body (n = 20000000)	7.328	36.896	36.969	23.939	7.398
Partialsums (n = 2500000)	1.610	2.395	2.422	1.648	1.314
Pidigits (n = 2500)	9.515	13.428	2.016	1.647	0.370
Recursive (n = 11)	3.172	3.515	4.797	4.664	1.541
Spectral-norm (n = 5500)	9.843	26.253	52.859	51.222	10.889
Startup	0.235	0.062	0.031	0.001	0.001

Ben Jackson.

Thanks for sharing the results of this study, while I agree that these kind of results should generally be taken if a pinch of salt, I think these kind of benmarks do serve a purpose, in that taking carefully taking a benchmark and refining your program then retaking the benchmark is pretty much the only way to address preformance issues.

It is interesting that pidigits test does not preform very well, I know the F# team were looking at improving this, it has already been discussed on this thread here:
http://cs.hubfs.net/forums/thread/1657.aspx

 

---

Robert Pickering, MVP
http://strangelights.com

Hi guys

Thanks for the analysis - and good caveats.

I took a look at pidigits last night and found numerous improvements that can be made to the BigInt implementation - neow about 40% faster. We'll get those fixes into the next release.

Kind regards

don

I've been using F#/Mono on OS X off and on. My understanding of the poor performance there revolves around three issues.

1. Mono Garbage Collection. The mono runtime still uses Boehm's conservative garbage collector. While that collector is well written, understood by industry, etc. it is not a compacting collector, which means it does not give you the "free" locality of reference boost that Microsoft's CLR does. I suppose the amount of garbage created by each benchmark would influence the delta between Mono & CLR; does this stand up? I am not sure what the mono team's plans on replacing the GC are.
   
2. Less impressive JIT compiler. Besides diving into very deep binary debugging sessions, I'm not sure how to evaluate the quality of the code produced by the mono JIT compiler, but this presentation suggests that it does very little optmization. This might make all the difference.
   
3. No Generics(?). Since ahead-of-time compilation with .NET 2.0 features is not yet supported on Mono, perhaps the F# compiler is working at a disadvantage here? (Question mark because this comment may be out of date. Things change so fast!)

I'd be curious to see your speculation on which of the above (or none of the above!) factors is affecting the difference between the CLR and Mono versions.

I'll translate my ray tracer benchmark into F# and post the results ASAP.

Hey Jon,

I apologize for being a little off topic and cheeky here but I was wondering how that Ray tracer benchmark was coming?

 

Are you so jazzed about Jocaml that you have forsaken us over here in the F# world?

 

I would also like to ask you about your thoughts about how the ocamlP3/jocaml multi-process implementations compare to F# (in terms of performance, ease of use)?

 

Is it true that jocaml relies on the Unix fork process command and thus cannot be ported to Windows?

 

Parnell

 

To cut to the chase, the Java and .NET approach to concurrency (using a single concurrent GC) works on Unix and Windows and offers the best performance for fine-grained concurrent applications but slows all allocations down by ~5x, does not scale to massive concurrency (millions of threads) and does not translate to distributed concurrency.

The OCaml/JoCaml approach to concurrency (forking processes using copy-on-write to give independent processes with their own GCs) works on Unix and offers the best performance for coarse-grained concurrent applications and is easily distributed but does not work on Windows, slows down fine-grained concurrent applications and also does not scale to massive concurrency.

The Erlang approach to concurrency (independent mini-GCs with unidirectional heaps) scales to massively concurrent applications but imposes more intrusive rules on the programmer (e.g. no cycles in the heap).

So there is no clear winner.

Note that functional programming languages compiling idiomatic functional programs to .NET will give pathologically bad performance due to their heavy allocation of short-lived objects.

I predict that the first F# translations on the Great Computer Language Shootout will be minimal translations of the OCaml implementations. As a consequence of the above, this will show F# in an unfairly-poor light because the programs will need reoptimizing (by avoiding rapid allocation at all costs). Another problem is that the shootout tasks are sufficiently ill-specified that most of the programs can be arbitrarily optimized away. Consequently, the maintainers are forced to subjectively reject submissions and, I suspect, they will reject most of the F#-specific optimizations.

Although it is interesting for a few people to compare the performance of OCaml and F#, the vast majority of people interested in F# are currently using C++ and C# rather than OCaml (statistics from the F# Journal survey). At least for the C# programmers, relative performance will be of little interest!

Finally, I should say that the factor of 5 that I referred to above is taken from the Sudoku Solver application articles in the OCaml and F# Journals. The implementations of the idiomatic purely functional solver algorithm are identical in both languages and the OCaml is 5x faster. The F# could be optimized by using a more sophisticated representation of a heap with more preallocated memory and larger chunks but this was not necessary in practice because all Sudoku puzzles can be solved in <0.01s, so the GUI remains responsive.

HTH.