on Lispkits performance

[bentxt]

trafficstars

Hi, I just run a simple and very limited benchmark to get idea where Lispkit score in terms of and startup and runtime performance. Below there are some mesurements where the test shows that Lispkit is slower.

Can you say something on Lispkits performance? Are there specific reasons? Is my testing setup wrong?

Thanks

Simple hello.scm

(display "hellooo")

Lispkit

time .build/debug/LispKitRepl -r Sources/LispKit/Resources -d LispKit hello.scm
hellooo
________________________________________________________
Executed in  689.86 millis    fish           external
   usr time  651.39 millis    0.20 millis  651.18 millis
   sys time   26.63 millis    1.53 millis   25.09 millis

Gauche

time gosh hello.scm
hellooo
________________________________________________________
Executed in   59.41 millis    fish           external
   usr time   28.57 millis    0.17 millis   28.40 millis
   sys time   18.11 millis    1.26 millis   16.85 millis

Simple fib.scm

(define (fib n)
  (if (<= n 1) n
      (+ (fib (- n 1))
	       (fib (- n 2)))))

(display (fib 30))
(newline)

Lispkit

time .build/debug/LispKitRepl -r Sources/LispKit/Resources -d LispKit fib.scm
75025

________________________________________________________
Executed in    4.62 secs    fish           external
   usr time    4.53 secs    0.18 millis    4.53 secs
   sys time    0.05 secs    1.49 millis    0.05 secs

Gosh

time gosh fib.scm
75025

________________________________________________________
Executed in   32.94 millis    fish           external
   usr time   32.40 millis    0.17 millis   32.23 millis
   sys time   15.04 millis    1.47 millis   13.57 millis

[objecthub]

If you care about performance then LispKit isn't really the right tool as it inherits quite a few performance penalties from Swift itself (e.g. it uses enumerations with associated values extensively which tend to be quite slow and also ARC as garbage collector). But if you do want to benchmark it, you should definitely only use release binaries. The difference is significant.

You can create a release binary with make release. This is what I get with release binaries for hello.scm:

[LispKit/swift-lispkit] time .build/release/LispKitRepl -r Sources/LispKit/Resources --- hello.scm
hellooo
================
CPU	91%
user	67ms
system	14ms
total	88ms

[LispKit/swift-lispkit] time gosh hello.scm
hellooo
================
CPU	142%
user	43ms
system	34ms
total	54ms

And these are the results for fib.scm:

[LispKit/swift-lispkit] time .build/release/LispKitRepl -r Sources/LispKit/Resources --- fib.scm
75025
================
CPU	96%
user	332ms
system	18ms
total	361ms

[LispKit/swift-lispkit] time gosh fib.scm 
75025
================
CPU	161%
user	51ms
system	22ms
total	45ms

The results are exactly what I would expect: LispKit should have a similar startup time compared to Gauche. But anything computationally expensive will be way slower compared to an optimizing compiler like Gauche (at least an order of magnitude). You can see that a significant amount of time is spent in the mathematical functions (which are all automatically turned into native integer arithmetic by Gauche) by replacing LispKit's polymorphic arithmetics with plain fixnum operations:

(define (fib n)
  (if (fx<= n 1) n
      (fx+ (fib (fx1- n))
	         (fib (fx- n 2)))))
(display (fib 25))
(newline)

Here's how that performs:

[LispKit/swift-lispkit] time .build/release/LispKitRepl -r Sources/LispKit/Resources --- fib2.scm
75025
================
CPU	96%
user	200ms
system	14ms
total	223ms

Another interesting observation — and this is why I gave up running microbenchmarks long ago — is that the performance of code compiled with Xcode IDE is significantly better than with the command-line tools. Using the time procedure from library (lispkit debug), I get the following via LispKitRepl:

> (time (fib 30))
elapsed time = 1.4443590420014516
832040

And this is what I get with the latest version of LispPad (compiled with the Xcode IDE):

➤ (time (fib 30))
elapsed time = 1.0428437919981661
832040