Some changes I did for my ClojureC REPL project

[bertfrees]

While playing around with ClojureC it occured to me that it would be cool to have a REPL for it. It seemed like a fun challenge and I gave it a shot. You can see where I am now at https://github.com/bertfrees/cljc.repl.

When I started, I hadn't seen the other effort from @ghubber yet, who tackled the problem by implementing a metacircular evaluator in ClojureC.

I didn't even think that far, I just did what seemed the most straightforward to me. The only way I knew how to dynamically generate and evaluate C code is to make a shared library and dlopen it. It may seem a bit brutal to make a library for each form you enter in the REPL, but the advantage is that it was fairly easy to implement, and you get "the real thing": everything you enter is actually compiled down to C. As far as I understand, in the metacircular evaluator approach there is no compilation involved, am I right?

Also, it seems to be reasonably fast.

Tell me what you think. And keep up the good work!

Cheers, Bert

[whilo]

The metacircular evalutor is just a sample program in ClojureC. I just wanted to use it to drive a potential compiler. I don't want to implement a Clojure REPL in it but use it as a vehicle to explore runtime behaviour. I don't know how far your runtime primitive support goes. My interest is mostly having a nice runtime construction environment for a functional Clojure runtime and potentially different emit targets (like clojure-scheme has with :gambit and :jvm e.g.) in the long run. This is why I have connected the metacircular evaluator to a cljs-in-cljs version of analyzer.cljs. This allows to analyze code from the ClojureC metacircular REPL without JVM. This might not be the best approach, I have to admit seeing that you can explore a C REPL from the JVM, it rather grew on me, I don't have enough experience in C, so I am keen on your code loading tricks. What are your plans/is your scope for the REPL?

[bertfrees]

OK, I must admit I only discovered your stuff recently and I hadn't really taken a closer look at it. But I think I'm starting to understand now. I can see that you've started to port the reader and the analyzer. That's great. Soon we'll have all the pieces to build a "pure C" REPL.

"I don't know how far your runtime primitive support goes."

I'm not sure I understand? Essentially everything you enter at the REPL is compiled and run as if you would've compiled and ran a normal ClojureC program, so everything that is supported in ClojureC should be supported at the REPL.

You should have a look at the code, there's really not much to it. Note that I'm not an experienced C hacker myself. The essential stuff (the dynamic loading of code with dlopen and dlsym) is in src/c/cljc/repl/runtime.c. The compilation from Clojure code to native libraries happens in src/clj/cljc/repl/compiler.clj. It uses cljc.compiler of course, not much special going on there. And that's basically it.

I don't really have any plans. I just seemed like a fun project. And I think a REPL is a good way to demonstrate ClojureC and get other people excited about it.

[whilo]

With primitives I meant protocols and other higher level Clojure-abstractions. I am pleased how well the REPL works, it is the most interactive and expressive way to write C code (with the c* special form) I have yet seen :-). A lispy syntactic hull for C was nice, like for SQL, so C-expressions and statements could be altered directly and transformed with all the Lisp-tooling instead of string concatenation.

[bertfrees]

I'm glad that it works good for you too!

it is the most interactive and expressive way to write C code (with the c* special form) I have yet seen

Could you elaborate a bit on that? How can you really write C code interactively? I know you can use the c* form to use existing C functions and convert between native values and ClojureC values with the various make_xxx and xxx_get functions, but this soon gets cumbersome:

typedef struct some_type {
  int a;
  int b;
} some_type;

some_type * allocate_some_type(int a, int b) {
  some_type *x = (some_type *)malloc(sizeof(some_type));
  x->a = a;
  x->b = b;
  return x;
}

int do_something(some_type *x) {
  return x->a + x->b;
}

(defn allocate-some-type [a b]
  (c* "make_raw_pointer(allocate_some_type(integer_get(~{}), integer_get(~{})))" a b))

(defn do-something [x]
  (c* "make_integer(do_something((some_type *)raw_pointer_get(~{})))" x)))

(do-something (allocate-some-type 1 2))

(defn do-something-else [x]
  (c* "some_type *x = (some_type *)raw_pointer_get(~{})" x)
  (c* "make_integer(x->a - x->b)" '()))

(do-something-else (allocate-some-type 1 2))

Is this what you had in mind?

[whilo]

Yes. Actually you helped me clearify what ClojureC is. Originally I wanted to use it to build a runtime, but this was then a different Clojure, since ClojureC emits C code and means it to be compiled by a C compiler, producing C primitives. Mixing in a dynamic runtime emitting different executable code wouldn't fit. Your REPL does what I think is best if you want to work in ClojureC with a C base and makes ClojureC a nice environment to work in dynamically. Writing a lot of C-code is not nice though, you are right (although a lispy C-wrapper could make the code cleaner and avoid string concatenation). But if one only wants to tweak some parts, it should be fun to explore the c* form.

On the other hand I am not sure how the ideal runtime/base for Clojure should look like. In many ways a C base is not optimal, an implemented runtime would either implement its own bytecode or use e.g. Lua. But newer VM's like V8, Dart and if I understood correctly also Dalvik spare the bytecode step and emit direct machine code jit-compiled from the AST. After all, the benefit of an intermediary representation only seems to be portability of the emitted bytecode (which the source code has anyway). The biggest drawback with native implementation is garbage collection, but then emitting machine code could make the compiler self-hosting and bootable on bare-metal. The goal for me is not to run on bare-metal though, but make the runtime dynamically changeable and adaptable, the smaller the statically compiled core, the better. I also don't like purism or love for technical details, so facing assembly and machine code should be well reflected. I could imagine though, that it is not impossible by hacking e.g. https://github.com/clojure/tools.emitter.jvm/blob/master/src/main/clojure/clojure/tools/emitter/jvm/emit.clj Haven't had time yet to investigate this.

Just as a longer term investment I thought working on some deeper infrastructure also to educate myself, this first attempt was interesting, yet I am still a novice (only doing Clojure for a year now, and some Java and C++ before). I wish Clojure was my OS's language as this probably would give me much more insight and power. Yet this is neither a realistic nor necessarily direct task imo. ClojureC moves in the direction, Clojure/Lua also shrinks the core, but Lua is also still bytecode. To eliminate the C compiler one either has to use such a bytecode VM or emit machine instructions directly. The idea to generalize Clojure's semantics seems to resonate with others as well. http://augustl.com/blog/2014/an_immutable_operating_system/ I think for example that immutable semantics might help JIT-compilation quite a bit and also profiling and tuning of the runtime was possible from inside (e.g. dynamically hook in a tweaked JIT-optimization).

What is missing from your side in ClojureC? I am just happy, you did it very well, better than I have could yet. Where are there flaws in my reasoning?

[bertfrees]

(oops almost forgot about this, sorry)

Okay, looks like you've already been putting much more thought in this then I did. I'm don't think I'm smart enough to give you any useful comments :|. Though thanks for clarifying your motivation behind this.

The main reason I picked up ClojureC was because I wanted to write a program that could be used both as a JavaScript library as well as a command line tool (preferably a native tool). Being a Lisp enthusiast, ClojureScript/ClojureC was a very natural choice.

The only thing I can think of right now that is missing (apart from porting more core functions), is macro expansion (with require-macros).