Can I go Fennel with luar?

[andreyorst]

I'd be interested in using Fennel as you use lua in lua %{} blocks. Fennel is a Lisp that compiles down to Lua, so perhaps it could be possible to make a wrapper that will use luar underneath. What do you think?

[gustavo-hms]

Hi, @andreyorst !

I use Fennel in some projects and was actually wondering what could be the best way to integrate it with luar. I'm open to suggestions.

The first thing to note is that you can already import some module written in Fennel and use it inside a lua %{} block.

Suppose you have a file called module.fnl with the following code:

(local um 17)
(fn outro [n] (+ 1 n))
{: um
 : outro}

Then you could use it inside a kak file this way:

lua %opt{some_option} %{
    local fennel = require "fennel"
    table.insert(package.loaders or package.searchers, fennel.searcher)

    local module = require "module"

    if arg[1] then
        return module.um
    end

   return module.outro(18)
}

Or you could compile your fennel code first with

fennel ---compile module.fnl > module.lua

and import module.lua directly.

One problem these solutions have is that inside module.fnl we have no access to the kak module (this is the same problem a plain lua module would have actually). This could be solved by injecting it in the fennel code. More or less like this:

; This is module.fnl

(fn [kak]
  (kak.set-option "buffer" "alingtab" false))

And then:

# This is the kak file
lua %{
    local fennel = require "fennel"
    table.insert(package.loaders or package.searchers, fennel.searcher)

    local main = require "module"
    main(kak)
}

We could also define a fennel command in the same way we have lua. But this raises some questions:

• is it needed or desired?
• if so, should we write the fennel command on top of lua or make it call the fennel interpreter just like we do with lua?

What do you think?

[gustavo-hms]

There's yet another option, using fennel.eval:

lua %{
    local fennel = require "fennel"
    fennel.eval [[
        (kak.set-option "buffer" "alingtab" false)
    ]]
}

If you think it could be useful having a fennel command, a minimal implementation (excluding proper error handling like in the lua version) could be:

define-command fennel -params 1.. %{ lua %arg{@} %{
    local code = arg[#arg]
    arg[#arg] = nil
    local fennel = require "fennel"
    fennel.eval(code)
} }

Then we could write:

fennel %val{bufname} %(
  (let [name (args)]
    (kak.echo name)))

[andreyorst]

I've mainly thought about using fennel directly from expansions, so I guess fennel command would be nice. Although I don't know if it will be possible to cache compiled Lua so we do not do reparsing of whole fennel code every time we invoke the same code.

[gustavo-hms]

OK! I'm gonna take a look at it.

[andreyorst]

theoretically, fennel compiler should be pure, so when we compile fennel code we should always receive the same result. So as a quick way we could compute fast hash sum of fennel code, and compile it down once. So when user calls fennel command it checks if there's a compiled file named after code hash-sum, and if it is none, it compiles fennel, saves the file, and runs the code. If there's a file, this means that we already have compiled code and can run it. If hash-sum changed we recompile. However this means a lot of housekeeping, because if hash-sum changed we have to somehow know old hash-sum so we could remove old compiled code to prevent space waste

[gustavo-hms]

I'd rather prefer to keep it as simple as possible. Additionally, I think it's a good idea to avoid premature optimisations.

When you say "reparsing whole fennel", do you mean the fennel.lua module? Or every code we pass to the :fennel command? If it's the first case, I think I've found a reasonable solution. You can check it in the fennel branch: 4862d70af7328b22e396d193b1968ea33cb16af0.

Instead of writing the fennel command on top of the lua command, it calls the fennel interpreter directly. I've extracted common code to a lib.lua module, and imported it both in luar.lua and luar.fnl.

I want to document here some design decisions:

Error handling

Error handling is the most laborious part. Specially considering that the output of the lua interpreter is different from the output of the fennel interpreter. So, I needed some common functions to both of them, but others are specific to each case.

Compiler output

Regarding compiler output, I've originally decided to print all the code of a lua %{} block when an error occurs because the compiler itself doesn't give enough context and so, if you have more than one lua %{} block, it was hard to tell from which one came the error. It's how things are currently implemented in master. Current Fennel implementation has the same behaviour, but things are about to change.

Having been working with Elm, I value good compiler messages and want to see them displayed when calling fennel %{}. The problem is that it makes the current print-all-the-code-block behaviour obsolete and quite noisy.

So, I've made a new switch (-name) to both :lua and :fennel to allow giving a name to a code block. This name will be referenced in the *debug* buffer when some error is printed. By doing so, the print-all-the-code-block behaviour is gone.

Fennel eval

I didn't found a way to evaluate a string inside fennel code. In lua, I can require "fennel" and then call fennel.eval. But there isn't such an API from the fennel side. Perhaps this could be done with macros, but I'm not used to Lisp macros and couldn't manage to do it. Do you know something about it?

Anyway, the solution I've found is to require "fennel" inside lib.lua and then use it in luar.fnl. Since the require function caches modules automatically and the fennel interpreter already requires it (although apparently it doesn't expose it), this doesn't impose any runtime penalty.

Edit: it turns out that I've missed something when experimenting with the fennel library. Trying again, I've found that it can be required from Fennel like any other lua library, matching the expected behaviour. I don't know exactly what I did wrong the first time, but it's good to know it works. See 573b6702adb8cd67383717a0ce3b0e368596ee90

Syntax highlighting

The last missing piece is syntax highlighting inside fennel %{}, but to have it we need highlighters for fennel files and currently Kakoune doesn't ship any. I have a personal fennel.kak file, but it uses :lua under the hood, so it would probably require rewriting it in plain kak to be shipped.

But, even if it is plain kak, what is the procedure to ship a filetype file to Kakoune? Do we try to submit it to the official repository? Or ship an independent one and make it a dependency for luar? But, if we do so, is it a good idea to require having a fennel filetype plugin to people interested only in the :lua command?

Further notes

Please, try the fennel branch and report any error or improvement you find. As always, I'm open to ideas.

Also, when time comes to use it to do more heavy-weighted work with fennel %{}, if you find any bottleneck, we can think about further performance optimisations.

[andreyorst]

I've tried some simple things (don't have a lot of time currently) and it seems to work. Hopefully, I'll be able to come back to this in a week or so.

[gustavo-hms]

I'm closing this issue by now because it seems there isn't enough demand for this feature. If you or someone else want reconsider it, please tell me. I won't delete the fennel branch just in case.

[andreyorst]

^{I forgot to mention that I live on Venus, and our week is about 1701 Earth days..}

Sorry, since I've recently decided to fully move to Emacs, I had very little time with Kakoune since, and couldn't do more testing.

[gustavo-hms]

Don't worry, @andreyorst! Even if you'd remained interested in such a feature, I still would wonder if it would be a good idea releasing it for just two potential users (you and me). And considering we can compile Fennel code to Lua ahead of time, perhaps it's just not necessary.

[gustavo-hms]

Hey, @andreyorst !

I decided to revisit this issue and make a new implementation. I extracted the common functions to a separate module both luar.lua and luar.fnl require. It was mostly a simple task. The most laborious part was handling execution errors (lua and fennel diverge in how they present errors to the user; fennel does a much better job here).

Are you still interested in a fennel command? If so, could you please test the implementation in the fennel branch?

[andreyorst]

Sure, will test this tomorrow!

[gustavo-hms]

Great!

I still need to update the docs, but it works mostly the same as the lua command, with the only exception that Kakoune commands with hyphens can now maintain them: (kak.execute-keys "xyp") in fennel compared to kak.execute_keys("xyp") in lua.

[andreyorst]

It does seem that I don't have fennel module on my installation of Kakoune v2020.09.01, but it's just repos having old release I guess. Other than that it does seem to work, but I need to refresh my kakoune skills, so I could do anything useful with it.

[gustavo-hms]

That's right: the last release (v2020.09.01) doesn't yet contain support for Fennel. So I'll probably hold the merge to master until Kakoune makes a new release.

[evanrelf]

I'm still interested in this, FWIW 🙂

[gustavo-hms]

Hi, @evanrelf !

Ok! Since demand is low and it would increase code size a bit, I was considering not integrating it. But I'm gonna do it soon. In the meanwhile, you can use the plugin in the fennel branch. It's already working. Please, report any bug you eventually find.