How to automatically import macros file?

[bangedorrunt]

@rktjmp I'm wondering how to make macros file available without using import-macros in each module/file?

For example, I have a :core.macros which defines some macros such as if-let, when-let.... I want those available through all modules without using import-macros. Is it possible?

[rktjmp]

Either the compilerEnv compiler option passed to setup or a plugin probably.

https://fennel-lang.org/api, see compilerEnv but how you define your macro and get it into that option is not obvious to me since there is a bit of a chicken and egg problem. I would first try and inject something simple into there, just a normal function, and see if I could make it work.

I think the example fennel plugin does something quite similar around missing syms, so that would be another route. You can hook onto parse-error I think and inject the symbol. Just be sure to return nil (I think) so other plugins continue to run.

You should be able to just add your plugin to the compiler plugins options in setup but not 100% sure how well that would work re: finding and loading the module etc.

I can't see the option immediately in the docs but I do just pass a plugins option internally, so perhaps it was undocumented but should be stable, it's just mirroring the --plugin option for the CLI.

https://github.com/rktjmp/hotpot.nvim/blob/64b64709f9fb25b7470012caa23f008b755e6284/fnl/hotpot/searcher/module.fnl#L73-L76

I would try to make that work with the fennel CLI then post back and we can look at how to configure hotpot to use it and any possible fixes needed.

Probably the Fennel chat would have some insights too, but primarily get a POC working with the CLI env - to be sure it's actually possible - and then it should be clear how to integrate that into Hotpot.

[bangedorrunt]

This could be a good reference then

https://github.com/Olical/aniseed/blob/4b760370c8e17fb32f6442a13c189e254ec1d5e9/fnl/aniseed/compile.fnl#L7-L42

[rktjmp]

I'd be curious how you go with a plugin or compileEnv option first as I think that is cleanest and most "fennel"-ly, but it could also be possible to expose a hook, before I call compile, where you could get say, code opts and return code opts after making any modifications you want (in this case pre-pending (import/require-macros ...).

I actually thought about this a while a long while ago (#37) but no one else ever gave a reason to do the work, perhaps its worth revisiting. I can't promise a timeline on this though. Diagnostics might be a burden too.

If the plugin doesn't work, or is too difficult you could always fork for now until (or if?) hooks appear.

Just patch

https://github.com/rktjmp/hotpot.nvim/blob/64b64709f9fb25b7470012caa23f008b755e6284/fnl/hotpot/compiler.fnl#L46

with something like

fnl-code (read-file! fnl-path) 
fnl-code (if (string.match fnl-path "some-path-i-know-is-my-code")
             (.. "(require-macros macro-file)" fnl-code)
             fnl-code)

[bangedorrunt]

I thank you so much for your insight. Honestly there is no rush on my side so please take your time for the implementation. I believe this worths it and really love your approach ❤️❤️❤️

[datwaft]

I have a question as I am also interested on at least trying to implement something like this.

I think the example fennel plugin does something quite similar around missing syms, so that would be another route.

Where can I find that default plugin, and where is the plugins documentation for Fennel?

I have only found this, and it doesn't show any example.

[rktjmp]

I think I was mixing linter.fnl with perhaps some chatter on matrix.

I poked at it for a minute or two but not 100% sure it's possible to return code that the compiler should insert, vs just inspecting code the compiler saw. I would think the best chance is with symbol-to-expression but I don't think you can really modify what the symbol expands to.

I think andreyorst has some plugin that lets you use @ named variables or something which might have some pointers (again this is just from memory, probably from a year ago or so now). That's probably clearer cut though as you'd be probably just inserting a new mangling/sym, not trying to inject a list. I would probably ask upstream if it's even possible to return new code to inject.

This doesn't work but might be useful, as well as grepping for (utils.hook : to find calls.

;; file.fnl
;; fennel --plugin plug.fnl -c file.fnl
(fn a-scope-fn [] (print "this exists to help see the scope"))

(map-seq [1 2 3] #(print &1))

;; plug.fnl
(fn map-seq-fn [seq f]
  (icollect [_ v (ipairs seq)] (f v)))

(macro map-seq-macro [seq f]
  `(icollect [_ v (ipairs ,seq)] (,f v)))

(fn parse-error [...]
  ;; not the way, our code is valid, just unrunnable/uncompilable
  (print :parse-error (view [...])))

(fn assert-compile [cond msg ast something-called-reset]
  ;; seems this cant modifiy scope, see compiler.fnl:50, utils.fnl:358
  (print :assert-compile ast))

(fn symbol-to-expression [ast scope]
  ;; Doing this will get you a "tried to ref macro at runtime", possible pathway here.
  ; (print (view ast {:metamethod? false}))
  (match ast
    [:map-seq] (do
                (tset scope :macros :map-seq (fn [seq f] (list)))
                (print :map-seq-post-scope (view scope))))
  (values nil))

(fn call [ast scope ...]
  ; (print :call (view [ast] {:metamethod? false}))
  (match ast
    [[:map-seq]] (do
                   (tset scope :macros :map-seq (fn [seq f] (list)))
                   (print :map-seq-post-scope (view scope))))
  (values nil))

(fn chunk [ast? scope]
  ; undocumented, compiler.fnl compile-stream
  ; (print :xy (view ast?) (view scope))
  (match ast?
    [[:map-seq]] (do
                   (tset scope :macros :map-seq (fn [seq f] (list)))
                   (print :map-seq-post-scope (view scope)))))

{:name :magic-map-seq
 ; :call call
 :chunk chunk
 ; :symbol-to-expression symbol-to-expression
 ; :parse-error parse-error
 ; :assert-compile assert-compile
 :versions [:1.1.0]}

[rktjmp]

Did you get anywhere with the plugin idea?

[datwaft]

Did you get anywhere with the plugin idea?

No, I couldn't find any way to make it work using plugins, so I scrapped the idea.

[rktjmp]

;; plugin.fnl

;; must define as function that returns a list
(fn map-seq-fn [seq f]
  `(icollect [_# v# (ipairs ,seq)] (,f v#)))

(fn call [ast scope ...]
  (match ast
    ;; match against symbol and capture arguments
    [[:map-seq] & other]
    ;; written as do for comment clarity
    (do
      ;; expand our macro as compiler would do, passing in capture arguments
      (local macro-ast (map-seq-fn (unpack other)))
      ;; now expand that ast again (this expands icollect etc, *other* macros)
      (local true-ast (macroexpand macro-ast))
      ;; change ast to match macro ast, note that we must
      ;; **modifiy** the ast, not return a new one, as we're
      ;; actually modifying the ast back in the compiler call-site.
      (each [i ex-ast (ipairs true-ast)]
        (tset ast i ex-ast))))
  ;; nil to continue other plugins
  (values nil))

{:name :magic-map-seq
 :call call
 :versions [:1.2.1]}

;; file.fnl
;; fennel --plugin plug.fnl -c file.fnl
(map-seq [1 2 3] #(print $))

-- fennel --plugin plugin.fnl -c file.fnl
local tbl_17_auto = {}
local i_18_auto = #tbl_17_auto
for __1_auto, v_2_auto in ipairs({1, 2, 3}) do
  local val_19_auto
  local function _1_(_241)
    return print(_241)
  end
  val_19_auto = _1_(v_2_auto)
  if (nil ~= val_19_auto) then
    i_18_auto = (i_18_auto + 1)
    do end (tbl_17_auto)[i_18_auto] = val_19_auto
  else
  end
end
return tbl_17_auto

# fennel --plugin plugin.fnl file.fnl
1
2
3

Will patch Hotpot to support custom plugins (should not be hard, just needs to pass through a {:plugins ...} option.

[rktjmp]

Going to close this as IMO the plugin method is more precise (and probably the intended path from upstream "if you really had to") than a hook that jams text everywhere but still open to that idea in the future. See cookbook for more details.

[datwaft]

I have a question regarding this (finally got time to test this).

I got your example working but when I try to import my macros Fennel cannot find the macros.

See this example:

; Originally I tried with `themis.var` and setting `--add-fennel-path 'fnl/?.fnl'`, but it didn't work
(local {: let!} (require :fnl.themis.var))

(fn call [ast scope ...]
  (match ast
    [[:let!] & args] (do
                       (local macro-ast (let! (unpack args)))
                       (local true-ast (macroexpand macro-ast))
                       (each [i ex-ast (ipairs true-ast)]
                         (tset ast i ex-ast))))
  (values nil))

{:name :themis
 :call call
 :versions [:1.2.1]}

When I run fennel --plugin fnl/themis/plugin.fnl the following error message is displayed:

lua: fnl/themis/plugin.fnl:1: module 'fnl.themis.var' not found:
        no field package.preload['fnl.themis.var']
        no file '/opt/homebrew/Cellar/luarocks/3.9.1/share/lua/5.4/fnl/themis/var.lua'
        no file '/opt/homebrew/share/lua/5.4/fnl/themis/var.lua'
        no file '/opt/homebrew/share/lua/5.4/fnl/themis/var/init.lua'
        no file '/opt/homebrew/lib/lua/5.4/fnl/themis/var.lua'
        no file '/opt/homebrew/lib/lua/5.4/fnl/themis/var/init.lua'
        no file './fnl/themis/var.lua'
        no file './fnl/themis/var/init.lua'
        no file '/Users/datwaft/.luarocks/share/lua/5.4/fnl/themis/var.lua'
        no file '/Users/datwaft/.luarocks/share/lua/5.4/fnl/themis/var/init.lua'
        no file '/opt/homebrew/lib/lua/5.4/fnl/themis/var.so'
        no file '/opt/homebrew/lib/lua/5.4/loadall.so'
        no file './fnl/themis/var.so'
        no file '/Users/datwaft/.luarocks/lib/lua/5.4/fnl/themis/var.so'
        no file '/opt/homebrew/lib/lua/5.4/fnl.so'
        no file '/opt/homebrew/lib/lua/5.4/loadall.so'
        no file './fnl.so'
        no file '/Users/datwaft/.luarocks/lib/lua/5.4/fnl.so'
stack traceback:
        [C]: in function 'require'
        fnl/themis/plugin.fnl:1: in main chunk
        (...tail calls...)
        /opt/homebrew/bin/fennel:6207: in main chunk
        [C]: in ?

Do you have any idea about how to fix this?

[rktjmp]

Not sure... Try dumping out package.path and fennel.path (might be named something else) in the plugin, make sure they're looking ok?

Possibly (probably?) plugins cant require any fennel code as ... the compiler is still bootstrapping? I admit I only tried embedded macros in the file, not a require - but that would make sense to me. I would try to load a lua file that's in the same dir as the plugin (bet that works), then try to load a normal fnl module too (bet that fails).

Perhaps the hook idea is worth returning to in the end. Probably it would look similar to the make api where you match on a pattern and get the file contents, alter it however you want and return the string.

[datwaft]

Here is the result of (print package.path):

/opt/homebrew/Cellar/luarocks/3.9.1/share/lua/5.4/?.lua;/opt/homebrew/share/lua/5.4/?.lua;/opt/homebrew/share/lua/5.4/?/init.lua;/opt/homebrew/lib/lua/5.4/?.lua;/opt/homebrew/lib/lua/5.4/?/init.lua;./?.lua;./?/init.lua;/Users/datwaft/.luarocks/share/lua/5.4/?.lua;/Users/datwaft/.luarocks/share/lua/5.4/?/init.lua

Here is the result of (print (. (require :fennel) :path)) (I couldn't use fennel.path as it doesn't exist):

./?.fnl;./?/init.fnl

A Lua file can be required without any problems but when I try to require a fennel file it fails.

[shaunsingh]

Hi, I wanted to do something similar and was following the cookbook

in core.patch.fnl

(fn map-seq-fn [seq f]
  `(icollect [_# v# (ipairs ,seq)] (,f v#)))

(fn call [ast scope ...]
  (match ast
    ;; match against symbol and capture arguments
    [[:map-seq] & other]
    ;; written as do for comment clarity
    (do
      ;; expand our macro as compiler would do, passing in capture arguments
      (local macro-ast (map-seq-fn (unpack other)))
      ;; now expand that ast again (this expands icollect etc, *other* macros)
      (local true-ast (macroexpand macro-ast))
      ;; change ast to match macro ast, note that we must
      ;; **modifiy** the ast, not return a new one, as we're
      ;; actually modifying the ast back in the compiler call-site.
      (each [i ex-ast (ipairs true-ast)]
        (tset ast i ex-ast))))
  ;; nil to continue other plugins
  (values nil))

{:name :nyoom : call :versions [:1.2.1]}

And then I load the plugin as follows;

if pcall(require, "hotpot") then
	require("hotpot").setup({
		provide_require_fennel = true,
		enable_hotpot_diagnostics = false,
		compiler = {
			modules = {
				correlate = true,
			},
			macros = {
				env = "_COMPILER",
				compilerEnv = _G,
				allowGlobals = true,
				plugins = { "core.patch" },
			},
		},
	})
	require("core")
else
	print("Unable to require hotpot")
end

However it errors out on compilation with a stack overflow (even if I don't use map-seq-fn anywhere):

Clearing cacheError detected while processing /Users/shauryasingh/.config/nvim/init.lua:
E5113: Error while calling lua chunk: /Users/shauryasingh/.config/nvim/init.lua:69: module 'core' not found:Compile error in /Users/shauryasingh/.config/nvim/fnl/core/init.fnl:24:6
  stack overflow

      ^[[7m(import-macros {: command! : let! : set!} :macros)^[[0m

[rktjmp]

Yeah I think it's a dead end for the idea, it's not really upstream supported and only useful for adding small forms probably.

Probably I will add a hook interface that is similar to the api.make, where you can specify maybe some kind of pattern to match mod-name against and a function-handler, or you specify just functions in a list that are executed in sequence for every compile.

The handler would accept at least the code as a string and should return a string or ... (nil err)? or throw error? (nil err) would match luas normal module loading style.

Probably hooks can only be set at "setup" and not adjusted dynamically unless there is a good reason to support that.

(setup
  {...
  :hooks ["config.stuff" (fn [code {:macro? true|false : some : helpers?}]
                           (.. "(print :hi)" code))]})

(setup
  {...
  :hooks [(fn [modname code {:macro? : some : helpers?}]
            (if (= modname "config.stuff")
              (.. "(print :hi)" code))]})

Not sure if the key should be something more like {:hooks {:on-compile [...]} or just :on-compile [...] directly. :pre-proccess?

Any input on the actual usecase (eg vague pseudo code or descriptions) would be helpful.

There is some muck involved as the compiler has a few entry points that need to keep options separate (e.g the "main" compiler, api.make compiler, api.compile-string) but I have been meaning to clean that up anyway.

[shaunsingh]

Looks good to me

Not sure if the key should be something more like {:hooks {:on-compile [...]}

I think just having :on-compile makes sense, I can't think of anywhere else you'd need a hook except when you're compiling.

Any input on the actual usecase (eg vague pseudo code or descriptions) would be helpful.

I personally would like my configuration macros to feel more "native", it's just a matter of convenience not having to import an autocmd! macro etc. Of course a cleaner way to do this would probably to have an additional macro that splats the rest of the macros in scope, but having hotpot handle feels cleaner to me

Similar to how aniseed appends (require-macros :macros) to every file before compiling it. It doesn't sound like the best way to go about it, which is why I was interested in using compiler plugins but as you said sounds like a dead-end :(

[rktjmp]

So there is a very beta option preprocessor now, in head (not in the tagged v0.6.0 release but in commits after it.)

It should be a function that accepts fnl-src for modification and a table that contains path (the file being compiled), modname, the modname if the file was required, otherwise it may be nil (eg during diagnostic compilation where the modname is not known!) and macro? which indicates whether the call is in the macro compiler env.

Since modname may be nil, its probably best to match on the path, which should generally be a full path, but also may be nil if a blank buffer has its ft=fennel set and diagnostics is enabled!

Not super happy with how its put together, but any changes will be posted here with some period of deprecation.

      :compiler {:modules {:correlate true}
                 :preprocessor (fn [src {: path : modname : macro?}]
                                 (print path modname (string.match path "config/nvim/"))
                                 (case (string.match path "config/nvim/")
                                   nil src
                                   _any (let [pre "(macro hello [] `(print :hello))"]
                                          (string.format "%s%s" pre src))))
                 :macros {:env :_COMPILER
                          :compilerEnv _G
                          :allowedGlobals false}}})