phoenixframework
Draft: use Elixir AST for macro components and make HEEx pipe-able
So after working on the engine internals a bit recently I had an idea that I've been experimenting with: what if we had an actual HEEx AST as an intermediary representation returned by sigil_H?
The idea is that you can do this:
defmacrop my_macro_component(ast) do
ast = Macro.expand(ast, __CALLER__)
Macro.prewalk(ast, fn
{:local_component, _, ["foo", _, _]} ->
# do something with <.foo>
other ->
other
end)
end
def my_component(assigns) do
~H"""
<.foo>
<:slot>bar</:slot>
</.foo>
"""
|> my_macro_component()
end
So I've made that work by making the TagEngine return MacroComponent AST and refactored the MacroComponent AST to be valid Elixir AST:
~H""
# gets transformed to
(
require Phoenix.LiveView.TagEngine
Phoenix.LiveView.TagEngine.finalize([...], [
# heex AST
])
)
Here, finalize is a macro that will compile the HEEx using the engine into a rendered struct as usual, but allowing macros to work on the intermediate AST.
I'm not sure if this is a good idea, but all tests are passing, so 🤷🏻♂️
An example of how the HEEx AST looks like this at the moment:
tag("div", [attribute("id", [], string_or_other_ast)]) do
# tag content goes here, e.g.
"some string\n"
local_component("live_component", [attribute("module", [], MyLiveComponent)], closing: :self)
end
So it's just regular Elixir AST and people familiar with macros can walk it and do whatever they want. The format of the AST is open for discussion. For example, right now attributes are kinda whacky, as they define some metadata as their second call argument {:attribute, _meta, [name, _more_meta, value]} but _more_meta could also live as a key under the regular meta. Also, attributes without value don't have extra_meta and are defined as {:attribute, _meta, [name, nil]}. So very much a proof of concept and not final.
The benefit of using Elixir AST is that we can put things like line numbers in the regular AST meta, so people don't need to care about them, we just need to be able to handle them not being present in some places.
I am worried this will add tons of complexity. Do we have any other use case besides scoped CSS which we could also handle with the scoped attributes we discussed in the past bit?
@josevalim for me, the main point is to address the current MacroComponent limitations as it allows you to also work with components, slots, expressions, etc.; I think @LostKobrakai wanted to experiment with automatically wrapping text into gettext calls (https://github.com/phoenixframework/phoenix_live_view/issues/3842), which you could probably do easily with the AST approach, but wouldn't work with the current MacroComponents as soon as anything more complex is involved. I could also see us having a config :phoenix_live_view, :heex_postprocess, [] compile time config where you could register custom code to execute for all components.
Can you expand on the kind of complexity issues you're thinking about?
The complexity is this PR itself, having to add a new representation for all of HEEx AST that looks like Elixir code but it isn't (for example, Code.eval_quoted doesn't actually work). I understand it is nice to piggyback on our notation but it comes with the downsides in that we need to do a lot of work going to and back from this notation.
Also, for something like translation, I would probably prefer heex_preprocess_tags indeed. As I think something like <t>will be translated</t> is less error prone than assuming all strings in the template are translated (but when you move those strings around, such as to an attribute, then you must wrap it in gettext).
Right, I'm not saying that actually applying gettext to all strings is necessarily a good idea - only doing that on specific tags would be much more reasonable.
And btw, you can absolutely eval_quoted that AST, you just cannot do that for all individual AST nodes by themselves.
[scratch/macro_component_playground.exs:14: DemoLive.my_macro_component/1]
Code.eval_quoted(ast, [assigns: %{}], __CALLER__) #=> {%Phoenix.LiveView.Rendered{
static: ["<!-- <DemoLive.render> scratch/macro_component_playground.exs:134 () --><style>\n h1 {\n color: red;\n }\n\n button {\n padding: 0.5em;\n font-size: 1.25em;\n font-weight: bold;\n background-color: \"#fafafa\";\n border: 2px solid gray;\n }\n</style>\n\n<button phx-click=\"randomize\">randomize</button>\n<button phx-click=\"change_0\">change first</button>\n\n<ul>\n ",
"\n</ul>\n\n", "\n\n", "\n\n",
"\n\n<!-- @caller scratch/macro_component_playground.exs:179 () -->",
"<!-- </DemoLive.render> -->"],
dynamic: #Function<42.113135111/1 in :erl_eval.expr/6>,
fingerprint: 246748478796526479488887298827552919271,
root: false,
caller: :not_available
},
[
{{:dynamic, Phoenix.LiveView.Engine},
#Function<42.113135111/1 in :erl_eval.expr/6>},
{:assigns, %{}}
]}
We could also go one step further and actually define tag, attribute, etc. macros, but I'm not sure about the benefits.
This PR is complex, yes. But maybe the effort to define a reasonable AST representation for all of HEEx is worth the complexity? I'm not sure yet myself, this was just an interesting idea I wanted to try out :)
Right, I'm not saying that actually applying gettext to all strings is necessarily a good idea - only doing that on specific tags would be much more reasonable.
Yes, exactly. It requires the wrapping and internally it expects certain nodes. I can't just treat it as a regular Elixir code. I think having an AST specific to HEEx would be less error prone, even if it may mean we need to implement our own traverse helpers (which should not be that hard anyway).
I'll want to leave my thoughts here:
To me having an elixir compatible AST isn't really necessary, but having a complete AST of heex would be quite useful from my POV, compared to the limitations of the current MacroComponent approach. By complete I mean that I can have code ingest as well as output most of – if not all of – what heex supports (including dynamic/elixir parts). E.g. the experiments with gettext stopped at me not being able to turn <anything>hello</anything> into <anything>{elixir code}</anything>.
I'm also interested in potentially bridging the gap between heex and solid.js jsx by compiling JS for hooks in a similar fashion as JSX for solid is compiled to js, which would work with just static children, but woul be more powerful if I could apply this to a heex tree, where there are dynamic parts included. Not needing to ask the DOM for certain nesting levels would also be quite interesting (to see if any parents provide data contexts).
For me, the beauty of it being Elixir AST is that it allows us to pipe ~H into macros and still have it completely compatible with what ~H currently compiles to. If it was not valid Elixir AST, the only way I can see to modify the HEEx AST of a whole component would be to wrap it inside a wrapper tag:
~H"""
<template-that-doesnt-matter-because-its-only-there-to-get-the-ast :type={MyMacroComponent}>
ANY CONTENT INSIDE
</template-that-doesnt-matter-because-its-only-there-to-get-the-ast>
"""
Being able to just work with Macro.prewalk/postwalk/traverse etc. and generating AST with quote is another benefit, but as was mentioned, building a custom traversal isn't too hard either way.
Being able to just work with Macro.prewalk/postwalk/traverse etc. and generating AST with quote is another benefit, but as was mentioned, building a custom traversal isn't too hard either way.
The problem is that we have to follow strict rules in order for this to work. I assume, for example, this would not be valid?
tag("div", [attribute("id", [], string_or_other_ast)]) do
# replaced the string by an expression
if condition?, do: "some string\n", else: "something else"
local_component("live_component", [attribute("module", [], MyLiveComponent)], closing: :self)
end
Or if it is valid, it can lead to subtle differences in behaviour compared to an if actually written in the template. And then, if there are exceptions, today we expect some metadata to be available, but it may not be if the user writes the AST by hand. So if we want to assume the AST can be written manually, then I think we need to fundamentally change our approach and make the whole metadata closer to what one would write and treat all fields in the metadata to be optional?
At the moment it would not be valid, since the tag_engine expects all expressions to be wrapped in {:expr, meta, ast} or {:body_expr, meta, ast}, but it could also have a fallback clause that just calls handle_expr of the engine for any other AST. For body_expr, we could definitely skip it, but for expr we need the EEx marker for the engine.
And yes, we would need to treat all metadata as optional (which this PR does). Right now, this only affects the line / column numbers for exceptions.
After sleeping on this, I am not really sure what we gain here is worth the complexity. I agree being able to write this is nice:
def my_component(assigns) do
~H"""
<.foo>
<:slot>bar</:slot>
</.foo>
"""
|> my_macro_component()
end
But you could also write this instead:
def my_component(assigns) do
~H"""
<div :type={MacroComponent}>
<.foo>
<:slot>bar</:slot>
</.foo>
</div>
"""
end
And I am ok with keeping the macro components within the template. In a way, it can be confusing if something outside of the template is changing what is happening inside.
But you could also write this instead
There are templates with multiple root level nodes as well though, where one might still want to target the whole template. Maybe something like<>…</> could be introduced to heex to work around that, but I guess that would be a more involved change to do so. It‘s - as you’re likely aware - how JSX enforces everything having a single root node.
Hi guys, I wanted to add my 2 cents here. When working with LiveVue, I was exploring HEEX introspection options - currently it's very limited. Ideas I'm toying with:
- Inlining Vue components inside the HEEX (right now done with custom ~V sigil, probably should rewrite it with MacroComponent). Accurate detection of used assigns is a challenge.
- Arbitrarily nesting Elixir and Vue slots (eg Vue component with Elixir slot with Vue slot inside)
In general, my goal is extreme interoperability of Vue and HEEX. I like the idea of HEEX AST, I think it might make these ideas easier to implement (although I must admit I didn't had time to play with MacroComponents, so it might be already possible).
On the other hand, currently piping or passing HEEX anywhere is not a thing. It might take some time for normal users to get used to it, I expect it would be mostly used by library authors. But then, how library authors might be able to use it without asking users to pipe their HEEX component? 🤔
Code Examples
Module.register_attribute(__MODULE__, :solid, accumulate: true)
defmacrop abc(ast) do
ast = Macro.expand(ast, __CALLER__)
Macro.prewalk(ast, fn
{:tag, meta, ["div", attr, content]} ->
# System.unique_integer([:positive, :monotonic])
id = 15
attr =
attr
|> Enum.filter(fn
{:attribute, _meta, ["$" <> attr, _delimiter, value]} ->
Module.put_attribute(__CALLER__.module, :solid, {id, {attr, value}})
false
attr ->
attr
end)
|> Kernel.++([{:attribute, [], ["data-solid-id", [], id]}])
|> Kernel.++([{:attribute, [], ["phx-hook", [], "Solid"]}])
{:tag, meta, ["div", attr, content]}
other ->
other
end)
end
def solid(assigns) do
~H"""
<div class="dropdown" id="abc" $class='{"dropdown-open": open()}'>
<button class="btn m-1">open or close</button>
<ul class="menu dropdown-content bg-base-100 rounded-box z-1 w-52 p-2 shadow-sm">
<li><a>Item 1</a></li>
<li><a>Item 2</a></li>
</ul>
</div>
"""
|> abc()
end
:ok =
File.write!(
"assets/js/test.js",
Enum.map_join(@solid, "\n\n", fn {id, {"class", value}} ->
"""
export function for_#{id}(el, open) {
const obj = #{value}
for(let c in obj) {
el.classList.toggle(c, obj[c]);
}
}
"""
end)
)
import { createEffect, createRoot, createSignal } from "solid-js";
import * as funs from "./test";
export const Solid = {
mounted() {
console.log(funs, this.el.dataset.solidId);
let fun = funs["for_" + this.el.dataset.solidId];
createRoot(() => {
let [open, _setOpen] = createSignal(true);
createEffect(() => {
fun(this.el, open);
});
});
},
};
I played a bit with with PR. There's a few things I found (even if minor):
- Having the attr delimiter in the AST feels weird. Not sure why exactly it's there.
- Macros in elixir live from using
quote, but while possible it feels weird to doquote do: attribute("class", …, "my-class") - Generating the JS I see why you want an AST though xD I wanted to generate code per js obj key, but that would've required parsing and dealing with the js code string. I did recently manage to POC using swc to turn js AST into JS code (using rustler), but haven't used that here.
- This is not too far fetched for JSX + solid.js compilation: https://playground.solidjs.com/anonymous/d0e87638-3fed-4c7f-9c74-fedb2b790baa looking at the output for client side rendering it creates a dom node and then adds an effect for
el.classList.toggle