zenscript
zenscript copied to clipboard
keean
Macros and DSLs
A homoiconic programming language can parse itself to an AST expressed as native data that can be transformed within the language, evaluated, or isomorphically serialized to human readable syntax.
I don’t think the sacrifices in syntactical and grammatical clarity caused by reduction to minimum consistent grammar such as hierarchical lists of lists s-expressions, is necessary to achieve the homoiconicity which enables macros and DSLs. Although Lisp's low-level semantics distills to Seven Primitive Operators, which makes its AST eval simple to implement, the complexity of higher-level semantics which can be expressed (and thus potential semantic complexity of a macro's semantics) is not limited any more so than for any Turing-complete language.
Requirements:
- Self-hosted compiler.
- Bracketed quotes.
- Type inference localized to expressions.
- Everything-as-an-expression syntax so that we can select the parser based on the data type the expression is assigned to.
- Incremental AST compilation whilst parsing so the compiler knows the data type for #4.
For macros we’d only need #1 - 3, because macros take some or all of their arguments as AST references. We wouldn’t want macros to depend on wide-scale inference as this could result in live or dead locks on inference decidability. K.I.S.S.. For DSLs such as a SQL query language, we need to select a customized parser. If we know the data type of the expression being parsed is for example SQL.Query and if that type has associated a customized parser and separately compiler, then that parser is run to create the customized AST of the said expression (and if it is not a macro then the associated compiler is run). There are cases where the expression determines its own type (i.e. the expression has side-effects and it is not assigned or the type of the reference it is assigned to it inferred from the expression's type), so the current parser is presumed. This strategy would not work with any wide-scale type inference, which is yet another reason I am against non-localized type inference.
A customized parser and compiler could invoke another parser and compiler for an expression within its grammar (perhaps using a similar strategy of selecting by target data type or by explicit delimiters, e.g. the use of curly braces in React JSX), and this could even be recursive.
For macros we only need #1 - 3
I don't want macros :-) I don't want a second weaker version of application in a language.
What you are basically saying is that the data format for expressing objects combined with the AST is the language syntax. I am not sure I want this - as the AST currently features things like application nodes which you would not want in the language syntax. I think homoiconicity is over-rated, and I would rather have a nice user friendly syntax without it compromising the AST, and vice versa.
I don't want macros :-)
Many programmers want them. Also DSLs.
a second weaker version of application
Huh?
as the AST currently features things like application nodes
Huh?
I would rather have a nice user friendly syntax without it compromising the AST, and vice versa
Me too. I didn't see any flaw in my proposal for macros which would prevent that. Perhaps you misread the OP.
Macro's are bad and are included in languages because normal function application is not powerful enough. If you want a function write a function.
Making things too powerful is often worse than a proper separation-of-concerns.
I doubt you can achieve the same things that macros and DSLs can achieve without making the type system so powerful and obtusely convoluted (and with mind bending and numbing type inference) that nobody can understand the code any more, not even the person who wrote it. K.I.S.S..
Macro's provide an escape hatch to bypass the type system (in some languages), this is a bad thing. Messing about with the program code itself using macro splicing is a bad idea, just write a function to do the thing in the first place. With generics you should never need a macro anyway.
Macros also make debugging harder as they are applied at compile time. Any errors reported at runtime will be in the post macro processed code, so the line numbers and context cannot be found in the program source. I have yet to see one compelling use of macros in the last 20 years (and this comes from someone that wrote their own macro-assembler and used to think that macros were all you needed to build a compiler and a high level language).
Maybe you can persuade me, if you can provide an example of a macro that cannot be more cleanly implemented in another way?
Macros enable editing the source code of the arguments before they are evaluated. A functional lazy evaluation of those arguments will not enable the transformation of their source code. Macros can thus perform some automated operations based on the structure of the source code within its arguments, not just the evaluated value of the arguments. That is turtles all the way down recursively.
For example, if we have list of functions we want to call with different numbers of arguments, they each have different types, and we want to add new common argument to the end of each of these calls, and collect the results in a list. Much easier to call a macro to do that, as it doesn't need to fight with the type system, as the type checking will be performed after the macro has done its code transformation. This example could also be done with partial application without a macro, except the typing can still get unwieldy, because those functions might have different type signatures for remaining default arguments or we have to manually specify the default arguments. Macros are basically a way the programmer can rescue himself from the fact that the language development moves too slowly and can't always do everything that you and I could imagine it might do one day in the distant future.
Macros are basically useful any where you need to do transpiling, so you don't have to write your own whole program compiler. And there have been many times I wanted to transform code, e.g. to convert the if as an expression to a lamba function compatible with TypeScript. If TypeScript had powerful macros, I might already be done with some of the features I wanted for my transpiler! Instead I will have to reinvent the wheel.
Also much of the functionality needed for macros is also needed for evaluating and running code sent over the wire, i.e. dynamic code compilation and modularity. We can't expect to link everything statically in this era of the Internet.
DSLs enable friendlier syntax such as SQL queries that are checked grammatically by the compiler.
Edit: a possible new idea for the utility of AST macros: https://github.com/keean/zenscript/issues/11#issuecomment-305997607
Macros enable editing the source code of the arguments before they are evaluated.
I know what macro's are, I am saying there is always a better way to do the same thing, without having source-to-source rewriting going on in the program code, which obfuscates errors and results in another case of write-only-code.
there is always a better way to do the same thing
Someday. But someday can be too many years (even decades) too late. Refresh.
I have not written a macro for about 20 years, so that someday was decades ago :-)
I don't limit my analysis to only your anecdotal limited needs. I’m interested in mine and those of others.
I think DSLs are the most important thing that can't be done reasonably without transforming code. I explained to enable the "macros" (parser + compiler transformation of code) for those based on detecting the type of the expression.
Show me how to write a SQL query DSL (without any noise in the syntax that isn't in the normal SQL syntax!) that is grammatically correct at compile (evaluation) time, that can be done without code transformation. Scala can sort of finagle it by use infix function names as keywords, but that is not compile-time parsing of the SQL specific syntax issues.
You do not need macro's to create DSLs.
Show me how to write a SQL query DSL
Sure I did it in Haskell back in 2004 :-)
Sure I did it in Haskell back in 2004 :-)
I am not interested in your obtuse HList gobbledygook.
A DSL should be elegant and easy to comprehend and work correctly. I asked you to show me how to do this, not asking you to brag.
What's obtuse about this:
moveContaminatedAnimal :: DAnimalType -> DCntdType -> DFarmId -> Query ()
moveContaminatedAnimal animalType contaminationType newLocation = do
a1 <- table animalTable
a2 <- restrict a1 (\r -> r!AnimalType `SQL.eq` animalType)
c1 <- table contaminatedTable
c2 <- restrict c1 (\r -> r!CntdType `SQL.eq` contaminationType)
j1 <- join SqlInnerJoin a2 c2 (\r -> r!AnimalId `SQL.eq` r!CntdAnimal)
j2 <- project j1 (CntdAnimal .*. HNil)
doUpdate animalTable
(\_ -> AnimalLocation .=. toSqlType newLocation .*. HNil)
(\r -> r!AnimalId `SQL.elem` relation j2)
That doesn't look like unadulterated SQL to me! And you are filling up my thread with (obtuse Haskell and braggart) noise!
You have no chance in hell of creating anything popular...
...if you think that gobbledygook is more readable than SQL syntax for the person who wants to express SQL.
That's relational algebra :-)
Here's an example:
R1 = join(CSG,SNAP)
R2 = select(CDH,Day="Monday" and Hour="Noon")
R3 = join(R1,R2)
R4 = select(R3,Name="Amy")
R5 = join(R4,CR)
R6 = project(R5,Room)
Taken from here: https://www.cs.rochester.edu/~nelson/courses/csc_173/relations/algebra.html
You have no chance in hell of creating anything popular.
C# LINQ seems pretty popular and its based on exactly the same ideas.
I don't care! Its not elegant.
Its very elegant. Relational algebra avoids several problems that SQL has as a language. SQL is not a proper algebra, its got a lot of ad-hoc elements that are pretty bad.
It's not SQL.
I asked you to show me how to do unadulterated SQL without a code transformation.
Its better then SQL. Why would I recreate a flawed query model along with all its problems. I took the opportunity to so something better.
In any case you are not going to be able to parse SQL syntax in the host language with macro's either because you cannot represent SQL in the languages AST.
Whether SQL is good or not is not germane to the point of this thread, which is whether it is possible to model the unadulterated syntax of another language without a code transformation. Yes or no?
In any case you are not going to be able to do that with macro's either because you cannot represent SQL in the languages AST
Incorrect. Re-read the OP.
That's not a proof, I want code :-)
Its not possible without custom parsers that can be implemented for new types of literals.
Its not possible without custom parsers that can be implemented for new types of literals.
Which is exactly what I wrote in the OP.
Back on examples where macros can’t be done with functions, there is no way to write a function will execute its argument expressions in the lexical scope of the caller and provide the name of the variables as a separate argument. There is no way to form that functional closure with the replacement lexical binding without a code transformation.
Paul Graham’s anaphoric macro eliminates the boilerplate of lambdas. It couldn’t be done without boilerplate nor macros with Scala _ shorthand for lambdas that take only one argument if the it is within another lambda.
Another example is when you want to declare numerous types and/or functions (in the lexical scope of the caller) based on some pattern (that would be an absolute pita to manually enter from the keyboard). A function can’t declare new types and functions. You can only do this with code generation.
There are instances where only code transformation will suffice.
there is no way to write a function will execute its argument expressions in the lexical scope of the caller and provide the name of the variables as a separate argument.
And why would you ever want to do that?
There is no way to form that functional closure with the replacement lexical binding without a code transformation.
Which is again the kind of trick I don't want to allow. If I wanted that kind of thing I could provide first class environments, but its just makes code harder to understand for no real reason.
What algorithm do you need the above to implement? Does it help me implement a sort function?
Another example is when you want to declare numerous types and/or functions (in the lexical scope of the caller) based on some pattern
Needing to do this is a symptom of a lack of sufficient abstractive power in the language itself. The 'numerous functions' should be a single generic function.
Generics remove the need for macros in the language by providing a type safe way to abstract code over parameters.