jscl
jscl copied to clipboard
jscl-project
Developer Documentation & On-boarding
Looking at the long tail of list of contributors and watchers I think it is fair to say that there are more than few developers interested in contributing to JSCL. We could improve the on-boarding experience by adding a small amount of (developer, not user) documentation. I'm thinking of adding a Doc/Overview.org, improving the Hacking.org file and add a page about medium sized projects in the spirit of SBCL's GSoC page to the wiki.
Overview.org
- [ ] Compiler Overview: A brief paragraph or two similar to chapter 3 in Design of CMU Common Lisp
- [ ] Bootstraping processs (Environment Dumping)
- [ ] Run-time: What does the run-time provide. What are the differences when running on the run-time when running on node or on the browser
- [ ] Compiler internals:
- [ ] What are the AST nodes
- [ ] What do the the macros like define-raw-builtn, define-builtin, define-compilation, define-transformation do.
- [ ] How does lisp code and lisp values map to Javascript?
- [ ] How is multiple-value-call handled?
- [ ] A compiled file js file layout
- [ ] Lambda-list desugaring: How does each 'feature' of CL's rich lambda lists (&key, &optional, &rest) maps to a fixed argument function call.
- [ ] Backquote/Quasiquote implementation and macro expansion
Hacking.org
Adding a couple of questions and maybe a 'TLA'/Glossary file listing some abbreviations used in the codebase (like ll for lambda-list)?
- [ ] How does one compiles a lisp application or lisp file with JSCL?
- [ ] I got a run-time error when running, How do I start to debug?
- [ ] How do I compile lisp code to the AST (aka with-compilation-unit+process-toplevel)? To JS?
- [ ] How does one use git bisect to track a regression? (This is not something that currently be done easily)
- [ ] What information should one put in a commit message contain?
Medium sized projects
Each project w/ links to information that could be useful for implementing, a link to the issue for discussion and branches with WIP if applicable.
Stuff like Type system, Build process improvements (ej incremental compilation), CLOS, Loop, reader, source maps and javacript pretty printer would go in there
What do you think?
I think I can write about the boostraping process and the AST an submit it on an issue for you to proof read & I'll add some projects to the Wiki tomorrow.
Sounds great!
Do you think it is better doing this in the code when possible (compiler.js, runtime.js (prelude)), or it should go directly in the Wiki or org files?
I think that the org file should give an overview and should not exceed two (3 at most) pages. Something that one can read with ease before diving into the source. Documentation that goes into the details should go in the code (so it is more likely to be updated when modified).
I've added the project page to the wiki, it is still a bit rough, feel free to polish it if needed or add another idea to it.
the page limit is appreciated, or it will become unreadable. Maybe a good material for ILS paper?
This is what I have about the bootstrap process. I am not happy with the 'flow' of the text. it reads liek a list of bullet points. Also I am not sure if the last sentence should be included. But as the saying goes perfect is the enemy of good.
Bootstrap Process
=================
First the compiler is loaded into the host lisp. The compiler now loaded in the
host lisp compiles (`!compile-file`) its own sources along with the
implementation of CL is compiled again to produce `jscl.js`. To ease
development, bootstraping also compiles the tests, a web-repl and
node-repls. The latter compilation also dumps the global environment at the end
of `jscl.js`. The build order is specified in the variable `jscl:*source*` that
can be found in `jscl.lisp`. During the compilation `:js-xc` is present in
\*features\*. To avoid clashing with the host compiler some functions are named
with a precedent bang, ej !proclaim. When the target jscl is compiled the names
are, ej `#+jscl(fset 'proclaim #'!proclaim)`. A function not yet defined can be
used in code as long as it is defined before it is called for the first time.
IMO - enumerated lists are very good for bootstraps, they give a numerical description of what has to happen, in order.
@pnathan thanks for the suggestion. You were right. I've rewritten it as an ordered list and am much happier with the result.
Bootstrap Process
=================
1. The compiler is loaded into the host lisp.
2. The compiler now loaded in the host lisp compiles (`!compile-file`) its own sources.
3. Along with the resulting JavaScript the environment is serialized and the result is saved `jscl.js`.
4. Although not strictly part of the bootstrapping itself, the tests, a web-repl and node-REPLs are also compiled and saved as `test.js`, `repl-web.js` and `repl-node.js` respectively.
Notes
=====
- The build order is specified in the variable `jscl:*source*` that can be found in `jscl.lisp`
- During the compilation `:js-xc` is present in \*features\*.
- To avoid clashing with the host compiler some functions are named with a precedent bang, ej !proclaim. When the target jscl is compiled the names are, ej `#+jscl(fset 'proclaim #'!proclaim)`.
- A function not yet defined can be used in code as long as it is defined before it is called for the first time.
Hi @PuercoPop, I think this is great.
As a jscl noob trying to understand the project, the most valuable information to have would be everything that you mention in Overview.org. These are all points essential to getting the project but which are difficult to learn just by glancing at the code. I can't think of anything else to add to your overview file, but I'd happily learn about all the points you mentioned.
In Hacking.org, it could also be interesting to have some information about what jscl is still missing or needs improvement (or maybe somewhere else, source code comments/Readme?). Right now, it's only really clear from the Readme and milestone that format, loop and clos are missing. Even if it is too much work to specify and maintain such list with everything in it, just having a few comments at less granularity would be helpful for people getting into jscl, I think. Maybe something like:
sequences
- A few of the implemented functions are missing argument keys, namely from-end.
- Some functions (e.g. count, count-if...) are not implemented, please take a look at http://www.lispworks.com/documentation/HyperSpec/Body/c_sequen.htm to see what is missing.
- Note that we are probably not ready to implement concatenate because first we need so and so.
Also, there could be a section somewhere explaining some strategies used in code that you think could confuse jscl beginners. I'm not sure if this is actually worthwhile or if i'm just dumb, but there is some code that I didn't yet figure out, like why we have (defun stringp (s) (stringp s)) and other definitions like this.
I think we can use github issues to keep track of what is missing. I created a ANSI Compliance milestone. Bugs and missing functionality from the standard can be assigned to such milestone so we can easily list them and track progress.
I am on holidays right now but I will write a small document with comments about the compiler and bootstrapping process next week. :)
Sure, github issues to track missing functionality sounds much better that my suggestion. Should I then start creating issues for every functionality that I find is not yet in accordance with the standard? Like an issue for the count function? Or is that too specific?
it depends. If the functionality is big, led's say type system, it is probably pointless to create all tickets in advance. Instead a "Implement type system" is better. We can create smaller tickets once that issue is started or converted in a milestone itself.
For missing general functions like count, yes, individual issues are preferred I think.
There are a few exceptions, for example implementing Setf for all the c[ad]+r family of functions.
But just go ahead and if someone prefers to break or group tickets we can discuss it there.
@davazp I've started to write some docs on the compiler, they are still far from complete. What can be of most value to you when you are writing the document on the compiler are the questions I'm left with after reading the compiler.lisp file.
= Backend compiler
The backend compiler is composed of two systems.
The first part rewrites s-expressions.
The second part takes s-expressions and generates the corresponding Javascript.
== Rewriting phase
The rewriting phase consists of +define-js-macro+, +define-compilation+, +define-transformation+, +define-raw-builtin+ and +define-builtin+.
+define-js-macro+ is how one defines macros that are going be present in the JSCL run-time (?). Ni idea, parece ser una implementación normal de macros. Tal vez no debería estar como parte del compilador.
+convert+
+define-compilation+ ... | The primitives are stored in +*compilations*+.
+define-transformation+ is similar to +define-compilation+. The difference is that the result is wrapped in a covert.
+define-raw-builtin+ ...
+define-builtin+ works in a similar manner than +define-raw-builtin+.
The only difference is that before executing the +body+, +convert+ is called on every argument.
As they share the storage place (should I use namespace?) the builtins and raw-builtins can nameclash.
The entry point is
See the file 'src/compiler/compiler.lisp' for more details.
=== Questions this section needs to answer
* Is +define-js-macro+ just how macros are implement in jscl or do they have a
special property that should be noted?
* What is the difference between define-compilation and a normal macro
definition? Is the only difference that they have access to the environment
through +*environment*+ variable instead of through the +&environment+
argument? Corroborate that the +*environment*+ is only the lexical
environment.
* Why does +with-compilation-enviroment+ not bind the +*environment*+ and
+*toplevel-compilations*+ variables?
* +define-raw-builtin+ and +define-compilation+ are identicall except for the
variable where the definitions are stored. What are the differences? Is it on
the phases they are used?
* Include a Graph following the entrypoint +js+ to each of the functions of
interest.
== Code generation pharse
See the file 'src/compiler/codegen.lisp' for more details.
@davazp Btw, is there any reason why some docstrings are written as a comment above the function? Or is just a legacy from before docstrings were supported?
Just bad discipline from my side.
I used to write comments for internal functions and do strings for public ones but I didn't stick to it.