mako
mako copied to clipboard
sqlalchemy
rearchitect the parser to be more modern, traditional, and extensible
Migrated issue, originally created by zaxebo1 zaxebo1 (@zaxebo1)
jinja2 has configurable syntax for delimiters. If you are generating LaTeX or other formats with Jinja2 you can change the delimiters to something that integrates better into the LaTeX markup. see at
http://svn.python.org/projects/external/Jinja-1.2/docs/src/altsyntax.txt
https://gist.github.com/lost-theory/3925738
for example: How Flask configures jinja templates for a configurable delimiter
jinja_options = Flask.jinja_options.copy()
jinja_options.update(dict(
block_start_string='<%',
block_end_string='%>',
variable_start_string='%%',
variable_end_string='%%',
comment_start_string='<#',
comment_end_string='#>',
))
=========================================================
**NOW, My Feature Request is that
FEATURE REQUEST A) can we have optionally configurable delimiters in mako too, like jinja above (VVIMP)
FEATURE REQUEST B) like Preprocess ( https://code.google.com/p/preprocess/ ) or cog ( http://nedbatchelder.com/code/cog/ ), can we have an enhancement where the mako statements can be hidden inside the host language(say,latex etc) comments like /* */ or, # . A mako statement is a comment (as appropriate for the language of the file being preprocessed). This way the mako statements do not make an unpreprocessed file syntactically incorrect for the IDEs. The exception to this can be ${variablename} statements , but other syntaxes like <% and % etc can be OPTIONALLY inside host language comments, if the mako user configures it as such.
**
I love mako very much and would like it embed it inside latex,sql and other host languages source code easily. :-)
Michael Bayer (@zzzeek) wrote:
we have a preprocessor option. I don't understand what B is asking for that's beyond that. if you want C-style "macros" then take the preprocess library and link that to "preprocessor": http://docs.makotemplates.org/en/latest/usage.html#mako.template.Template.params.preprocessor. Don't see what feature needs to be added here.
The whole Lexer is pluggable also.
I prefer options that are more generic and extensible rather than a list of specific delimiters in specific spots. I can make Mako do all the things you describe right now without changing Mako itself.
zaxebo1 zaxebo1 (@zaxebo1) wrote:
Thanks for reply. I will try to explain my viewpoint again
**for feature request B) ** it was not about preprocessor feature as per se.
stepB.1) Say, i have a multi hundred lines C++ source file , which I am writing in VisualStudio IDE. OR, a big latex file which I am writing in TexStudio IDE .
Step B.2) Now, if i write small mako code in (mako basically itself being used as a uniform flexible preprocessor) in these C++ /latex source. Then this mako source code block inside C++/latex file will be shown in IDE as syntax errors (even before running the code, it will be shown as syntax error, even while typing this mako code inside C++/latex source)
So, if "optionally" i can put mako source code inside a configurable multiline comment syntax say /* */ (in C++), then IDE will not show me errors on mako code blocks . IDE will continue in natural way with autocompletion etc tasks. And when i have to run mako finally , i can always run and strip it. HENCE feature request B, It will enable me to use mako as preprocessor inside C++ files, without violating C++ syntax for IDE.
BUT, on second thoughts I too see it as a corner special case as per your suggestion, and WITHDRAW my request of feature B
zaxebo1 zaxebo1 (@zaxebo1) wrote:
for feature request A) you had said that "The whole Lexer is pluggable also. I prefer options that are more generic and extensible rather than a list of specific delimiters in specific spots. I can make Mako do all the things you describe right now without changing Mako itself."
Can you guide/point me that How can I change delimiters by pluggable lexer or any other method, as to finally get the configurable delimiters.
apart from jinja2, in cheetah templating engine also , we could change delimiters
========================
**conclusion: May be I can live WITHOUT feature request B, but I would love to get feature request A . ** Without feature A - i will not be able to dump jinja altogether. I want to use only one templating that is mako, currently I have to bear with jinja also" .
Please reconsider ATLEAST ABOUT implementing atleast feature A (configurable delimiters). If it is in jinja, then obviously it must be a real requirement for many many people like me. They show it off "configurable syntax" as their important features on top of their homepage "http://jinja.pocoo.org/". Obviously , configurable delimiters by default in the language is a real life feature requirement , **I will be really very very grateful if you consider my request of atleast implementing feature A. **
zaxebo1 zaxebo1 (@zaxebo1) wrote:
https://groups.google.com/forum/#!topic/bottlepy/pJaU3vzVXuU
refer to this above link to understand various people , who really want to customize the delimiters to template language (although the example link does not use mako, but still... ) if we want to use mako for that as a preprocessor for other languages. then {{var}}, ${var}, etc may be harder to distinguish from other syntax in case of perl , latex etc. This IS a real world production level requirement, at lot of places.
Michael Bayer (@zzzeek) wrote:
I'm in full support of the features you want just not in exactly the way you see them being implemented.
e.g. instead of just adding a zillion flags and switches which still only cover a very specific use case, why not just provide mako.ext.latex that gives you a new Lexer that works very well for latex files ?
zaxebo1 zaxebo1 (@zaxebo1) wrote:
I actually do not know about implementing lexers in mako/python, nor could i find any documentation on the topic. I am a user using mako as preprocessor on other language(like latex, rst ) sourcefiles
Can you please kindly implement "mako.ext.flexiLexer" that **gives me a new flexible Lexer which implements configurable delimiters ** itself . That would be a really great help.
Thanks in advance.
Michael Bayer (@zzzeek) wrote:
jinja's parser is more traditionally architected, which is partially because jinja has the advantage of being able to parse the text completely. Mako's parser has to work around sections of Python code, so has a little more of an ad-hoc way of getting around that. It has a lot of regular expressions that get pretty thorny. To accommodate switching of delimiters around would require all of these regexps be injected with those special symbols, which would make the existing regexps that are already way too crazy that much more difficult to work with. It's also likely that if a user puts particular symbols in there which conflict with how we work around the Python code, it'll break. Mako's syntax is more complicated than Jinja's for this reason.
Jinja's parser is definitely nicer and more extensible but we'd have to first modernize and rewrite ours to be more like Jinja's, which I'd need a volunteer to work on, it's nothing I have any availability to work on. It would have to be very smart about extracting the blocks of real Python code which is where it gets difficult.
The good news is that if someone wants to work on reworking the parser, it at least can be dropped into Mako dynamically without adding it to the source to start with.
The preprocessor feature can be used to get simplistic versions of these features. You can take any file and search-and-replace out whatever delimiters you want into Mako's, and take all the existing symbols that would be confused with mako and turn them into ${"<%"} or place them inside of <%text>. It's not ideal but it will get the job done.
zaxebo1 zaxebo1 (@zaxebo1) wrote:
ok, now i see your point and agree with you. the preprocessor way you suggested is fine for me. but the link you had mentioned earlier (for preprocessor) was too brief http://docs.makotemplates.org/en/latest/usage.html#mako.template.Template.params.preprocessor
so i even further scale down my requirement to the following: :-)
"Can you suggest some example source code of a fully working "helloworld kind of a simple preprocessor actually plugged into mako", so that I can extend it from there."
Michael Bayer (@zzzeek) wrote:
wondering if @CodeMan99 has any thoughts on this issue. Source code generation in arbitrary languages is a special niche for Mako and making it more flexible wrt languages with heavy overlap like latex is probably an important feature. The preprocessor can do these kinds of things but not totally nicely, for one it generates a Mako template that is not the one you see, so it can be more difficult to debug. A new lexer design that's more extensible would be nice. the existing one is essentially descended from the HTML::Mason lexer (yes that's perl).
zaxebo1 zaxebo1 (@zaxebo1) wrote:
Really Thanks and indebted from my heart for acceptance and marking of this really important feature for me.
Michael Bayer (@zzzeek) wrote:
see http://pygments.org/docs/lexerdevelopment/ for inspiration. we should be going along those lines in some way, though at the same time if our own default Lexer can accept rules like those without writing a whole new one from scratch that would be great.
jvanasco (@jvanasco) wrote:
I just wanted to note that an added benefit of Jinja's behavior is that you can fake a lot of other templating languages with it. I had a prototype a while back of a sandboxed Jinja environment that was rendering most of the functionality of Mako and Dust templates.
Changes by Michael Bayer (@zzzeek):
- changed title from "configurable delimiters like jinja2" to "rearchitect the parser to be more modern tradition"