lua-filters
lua-filters copied to clipboard
pandoc
Change this repository into a collection of links?
One drawback of the current structure is that people submit code here but then don't monitor the repository, and issues are neglected. Perhaps it would be better to make this simply a collection of links to lua filters that are maintained in independent repositories?
I kind of like this idea. Maybe this repo could serve as a kind of collection of "official" scripts from the Pandoc creators and all other filters could be linked in the README (sorted by topic)? That would reduce the maintenance to checking the links every year. In addition, the "official" code provided could serve as a live demo / live documentation of the Lua API.
I'm very much in favor of that; it would save me a lot of time. It takes a significant amount of effort, on almost each new pandoc release, to adjust the tests and filters to the changes. It's tiresome, and pinging all authors and waiting for them to change the code would take just as long. I'd be glad to get out of that obligation.
We could still do occasional automatic "releases", which pack the filters into a single archive. This shouldn't be too hard if the individual repos use a common structure.
We could still do occasional automatic "releases", which pack the filters into a single archive.
That sounds interesting, but might not be quite easy with regard to the then presumably different licences in different repos?
This shouldn't be too hard if the individual repos use a common structure.
Would maintaining a template repo help with this?
Pinging everyone who contributed a filter so far: what do you think of this idea? What would you need to make this as painless as possible for you?
@jdutant @tolot27 @blake-riley @not-my-profile @svenevs @b3 @jkr @cole-miller @sokotim @korintje @gtuckerkellogg @stroobandt @frederik-elwert @odkr
Since each filter belongs to a subfolder, it should be easy to split your repository into several individual repositories using git filter-branch and retain the individual history :)
I am not sure about this. There are advantages to having a common repository. Having them all here guarantees that other pepole can contribute improvements even when the original author has ceased maintenance.
It takes a significant amount of effort, on almost each new pandoc release, to adjust the tests and filters to the changes. It's tiresome, and pinging all authors and waiting for them to change the code would take just as long. I'd be glad to get out of that obligation.
I think an easy fix for that would be to have a latest_pandoc_supported variable for each filter. When a new pandoc version is released that variable could be automatically bumped for each filter which tests pass with the new version. And the script could automatically update a table in the README of this repository that lists all scripts known to work with the latest version. Even the pinging of authors when their filter is no longer compatible with the latest version of pandoc could be easily automated.
Especially if you still occasionally want to release filter bundles having all filters in a single repository should make that easier. Otherwise you just have new potential problems to deal with (e.g. some repository went offline, some repository suddenly has an unexpected file structure, etc.).
I am in favor, but what about having this repository contain submodules/subtrees/subrepos linking to contributors' repositories so that people can still pull this repository and get all filters? I suppose a cronjob or action could be set up to update the links every day/week/month.
Den fre 31 dec. 2021 09:56Martin Fischer @.***> skrev:
I am not sure about this. There are advantages to having a common repository. Having them all here guarantees that other pepole can contribute improvements even when the original author has ceased maintenance.
It takes a significant amount of effort, on almost each new pandoc release, to adjust the tests and filters to the changes. It's tiresome, and pinging all authors and waiting for them to change the code would take just as long. I'd be glad to get out of that obligation.
I think an easy fix for that would be to have a latest_pandoc_supported variable for each filter. When a new pandoc version is released that variable could be automatically bumped for each filter which tests pass with the new version. And the script could automatically generate a table for the README that lists all scripts known to work with the latest version. Even the pinging of authors when their filter is no longer compatible with the latest version of pandoc could be easily automated.
— Reply to this email directly, view it on GitHub https://github.com/pandoc/lua-filters/issues/207#issuecomment-1003313160, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAI3OU5UGRQWJSEWCOAMAKDUTVV2ZANCNFSM5LAX3HCQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>
I agree with this idea. It is because the roles of the filters are highly independent, so we cannot expect large synergy effects by collecting them at one repository. Instead, I think it is better to focus on keeping accessibility, readability, and consistency of each filter documents. Listing the filters at Pandoc official web page or GitHub Pages would be nice. As far as I know, one ideal example is crates.io, which is a kind of library repository for Rust language, is known to have well-formatted and easily-accessible documents.
There are advantages to having a common repository. Having them all here guarantees that other pepole can contribute improvements even when the original author has ceased maintenance.
In that case you could always fork the original repository, make your changes, and submit a PR here for an updated link to the fork.
I am in favor, but what about having this repository contain submodules/subtrees/subrepos linking to contributors' repositories so that people can still pull this repository and get all filters?
I'm not sure it's all that valuable to be able to get all the filters in one repository. Generally you only need some of them; why not just clone those separately?
I like the idea of submodules and switching to them should be easy because we have subdirectories, already. submodules can be checked out from their origin and individually. Having this repository as the main repository has the advantage that checks (i. e. in case of a new pandoc version) can be maintained at a central place.
I don't have any preference either way, if it makes things easier for maintainers then I'm all for it :heart: I'm pretty sure my filter is feature complete, but I'm sorry if I've missed any issues related to it.
I'm really not convinced this would be an adventitious move. Having some complex filters that see a lot of development in their own repos is a good thing perhaps (and we have a history of suggesting that) but for small one-off ones that tend to be submitted, used by them a few times, and then the submitter moves on I think a large chunk of them would fall below some minimum threshold that would make them viable FOSS projects on their own. Having a team of maintainers at least reviewing submissions here adds some amount of normalizing and consistency that makes filters in this repo much more attractive than random ones out of people's Gists, and for maintenance not having the bottle neck of one maintainer that got it working form themselves and then is never motivated to tweak it to be more generally useful seems seems like a benefit to most simpler filters.
@alerque I think the filters could still be reviewed -- at any rate, we wouldn't need to include links to filters that didn't look good. The aim would be to change where bug reports and enhancement or support requests go. They should go to the author of the filter, not to the pandoc maintainers.
As a first-time, one-off contributor, I have to admit that the current process together with the suggestions and help provided by @tarleb rendered my contribution more worthwhile and universally applicable. That would not have happened without the "editorial work" of @tarleb. The current process could be considered as a very valuable peer-review, where the value eventually goes to the end user.
Another admission of mine is that my extended family and I usually employ my filter only with the version of pandoc that comes packed with the latest Ubuntu LTS release and upgrades. The reason for this being the fact that too many of my users on too many machines require a stable system environment for work/study.
This certainly does not mean that I would not maintain my filter. However, if the user community at large fails to prod me, I would typically notice a version compatibility problem with my filter only when a new version of pandoc eventually lands in the Ubuntu LTS repositories.
I hope this straightforwardness helps with reaching a consensus about how to proceed with this great, curated collection of filters.
I do not have any smart definitive answer to this good question. I added thumb up to comments given ideas that I like.
It is a fact that I didn't follow issues here for my small filters (thanks to @jgm I will now try to check them).
It is also a fact that, as @stroobandt states, @tarleb work rendered my small contributions more worthwhile and universally applicable.
IMHO I think that keeping a common framework (at least for tests and description for instance) need however to be kept.
Being able to fetch all code at once is also a nice facility (which helps me being inspired) but still can be offered if this repo is changed to a simple list of links.
Sorry not being able to help more concretely.
The aim would be to change where bug reports and enhancement or support requests go. They should go to the author of the filter, not to the pandoc maintainers.
To some extent, we can get the best of both worlds. If the code stays here and we add contributors to a GitHub team with limited access to this repo, we can use .gitattributes to specify GitHub users as code owners for the filters they contribute. That way they would not only get asked to be involved in code review if somebody touched their code, but they could be assigned to related issues and such.
My experience is that people are even more likely to stay involved and take some ownership over their code if it has the publicity of being in an official repository rather than being in their own ad-hoc repos. Anybody that is going to keep on top of issue reports on their own repo is also likely to stay involved with it if they have some ownership in a bigger project.
I really like this repository and it is a great source of inspiration when writing filters. It is very useful to have all these filters in one place. Of course, I agree the plugins creators should maintain their plugins( if they have the time to). Some plugins could be placed in a unmaintained folder or repo if they can't. Also a table in the README would be useful indicating filters name, description and formats processed.
we can use .gitattributes to specify GitHub users as code owners for the filters they contribute.
Can you elaborate? What would the syntax be? That would certainly be an improvement, as now there's no way to figure out who contributed which filter other than looking at git history.
Here is an example CODEOWNERS file that uses .gitattributes syntax to assign code is a repository to different people. The @... names can be individual accounts or teams (or mix and match) that can have multiple members. This will automatically request they review any PRs touching those code paths as well as open the door to other GitHub tooling like allowing them to approve PRs if code-owners approve them.
What it doesn't do is triage bug reports and assign them to those owners. That would still need to be done manually, only PRs are automatically assigned.
I've added .github/CODEOWNERS, but I don't know the github handles of the contributors. Maybe people can update this themselves with PRs?
As a side note, I think this is really about having a packing index and a package manager.
people like this and pandocfilters (the Python one) because they act like both. It is a centralized location that once cloned, someone else is maintaining that for you which should guarantee it is working with the latest-ish pandoc.
The problem of this repo is that it isn't going to scale well (into many filters) and the work of maintenance is transferred to the maintainer.
Years ago some of us proposed to have a package manager, and there was a prototype. But there was a few problems. First we mixed the 2 related concept in one solution, and second it isn't official.
In short I think the right direction would be to have an official package index (like CTAN). This is similar to the "link" concept above, but more formal. May be a YAML file with a certain spec. The official pandoc community advertise this as the pandoc packaging index that people should submit to as authors and discover as users.
Then we can let 3rd parties to build a ecosystem around it. Eg a package manager (similar to 3rd party filter framework), or a website (like the 3rd party Mac AppStore-like website for homebrew).
Allow me to think out loud for a moment; this gets a bit fundamental and includes some of the good points others already made above.
What I like about this repo:
- It has become a beautiful collection of useful filters. Users can browse it, use the filters, and hopefully learn from the code.
- Filters share a similar structure and allow customization in a similar manner.
- I get to work with contributors, which is a chance for me to bring in the experience from writing many filteres and large parts of the Lua subsystem.
What I dislike:
- Filter authors don't really get the recognition they deserve. I'm not at all a fan of "GitHub is my resumé", but that's often the way it is. A filter in a personal repository is easier to show off as achievement than a contribution in a repo like this one. It seems fair to encourage people to highlight their work.
- We are currently excluding people who prefer other platforms like GitLab.
- My code standards are opinionated and often high, quite possibly too high. This effectively prevents useful filters from becoming available as they are stuck in code review for far too long. (But I appreciate the kind words noting the positive side of this.)
- The previous point is made worse by my time constraints. It felt in the past like I was often the bottleneck for changes and new filters. Not a good situation, neither for contributors, nor for me.
- Tests are often difficult to write, and most tests depend on the specific output created by pandoc. That makes them, and the repo, high maintenance.
In conclusion, I'd still rather turn this repository into a collection of links. My proposal:
- Create a template repository for Lua filters. This way we can still encourage a certain standard layout, but filter authors have the freedom to do whatever they feel is right.
- Add an issue template for adding new links: this should include a checkbox to select if the author wishes for a detailed code review of their filter. We could go as far as to encourage community review by sending an automated mail to pandoc-discuss whenever such an issue is opened.
- Slowly move filters to separate repos, but explore ways to create collections of all filters listed here and adhering to certain conventions.
Edit: Forgot to make my main point: it seems unreasonable to expect people to maintain code that they no longer control; the sense of ownership is much stronger if authors can retain full control over their code.
I started work on a template repository. It's not quite done yet, but feedback is welcome, especially if it takes the form of a PR ;)
The template contains code to create a documentation page, but I'm not happy with having the HTML and CSS in the main branch. If anyone has some ideas on how this could be avoided, then please let me know.
BTW I wonder whether my filters written in MoonScript which is then compiled into pure Lua code would be acceptable if/when this repository turns into a catalog?
(FWIW I also have a helper library for Lua/MoonScript filters written in MoonScript, which perhaps could be linked to. I already have begun writing a README including instructions on how how to setup the LUA_PATH environment variable, based on an excellent [blog post][] which now sadly can only be found on the Wayback Machine.)
[blog post][]: https://web.archive.org/web/20210127030419/http://www.thijsschreijer.nl/blog/?p=1025
Den fre 21 jan. 2022 09:37Albert Krewinkel @.***> skrev:
I started work on a template repository https://github.com/tarleb/lua-filter-template. It's not quite done yet, but feedback is welcome, especially if it takes the form of a PR ;)
The template contains code to create a documentation page https://ztkrt.de, but I'm not happy with having the HTML and CSS in the main branch. If anyone has some ideas on how this could be avoided, then please let me know.
— Reply to this email directly, view it on GitHub https://github.com/pandoc/lua-filters/issues/207#issuecomment-1018292762, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAI3OU2DUQAUGU47EWSJ76TUXELIFANCNFSM5LAX3HCQ . You are receiving this because you commented.Message ID: @.***>
I'd think that'd be perfectly fine, esp. if the transpiled Lua script can be downloaded somewhere.
Off topic: my long-term goal is to write filters with teal.
The template contains code to create a documentation page, but I'm not happy with having the HTML and CSS in the main branch. If anyone has some ideas on how this could be avoided, then please let me know.
Hmmm, if the goal is to provide nicely rendered documentation, you could write the documentation in Markdown and use a simple workflow that uses Jekyll or Hugo, deploying the result as Github pages.
Alternatively, you could use a workflow where Jupyter notebooks are generated from the Markdown with Pandoc and made available as Github pages (.ipynb files will be rendered as preview directly by Github).
Den fre 21 jan. 2022 10:29Albert Krewinkel @.***> skrev:
I'd think that'd be perfectly fine, esp. if the transpiled Lua script can be downloaded somewhere.
Of course they are, in the very same directory, where the moonc compiler puts them by default.
Off topic: my long-term goal is to write filters with teal
https://github.com/teal-language/tl.
— Reply to this email directly, view it on GitHub https://github.com/pandoc/lua-filters/issues/207#issuecomment-1018332250, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAI3OU2NYW67HQGXW7YKN7LUXERLVANCNFSM5LAX3HCQ . You are receiving this because you commented.Message ID: @.***>
Sorry I haven't had time to jump in and help with this yet. A template repo is a great idea. Including some CI to test pandoc interactions would be good to include there too.
Before this gets too far though I just wanted to throw in the idea that if filters are going to be independent, it might actually be useful to package them as Lua Rocks. The luarocks infrastructure can actually be used for this (plugins for some Lua app as opposed to stand alone packages) and has a concept of manifests to organize them. This would bring in free tooling for versioning, distribution/packaging, dependency management (including both on other Pandoc filters or other LuaRocks), etc.
