Removing dynamic plugins

[zaxtax]

I've noticed that the bundled plugins are severely bit-rotted. The code for dynamically loading plugins is very brittle and does not play well with the sandboxed method Haskell libraries are installed these days.

@jgm would you be opposed to a PR that ripped out dynamic plugin capabilities and replaced it with something more akin to a contrib library or just bundled the existing plugins in a way that they are compiled with the rest of the package?

[jefdaj]

I've been doing the latter myself for a while. Here are the relevant parts of my Plugins.hs:

import qualified Network.Gitit.Plugins.Csv as Csv import qualified Network.Gitit.Plugins.CiteLinks as CiteLinks import qualified Network.Gitit.Plugins.CiteProc as CiteProc import qualified Network.Gitit.Plugins.Dot as Dot import qualified Network.Gitit.Plugins.External as External import qualified Network.Gitit.Plugins.Files as Files import qualified Network.Gitit.Plugins.RelatedFiles as RelatedFiles

compiledPlugins :: [Plugin] compiledPlugins = [ Csv.plugin , Dot.plugin , External.plugin , Files.plugin , RelatedFiles.plugin , CiteLinks.plugin , CiteProc.plugin ]

loadPlugins :: FilePath -> [FilePath] -> IO [Plugin] loadPlugins pluginDir pluginNames = do ... return $ compiledPlugins ++ plugins'

Jeff

On Sun, May 23, 2021 at 2:55 PM zaxtax @.***> wrote:

I've noticed that the bundled plugins are severely bit-rotted. The code for dynamically loading plugins is very brittle and does not play well with the sandboxed method Haskell libraries are installed this way.

@jgm https://github.com/jgm would you be opposed to a PR that ripped out dynamic plugin capabilities and replaced it with something more akin to a contrib library or just bundled the existing plugins in a way that they are compiled with the rest of the package?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jgm/gitit/issues/672, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJEP4MGQJNO2EDOXAK4TB3TPF2WRANCNFSM45MDYLKQ .

[jgm]

I'm open to this. I'm not sure what the best way would be.

One advantage to the dynamically loaded plugins is that compiled plugins are probably going to be quite large (since they link to Gitit itself). (That could be mitigated with dynamic linking, but that's not the default at least with stack.)

[zaxtax]

I've compared the size of the gitit binary with and without compiled plugins and the difference is negligible. If we are concerned about binary size I've had reasonable success using upx and strip reducing the gitit binary size from 222M to 42M.

[jgm]

Oh, I see -- do you mean you're including the plugins in the main binary? I had thought you meant compiling them separately.

[jgm]

Part of the idea of the plugins directory was to give examples showing how people could create their own dynamically loadable plugins -- this objective wouldn't be met just by providing some baked in ones.

[zaxtax]

I'm happy with either suggestion. I am suggesting including the existing plugins in the main binary as the simplest way to prevent and reverse bit-rot. It also doesn't seem totally unreasonable as it seems the main way people are running gitit is installing it from source. If you are creating your own plugins, you are probably skilled enough to patch gitit to bundle the plugin.

[jgm]

On the other hand, if you can patch gitit, then you don't really need plugins. You can just patch your transformation into the main content transformer chain.

[zaxtax]

I like the idea of having something like pandoc filters but optionally configurable. Right now, I'm doing what @jefdaj does for his plugins since the mechanism is really hard to get working with the way cabal sandboxes packages these days. In my current setup, effectively Network.Gitit.Plugins is where I include my plugins and not gitit.conf.

[jgm]

So, is the main problem that without a global cabal package store, the current dynamic loading mechanism can't find the dependencies? Or is the problem installing the dependencies in the first place? If it's the first problem, we might be able to do something like

export GHC_PACKAGE_PATH=`stack path --snapshot-pkg-db`:
gitit

[zaxtax]

It's the first problem. There is a way to fix this by explicitly adding gitit to

~/.ghc/x86_64-linux-8.10.2/environments/default which is a bit hacky

[zaxtax]

The larger issue is that either solution is fairly brittle, and it doesn't solve the bit-rot problem that the included plugins in the repo don't even compile as they were built to work with an older version of pandoc.

I think the simplest solution would bundle all the plugins as compiled modules and the configuration file would change to just select amongst those already included in the package. If a user wanted to add their own plugin they would use the bundled ones as examples and modify the codebase to add them. This also has the advantage that it's a small step from there to push it upstream where it's visible to all other users of gitit.

[jgm]

The whole idea of plugins, though, was that you don't need to recompile the gitit code base to use them. I'd prefer to keep that. It would be helpful to hear from others who use gitit plugins, to see how they deal with these issues.

[zaxtax]

Certainly. In the meanwhile, let me fix up the existing plugins. Whatever gets decided that needs to happen regardless.

[jgm]

That would be great!

[j-loreaux]

Just to throw in my 2 cents: I also compile my plugins into the main gitit binary. My feeling is that in gitit's current state (i.e., many people building from source; I had to for FreeBSD), anyone who can both compile the software and knows enough Haskell to write a basic plugin probably has enough knowledge (and likely won't mind) compiling the plugin directly into the binary.