Improve ability to support custom arguments for Fuseki Servers

[rvesse]

Version

5.1.0

Feature

The Fuseki Modules mechanism makes it possible to add extensions to Fuseki but modules have to rely upon external configuration sources e.g. System Properties, Environment Variables, Fuseki Configuration file etc. because there isn't a meaningful way to pass configuration via the command line.

FusekiMain is built on Jena's general CLI framework which means we can in principal extend FusekiMain and add additional arguments via the constructor, or use the static FusekiMain.addArgModule() method to register global extensions.

However, it's hard to do anything meaningful with these custom arguments, processModulesAndArgs() currently populates a private ServerConfig class which is then used to actually populate a FusekiServer.builder() via a private static applyServerArgs() which cannot be modified. Thus even if a custom argument is introduced the only way for it to pass configuration to a module would be by placing it into System Properties, static variable or another JVM wide store that the module can later read back.

As a practical example of this we're looking to introduce support for specifying Fuseki configuration via a YAML configuration file for which we'd like to add a --yaml-config <yaml-file> argument and then have a FusekiModule do a prepare() step which reads in the <yaml-file> and modifies the FusekiServer.Builder accordingly. Right now we can't do that cleanly because there's no way to get the value of <yaml-file> passed into the builder for later access without resorting to passing it out-of-band.

It would be better if we were able to carry extra configuration information via ServerConfig and override applyServerArgs() so that we can apply that to the builder this would offer users who want to extend FusekiMain much more flexibility.

Are you interested in contributing a solution yourself?

Yes

[afs]

General comment - when (if!) there is a switch to Fuseki Main + UI + Admin for the standalone server (jena-fuseki-fulljar and the jars file in the download), there maybe other set of arguments that are optional. Maybe there should be an extension point for this.

If that comes along, PR #2754 would be fitted into a formal lifecycle for the command line.

Do you think adding a new interface FusekiModule would fit your customization start flow? (FusekiModule collects several interfaces already)

[rvesse]

@afs Yeah when we started on this work I had it in my head that we'd previously discussed an extension point for adding custom arguments into Fuseki but then I found what was there didn't really work for the reasons described here.

Ideally making the arguments a more formal integration point would be useful. However not sure what that looks like in practise because there's some separation of concerns issues. Currently arguments are used to populate a ServerConfig which is then applied to the builder, and it's only when the builder has build() called that modules actually kick in currently. So it seems like this would need to be an independent extension point with its own interface that gets called prior to argument processing.

I'd almost be tempted to move argument processing concerns more directly into the FusekiServer.Builder thought then that conflates concerns between CLI and API usage. Or possibly remove ServerConfig entirely and just have a builder as an instance variable of FusekiMain and let it populate it directly from its processModulesAndArgs() method. This would enable the extension style approach I showed in the PR, or more API driven extension points where you could register interfaces/functions that worked on the builder instance and used that

I'm not in a rush to get this into 5.2.0, we have a workaround for now which is basically to pre-process the arguments and effectively just overwrite the value of the --conf argument if it has a .yaml extension with our generated .ttl file.

Let me have a play around in a separate PR for comparison purposes, this will likely be Tues next week at the earliest before I can get to this.

[afs]

The builder is available in FusekiModule.prepare but builder is limited as it has a fixed contract. Some custom arguments may be complete separate from building.

The extensions would ideally not subclass FusekiMain or any Fuseki cmd/builder class because multiple sets of argument don't "play nice" by subclassing.

It needs a "prepare args" via some provided ArgModules with a lifecycle "setup - parse - respond to args - ....".

[afs]

@rvesse -- I'm working on a PR with material taken from #2773 that focuses on just a customised command line.

I think there needs to more steps in the customisation process

• add custom arguments
• process arguments to get any flags and arguments - this happens after the standard arguments have filled ServerConfig
• after the server configuration has been setup before building starts - this can react to the configuration model before the build process uses it.

ServerConfig becomes public (it is just a mutable record - may rename as ServerArgs) and now has both the file name from --config and also a model slot initialized to null. At the start of building and there no model (the normal case), then the filename is read as RDF. A customizer can get the file name, and itself set the configuration model.

• The configuration file name can be checked before use (e.g. either an allow-list or a deny-list implemented, or string for a customiser to interpret.
• --config=custom.yaml can be made to work, as well as --yaml-config.

[rvesse]

@afs You're much more familiar with this area of the codebase so if you have a cleaner approach then my initial attempt happy to test that once it's ready

[afs]

@rvesse PR #2845 - in TestFusekiMainCmdCustomArguments, tests cover ways to control the configuration model.

test_custom_confModel_noFlag()
test_custom_confModel_replace()
test_custom_confModel_different()
test_custom_confModel_different_ignore()