Arbitrary SearchFields in ImportConfigurations for OAI interfaces

[solth]

Describe the bug Currently, all ImportConfigurations of type Catalog search allow configuring arbitrary SearchFields for search queries. This makes sense for SRU interfaces, but not for OAI interfaces which are designed not for searching but for harvesting. OAI interfaces in Kitodo.Production only use the predefined OAI verb GetRecord to retrieve individual records and therefore cannot generate search result lists, so defining custom SearchFields does not make sense for OAI configurations in Kitodo.Production.

To Reproduce Steps to reproduce the behavior:

1. Go to 'Projects'
2. Click on 'New' and then 'ImportConfiguration'
3. Select 'Catalog search' in the 'Configuration type' menu
4. Select 'OAI' in the 'Search interface type' menu
5. Scroll down to 'Search fields'. Here, custom fields can be added, which should not be allowed for OAI

Expected behavior Adding custom SearchFields should not be possible for OAI ImportConfigurations.

Release 3.4.4-SNAPSHOT

[BartChris]

You are probably correct. One problem i see is that there might be a custom interface, which responds in an OAI-conformant way but also supports other search fields. I used custom "searchfields" (ID) to return an OAI conformant response. So fixing this issue would reduce flexibility which is on the other hand re-introduced by e.g. https://github.com/kitodo/kitodo-production/issues/5257

The question is if there is a real disadvantage in allowing for more flexibility.

[solth]

I guess the best solution would be to add another search interface type "Custom", which does allow freely adding search fields but does not impose the restrictions that real OAI or SRU interfaces have (OAI interfaces need a configured metadataPrefix, but do not allow for custom search fields, for example, while SRU interfaces always need a recordSchema and also support - or even require! - configuring arbitrary search fields).

Such a "Custom" search interface could also be used for other APIs like Solr or ElasticSearch servers or direct access to databases. In addition, it should probably also be the option where the user can add the custom URL parameters described in #5257.

One of the motivations behind replacing the opac XML configurations with the new ImportConfiguration class was to reduce the risk of faulty configurations by having a tighter control over which type of configuration can contain which type of setting. I would prefer not to weaken this newly gained "stability" by allowing parameters for interface types that normally do not support them.

The proposed "Custom" interface type seems like the best way to allow for more "unconventional" search interfaces.

@BartChris would this approach cover all your requirements?

[BartChris]

This sounds reasonable. The custom importer probably requires a mechnism to customize import behaviour as descibed here as well: https://github.com/kitodo/kitodo-production/issues/5259 Alternatively one would need a way to use the custom import query logic but still reuse the OAI or SRU settings for the processing of the data after the query.

[solth]

This sounds reasonable. The custom importer probably requires a mechnism to customize import behaviour as descibed here as well: #5259

#5259 and #5049 will be fixed separately from this issue by allowing the user to customize the record ID XPath of imported MODS documents in import configurations of type "Catalog import".

Do you have an example for a publicly available search interface where configuration of custom URL parameters is necessary, which I can use for testing?

[BartChris]

@solth You could try http://search.ub.uni-koeln.de/portal/usbportal/connector/unapi?id=inst001:805615&format=oai_mods

[solth]

@solth You could try http://search.ub.uni-koeln.de/portal/usbportal/connector/unapi?id=inst001:805615&format=oai_mods

Thanks for the link. This does not seem to be valid OAI syntax, though, because in OAI the id parameter is always called identifier and afaik that is not configurable (see http://www.openarchives.org/OAI/openarchivesprotocol.html#GetRecord) Do you know if it possible to also allow identifier as an alias parameter for id in that interface in order to use a standard OAI query syntax?

[BartChris]

Thank you very much for looking into that. Yes, i know this is not valid OAI interface structure, it is more of a fake OAI interface to allow for easy import to Kitodo. It is designed this way to avoid having to add a lot of additional processing logic. It might be possible to add the alias and it is also possible (but not very convinient) to add an proxy interface which forwards the request (this is what i do right now). Right now the interface depends on custom paramters (format=) and query filters (id=), so it would allow for testing the general case of a custom query interface if you are willing to go down this road.

[solth]

This issue was fixed by #5313

@BartChris if I understand correctly, implementing #5257 for custom type import configurations would solve the issue you reported and enable you again to query the "fake" OAI interface you mentioned above, right?

As previously explained, we want to stick to our goal to minimize the risk of creating faulty configurations. Allowing for arbitrary url search fields for OAI interfaces while in reality OAI interfaces do not support such additional search fields and URL parameters would not be in line with that goal.