ckanext.scheming.errors.SchemingException: validator/converter not found: u'scheming_multiple_choice_output'

[kimepe]

This might be related in some ways to #199.

I have a CKAN installation, coupled with Wordpress, on top of which is activated the ckanext-scheming plugin. I have a custom scheming json which is used to enrich the resources fields.

While trying to perform a search index rebuild, I get the following errors :

Indexing dataset 1/1113Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 11, in <module>
    sys.exit(run())
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 579, in command
    self.rebuild()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 602, in rebuild
    quiet=self.options.quiet)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 194, in rebuild
    defer_commit
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 102, in update_dict
    self.index_package(pkg_dict, defer_commit)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 122, in index_package
    pkg_dict, schema, 'package_show')
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/plugins.py", line 268, in plugin_validate
    result = plugin.validate(context, data_dict, schema, action)
  File "/usr/lib/ckan/default/src/ckanext-scheming/ckanext/scheming/plugins.py", line 268, in validate
    f, scheming_schema, False)
  File "/usr/lib/ckan/default/src/ckanext-scheming/ckanext/scheming/plugins.py", line 417, in _field_output_validators
    f['output_validators'], f, schema)
  File "/usr/lib/ckan/default/src/ckanext-scheming/ckanext/scheming/validation.py", line 301, in validators_from_string
    v = get_validator_or_converter(p)
  File "/usr/lib/ckan/default/src/ckanext-scheming/ckanext/scheming/validation.py", line 319, in get_validator_or_converter
    raise SchemingException('validator/converter not found: %r' % name)
ckanext.scheming.errors.SchemingException: validator/converter not found: u'scheming_multiple_choice_output'

What am I missing? Where should I look for the validator/converted mentionned above? Is it a specific file which is expected somewhere in the system? wonderful_metadata_scheming.txt #199

[wardi]

The scheming_multiple_choice_output validator is installed when you enable one of the scheming plugins. What plugins are enabled on your site?

[kimepe]

Thanks for your quick answer. here are the plugins currently activated :

• stats
• text_view
• image_view
• recline_view
• datastore
• datapusher
• resource_proxy
• recline_grid_view
• recline_graph_view
• recline_map_view
• webpage_view
• spatial_metadata
• spatial_query
• harvest
• ckan_harvester
• geo_view
• geojson_view
• wmts_view
• geonetwork_harvester
• scheming_datasets
• sitemap

[wardi]

which version of ckan and which version of ckanext-scheming are you using?

[kimepe]

Oh, sorry about that. I should have started with this :

• CKAN : 2.8.1
• Ckanext-scheming: 1.1.0

[wardi]

scheming may just be incompatible with 2.8 right now. We'll need to investigate. @amercader are you aware of any sites using scheming with 2.8?

[amercader]

I'm using scheming on 2.8 sites without issue so far but of course I'm not using all functionality. I think tests run with 2.8 on Travis as well

[shevron]

We are experiencing a similar issue (but with scheming_load_json), with CKAN 2.8.3.

@kimepe I will post more information as we try to debug this, but in the mean time was wondering if you ever got this solved.

[kimepe]

No, unfortunately, I've not yet been able to solve this. To be honest, I haven't tried since.

Le jeu. 26 sept. 2019 09 h 26, Shahar Evron [email protected] a écrit :

We are experiencing a similar issue (but with scheming_load_json), with CKAN 2.8.3.

@kimepe https://github.com/kimepe I will post more information as we try to debug this, but in the mean time was wondering if you ever got this solved.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ckan/ckanext-scheming/issues/200?email_source=notifications&email_token=ACZM5E6T3QYZKVV7NV6UIFDQLSZ7VA5CNFSM4GIEIXSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7VRSLY#issuecomment-535501103, or mute the thread https://github.com/notifications/unsubscribe-auth/ACZM5E4BIQP3YDAG5YRJKCLQLSZ7VANCNFSM4GIEIXSA .

[shevron]

For the sake of record keeping, we have seen this issue manifests (in our case) only in an environment where there were old style (v1) ckan_harvester harvesters saved in the DB, and an upgrade process was needed.

While our quick workaround was to remove these old (unused) harvesters, I do not believe this issue is specific to interaction with ckan_harvester but that ckan_harvester will do something to trigger this issue which is related to ckanext-scheming's IValidators.get_validators() implementation:

• ckanext-harvest will run it's old data upgrade process during the extension's configuration phase (IConfigurable.configure() calls model.setup). If old DB entires are found, new datasets will be created - that happens before other extensions have finished initializing in my cases.
• Creating new datasets will populate the validator cache, which means ckanext-scheming validators are "loaded" for the first time (assuming ckanext-scheming is already loaded)
• However, loading more extensions will cause the validator cache to be wiped
• When the next call to load the validators comes along (during normal operation), the validator cache is re-populated; However, the code in ckanext-scheming's get_validators() implementation which defends against returning the same validators twice will cause it to return an empty dict of validators
• scheming validators are no longer defined - bug is manifested.

Unfortunately, I cannot debug / test / proof this analysis so I cannot offer a tested fix.

That said, I am not sure why the defense against returning the same validators twice is even needed - it seems a bit like a premature optimization to me. Validators are merged as a dictionary, so even if someone calls get_validators twice from two separate scheming plugins, the end result is going to be the same as old validators cache entries will be overwritten with the same ones. However this adds a lot of complexity and potential bugs. I believe the best solution here would be to do away with the defensive code around get_validators.

[kimepe]

Waouh! What a deep analysis. In our case, we also have an old ckan harvester. We'll try out your process.

Thanks once again.

Le dim. 6 oct. 2019 04 h 24, Shahar Evron [email protected] a écrit :

For the sake of record keeping, we have seen this issue manifests (in our case) only in an environment where there were old style (v1) ckan_harvester harvesters saved in the DB, and an upgrade process was needed.

While our quick workaround was to remove these old (unused) harvesters, I do not believe this issue is specific to interaction with ckan_harvester but that ckan_harvester will do something to trigger this issue which is related to ckanext-scheming's IValidators.get_validators() implementation:

• ckanext-harvest will run it's old data upgrade process during the extension's configuration phase (IConfigurable.configure() calls model.setup). If old DB entires are found, new datasets will be created - that happens before other extensions have finished initializing in my cases.
• Creating new datasets will populate the validator cache, which means ckanext-scheming validators are "loaded" for the first time (assuming ckanext-scheming is already loaded)
• However, loading more extensions will cause the validator cache to be wiped
• When the next call to load the validators comes along (during normal operation), the validator cache is re-populated; However, the code in ckanext-scheming's get_validators() implementation which defends against returning the same validators twice will cause it to return an empty dict of validators
• scheming validators are no longer defined - bug is manifested.

Unfortunately, I cannot debug / test / proof this analysis so I cannot offer a tested fix.

That said, I am not sure why the defense against returning the same validators twice is even needed - it seems a bit like a premature optimization to me. Validators are merged as a dictionary, so even if someone calls get_validators twice from two separate scheming plugins, the end result is going to be the same as old validators cache entries will be overwritten with the same ones. However this adds a lot of complexity and potential bugs. I believe the best solution here would be to do away with the defensive code around get_validators.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ckan/ckanext-scheming/issues/200?email_source=notifications&email_token=ACZM5E75AF5GVFQT3TZ5K6DQNGOE3A5CNFSM4GIEIXSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAOEGTQ#issuecomment-538723150, or mute the thread https://github.com/notifications/unsubscribe-auth/ACZM5EYG5LXTUGNR2ZP5PALQNGOE3ANCNFSM4GIEIXSA .

[wardi]

@shevron Thank you for that. The defensive validator check was for supporting ckan versions before 2.8 where we weren't allowed to redefine validators. We can either fix the defensive check or drop support for ckan < 2.8. I'd prefer to fix the check because sites I run aren't on 2.8 yet :-)

[shevron]

Ok, in that case if you still need to defend against multiple extensions returning the same list of validators, I can suggest a couple of different approaches to fixing:

1. Be explicit and simple ("Zen of Python" style): only have one CKAN plugin in the extension that provides validators, call it something like scheming_validators and tell users to load that if they need to use the validators (in addition to loading any other scheming plugin. I believe this is the "best" solution but I can see why this is not desired due to backwards compatibility being broken.

Or

2. Implement IPluginObserver to hook into plugin load events in a way that will allow you to return the full list of validators most of the time but return an empty list if the plugin is a scheming plugin. I'm thinking something like:

    def before_load(self, plugin):
        if plugin._validators is self._validators:  # both plugins share the same _validators object
            plugin._validators = {}
   

This is a bit crude and I have not tested any of this (as I said, can't really test it) - just sharing some thoughts I had.

[Chealer]

Does this affect CKAN versions prior to 2.7? If not, can this not be fixed by simply disabling the defensive check when running with CKAN >= 2.8?