Adding Subcategories Breaks ES Bootstrap

[sion42x]

trafficstars

Bug Report

Expected behavior/code Categories should be added under parent categories ("kinds") to be available for linking. When a category is added under another category, the ./sandbox/manage.py bootstrap_elasticsearch command should continue to work without error.

Actual Behavior Adding a category under another category results in the below error when bootstrap is run. This will also cause tutor local quickstart to fail.

Traceback (most recent call last):
  File "./sandbox/manage.py", line 14, in <module>
    execute_from_command_line(sys.argv)
  File "/app/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/app/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/app/venv/lib/python3.7/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/app/venv/lib/python3.7/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/app/richie/src/richie/apps/search/management/commands/bootstrap_elasticsearch.py", line 29, in handle
    regenerate_indices(logger)
  File "/app/richie/src/richie/apps/search/index_manager.py", line 80, in regenerate_indices
    list(map(lambda ix: perform_create_index(ix, logger), ES_INDICES)), ES_INDICES
  File "/app/richie/src/richie/apps/search/index_manager.py", line 80, in <lambda>
    list(map(lambda ix: perform_create_index(ix, logger), ES_INDICES)), ES_INDICES
  File "/app/richie/src/richie/apps/search/index_manager.py", line 59, in perform_create_index
    richie_bulk(indexable.get_es_documents(new_index))
  File "/app/richie/src/richie/apps/search/index_manager.py", line 25, in richie_bulk
    stats_only=True,
  File "/app/richie/src/richie/apps/search/elasticsearch.py", line 108, in bulk_compat_7_to_6
    bulk(client, actions, stats_only=stats_only, *args, **kwargs)
  File "/app/venv/lib/python3.7/site-packages/elasticsearch/helpers/actions.py", line 314, in bulk
    for ok, item in streaming_bulk(client, actions, *args, **kwargs):
  File "/app/venv/lib/python3.7/site-packages/elasticsearch/helpers/actions.py", line 244, in streaming_bulk
    **kwargs
  File "/app/venv/lib/python3.7/site-packages/elasticsearch/helpers/actions.py", line 172, in _process_bulk_chunk
    raise BulkIndexError("%i document(s) failed to index." % len(errors), errors)
elasticsearch.helpers.errors.BulkIndexError: ('1 document(s) failed to index.', [{'index': {'_index': 'richie_categories_2021-12-16-13h25m48.969081s', '_type': '_doc', '_id': 'L-000300010001', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Contexts are mandatory in context enabled completion field [complete.en]'}, 'data': {'absolute_url': {'en': '/en/categories/blockchain-topics/introduction-to-blockchain-and-crypto/'}, 'complete': {'en': ['Introduction to Blockchain and Crypto', 'to Blockchain and Crypto', 'Blockchain and Crypto', 'and Crypto', 'Crypto']}, 'description': {}, 'icon': {}, 'is_meta': False, 'kind': None, 'logo': {'en': {'src': '/media/filer_public_thumbnails/filer_public/ef/4f/ef4fb72b-b829-45bc-ae7f-155220764978/launchpresso-h5iazr-wlju-unsplash.jpg__500x500_q85_crop-smart_subsampling-2.jpg', 'srcset': '/media/filer_public_thumbnails/filer_public/ef/4f/ef4fb72b-b829-45bc-ae7f-155220764978/launchpresso-h5iazr-wlju-unsplash.jpg__1000x1000_q85_crop-smart_subsampling-2.jpg 1000w, /media/filer_public_thumbnails/filer_public/ef/4f/ef4fb72b-b829-45bc-ae7f-155220764978/launchpresso-h5iazr-wlju-unsplash.jpg__2000x2000_q85_crop-smart_subsampling-2.jpg 2000w', 'sizes': '100vw'}}, 'nb_children': 0, 'path': '000300010001', 'title': {'en': 'Introduction to Blockchain and Crypto'}, 'title_raw': {'en': 'Introduction to Blockchain and Crypto'}}}}])
Error: Command failed with status 1: docker-compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm richie ./sandbox/manage.py bootstrap_elasticsearch

Steps to Reproduce

1. Create a category
2. From that category, create one under it
3. Run: tutor local run richie ./sandbox/manage.py bootstrap_elasticsearch

Environment

• Richie version: 2.8.2 + 2.9.1 (tried both versions)
• Platform: tutor-richie (lilac)

Possible Solution

Additional context/Screenshots The specifics seem to be in this portion here:

'error': {'type': 'illegal_argument_exception', 'reason': 'Contexts are mandatory in context enabled completion field [complete.en]'}, 'data': {'absolute_url': {'en': '/en/categories/blockchain-topics/introduction-to-blockchain-and-crypto/'}, 'complete': {'en': ['Introduction to Blockchain and Crypto', 'to Blockchain and Crypto', 'Blockchain and Crypto', 'and Crypto', 'Crypto']},

[sampaccoud]

@sion42x This looks like running an incompatible version of Elasticsearch. What version are you running @regisb? I didn't find the information in https://github.com/overhangio/tutor-richie

[sion42x]

@sion42x This looks like running an incompatible version of Elasticsearch. What version are you running @regisb? I didn't find the information in https://github.com/overhangio/tutor-richie

Looks like it's running 7.8.1:

ubuntu@ip-172-31-38-207:~$ docker ps -a | grep elastic
01d263b62a29   elasticsearch:7.8.1                          "/tini -- /usr/local…"   13 days ago    Up 21 hours                 9200/tcp, 9300/tcp                                                                   tutor_local_elasticsearch_1
ubuntu@ip-172-31-38-207:~$

[sampaccoud]

Then it should be compatible.... any idea @mbenadda?

[regisb]

Note that in Maple, we upgraded Elasticsearch to 7.10.1. Is that version also compatible? If yes, do you manage to discern if a change is required in tutor-richie to bootstrap Elasticsearch?

[sampaccoud]

Yes richie is compatible with both ES 6 and 7 versions. No change is required as the version is automatically detected by a compatibility layer: https://github.com/openfun/richie/blob/master/src/richie/apps/search/elasticsearch.py#L17

[sion42x]

I now have a working tutor/tutor-richie config on Maple in production, I'll add a subcategory and see if the issue is resolved.

[sion42x]

Unfortunately the same issue occurred on tutor 13.0.2 and tutor-richie 13.0.0:

richie_1                     |     raise BulkIndexError("%i document(s) failed to index." % len(errors), errors)
richie_1                     | elasticsearch.helpers.errors.BulkIndexError: ('1 document(s) failed to index.', [{'index': {'_index': 'richie_categories_2022-01-03-16h57m34.838532s', '_type': '_doc', '_id': 'L-000300010001', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Contexts are mandatory in context enabled completion field [complete.en]'}, 'data': {'absolute_url': {'en': '/en/categories/blockchain-topics/introduction-and-overview/'}, 'complete': {'en': ['Introduction and Overview', 'and Overview', 'Overview']}, 'description': {}, 'icon': {}, 'is_meta': False, 'kind': None, 'logo': {}, 'nb_children': 0, 'path': '000300010001', 'title': {'en': 'Introduction and Overview'}, 'title_raw': {'en': 'Introduction and Overview'}}}}])

Removing the subcategory resolves the issue.