Missing really important error data when failing to creating index

[peterbe]

I'm working on a project where I'm upgrading from Elasticsearch 6.2 (and elasticsearch==6.3.1 and elasticsearch-dsl==6.4.0). So I just dig in and admit that I didn't carefully read changelogs. Anyway, I make some small changes and try to create the index:

blog_item_index = BlogItemDoc._index
blog_item_index.delete(ignore=404)
blog_item_index.create()

But it fails like this:

Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/peterbe/dev/PETERBECOM/django-peterbecom/peterbecom/base/basecommand.py", line 82, in execute
    return super(BaseCommand, self).execute(*args, **kwargs)
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/Users/peterbe/dev/PETERBECOM/django-peterbecom/peterbecom/base/basecommand.py", line 50, in handle
    self._handle(**options)
  File "/Users/peterbe/dev/PETERBECOM/django-peterbecom/peterbecom/plog/management/commands/index-blogitems.py", line 26, in _handle
    blog_item_index.create()
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/elasticsearch_dsl/index.py", line 259, in create
    return self._get_connection(using).indices.create(index=self._name, body=self.to_dict(), **kwargs)
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/elasticsearch/client/indices.py", line 102, in create
    return self.transport.perform_request(
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/elasticsearch/transport.py", line 355, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 252, in perform_request
    self._raise_error(response.status, raw_data)
  File "/Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 283, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'failed to build synonyms')

The most important message is on the last line:

elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'failed to build synonyms')

Hmm... Is there something wrong with my list of synonyms? Perhaps something that used to work in there doesn't work any more. So I start debugging my synonyms file and waste a bunch of time.

The problem is with the synonyms. It's something else. To find out what's really wrong, I open /Users/peterbe/.pyenv/versions/3.8.1/envs/django-peterbecom/lib/python3.8/site-packages/elasticsearch/connection/base.py and put in this:

    def _raise_error(self, status_code, raw_data):
        """ Locate appropriate exception and raise it. """
        error_message = raw_data
        additional_info = None
        try:
            if raw_data:
                additional_info = json.loads(raw_data)
                error_message = additional_info.get("error", error_message)
                if isinstance(error_message, dict) and "type" in error_message:
                    error_message = error_message["type"]
        except (ValueError, TypeError) as err:
            logger.warning("Undecodable raw error response from server: %s", err)

+       print(raw_data)
        raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
            status_code, error_message, additional_info
        )

Now I can get the truth:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"failed to build synonyms"}],"type":"illegal_argument_exception","reason":"failed to build synonyms","caused_by":{"type":"parse_exception","reason":"Invalid synonym rule at line 1","caused_by":{"type":"illegal_argument_exception","reason":"The [standard] token filter has been removed."}}},"status":400}

I can figure that out but what's really sad is that the caused_by was not in the RequestError that was raised.

[tmreay]

I would like to second this, but for any sort of error, not just creating the index.

Any time I encounter a search phase execution exception, I always have to drop a debugger or print statement inside the elasticsearch_dsl code to see what the caused_by data is.

[miguelgrinberg]

I'm testing this on 8.13 and index errors seem to have gotten better.

elasticsearch.BadRequestError: BadRequestError(400, 'illegal_argument_exception', 'The [nGram] tokenizer name was deprecated in 7.6. Please use the tokenizer name to [ngram] for indices created in versions 8 or higher instead.')

Closing. If you have a specific instance in which errors are still obscure please provide an up to date example in a new issue.