Inomplete populate

[changuelsami]

Hi, I'am a beginner, I'm using ES 6.1.1 When I run populate command the progress bar stops at 75% :

root@ce30fac77888/app# php bin/console fos:elastica:populate
Resetting ndx_etablissement
12763/16797 [=====================>------]  75%
Populating ndx_etablissement/etablissementRefreshing ndx_etablissement
Refreshing ndx_etablissement
root@ce30fac77777:/app# 

This is my fos_elastica.yml :

parameters:
    env(ELASTICSEARCH_URL): ''

fos_elastica:
    clients:
        default:
            url: '%env(string:ELASTICSEARCH_URL)%'
    indexes:
        ndx_etablissement:
            index_name: ndx_etablissement
            finder: ~
            settings:
                index:
                    max_result_window: 1000
            types:
                etablissement:
                    persistence:
                        driver: orm
                        model: App\Entity\Etablissement
                        provider:
                            debug_logging: false
                        listener:
                            defer: true
                    indexable_callback: isEtablissementActif
                    properties:
                        texte:         { type: text,      property_path: elasticTexte, analyzer: french }
                        type:          { type: keyword,   property_path: type }
                        ville:         { type: keyword,   property_path: ville.id }
                        participant:   { type: keyword,   property_path: participant }
                        graine:        { type: keyword,   property_path: elasticGraine }
                        coordonnees:   { type: geo_point, property_path: elasticCoordonees }
                        nombreEleves:  { type: short,     property_path: nombreEleves }
                        formations:    { type: text,      property_path: elasticFormations, analyzer: french }
                        filtres:       { type: text,      property_path: elasticFiltres }
                        region:        { type: keyword,   property_path: region.id }
                        departement:   { type: keyword,   property_path: departement.id }
                        typesInternat: { type: text,      property_path: elasticTypesInternat }
                        ageAdmission:  { type: short,     property_path: ageAdmission }
                        catholique:    { type: keyword,   property_path: catholique }
                        sousContrat:   { type: keyword,   property_path: sousContrat }
                        alternance:    { type: keyword,   property_path: alternance }
                        typeClient:    { type: keyword,   property_path: typeClient }

ES is running inside a docker container, and the app is running on a different one. I can check ES logs with docker logs es_container_id but see no errors. There are also no errors in Symfony's logs !

How can I debug and find what's wrong ?

Regards :)

[dsuurlant]

We're encountering the same problem -- it gets to 75% and drops. Other applications using the same ES instance are not affected by the problem. We can replicate the problem in local development as well, but no problem shows in the logs.

[gisostallenberg]

@changuelsami can you run the command using 'very verbose' to find out what Elastica is doing at 75%? php bin/console fos:elastica:populate -vvv It might help you track down the problem

[dsuurlant]

I've determined that the error in our case has to do with the page count and objects that count as 'unindexable'; at one point, many objects are filtered out, but this doesn't affect the progress bar. So the progress bar gives the illusion of the indexing process going wrong, when in fact less objects are indexable than you might expect (about 25% less).

[changuelsami]

@dsuurlant thanks for the details, how can I check the page count and unindexable objects ?

[changuelsami]

@gisostallenberg thanks for the response, here what I have with -vvv option, I really don't get it :(

...
    "sousContrat" => false,
    "alternance" => false,
    "typeClient" => "internat"
  ]
 12761/16797 [=====================>------]  75%  1 min/2 mins 226.0 MiB
Populating ndx_etablissement/etablissementRefreshing ndx_etablissement
10:05:12 DEBUG     [elastica] Elastica Request
[
  "request" => [
    "path" => "ndx_etablissement/_refresh",
    "method" => "GET",
    "data" => [],
    "query" => [],
    "contentType" => "application/json",
    "connection" => [
      "config" => [
        "url" => "http://172.17.0.1:9200/",
        "headers" => [],
        "curl" => []
      ],
      "http_error_codes" => [
        400,
        403,
        404
      ],
      "ssl" => false,
      "logger" => "fos_elastica.logger",
      "compression" => false,
      "retryOnConflict" => 0,
      "persistent" => true,
      "enabled" => true
    ]
  ],
  "response" => [
    "_shards" => [
      "total" => 10,
      "successful" => 5,
      "failed" => 0
    ]
  ],
  "responseStatus" => 200
]
10:05:12 INFO      [elastica] ndx_etablissement/_refresh (GET) 1.66 ms
[
  []
]
Refreshing ndx_etablissement
10:05:12 DEBUG     [elastica] Elastica Request
[
  "request" => [
    "path" => "ndx_etablissement/_refresh",
    "method" => "GET",
    "data" => [],
    "query" => [],
    "contentType" => "application/json",
    "connection" => [
      "config" => [
        "url" => "http://172.17.0.1:9200/",
        "headers" => [],
        "curl" => []
      ],
      "http_error_codes" => [
        400,
        403,
        404
      ],
      "ssl" => false,
      "logger" => "fos_elastica.logger",
      "compression" => false,
      "retryOnConflict" => 0,
      "persistent" => true,
      "enabled" => true
    ]
  ],
  "response" => [
    "_shards" => [
      "total" => 10,
      "successful" => 5,
      "failed" => 0
    ]
  ],
  "responseStatus" => 200
]
10:05:12 INFO      [elastica] ndx_etablissement/_refresh (GET) 1.19 ms
[
  []
]