Index-rebuild failure after upgrading from 2.0.4 to 2.2.0

[jilen]

Every time while I am upgrading the web console always remind to run rethinkdb index-rebuild

But the release note just states that it was not necessary to do this, why ?

Anyway I perform the index rebuild but the python driver complains following

ReQL error during 'rename `aaa.bbb` index `idx_ccc`': The server(s) hosting table `weixin.$reql_temp_index$_idx_ccc` are currently unreachable. The secondary index was not renamed.

[mlucy]

@jilen -- Does the Web UI show any problems with your cluster when you go to it, such as unavailable servers? Or is the cluster otherwise healthy aside from the index rebuild failing?

[jilen]

@mlucy The Web UI shows the table replication is backfilling of one shard of that table. But I never change replication config.

[mlucy]

@jilen -- that's interesting. Does the backfilling look like it's going to complete, or is it stuck in that state?

[jilen]

@mlucy It stuck in that state and seems won't finish. But that table has 1.4M docs, maybe it just need some time.

[mlucy]

@jilen -- I'd also consider checking on the health of the server it's backfilling to, just to make sure it isn't out of RAM or hard drive space or something. (RethinkDB using up all the RAM and getting pushed into swap can sometimes cause very slow backfilling.)

[jilen]

@mlucy I reduce replicas per shard . And the backfill finished soon. But there are indexes like $reql_temp_index$_xxx should I drop the old index, and rename the temp index as the new index ?

[mlucy]

@jilen -- I think that should work (@danielmewes, @Tryneus, am I right?).

I would rename the old index and not drop it until you're sure the new index is working as intended, though.

[danielmewes]

@jilen Yes, renaming the $reql_temp_index$_... indexes to their proper names (without the $reql_temp_index$ prefix) should work and complete the index migration.

[jilen]

@mlucy @danielmewes After the backfilling finish, it could do rename properly

[danielmewes]

It's very likely that the original index rebuild failure was caused by the table still being backfilling.

I think we should put a table.wait() into the index rebuild script. Probably as a separate step with a message "Waiting for tables to become ready" or something, in case the wait takes a while? @Tryneus any thoughts on that?

[danielmewes]

We added the table.wait() to the import/restore scripts, but not to index-rebuild. I think we should do it in index-rebuild as well.