Residual bug in Way_Updater

[drolbr]

The public database currently shows for example on the query [date:"2023-10-25T00:00:00Z"];way(404135463); the error

Way 404135463 cannot be expanded at timestamp 2023-10-25T18:12:34Z.

After some investigation it is clear that there is indeed a broken way entry in the database, but also it looks like currently no other way is affected. There is a delta compressed way version of this way for which the reference sits on a different index. The most likely cause is a rather arcane bug in way_updater that cause the bad data to be written. The precise bug is yet unknown.

[mmd-osm]

Many but not all other instances I'm aware of are also impacted:

Host	Version	Status
//overpass.osmcha.org/api/	0.7.57.1 74a55df1	broken
//maps.mail.ru/osm/tools/overpass/api/	0.7.57.1 74a55df1	broken
//overpass.m....dev/api/	0.7.61.8 b1080abd	broken
//overpass.kumi.systems/api/	0.7.60.6 e2dc3e5b	ok
dev drolbr	0.7.60.6 e2dc3e5b	broken
dev mmd	0.7.59.120 (mmd) 16d06a6e	ok

[drolbr]

It looks like this affects only one version of one way. The following patch program has been applied to the public databases to fix the defect version:

#include "overpass_api/core/datatypes.h"
#include "overpass_api/core/settings.h"
#include "template_db/block_backend_write.h"
#include "template_db/dispatcher_client.h"
#include "template_db/transaction.h"

int main(int argc, char* args[])
{
  Attic< Way_Delta > way(
      Way_Delta(Way_Skeleton(), Way_Skeleton(404135463,
          { 1571656126ull, 4520945912ull, 1571656431ull, 1571656414ull, 1571656506ull, 1571656037ull,
            1571656516ull, 1571656432ull, 1571656248ull, 4520945911ull, 1571656108ull, 1571656126ull
          }, {})),
      Timestamp(2023, 10, 25, 18, 12, 34).timestamp);
  
  std::cout<<Timestamp(way.timestamp).str()<<' '<<way.full;
  for (auto i : way.nds_added)
    std::cout<<'\t'<<i.first<<','<<i.second.val();
  std::cout<<'\n';
  
  std::map< Uint31_Index, std::set< Attic< Way_Delta > > > to_insert;
  to_insert.insert({ Uint31_Index(0xe192676d), { way } });
  
  Dispatcher_Client dispatcher_client(osm_base_settings().shared_name);
  dispatcher_client.migrate_start();

  {
    Nonsynced_Transaction transaction(true, false, dispatcher_client.get_db_dir(), "");
  
    Block_Backend< Uint31_Index, Attic< Way_Delta > >
      into_db(transaction.data_index(attic_settings().WAYS));
    into_db.update(to_insert, to_insert);
  }
  
  dispatcher_client.migrate_commit();

  return 0;
}

It con be compiled with g++ -o patch_way -DHAVE_LZ4 -I./ ~/patch_way.cc overpass_api/core/settings.cc template_db/dispatcher_client.cc template_db/types.cc template_db/zlib_wrapper.cc template_db/lz4_wrapper.cc -lz -llz4 -lrt. Then execute it once while not update process is active.

[mmd-osm]

I believe migrate_start and migrate_commit were added fairly recently only, maybe in 0.7.60. Is there an easy way to also apply the update on older releases?