checklistbank icon indicating copy to clipboard operation
checklistbank copied to clipboard

Published 20 hours ago • verified publisher icon gbif

field taxonID missing in WoRMS dataset

Open bart-v opened this issue 3 years ago • 10 comments

The field taxonID is no longer present in the species/search API result, only for the WoRMS dataset (?) This only happened in the last few months.

i.e. compare https://api.gbif.org/v1/species/search?datasetKey=2d59e5db-57ad-41ff-97d6-11f5fb264527&offset=60000&limit=1 WoRMS, missing taxonID

vs.

https://api.gbif.org/v1/species/search?datasetKey=9ca92552-f23a-41a8-a140-01abaa31c931&offset=60000&limit=1 ITIS includes taxonID

bart-v avatar Sep 20 '21 10:09 bart-v

If you lookup the species directly via the database it still has the taxonID: http://api.gbif.org/v1/species/154667321

This must have happened when we rebuilt the search index while deploying the new backbone last week. Why WoRMS and if there are other datasets involved with the same problem I can't tell right now.

mdoering avatar Sep 20 '21 10:09 mdoering

Yes I know, but we want to get all of the nub keys for all WoRMS entries. Sending 1.2M requests is just not what we want...

bart-v avatar Sep 20 '21 11:09 bart-v

Understood. This is a bug. Meanwhile you can page through the database like this, even though paging is a terrible thing for accessing all records. This is true for both a database and solr/elastic: http://api.gbif.org/v1/species?datasetKey=2d59e5db-57ad-41ff-97d6-11f5fb264527&limit=1000&offset=1000

I could probably easily expose a new service that returns just the mappings very quickly for all records of a dataset. Useful?

mdoering avatar Sep 20 '21 11:09 mdoering

Definitely useful, since currently we need to apply voodoo magic to work around the 100k offset limitation :)

bart-v avatar Sep 20 '21 11:09 bart-v

... the limits are there for good reasons :)

mdoering avatar Sep 20 '21 11:09 mdoering

Any ETA on fixing the taxonID field or creating a mapping service?

bart-v avatar Oct 11 '21 14:10 bart-v

The taxonID field seems back now. Excellent & thanks.

bart-v avatar Oct 28 '21 09:10 bart-v

Problem is back: taxonID missing Reopening

bart-v avatar Jan 03 '22 20:01 bart-v

@mdoering when will this be fixed? Now already 9 months without taxonID

bart-v avatar Sep 14 '22 15:09 bart-v

@marcos-lg @fmendezh maybe you have an idea why the search is missing the taxonID field? It apparently was there before. The field is in NameUsageSearchResult, but not in NameUsageAvro if we use that to update ES?

mdoering avatar Sep 14 '22 16:09 mdoering