the_od_bods icon indicating copy to clipboard operation
the_od_bods copied to clipboard
verified publisher icon OpenDataScotland

NLS scraper has inaccurate number of records calculation

Open JackGilmore opened this issue 2 years ago • 0 comments

Describe the bug The NLS scraper appears to incorrectly calculate the number of records field on opendata.scot based on the file contents description which just gives you the number of files in a zip. It also judges this based on the first file upload on a page and doesn't take into account multiple file uploads.

To Reproduce See fetch_num_recs() method in nls_scraper.py

Expected behavior Number of records should reflect the number of records in a dataset (e.g. number of rows in a CSV)

Screenshots image image

Hardware and software used N/A

Additional context This functionality was patched out in 59dca44 but the function still remains

JackGilmore avatar Nov 10 '23 16:11 JackGilmore