arche
arche copied to clipboard
scrapinghub
Analyze scraped data
Fixes #144 I created this spider to improve the examples at documentation. Not sure if we have to merge or just keep it here in an open PR for future...
Current DQR https://arche.readthedocs.io/en/latest/nbs/DQR.html is: * scores based on schema validation and some rules\stats https://github.com/scrapinghub/arche/blob/master/src/arche/quality_estimation_algorithm.py * table of job stats * some rules summary * coverage graph (same as in the...
Refactor compare_was_now, compare_prices_for_same_urls, compare_names_for_same_urls, compare_prices_for_same_names compare_was_now: * compares numeric value between two job fields * Outputs >,
Wouldn't it be nice to have samples each time you launch?
As per https://github.com/scrapinghub/arche/pull/167#issuecomment-533210953 ```res = difference(left_df, right_df, key) res.show() >>>55 items hasn't changed 800 items were added 200 items changed res.additional_stats.added >>>key column_x ... column_f ... res.additional_stats.changed key column_x_left column_x_right...
I want to go away from schema, mostly because now schema-based features are all mixed together between Arche, Matt and standard schema. I am thinking about fastai-like parameters https://docs.fast.ai/tabular.data.html#TabularList: `a...
#157 We need to compare values between items found by key. ``` compare(source_df, target_df, fields=["price", "name"], key=["id"], num_threshold = 0.1) >>>Compare: FAILED think about what to display ```
``` ghat = Gatf(source='364692/1/17', schema='schemas/Global Strategies/amazon_product.json', target='364692/1/14', # schema=schema ) ``` The price fields here differ on 15% and yet no error in log. Perhaps we don't really want to...
https://www.kaggle.com/himanshupoddar/zomato-bangalore-restaurants @ejulio suggested that we can use something like dataset stats from kaggle .com. There are lots of data, but perhaps we can really use something particular.
There is a bug, maybe this one - https://github.com/tqdm/tqdm/issues/485 which prevents from using tqdm_notebook in JupyterHub, Lab or Notebook. At the moment the output is blank. The easiest way seemed...