datapackage-pipelines
datapackage-pipelines copied to clipboard
Expected to see 0 resource(s) but spewed N when trying to load data with flows
- python version 3.6
- operating system Ubuntu
- Dpp version 1.7.2
Trying to build simple processor that runs flows but getting following error:
./datasets/finance-vix/flows/finance-vix-flow: FAILURE, processed 0 rows
INFO :RESULTS:
INFO :FAILURE: ./datasets/finance-vix/flows/finance-vix-flow
ERROR log from processor flow:
+--------
| ERROR :Expected to see 0 resource(s) but spewed 1
| Traceback (most recent call last):
| File "/home/zelima/.virtualenvs/data-factory/lib/python3.6/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 38, in <module>
| spew(datastream.dp.descriptor, datastream.res_iter, stats)
| File "/home/zelima/.virtualenvs/data-factory/lib/python3.6/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 89, in spew
| assert num_resources == expected_resources
| AssertionError
+--------
my_flow.py
from dataflows import Flow
def flow(parameters, datapackage, resources, stats):
data = [
{'data': 'Hello'},
{'data': 'World'}
]
def lowerData(row):
row['data'] = row['data'].lower()
return Flow(data,lowerData)
pipelines-spec.yaml
simple-flow:
pipeline:
- flow: my_flow
Removing "dpp:streaming"=true from datapackage.json in my case solved this problem.