datapackage-pipelines icon indicating copy to clipboard operation
datapackage-pipelines copied to clipboard

Expected to see 0 resource(s) but spewed N when trying to load data with flows

Open zelima opened this issue 6 years ago • 1 comments

  • python version 3.6
  • operating system Ubuntu
  • Dpp version 1.7.2

Trying to build simple processor that runs flows but getting following error:

./datasets/finance-vix/flows/finance-vix-flow: FAILURE, processed 0 rows
INFO    :RESULTS:
INFO    :FAILURE: ./datasets/finance-vix/flows/finance-vix-flow 
ERROR log from processor flow:
+--------
| ERROR   :Expected to see 0 resource(s) but spewed 1
| Traceback (most recent call last):
|   File "/home/zelima/.virtualenvs/data-factory/lib/python3.6/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 38, in <module>
|     spew(datastream.dp.descriptor, datastream.res_iter, stats)
|   File "/home/zelima/.virtualenvs/data-factory/lib/python3.6/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 89, in spew
|     assert num_resources == expected_resources
| AssertionError
+--------

my_flow.py

from dataflows import Flow

def flow(parameters, datapackage, resources, stats):
    data = [
      {'data': 'Hello'},
      {'data': 'World'}
    ]

    def lowerData(row):
        row['data'] = row['data'].lower()

    return Flow(data,lowerData)

pipelines-spec.yaml

simple-flow:
  pipeline:
    - flow: my_flow

zelima avatar Oct 17 '18 11:10 zelima

Removing "dpp:streaming"=true from datapackage.json in my case solved this problem.

svetozarstojkovic avatar Jun 18 '19 14:06 svetozarstojkovic