Key Error: Modules

[Miserlou]

Traceback (most recent call last): File "bin/compile", line 82, in pipe = parse_pipe_def(pipe_def, pipe_name) File "/Users/rjones/Projects/GlanceSWF/zappa-pipes/env2/lib/python2.7/site-packages/pipe2py-0.23.7-py2.7.egg/pipe2py/compile.py", line 260, in parse_pipe_def File "/Users/rjones/Projects/GlanceSWF/zappa-pipes/env2/lib/python2.7/site-packages/pipe2py-0.23.7-py2.7.egg/pipe2py/lib/utils.py", line 641, in gen_graph1 KeyError: 'modules'

It works for your test pipelines, but not for the ones I actually need to convert.

[reubano]

Can you please provide the pipeline code you are converting?

[Miserlou]

Oh, of course!

Here are two samples:

{"PIPE":{"description":" ","clones":1,"working":"{\"layout\":[{\"id\":\"sw-55\",\"xy\":[210,86]},{\"id\":\"_OUTPUT\",\"xy\":[341,568]},{\"id\":\"sw-137\",\"xy\":[616,235]}],\"modules\":[{\"type\":\"fetch\",\"id\":\"sw-55\",\"conf\":{\"URL\":{\"value\":\"http:\/\/www.feedbooks.com\/books\/top.atom?category=FBFIC019000&lang=en\",\"type\":\"url\"}}},{\"type\":\"output\",\"id\":\"_OUTPUT\",\"conf\":{}},{\"type\":\"rename\",\"id\":\"sw-137\",\"conf\":{\"RULE\":[{\"field\":{\"value\":\"\",\"type\":\"text\"},\"op\":{\"type\":\"text\",\"value\":\"rename\"},\"newval\":{\"value\":\"\",\"type\":\"text\"}}]}}],\"terminaldata\":[],\"wires\":[{\"id\":\"_w0\",\"src\":{\"id\":\"_OUTPUT\",\"moduleid\":\"sw-55\"},\"tgt\":{\"id\":\"_INPUT\",\"moduleid\":\"sw-137\"}},{\"id\":\"_w1\",\"src\":{\"id\":\"_OUTPUT\",\"moduleid\":\"sw-137\"},\"tgt\":{\"id\":\"_INPUT\",\"moduleid\":\"_OUTPUT\"}}]}","guid":"U6NQLJFDD3DHBIHDXRQDM4SQS4","published":null,"createtime":"2015-06-28 12:10:45","runs":null,"domain":["feedbooks.com","www.feedbooks.com","ycombinator.com","news.ycombinator.com","ihackernews.com","api.ihackernews.com","feedburner.com","feeds.feedburner.com","digg.com"],"status":"A","favorited":null,"engineversion":2,"module":["rename","fetch"],"id":"2e861ac39ecbefbe827d5d6c6cf85c75","modifiedtime":"2015-06-28 12:10:47","name":"BOOKS - Literary","abuse":null,"pubtime":"1970-01-01 12:00:00","modified":1,"modifiedtime_ts":1435450247,"is_favorite":0,"is_owner":0,"tags":{"module":["rename","fetch"],"domain":["feedbooks.com","www.feedbooks.com","ycombinator.com","news.ycombinator.com","ihackernews.com","api.ihackernews.com","feedburner.com","feeds.feedburner.com","digg.com"]},"ui":[],"creatorname":"RichJ","creatorimage_medium":"https:\/\/s.yimg.com\/wv\/images\/943182f1a1d37105e8cdac60cb99cb6e_48.jpg","creatorimage_small":"https:\/\/s.yimg.com\/wv\/images\/943182f1a1d37105e8cdac60cb99cb6e_16.jpg"},"PIPE_ID":"2e861ac39ecbefbe827d5d6c6cf85c75","ACTIONS":{"View Results":"\/pipes\/pipe.info?_id=2e861ac39ecbefbe827d5d6c6cf85c75","View Source":"\/pipes\/pipe.edit?_id=2e861ac39ecbefbe827d5d6c6cf85c75","Clone":"\/pipes\/pipe.clone?_crumb=0&_id=2e861ac39ecbefbe827d5d6c6cf85c75"}}

And

{"PIPE":{"name":"NEWSJSON","domain":["economist.com","www.economist.com","theguardian.com","www.theguardian.com","thebaffler.com","www.thebaffler.com","feedburner.com","feeds.feedburner.com","newyorker.com","www.newyorker.com","bbci.co.uk","feeds.bbci.co.uk","aljazeera.com","america.aljazeera.com","nytimes.com","rss.nytimes.com"],"published":null,"format":["media-image","media"],"description":null,"status":"A","runs":null,"guid":"U6NQLJFDD3DHBIHDXRQDM4SQS4","engineversion":2,"id":"40805955111ac2e85631facfb362f067","modifiedtime":"2014-08-13 08:22:15","createtime":"2014-08-09 12:28:24","favorited":null,"abuse":null,"module":["truncate","tail","sort","fetch"],"working":"{\"layout\":[{\"id\":\"sw-52\",\"xy\":[177,85]},{\"id\":\"_OUTPUT\",\"xy\":[631,581]},{\"id\":\"sw-71\",\"xy\":[496,304]},{\"id\":\"sw-94\",\"xy\":[611,422]}],\"modules\":[{\"type\":\"fetch\",\"id\":\"sw-52\",\"conf\":{\"URL\":[{\"value\":\"http:\/\/rss.nytimes.com\/services\/xml\/rss\/nyt\/InternationalHome.xml\",\"type\":\"url\"},{\"value\":\"http:\/\/feeds.bbci.co.uk\/news\/rss.xml\",\"type\":\"url\"},{\"value\":\"http:\/\/america.aljazeera.com\/content\/ajam\/articles.rss\",\"type\":\"url\"},{\"value\":\"http:\/\/www.theguardian.com\/world\/rss\",\"type\":\"url\"}]}},{\"type\":\"output\",\"id\":\"_OUTPUT\",\"conf\":{}},{\"type\":\"sort\",\"id\":\"sw-71\",\"conf\":{\"KEY\":[{\"field\":{\"value\":\"pubDate\",\"type\":\"text\"},\"dir\":{\"type\":\"text\",\"value\":\"DESC\"}}]}},{\"type\":\"truncate\",\"id\":\"sw-94\",\"conf\":{\"count\":{\"value\":\"45\",\"type\":\"number\"}}}],\"terminaldata\":[],\"wires\":[{\"id\":\"_w0\",\"src\":{\"id\":\"_OUTPUT\",\"moduleid\":\"sw-52\"},\"tgt\":{\"id\":\"_INPUT\",\"moduleid\":\"sw-71\"}},{\"id\":\"_w6\",\"src\":{\"id\":\"_OUTPUT\",\"moduleid\":\"sw-71\"},\"tgt\":{\"id\":\"_INPUT\",\"moduleid\":\"sw-94\"}},{\"id\":\"_w4\",\"src\":{\"id\":\"_OUTPUT\",\"moduleid\":\"sw-94\"},\"tgt\":{\"id\":\"_INPUT\",\"moduleid\":\"_OUTPUT\"}}]}","clones":null,"pubtime":"1970-01-01 12:00:00","modified":1,"modifiedtime_ts":1407961335,"is_favorite":0,"is_owner":0,"tags":{"format":["media-image","media"],"domain":["economist.com","www.economist.com","theguardian.com","www.theguardian.com","thebaffler.com","www.thebaffler.com","feedburner.com","feeds.feedburner.com","newyorker.com","www.newyorker.com","bbci.co.uk","feeds.bbci.co.uk","aljazeera.com","america.aljazeera.com","nytimes.com","rss.nytimes.com"],"module":["truncate","tail","sort","fetch"]},"ui":[],"creatorname":"a contributor","creatorimage_medium":"http:\/\/l.yimg.com\/a\/i\/identity\/nopic_48.gif","creatorimage_small":"http:\/\/l.yimg.com\/a\/i\/identity\/nopic_16.gif"},"PIPE_ID":"40805955111ac2e85631facfb362f067","ACTIONS":{"View Results":"\/pipes\/pipe.info?_id=40805955111ac2e85631facfb362f067","View Source":"\/pipes\/pipe.edit?_id=40805955111ac2e85631facfb362f067","Clone":"\/pipes\/pipe.clone?_crumb=0&_id=40805955111ac2e85631facfb362f067"}}

[reubano]

Sorry for the delay... so if you compare the structure of a test json file, you will see it has the keys: layout, modules, terminal_data, etc. Whereas the structure of your pipe has these fields nested under ['PIPE']['working']. My guess is that you didn't export the correct subsection.