openaustralia
Warn user if they generate a database with the naming convention of the original scraperwiki library
This is to guard against people accidentally using the original version which writes to an sqlite database with a different name.
Or, alternatively we could just grab both the files data.sqlite and swdata.sqlite and if only one of those two exists use that as our sqlite database and if both exist use data.sqlite. This way we can make everything work with both versions of the scraperwiki library.
This way we can make everything work with both versions of the scraperwiki library.
I think this is a good approach. I initially found it very confusing trying to work out which version of the library I had, and why that was causing a difference between my scraper running on morph.io and running on my machine. I think if we can remove that confusion, in a way that doesn't require people to read docs, that removes a barrier to getting started.
If possible, it might be an option to preconfigure?
The python scraperwiki library doesn't really offer a way to configure the db filename, apart from the SCRAPERWIKI_DATABASE_NAME environment variable. Maybe if the is set as part of the container setup, this'll solve things for python scrapers? (That's what I use for my scrapers...)
It doesn't look like the ruby library looks for this environment var, but it does try looking at its config. Unsure if that helps or not. A plus for the ruby version is that it also lets you set the table name, where the python library gives no such option.
Ah! I wasn't aware that the python library used an environment variable. If only the scraperwiki libraries for the different languages did this consistently and then we wouldn't need our own forks of the libraries with the morph naming conventions.
On Fri, Jun 19, 2015 at 12:34 PM, Chris Nilsson [email protected] wrote:
If possible, it might be an option to preconfigure?
The python scraperwiki library doesn't really offer a way to configure the db filename, apart from the SCRAPERWIKI_DATABASE_NAME https://github.com/scraperwiki/scraperwiki-python/blob/master/scraperwiki/sql.py#L15 environment variable. Maybe if the is set as part of the container setup, this'll solve things for python scrapers? (That's what I use for my scrapers https://github.com/otherchirps/orange_city_council/blob/master/scraper.py#L7 ...)
It doesn't look like the ruby library looks for this environment var, but it does try looking at its config https://github.com/scraperwiki/scraperwiki-ruby/blob/master/lib/scraperwiki.rb#L192. Unsure if that helps or not. A plus for the ruby version is that it also lets you set the table name https://github.com/scraperwiki/scraperwiki-ruby/blob/master/lib/scraperwiki.rb#L197, where the python library gives no such option.
— Reply to this email directly or view it on GitHub https://github.com/openaustralia/morph/issues/799#issuecomment-113346632 .
