dbhub.io
dbhub.io copied to clipboard
sqlitebrowser
Any chance of a bigger size limit?
I wonder, is it possible to have a bigger size limit? There are a couple of databases that I would love to upload, but they're bigger than the current limit of 512 MB. Trying to upload, I get an error like:
Database is too large. Maximum database upload size is 512 MB, yours is 23840 MB
If it makes a difference, this is one of the datasets - https://data.gov.uk/dataset/e3939ef8-30c7-4ca8-9c7c-ad9475cc9b2f/anonymised-mot-tests-and-results
(I'm not in any way officially associated with this dataset)
Interesting. Any idea if the data set changes much?
If it doesn't change very often, then we can probably upload that (and subsequent) changes on an adhoc basis to see how well things perform. We moved most of the backend to reasonably sized dedicated servers about a month ago, so a ~20GB file will probably be a decent test to stress things.
That being said, our rest API is still running on low spec vm's so might need moving over as well. :thinking:
Looking at the dates, it looks like once a year... but I can't be absolutely sure if updates don't sneak in more frequently.
There potentially might need to be some cleaning-related improvements though - the data comes as a combination of comma separated and pipe separated files, but looks like pipes or commas in some of the rows are not quoted/escaped, so I've had to guess as to the "right" thing to do, and my guess could be wrong...
@michalc Is this still of interest to you?
We recently added the ability to white list specific users so they can bypass this upload limit. Can add yourself to this list if you're interested. :smile:
Closing this, as @michalc seems to have moved on to other things. It's been a while after all... :wink: