datapusher-plus
datapusher-plus copied to clipboard
dathere
A standalone web service that pushes data into the CKAN Datastore fast & reliably. It pushes real good!
Currently, CKAN's Data Dictionary is limited to data type, label and description. With `qsv stats` we collect descriptive statistics when we infer each column's data type during the Analysis phase...
Could not get the containerfile for version 0.10.1 to run, issue is here: ``` Traceback (most recent call last): File "/usr/lib/ckan/datapusher/venv/bin/datapusher_initdb", line 11, in load_entry_point('datapusher-plus', 'console_scripts', 'datapusher_initdb')() File "/usr/lib/ckan/datapusher/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line...
The browser-based Management Console will: - allow admins to view Datastore entries on a per Org/Instance basis based on their CKAN access rights - be able to see the upload...
Starting CKAN 2.10 its required to add api_token when setting up datapusher/datapusher_plus Add the [`ckan.datapusher.api_token`](https://docs.ckan.org/en/2.10/maintaining/configuration.html#ckan-datapusher-api-token) configuration variable in the README file
With 0.11, we already validate and normalize the CSV during analysis. Will need to add `schema` support, perhaps, by looking for a `schema.json` attribute in the package? If that's set,...
**Describe the bug** If databases connections are lost for some reason, like restarting database server, the connections are lost and datapusher+ produces an error. The logs show the following: ```...
If DP+ is deployed on a resource-constrained host (low memory and low working disk space), add a warning when starting up. Continue monitoring disk space and datastore utilization as well...
Psycopg3 has been stable for ~1.5 years (https://www.psycopg.org/articles/2021/10/13/psycopg-30-released/) and one of its headline features is [async support](https://www.psycopg.org/psycopg3/docs/advanced/async.html#async). With it, perhaps we can do async COPY so we can return quickly...
DP+ is currently being containerized and that will make for an easier deployment. Independent of that effort, DP+ should have a near bulletproof installer script that: * check if python...
Create a DOWNLOAD_ALWAYS_WHITELIST - a list of hosts from which DP+ will always download the entire dataset even if DOWNLOAD_PREVIEW_ONLY is true. DOWNLOAD_ALWAYS_WHITELIST are typically local hosts or hosts where...