OpenOversight icon indicating copy to clipboard operation
OpenOversight copied to clipboard

CSV Import improvements

Open abandoned-prototype opened this issue 3 years ago • 0 comments

Having used the advanced csv import feature multiple times for various different departments now I have a few things I think we can improve on the tool.

  1. Automatically create job titles that do not exist yet, instead of failing. They can just be created in a random order, or maybe based on how many values we have per job title. By default all should be marked "sworn". An admin or area coordinator can then sort them in the correct order with the web ui (the order also doesn't seem to have any impact expect for the sort order in the selection menu).

  2. Instead of providing unit-id just provide unit names and create all the units that are provided in the data set. Currently we have no way of creating units via commands and relying on unit-ids means generating different csv files for the different environments (like local dev, staging and production).

  3. Add another flag for local development that will truncate the existing tables for the given department

  4. Either relax the condition on the department name column when having --force-create set or add a department name column to the csv download. Also make sure that all other fields of the csv download match fields expected by the csv import command with the goal that downloading the csv files and then importing locally gives you a basically exact copy of the production data for that department

  5. Dive deeper into different fields and decide if they can actually be optional or not. We have a lot of fields in our tables that are allowed to be NULL but if you actually set them NULL some script breaks because it can't handle None. For example

May 04 04:07:33 staging gunicorn[1350]:     {% for type, list in obj.links|groupby('link_type') %}
May 04 04:07:33 staging gunicorn[1350]:   File "[..]/lib/python3.5/site-packages/jinja2/filters.py", line 852, in do_groupby
May 04 04:07:33 staging gunicorn[1350]:     in groupby(sorted(value, key=expr), expr)]
May 04 04:07:33 staging gunicorn[1350]: TypeError: unorderable types: NoneType() < NoneType()

A result of not setting link_type.

abandoned-prototype avatar May 04 '21 04:05 abandoned-prototype