Improved import modules

[jkall]

Solved see comment Improve sanity checks, number of columns etc., and give user more relevant error messages.

Solved see comment High priority: Better way to remove null and empty values when importing data -All import functions are trying to remove null and empty values. This method needs revision. See for example the result when importing empty comments to the stratigraphy table - this will lead to the text "NULL" (at least under windows).

Solved see comment Allow different importing formats (specifically w_levels should be allowed to have varying number of columns)

Sanity check for existing loggers during logger data import - Whenever logger data is to be imported, perform a sanity check (by querying table ”loggers”) to verify that there actually is supposed to be a logger at this position, during this period.

Automatically swap international characters to ascii in OBSID during import

Solved see comment Validation of stratigraphy import, stop if stratid incorrect - Current import accepts any erorrs in stratid which is not satisfactory. Here should be a number of validation checks. .

Probably solved see comment Also make sure to validate obsid during import.

[HenrikSpa]

"Improve sanity checks, number of columns etc., and give user more relevant error messages."

• The importer now checks if all "not null" columns have values and remove the rows which doesn't. It also gives the user information about how many rows are finally imported.

"High priority: Better way to remove null and empty values when importing data -All import functions are trying to remove null and empty values. This method needs revision. See for example the result when importing empty comments to the stratigraphy table - this will lead to the text "NULL" (at least under windows)."

• Solved with: case when ("%s"!='' and "%s"!=' ') then CAST("%s" as "%s") else null end

"Allow different importing formats (specifically w_levels should be allowed to have varying number of columns)"

• Solved by the general csv importer.

"Validation of stratigraphy import, stop if stratid incorrect - Current import accepts any erorrs in stratid which is not satisfactory. Here should be a number of validation checks. ."

• Solved in import_data_to_db.midv_data_importer.general_csv_import

"Also make sure to validate obsid during import."

• This is not generally solved. It's solved for the general csv import and other new imports (different imports have different places in the code where it's best to do the check).

[HenrikSpa]

These two remains: Sanity check for existing loggers during logger data import - Whenever logger data is to be imported, perform a sanity check (by querying table ”loggers”) to verify that there actually is supposed to be a logger at this position, during this period.

Automatically swap international characters to ascii in OBSID during import //This one hasn't caused a problem for a while. returnunicode is used a lot so that seems to have solved many issues.