Empty type known issues [epic]

[vasil-pashov]

A list of known issues, limitation and things to check related to the empty type.

Things to check

• [ ] Check how empty type behaves when compaction is happening. Currently compaction is allowed only for static schema. It creates a new version in which consecutive small segments are compacted into one or more larger segments. The segments must be of exactly the same type (including byte size). Empty typed segments should not be compacted with non-empty typed segments. Several consecutive empty typed segments, however, must be compacted into one "large" empty typed segment. Note: no need for more than one segment as empty typed segments do not store anything in storage.
• [x] #1226
• [ ] How empty type behaves with arrays and nullable booleans. (New types are not released)

Limitations

• [ ] Pandas 1 doesn't support sparse types which means that some types which don't have "null" values cannot be properly represented by the empty type. One such type is int. Currently it is replaced with 0. A way to improve the user experience is to allow the user to pass custom value. Note: the default value is not stored as the expansion of empty type happens during a read.

Known issues

• [ ] #1056
• [x] When the initial dataframe contains no rows at all the types of each column (including the index) should be subject to change. The first append containing actual data should set the types in stone. There are two separate cases for regular columns and for the index column
  • [x] #1377
  • [x] #1428
  • Add special cases in code for appending to a symbol with 0 rows
  • Add empty index type similar to the empty type
  • Use the empty type for the index column
• [ ] #1562

Summary of running test_missing_empty.py against #1227

Total test cases 660 428 passed 232 failed

Failures due to feature flagging nullable booleans (12)

• [ ] Needs to be run again after nullable booleans are enabled

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v1-no_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v1-int_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v1-ts_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v2-no_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v2-int_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v2-ts_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V1-no_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V1-int_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V2-no_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V2-int_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\bool_single_none] - Failed: Write error: E_UNIMPLEMENTED_INPUT_TYPE Nullable booleans are not supported at the moment ```

Failures due to not having proper sparseness in Pandas (68)

• [ ] Due to constraints of Pandas 1 we cannot store None and integer values in the same array. Thus instead of None the array holds 0, which is in line with how dynamic schema operates. It is possible to add read-time override so that instead of 0 the empty-typed values are substituted by an user defined value. (30)

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\int_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\int_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-int_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-int_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-int_index\int_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\int_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-int_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-int_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-int_index\int_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\int_all_append_none] - Failed: no match: data values differ for column col: [1 2 3 None None None] vs [1 2 3 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\none_all_append_int] - Failed: no match: data values differ for column col: [None None None 11 12 13] vs [ 0 0 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\int_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [0 0 0] ```

• [ ] Due to the same constraint if the array is of type bool the backfill value will be 0. This means that if the column is of type bool and we append empty or update with empty the empty entries will be backfilled with False. Note if the type is nullable boolean it should use None. The behavior must be discussed as it might be missleading. There is an option to check the type descriptor on update/append with empty and in case it's bool promote it to nullable boolean. Note this is also related to how Pandas 1 treats bool dtypes. If the column type is bool and the column contains None, pandas will convert the None to False. Thus ArcticDB will not even see the None. In order to create a column of nullable booleans the dtype must be object and all values in the colum must be either True, False or None. Needs further discussion. (38)

FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v1-ts_index\bool_all_update_none] - Failed: no match: data values differ for column col: [False True False nan] vs [False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\bool_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-int_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-int_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-int_index\bool_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\none_all_update_bool] - Failed: no match: data values differ for column col: [nan False True False] vs [False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\none_all_update_bool] - Failed: no match: data values differ for column col: [nan False True False] vs [False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\bool_all_update_none] - Failed: no match: data values differ for column col: [False True False nan] vs [False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v2-ts_index\bool_all_update_none] - Failed: no match: data values differ for column col: [False True False nan] vs [False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\bool_all_update_none] - Failed: no match: data values differ for column col: [False True False nan] vs [False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v2-ts_index\none_all_update_bool] - Failed: no match: data values differ for column col: [nan False True False] vs [False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v1-ts_index\none_all_update_bool] - Failed: no match: data values differ for column col: [nan False True False] vs [False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\bool_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-int_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-int_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-int_index\bool_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\bool_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\bool_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\bool_all_append_none] - Failed: no match: data values differ for column col: [False True False None None None] vs [False True False False False False] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\none_all_append_bool] - Failed: no match: data values differ for column col: [None None None False True False] vs [False False False False True False]

Failures due to setting the default type of a column with 0 rows to float (12)

When a column with 0 rows is added its type is default assigned to float rather than empty type. The observed result is that the values read back are all NaNs rather than None.

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\no_type_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\no_type_empty_append] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\no_type_empty_append] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\no_type_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-int_index\no_type_empty_append] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-int_index\no_type_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\no_type_empty_append] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\no_type_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-int_index\no_type_empty_append] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-int_index\no_type_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\no_type_empty_append] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\no_type_empty_append_none] - Failed: no match: data values differ for column col: [None None None] vs [nan nan nan] ```

False negatives (48)

Failing tests where code is exerting proper behavior, but the code asserting the behavior is wrong.

• [ ] Reading an integer column but comparing it against float (6)

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v1-ts_index\none_all_update_int] - Failed: no match: data values differ for column col: [nan 11.0 12.0 13.0] vs [ 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v2-ts_index\int_all_update_none] - Failed: no match: data values differ for column col: [1.0 2.0 3.0 nan] vs [1 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v2-ts_index\none_all_update_int] - Failed: no match: data values differ for column col: [nan 11.0 12.0 13.0] vs [ 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\none_all_update_int] - Failed: no match: data values differ for column col: [nan 11.0 12.0 13.0] vs [ 0 11 12 13] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\int_all_update_none] - Failed: no match: data values differ for column col: [1.0 2.0 3.0 nan] vs [1 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\none_all_update_int] - Failed: no match: data values differ for column col: [nan 11.0 12.0 13.0] vs [ 0 11 12 13] ```

• [ ] Comparing dfs with different index types. The dataframe being passed to arctic has index type of datetime, however the test compares it against df with row-range index (24)

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\bool_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\int_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\float_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\str_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\datetime_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-ts_index\none_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\bool_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\int_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\float_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\str_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\datetime_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-ts_index\none_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\bool_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\int_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\float_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\str_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\datetime_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\none_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\bool_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\int_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\float_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\str_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\datetime_all_append_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\none_all_append_empty] - Failed: no match: index types differ vs ```

• [ ] The difference occurs when updating non-empty rows with empty data. The tests assume that the overlapping rows stay unchanged while the code overrides overlapping rows with None. Example: Initial dataframe has index pd.date_range('20231201', '20231203') and values [1, 2, 3]. Update range is pd.date_range('20231202', '20231204') (note that there is one new value added with the update 20231204). Test expected result: [1, 2, 3, None], actual result [1, None, None, None] Both tests properly add the new value. The difference is in the intersecting parts of the index (14)

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v1-ts_index\int_all_update_none] - Failed: no match: data values differ for column col: [1.0 2.0 3.0 nan] vs [1 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v1-ts_index\float_all_update_none] - Failed: no match: data values differ for column col: [1.1 2.1 3.1 nan] vs [1.1 nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v1-ts_index\str_all_update_none] - Failed: no match: data values differ for column col: ['a1' 'a2' 'a3' nan] vs ['a1' None None None] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v2-ts_index\float_all_update_none] - Failed: no match: data values differ for column col: [1.1 2.1 3.1 nan] vs [1.1 nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\float_all_update_none] - Failed: no match: data values differ for column col: [1.1 2.1 3.1 nan] vs [1.1 nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\int_all_update_none] - Failed: no match: data values differ for column col: [1.0 2.0 3.0 nan] vs [1 0 0 0] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\str_all_update_none] - Failed: no match: data values differ for column col: ['a1' 'a2' 'a3' nan] vs ['a1' None None None] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\str_all_update_none] - Failed: no match: data values differ for column col: ['a1' 'a2' 'a3' nan] vs ['a1' None None None] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v2-ts_index\str_all_update_none] - Failed: no match: data values differ for column col: ['a1' 'a2' 'a3' nan] vs ['a1' None None None] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\float_all_update_none] - Failed: no match: data values differ for column col: [1.1 2.1 3.1 nan] vs [1.1 nan nan nan] FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v1-ts_index\datetime_all_update_none] - Failed: no match: data values differ for column col: ['2022-06-01T00:00:00.000000000' '2022-06-02T00:00:00.000000000' FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v2-ts_index\datetime_all_update_none] - Failed: no match: data values differ for column col: ['2022-06-01T00:00:00.000000000' '2022-06-02T00:00:00.000000000' FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\datetime_all_update_none] - Failed: no match: data values differ for column col: ['2022-06-01T00:00:00.000000000' '2022-06-02T00:00:00.000000000' FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\datetime_all_update_none] - Failed: no match: data values differ for column col: ['2022-06-01T00:00:00.000000000' '2022-06-02T00:00:00.000000000' ```

• [ ] Currently the test is failing due to error in the testing case. It is trying to update datetime column with a float column which is wrong (4)

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v1-ts_index\none_all_update_datetime] - Failed: Update error: The columns (names and types) in the argument are not identical to that of the existing version: UPDATE FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb[lmdb_version_store_v2-ts_index\none_all_update_datetime] - Failed: Update error: The columns (names and types) in the argument are not identical to that of the existing version: UPDATE FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V1-ts_index\none_all_update_datetime] - Failed: Update error: E_ASSERTION_FAILURE No valid common type between TD and TD for column col FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_update_lmdb_dynamic_schema[EncodingVersion.V2-ts_index\none_all_update_datetime] - Failed: Update error: E_ASSERTION_FAILURE No valid common type between TD and TD for column col ```

Failures due to index default value being datetime (92)

• [ ] When a dataframe with 0 rows is added the type of the dataframe is set to be datetime. This makes it impossible to append dataframe with row range index to it. In case of 0 row dataframe the index should be set by the first update/append (22)

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\bool_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\int_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\float_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\str_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\datetime_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\bool_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\int_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\float_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\str_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\datetime_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-no_index\no_type_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\bool_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\int_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\float_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\str_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\datetime_empty_append] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\bool_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\int_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\float_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\str_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\datetime_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-no_index\no_type_empty_append_none] - Failed: Append error: E_INCOMPATIBLE_INDEX The argument uses a range-style index which is incompatible with the existing version. Please convert both to use Int64Index if you need this to work. ```

• [ ] The dynamic schema is also not working but the problem is not so obvious from the error message. This happens because the testing code swallows exceptions and sometimes discards some exceptions. This is why we don's see E_INCOMPATIBLE_INDEX exception but a a comparison error. A suggestion is to remove all try catch(all) code and leave arctic crashing (22)

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\bool_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\int_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\float_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\str_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\datetime_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\bool_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\int_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\float_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\str_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\datetime_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V1-no_index\no_type_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\bool_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\int_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\float_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\str_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\datetime_empty_append] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\bool_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\int_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\float_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\str_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\datetime_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb_dynamic_schema[EncodingVersion.V2-no_index\no_type_empty_append_none] - Failed: no match: number of columns differs 1:Index(['col'], dtype='object') vs 2:Index(['col', 'index'], dtype='object') ```

• [ ] When a dataframe with 0 rows is saved the index is set to be datetime. If the dataframe had row-range index then comparing the initial df and the one returned by arctic will fail. (24)

``` FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v1-no_index\bool_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v1-no_index\int_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v1-no_index\float_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v1-no_index\str_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v1-no_index\datetime_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v1-no_index\no_type_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v2-no_index\bool_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v2-no_index\int_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v2-no_index\float_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v2-no_index\str_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v2-no_index\datetime_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb[lmdb_version_store_v2-no_index\no_type_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V1-no_index\bool_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V1-no_index\int_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V1-no_index\float_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V1-no_index\str_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V1-no_index\datetime_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V1-no_index\no_type_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V2-no_index\bool_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V2-no_index\int_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V2-no_index\float_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V2-no_index\str_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V2-no_index\datetime_empty] - Failed: no match: index types differ vs FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_round_trip_lmdb_dynamic_schema[EncodingVersion.V2-no_index\no_type_empty] - Failed: no match: index types differ vs ```

• [ ] When a dataframe with non-zero rows is saved with any kind of index and a dataframes with 0 rows is appended to it but the index of the latter df is specified explicitly not to be the same as the index of the df in arctic the append fails. Appending a DF with 0 rows should never fail as long as it has the required columns. Same goes for column types. By default pandas will assign dtype float to columns with 0 rows. (24)

``` has the required columns. Same goes for column types. By default pandas will assign dtype float to columns with 0 rows. 24 FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\bool_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\int_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\float_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\str_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\datetime_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\no_type_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\bool_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\int_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\float_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\str_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\datetime_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v1-int_index\no_type_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\bool_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\int_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\float_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\str_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\datetime_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\no_type_empty_append] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\bool_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\int_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\float_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\str_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\datetime_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND FAILED tests\unit\arcticdb\version_store\test_missing_empty.py::test_empty_missing_append_lmdb[lmdb_version_store_v2-int_index\no_type_empty_append_none] - Failed: Append error: The columns (names and types) in the argument are not identical to that of the existing version: APPEND ```

[DrNickClarke]

Here are a few points but generally I agree with the above

• on the compaction, I would also like to test missing data. eg. we can write [1,2] then append [None, None]. Can this end up being compacted into a single segment? I believe that should not happen but it would be good to test it. Before we release 4.4.0 so at least we know if we have a problem.
• known issues: please test update as well as append.
• the tests that start with values and update with None should be fixed by the PR. eg ts_index/float_all_update_none.
• it would be nice to have empty dataframes round-trip correctly.

[alexowens90]

I think all this stuff is covered above, but to summarise more concisely, I believe there are 5 underlying issues that should take care of everything:

• Writing a column with zero rows must be stored as empty type (currently float64, Pandas default)
• If a dataframe with no rows is written, must be able to append either a rowcount or timestamp indexed df to this
• If a dataframe with no rows is written, and the index dtype is specified, this should round-trip correctly
• compact_incompletes must be able to set the correct global type of the column in the index segment if 1, several, or all of the segments contain column of empty type
• defragmentation must not compact empty and non-empty columns together

Plus one thing to test:

• https://github.com/man-group/ArcticDB/issues/1226

[jamesmunro]

Should this issue be included in this epic?

• [ ] #1312