Indexing with dask distributed client does not work

[aidanheerdegen]

Indexing serially works fine, but with a dask.distributed client is supported with build_index, but doesn't seem to work

<Client: 'tcp://127.0.0.1:40089' processes=8 threads=48, memory=268.73 GB>                                           Indexing: /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_weddell_up1
Indexing experiment: 01deg_jra55v13_ryf9091_weddell_up1                                                             
Traceback (most recent call last):                                                                                     File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/engine/base.p
y", line 1276, in _execute_context
    self.dialect.do_execute(                                                                                           File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/engine/defaul
t.py", line 609, in do_execute   
    cursor.execute(statement, parameters)                                                                            sqlite3.IntegrityError: UNIQUE constraint failed: experiments.experiment, experiments.root_dir

I have tried client = Client(processes=False) as well as with full processes, both give the same error. I guess the DB is objecting to more than one process trying to create the experiment.

If I index serially, then manually delete all the entries in the ncfiles table, so the experiment already exists, it then throws this error:

<Client: 'tcp://127.0.0.1:44017' processes=8 threads=48, memory=268.73 GB>
Indexing: /g/data/ik11/outputs/access-om2-01/01deg_jra55v13_ryf9091_weddell_up1
Indexing experiment: 01deg_jra55v13_ryf9091_weddell_up1
Traceback (most recent call last):
  File "index.py", line 20, in <module>
    cc.database.build_index(dir, session, client=client, update=True, prune=True, delete=True)
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/cosima_cookbook/database.py", line 451, in build_index
    indexed += index_experiment(directory, session, client, update, prune, 
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/cosima_cookbook/database.py", line 425, in index_experiment
    ncvar.variable = CFVariable.as_unique(session,
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 279, in __set__
    self.impl.set(
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 1024, in set
    value = self.fire_replace_event(state, dict_, value, old, initiator)
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 1046, in fire_replace_event
    value = fn(
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 1479, in emit_backref_from_scalar_set_event
    child_impl.append(
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 1245, in append
    value = self.fire_append_event(state, dict_, value, initiator)
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 1183, in fire_append_event
    value = fn(state, value, initiator or self._append_token)
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 51, in append
    sess._save_or_update_state(item_state)
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2042, in _save_or_update_state
    self._save_or_update_impl(st_)
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2389, in _save_or_update_impl
    self._update_impl(state)
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2378, in _update_impl
    self.identity_map.add(state)
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/lib/python3.8/site-packages/sqlalchemy/orm/identity.py", line 149, in add
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Can't attach instance <NCExperiment at 0x1525dec23700>; another instance with key (<class 'cosima_cookbook.database.NCExperiment'>, (1,), None) is already present in this session.