bigmetadata icon indicating copy to clipboard operation
bigmetadata copied to clipboard

[Canada] DA Interpolation for NHS gives error in Topic 006

Open ethervoid opened this issue 6 years ago • 1 comments

This only happens in Topic 006

I've tried to make batches, VACUUM FULL, remove problematic columns and nothing. Maybe the underlying data is corrupted

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/luigi/worker.py", line 203, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python3.5/dist-packages/luigi/worker.py", line 140, in _run_get_new_deps
    task_gen = self.task.run()
  File "/bigmetadata/tasks/base_tasks.py", line 927, in run
    self.populate()
  File "/bigmetadata/tasks/ca/statcan/data.py", line 224, in populate
    self.populate_da_from_cd()
  File "/bigmetadata/tasks/ca/statcan/data.py", line 289, in populate_da_from_cd
    session.execute(update_query)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py", line 1044, in execute
    bind, close_with_result=True).execute(clause, params or {})
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 947, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py", line 262, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1055, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1191, in _execute_context
    context)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1386, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py", line 185, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1184, in _execute_context
    context)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 462, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) row is too big: size 8168, maximum size 8160
 [SQL: '\n                            UPDATE observatory.obs_147224437ad4d1dc738b46c83c72b74c7c287512 da_orig SET (t006c221_f) = (\n                                SELECT (t006c221_f * (ST_Area(da.the_geom)/ST_Area(cd.the_geom))) t006c221_f FROM observatory.obs_96e9d2db7275d99670e8b09401438117d8bbbcf3 da\n                                INNER JOIN observatory.obs_375053d0769d282bd0ce966038a4dbcbda7bb2b6 cd ON (cd.geom_id = left(da.geom_id,4))\n                                INNER JOIN observatory.obs_683bab54d0f41fd84ab5223f114d7ffa9fa13e62 data ON (cd.geom_id = data.geo_code)\n                                WHERE da.geom_id = da_orig.geo_code\n                            )\n                            ']

ethervoid avatar Jul 11 '18 13:07 ethervoid

Related https://www.postgresql.org/message-id/CAFYwGJ0Sphm9m3noXMgYf%3D1iDCn1%3DV2%2B7%2BVCD-nRg%2BC21Ee-gA%40mail.gmail.com

ethervoid avatar Jul 11 '18 19:07 ethervoid