Traceback when importing a very long source string

[friedelwolff]

When importing a very long source string (the one causing this was over 18 000 characters), I get the following traceback from the import script (caused by PostgreSQL):

Traceback (most recent call last): File "bin/amagama-manage", line 45, in manager.run() File "env/lib/python2.7/site-packages/flask_script.py", line 731, in run result = self.handle(sys.argv[0], command, sys.argv[2:]) File "env/lib/python2.7/site-packages/flask_script.py", line 704, in handle return command.handle(app, _positional_args, *_command_namespace.dict) File "env/lib/python2.7/site-packages/flask_script.py", line 193, in handle return self.run(_args, *_kwargs) File "amagama/amagama/commands.py", line 123, in run self.real_run(slang, tlang, filename) File "amagama/amagama/commands.py", line 138, in real_run self.handlefile(filename) File "amagama/amagama/commands.py", line 161, in handlefile current_app.tmdb.add_store(store, source_lang, target_lang, commit=True) File "amagama/amagama/tmdb.py", line 302, in add_store return self.add_list(units, source_lang, target_lang, commit) File "amagama/amagama/tmdb.py", line 321, in add_list self.get_all_sids(units, source_lang) File "amagama/amagama/tmdb.py", line 283, in get_all_sids cursor.executemany(insert_query, params) psycopg2.OperationalError: index row requires 8960 bytes, maximum size is 8191

I guess the issue is the unique index. Maybe we could simply decide not to support such long strings, but a translator hitting that (and this was from a real life file) would definitely appreciate a TM match for it. (The string was the whole GPL and was just copied from source for the file I had, though.)

We will probably benefit from partly going back to using hashes, but only for indexes (not in a column as we did before): http://gjsql.wordpress.com/2009/04/19/how-to-speed-up-index-on-bytea-text-etc/ I'm not sure if that is the right solution, though.