Import breaks on invalid metadata

[bobobo1618]

Describe the bug When invalid metadata is encountered, an exception is thrown and the entire scanning process crashes.

To Reproduce Steps to reproduce the behavior:

1. Have a photo with "invalid" metadata (in my case, a timezone in one of the date fields)
2. Run Photonix
3. Encounter ValueError: unconverted data remains: +01:00, leading to exited: rescan_photos_periodically (exit status 1; not expected)

Expected behavior Invalid metadata is ignored and scan continues.

Additional context

Logs:

photonix    | 2019-10-19 13:54:00,463 INFO spawned: 'rescan_photos_periodically' with pid 239
photonix    | Traceback (most recent call last):
photonix    |   File "/srv/photonix/manage.py", line 22, in <module>
photonix    |     execute_from_command_line(sys.argv)
photonix    |   File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
photonix    |     utility.execute()
photonix    |   File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
photonix    |     self.fetch_command(subcommand).run_from_argv(self.argv)
photonix    |   File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
photonix    |     self.execute(*args, **cmd_options)
photonix    |   File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
photonix    |     output = self.handle(*args, **options)
photonix    |   File "/srv/photonix/photos/management/commands/rescan_photos_periodically.py", line 27, in handle
photonix    |     self.rescan_photos(options['paths'])
photonix    |   File "/srv/photonix/photos/management/commands/rescan_photos_periodically.py", line 22, in rescan_photos
photonix    |     import_photos_in_place(path)
photonix    |   File "/srv/photonix/photos/utils/organise.py", line 175, in import_photos_in_place
photonix    |     modified = record_photo(filepath)
photonix    |   File "/srv/photonix/photos/utils/db.py", line 24, in record_photo
photonix    |     date_taken = parse_datetime(metadata.get('Date/Time Original'))
photonix    |   File "/srv/photonix/photos/utils/metadata.py", line 27, in parse_datetime
photonix    |     return datetime.strptime(date_str, '%Y:%m:%d %H:%M:%S').replace(tzinfo=utc)
photonix    |   File "/usr/local/lib/python3.6/_strptime.py", line 565, in _strptime_datetime
photonix    |     tt, fraction = _strptime(data_string, format)
photonix    |   File "/usr/local/lib/python3.6/_strptime.py", line 365, in _strptime
photonix    |     data_string[found.end():])
photonix    | ValueError: unconverted data remains: +01:00
photonix    | 2019-10-19 13:55:01,653 INFO exited: rescan_photos_periodically (exit status 1; not expected)
photonix    | 2019-10-19 13:55:02,654 INFO gave up: rescan_photos_periodically entered FATAL state, too many start retries too quickly

I think there should be a catchall somewhere in this stack which should log the error instead of crashing.

[damianmoore]

I'm hoping this is fixed now but we should test again before releasing 1.0

[Hatef-Git]

Not sure if this is related to "invalid" but I have tried importing 68,000 photos from over 20 years of photography and after 7,000 the import stopped. Looking at the logs the last message I see is:

photonix-postgres | 2021-08-10 11:46:03.454 UTC [51] ERROR: value too long for type character varying(128) photonix-postgres | 2021-08-10 11:46:03.454 UTC [51] STATEMENT: INSERT INTO "photos_lens" ("id", "created_at", "updated_at", "library_id", "name", "earliest_photo", " latest_photo") VALUES ('92c0d673-7304-43bf-8de6-9dbf85763732'::uuid, '2021-08-10T11:46:03.453339+00:00'::timestamptz, '2021-08-10T11:46:03.453339+00:00'::timestamptz, '4532d2a4-bab1-4914-b0f9-aefba16dd19b'::uuid, 'Sony FE 35mm F2.8 ZA or Sony FE 24-70mm F2.8 GM or Sony FE 16-35mm F2.8 GM or Samyang AF 35mm F2.8 FE or Tamron 28-75mm F2.8 Di III RXD or Arax MC 35mm F2.8 Tilt+Shift', '2019-12-25T21:08:52+00:00'::timestamptz, '2019-12-25T21:08:52+00:00'::timestamptz)

Does this need a to be a different bug request?

I guess in this case if it can truncate or skip this metadata rather than crashing on it?

[damianmoore]

Thanks for this report @Hatef-Git. Fine to keep this in the same issue. I can see the problem is your lens name is really long (168 characters) and the max length we allow is 128 characters so I'll fix that soon.