Add ingestion of large DK or SK records

Open spacesnottabs opened this issue 2 years ago • 5 comments

Currently DK and SK source records larger than 1Mb are not being processed and instead stored in a separate S3 folder with a link provided in the output stream.

These need to also be imported.

Aug 28 '23 11:08 spacesnottabs

Blocks #164 QA.

Sep 02 '23 08:09 tiredpixel

@spacesnottabs, @StephenAbbott, what is the testing status of this ticket, please?

Oct 18 '23 12:10 tiredpixel

@spacesnottabs Are you happy that the large DK and SK records have been reinserted now?

Oct 19 '23 06:10 StephenAbbott

Outstanding work to do here...

Large DK Records.pdf

Nov 02 '23 10:11 StephenAbbott

I'm not sure whether this should be in testing, @StephenAbbott, if there's still work to do. But maybe this relates to other tickets (or other tickets will be created)?

Nov 03 '23 14:11 tiredpixel