Nicolas Williams
Nicolas Williams
Ingester PSC stream misses events. This has been discussed at length elsewhere, including in https://github.com/openownership/register/issues/264 . However, I'm creating this new ticket, to make it clearer what the resolutions were.
Since the rewrite of Register Files Combiner (https://github.com/openownership/register/issues/213), it has been possible to run the combination process locally, rather than requiring AWS services directly. However, in order for this to...
- Perform live transformation on all existing S3 files produced by the Kinesis Firehose attached to the ingester - Start the live transformer (which will default to the start of...
Since it should be running continuously, it is important to get notifications of when it has crashed. It ought to be recovering if it is a brief crash, since the...
Instead of a monthly job run on an EC2 instance, this will need to run continuously, so that new records published to the PSC Ingester Stream (Kinesis) are consumed immediately....
It is not ideal to process the same records multiple times, since it may keep replacing statements. When we are consuming from S3, we only transform each file once, and...
The main work recently (and monthly import) have involved running the bulk transformer, which transforms from the S3 files produced from buffering the Kinesis stream. This means the app which...
Transformation is very slow. It takes around 2.03 s/statement (*): ``` I, [2024-04-30T12:57:31.688140 #1] INFO -- : [shardId-000000000002] [49651600875111508319672393859288357183009020135870562338] /company/15694555/persons-with-significant-control/individual/7AYbJehiTTD91IcUhbXdBvumeaY I, [2024-04-30T12:57:33.697593 #1] INFO -- : [shardId-000000000002] [49651600875111508319672393859531351272751560599986503714] /company/12462557/persons-with-significant-control/individual/jjdgflZphXeM0pPxRkQkZs5kbHM I,...
If processing the Kinesis Stream and the stream pointer given is invalid, it should refetch the latest stream pointer per shard. In practice, this should not happen unless the app...
This is similar to Task T3, where a stream pointer will need to be stored and retrieved when processing from the Kinesis Stream. A slight complexity is that stream pointers...