Brian Thorne

Dealing with large results

1

When using the entity service to deduplicate a ~4M record file I get 1.3TiB of results spread over ~15k files stored in the object store. Various tasks fail without the...

This would be very easy on kubernetes so this depends on automated k8s tests

Export metrics

1

Thinking something similar to how the redis helm chart allows loading a "sidecar" prometheus exporter of metrics. This optional component would export regular updates into our monitoring solution of choice...

As found by @wilko77 in https://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api#rate-limiting: To prevent abuse, it is standard practice to add some sort of rate limiting to an API. RFC 6585 introduced a HTTP status code...

Ingress resources can specify URL rewriting and paths as well as domains/virtual host names. It would be nice to have multiple versions of the entity service available behind the same...

There is a big push for public helm charts to have proper RBAC setups with locked down privilages. This issue is to investigate and modify the helm charts to include...

Put together a load testing test suite. The tool https://locust.io/ has proven good for this kind of thing.

In Feb 2017 Dongxi Liu from Data61 Marsfield have proposed a method for doing secure division using paillier to calculate the dice coefficient. Meaning the entity matching could be worked...

Need to look at best practise. Probably implement via a shared `BaseTask`.

For long term maintainability it would be good to refactor the database code to use `sqlalchamy`.