anonlink-entity-service
anonlink-entity-service copied to clipboard
data61
Privacy Preserving Record Linkage Service
In [/backend/entityservice/serialization.py#L166](https://github.com/n1analytics/entity-service/blob/c1cd83a9ee2a0f73a5c4bc029a2c499eb757b8d7/backend/entityservice/serialization.py#L166) we stream the results to avoid having to load the whole dataset into memory at once. However as in #184, connexion is very strict about json validation. It...
At the moment results are json and must fit in memory to keep connexion happy. We could instead serve static files directly using nginx. Perhaps the [X-Accel](https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/) headers could be...
When using the entity service to deduplicate a ~4M record file I get 1.3TiB of results spread over ~15k files stored in the object store. Various tasks fail without the...
Users experience problems with the current file upload if the internet connection is not that great. Eventually there will be a timeout and all the progress is for naught. Related...
Currently, Anonlink requires a Clkhash schema for a project, even though it doesn’t use it. This is an opportunity to enforce consistent hashing schemas between parties. I propose that we...
#### endpoint `/projects/{project_id}: get` currently, data providers can only access this endpoint if the result type of the project is `permutations`, but not if the type is `mapping` or `similarity_scores`....
Running the entity service locally: ``` $ docker-compose -f tools/docker-compose.yml up ``` and ``` port=`docker ps | grep nginx | sed 's/^.*0.0.0.0:\([0-9]\+\)->.*$/\1/'` if [ "$port" -lt 32000 ]; then echo...
Got the following timeout while investigating #125. ``` $ docker run -i \ -e ENTITY_SERVICE_URL=http://localhost:${port}/api/v1 \ -e ENTITY_SERVICE_PERMUTATION=1 \ -e ENTITY_SERVICE_TEST_SIZE=8000 \ --net=host \ quay.io/n1analytics/entity-app python - < backend/test_service.py 04:27:03.765...
After a recent rewrite, the `test_service.py` file has been decommissioned and is incompatible with the current API; there is no plan to update it. However, triggering issues #125 and #128...
This would be very easy on kubernetes so this depends on automated k8s tests