probe-cli
probe-cli copied to clipboard
ooni
Start implementing measurement re-upload
The file format was not friendly to upload files, so we should change the file format to have an individual file for each measurement so that it can be updated easier.
Not really! (I have clarified the scope of the PR with a better title, BTW)
Not really! (I have clarified the scope of the PR with a better title, BTW)
@bassosimone, thanks for replying!
So probe-cli is already able to upload measurements automatically during the scan process?
So probe-cli is already able to upload measurements automatically during the scan process?
Indeed. What is missing here is when you (1) opt-out of upload and then want to upload later and (2) upload fails and you wanna retry upload. The latter use case is important.
Yet, when working on this, I realised our file format was a bit annoying to work with. The issue is more or less that a measurement is modified when you upload it. Hence, one needs to reinsert it again into the measurement file. This means scanning the file again and changing only a specific line. This seems conducive to tech debt and complexity, so we paused a bit.
We either want to store each measurement in a different file, or just trust the fact that now the backend pipeline is blazingly fast and only keep around non-submitted measurements. Such that, if a user wants to see an already submitted measurement (not the common case, but useful when debugging and digging into censorship), they can just fetch from the backend.
Thanks for asking the question. It did not occur to me we can take advantage of the pipeline being faster, so probably the right solution to move forward is to delete submitted measurements, and we don't need to refactor the way in which we store them on disk.