Jonathan Gray
Jonathan Gray
This is nearly there, but seems to be timing out in the flowetl integration tests.
Smarter queue would be good really, because you probably don't want to be chucking stuff out of the cache if there's something about to use it. I suppose one could...
Ah this is going to be tricky isn't it, because we rely on `get_query` being blocking. So you can imagine a situation where there are two calls to `get_query` one...
So you need to track everything currently in use, and prevent it from being evicted.
I think #2 isn't actually guaranteed to be a good idea tho, right? The queue might be very long, it might not be sensible to hold on to the cache....
We can actually lock tables ourselves, which is a possible solution. My worry with locking from get_query is getting things stuck if the managing instance goes away for any reason....
I was looking at this again - the window is actually much smaller than I'd initially thought, because `_make_sql` isn't called until https://github.com/Flowminder/FlowKit/blob/master/flowmachine/flowmachine/core/cache.py#L191
My suggestion from that thread is that we do something using `pg_ls_dir` to allow for checking for multiple files.
So seems like the issue then is that we're getting a null from `pg_total_relation_size`, which means that the oid no longer exists, i.e. the actual table no longer exists?
_A_ fix would presumably be to return 0 for the size when that happens, but it slightly begs the question of how one arrived at a situation where there was...