06chaynes
SQLite manager?
I have an http cache that I've built up using Python requests-cache. It's in an SQLite database, as is the default for that project. I'm considering porting the scraper to Rust, but I don't want to lose the gigabytes of cached responses I've already fetched.
Is there a staightforward way to use this SQLite database with http-cache? It looks like the existing DB uses Python pickle to serialize the data, so… I'm guessing not. But I thought I'd ask just in case anyone has addressed this use case before.
A cache manager implementation that uses SQLite for the backend can be created, and would probably be fairly easy to implement. The cache manager handles serialization/deserialization so you would need to make sure that logic reflects your existing structure properly. I think this would be achievable but there could always be some unknown I have yet to stumble upon.
Also this may be useful for the serialization/deserialization logic https://docs.rs/serde-pickle/latest/serde_pickle/. I haven't tried this myself as I don't use pickle but it looks promising.
I know this isn't exactly what you're looking for but for others looking for an SQLite cache manager, I previously implemented an MVP to trial using SQLite https://github.com/nethunterslabs/http-cache/tree/manager-sqlite.
That's a huge help, thank you! I'll have a look at modifying your code for my purpose.
I haven't done anything for this, unfortunately. I still think it's a great idea, but I don't need it enough for my current project to put in the work.