dgtlmoon
Eliminate globals usage
This will help running tests because the app isn't initialized automatically by touching the "changedetectionio" package. Moving things out of the __init__.py removes the side-effect of "import changedetection" which means tests can control the state without restarting.
This is the first step in making the tests run with only calling "pytest". The fixture use and test setup need to be adjusted to not depend on test ordering.
Would be awesome if you can add this on another PR and base it on this branch/PR https://github.com/dgtlmoon/changedetection.io/pull/2760
eventlet @ git+https://github.com/eventlet/eventlet@4d48d10b990910cc87ec6bbeeaea63b295fe314d
thanks again - tests are being a bit flakey, i'll keep pushing and see whats up
The fixture use and test setup need to be adjusted to not depend on test ordering.
can you branch from this branch and also do that work on another PR?
The fixture use and test setup need to be adjusted to not depend on test ordering.
can you branch from this branch and also do that work on another PR?
Yes, it will take a while because there are many tests and I don't quite understand the test fixture setup yet.
I actually have another branch in my local repo where I changed the entire codebase to use Quart, but the change is gigantic. Doing these smaller changes first will help make the transition to asyncio more smooth if you want to go that direction.
could this run the basic tests in parallel then too? (faster)
It should be able to using pytest-xdist.
The fixture use and test setup need to be adjusted to not depend on test ordering.
can you branch from this branch and also do that work on another PR?
Yes, it will take a while because there are many tests and I don't quite understand the test fixture setup yet.
I actually have another branch in my local repo where I changed the entire codebase to use Quart, but the change is gigantic. Doing these smaller changes first will help make the transition to asyncio more smooth if you want to go that direction.
amazing :) first question, what is Quart?
'It should be able to using pytest-xdist." that would be neet, also if the run-basic-test.sh was updated
Doing these smaller changes first will help make the transition to asyncio more smooth if you want to go that direction.
problem is the architecture of the app, it's basically just a single threaded app because it has this kind of global dictionary of data "datastore" that cant be accessed from different threads if the flask wrapper launched a new thread per request
which is why i guess mysql/couchdb/etc exist - tho i initially started this app in the fastest easiest way i could without worrying about huge database setups, i really like the flat JSON way
I would really prefer to have a "settings.json" in each "datastore" watch UUID directory rather than in one huge url-watches.json tho, i tried to deal with that earlier
Doing these smaller changes first will help make the transition to asyncio more smooth if you want to go that direction.
problem is the architecture of the app, it's basically just a single threaded app because it has this kind of global dictionary of data "datastore" that cant be accessed from different threads if the flask wrapper launched a new thread per request
which is why i guess mysql/couchdb/etc exist - tho i initially started this app in the fastest easiest way i could without worrying about huge database setups, i really like the flat JSON way
I would really prefer to have a "settings.json" in each "datastore" watch UUID directory rather than in one huge
url-watches.jsontho, i tried to deal with that earlier
Yes, I noticed that JSON files were preferred from looking at the code.
Quart is Flask but using asyncio. So everything is run in a single thread but it yields when it will do something that blocks the processor. In my quart branch, I've removed all the threads and used asyncio for everything.
For instance, when sync_to_json is called, it uses aiofiles to write the JSON database to make sure that the file operation doesn't block the other tasks in the asyncio event loop. Files are actually written in a subprocess, but that detail is hidden from your view.
Quart looks amazing, because i'de also like to add some kind of websocket for better live updates too
heya, whats your thoughts on this PR? as in, how much more work should go into it? what else are you hoping to solve here?
if functions like this in Watch could access g globals for getting default values of the datastore settings, that would be AMAZING, then I can throw away a lot of parsing and special switch code across the whole application
https://github.com/dgtlmoon/changedetection.io/blob/feccb18cdc01d315335ef12d31ebe7c037863c21/changedetectionio/model/Watch.py#L130-L140
is it possible? Because "Watch" object cant "see" the global datastore values in the case that it needs to default to some value if the 'watch' is to default/null etc
Hello, this is a very long PR and it needs to be resolved ASAP or it will get more and more conflicts and probably get closed/lost - i would really love to resolve this