Long Vu

Just an update, the intermittent failure is detected very frequently with the `stress-test.ipynb` during our Jenkins nightly runs. 

> @tlvu If you could extract the docker-compose logs with Magpie/Twitcher outputs corresponding to those failing stress-test runs, that would most probably help pinpoint where the problem occurs. @fmigneault Waiting...

In `docker logs twitcher`: `Could not get service with name thredds` ``` 2023-05-09 01:24:11,868 INFO [TWITCHER:57][ThreadPoolExecutor-0_0] Using adapter: '' 2023-05-09 01:24:11,914 INFO [TWITCHER:57][ThreadPoolExecutor-0_0] Using adapter: '' 2023-05-09 01:24:11,930 INFO [TWITCHER|magpie.adapter.magpieowssecurity:245][ThreadPoolExecutor-0_3]...

Matching logs from thredds container, `/usr/local/tomcat/logs/localhost_access_log.2023-05-09.txt` file: ``` 172.19.0.1 - - [09/May/2023:01:24:11 +0000] "GET /twitcher/ows/proxy/thredds/catalog/birdhouse/testdata/catalog.html?dataset=birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc HTTP/1. 1" 200 4541 172.19.0.1 - - [09/May/2023:01:24:11 +0000] "GET /twitcher/ows/proxy/thredds/catalog/birdhouse/testdata/catalog.html?dataset=birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc HTTP/1. 1" 200 4541...

No errors in `docker logs magpie` and `access_log` from proxy container already gone. So @fmigneault that's all I've got, sorry no stack-trace for you. If you need something else, please...

Below all the "could not get service name" type of error in `docker logs twitcher`. There are no stack-trace around those errors: ``` $ docker logs twitcher 2>&1 | grep...

> Do you have Twitcher logger running in DEBUG level? > Default is INFO: https://github.com/bird-house/birdhouse-deploy/blob/master/birdhouse/config/twitcher/default.env#L4 @fmigneault Tried setting `export TWITCHER_LOG_LEVEL=DEBUG` in `env.local` on my dev VM, run a Jenkins on...

DEBUG logging enabled in prod, see commit https://github.com/Ouranosinc/birdhouse-deploy/commit/9d1e37a1a091346a749f84f13a539fabbe7a7fad

``` 2023-05-17 02:21:20,610 DEBUG [TWITCHER:49][ThreadPoolExecutor-0_3] Could not get service with name thredds Traceback (most recent call last): File "/opt/birdhouse/src/twitcher/twitcher/owsregistry.py", line 46, in get_service_by_name service = self.store.fetch_by_name(name=name) File "/opt/local/src/magpie/magpie/adapter/magpieservice.py", line 173,...

> @fmigneault script was working early to mid february. @tlvu Is there a history of deployment changes we can track? @tlogan2000 Yes, the Twitcher/Magpie upgrade was done in this PR...