JSONDecodeError after running Waterinfo("vmm")

[frievb]

• pywaterinfo version: 0.7.0
• Python version: 3.9.7
• Operating System: Windows

Description

I was trying to download data via pywaterinfo, but it already went wrong during connecting to the Kiwis-database. A JSONDecodeError came up. Maybe something changed in the output coming from the database?

What I Did

from pywaterinfo import Waterinfo
import logging

logging.basicConfig(level = logging.DEBUG, format = "%(asctime)s %(name)s %(filename)s %(levelname)s: %(message)s")
logger = logging.getLogger(__name__)

provider = Waterinfo("vmm")

The traceback:

2023-07-31 09:37:19,301 requests_cache.backends __init__.py DEBUG: Initializing backend: None pywaterinfo_cache.sqlite
2023-07-31 09:37:19,302 requests_cache.backends.base base.py DEBUG: Initialized SQLiteDict with serializer: SerializerPipeline(name=pickle, n_stages=2)
2023-07-31 09:37:19,327 requests_cache.backends.sqlite sqlite.py DEBUG: Opening connection to C:\Users\USERNAME\AppData\Local\pywaterinfo_cache.sqlite:responses
2023-07-31 09:37:19,424 requests_cache.backends.base base.py DEBUG: Initialized SQLiteDict with serializer: None
2023-07-31 09:37:19,431 requests_cache.backends.sqlite sqlite.py DEBUG: Opening connection to C:\Users\USERNAME\AppData\Local\pywaterinfo_cache.sqlite:redirects
2023-07-31 09:37:19,516 requests_cache.policy.actions actions.py DEBUG: Cache directives from request headers: CacheDirectives()
2023-07-31 09:37:19,520 requests_cache.policy.actions actions.py DEBUG: Pre-read cache checks: Passed
2023-07-31 09:37:19,526 requests_cache.policy.actions actions.py DEBUG: Post-read cache actions: CacheActions(expire_after=datetime.timedelta(days=7), send_request=True)
2023-07-31 09:37:19,549 urllib3.connectionpool connectionpool.py DEBUG: Starting new HTTPS connection (1): download.waterinfo.be:443
2023-07-31 09:37:20,276 urllib3.connectionpool connectionpool.py DEBUG: https://download.waterinfo.be:443 "GET /tsmdownload/KiWIS/KiWIS?request=getRequestInfo&service=kisters&type=QueryServices&format=json&datasource=1&timezone=UTC HTTP/1.1" 200 14123
2023-07-31 09:37:20,282 requests_cache.policy.actions actions.py DEBUG: Cache directives from response headers: CacheDirectives(expires='Mon, 31 Jul 2023 08:20:13 GMT', no_cache=True, last_modified='Mon, 31 Jul 2023 07:20:13 GMT')
2023-07-31 09:37:20,284 requests_cache.policy.actions actions.py DEBUG: Pre-write cache checks: Passed
2023-07-31 09:37:20,332 root waterinfo.py INFO: Successful waterinfo API request with call https://download.waterinfo.be/tsmdownload/KiWIS/KiWIS?request=getRequestInfo&service=kisters&type=QueryServices&format=json&datasource=1&timezone=UTC (call to waterinfo.be with without cache activated).

I also tried following commands. These had the same outcome as above.

provider = Waterinfo("hic")
provider = Waterinfo("vmm", token = "token obtained from VMM")

[johanvdw]

I can confirm the issue, the url https://download.waterinfo.be/tsmdownload/KiWIS/KiWIS?request=getRequestInfo&service=kisters&type=QueryServices&format=json&datasource=1&timezone=UTC returns invalid json

[frievb]

I contacted VMM in the meantime as well. Apparently something went wrong during an update of their webservice, changing the JSON-formatting. They asked Kisters to correct this.

Thus, this will probably solve itself once the error is corrected at their side.

[johanvdw]

at column 84900: "attributes": ["Aggregation Start (MANDATORY): Relative or absolute time- "(0)-(0)-(0)T(0):(0):(0)" / yyyy-MM-dd'T'HH:mm:ss, decadal, yearly, year, monthly, month, daily, day, hourly, hour, hydrology, meteorology", "

the quotes inside this first attribute are not escaped, this is why the parsing fails.

[frievb]

Since it looks like it is going to take a while before Kisters is fixing the mistake. I was looking a bit more closely what was going wrong an realized a workaround is possible, but then I saw this was already dealt with as this issue was mentioned in #63. Thank you for addressing this!

If I understand correctly, I cannot pull the code changes since I do not have write access. Is the easiest way then to clone the package and change the code manually?

[johanvdw]

It should not be needed to make manual changes. If you clone the package you can specify a branch:

git clone https://github.com/fluves/pywaterinfo.git
git branch -a # will show you all the branches
git checkout static-request-info

alternatively, you may be able to use pip to install from this branch directly (requires git to be present)

pip install git+https://github.com/fluves/pywaterinfo@static-request-info-file

[FrederikWynants]

Hello,

Just a quick question. As i'm new to waterinfo and wanted to learn by experimenting with it i did:

from pywaterinfo import Waterinfo vmm = Waterinfo("vmm") df = vmm.get_timeseries_values(282546042, period="P1D")

But i got a JSON error:

JSONDecodeError: Expecting ',' delimiter: line 1 column 84885 (char 84884)

I assume this error relates to the topic you are discussing and i conclude out of it that it is better for me to wait a little bit with experimenting until the issue is resolved. Am I right?

Thank you!

Frederik

[Beramos]

@FrederikWynants, this is indeed related to this issue, which is a bug in the API and not in our codebase. This will definitely be fixed but the duration is unknown. For now you can always work with @johanvdw's work around by installing pywaterinfo from the static-request-info-file-branch, I will update this thread whenever the API is fixed.

alternatively, you may be able to use pip to install from this branch directly (requires git to be present)

pip install git+https://github.com/fluves/pywaterinfo@static-request-info-file

[FrederikWynants]

@Beramos, thank you for your reaction!

[frievb]

It should not be needed to make manual changes. If you clone the package you can specify a branch:

git clone https://github.com/fluves/pywaterinfo.git
git branch -a # will show you all the branches
git checkout static-request-info

alternatively, you may be able to use pip to install from this branch directly (requires git to be present)

pip install git+https://github.com/fluves/pywaterinfo@static-request-info-file

Thanks! A much easier way to use the modification, it worked easily. Maybe useful to mention that the requests package needs to be of version 2.27 or higher for the code to work. (In previous versions requests.JSONDecodeError is not included yet and causes a attribute error)

[binomaiheu]

Possibly a related issue : It would seem that the "ReturnPeriod" returnfield is no longer valid in the HIC getTimeseriesValues request, which is looks like a side effect of using a static request info file. I've prepared a new json file, following @johanvdw 's example for this temporary fix with the "ReturnPeriod" removed from the getTimeseriesValues call. Probably this changed as well during the update.

We encountered this issue for a HIC station with ts_id 63718010

https://hicws.vlaanderen.be/KiWIS/KiWIS?datasource=4&service=kisters&type=queryServices&request=gettimeseriesvalues&ts_id=63718010&period=P10D&returnfields=Timestamp,Value

works

https://hicws.vlaanderen.be/KiWIS/KiWIS?datasource=4&service=kisters&type=queryServices&request=gettimeseriesvalues&ts_id=63718010&period=P10D&returnfields=Timestamp,Value,ReturnPeriod

doesn't and yields :

<ExceptionReport>
<Exception exceptionCode="InvalidParameterValue">
<ExceptionText>Parameter 'returnfields' contains at least one invalid field.</ExceptionText>
</Exception>
</ExceptionReport>

it is of course possible to use the get_timeseries_values call by explicitly mentioning the returnfields kwarg.

[MaartenVL]

Any updates if the API is already fixed?

meanwhile this fix still works, thanks for this! pip install git+https://github.com/fluves/pywaterinfo@static-request-info-file

[johanvdw]

I'm under the impression that the service was fixed. At least the url mentioned in my comment last year is now returning correct json. If someone else can confirm, I'll close this issue.

[stijnvanhoey]

I can confirm, we can close this issue.