danieldotnl
An empty response as a result of the request
Version of the custom_component
v7.2.0
Configuration
# Трамвай №6
- resource: https://mu-kgt.ru/informing/wap/marsh/?action=getMarshData&m=6%f2%f0
name: Трамвай 6
scan_interval: 120
parser: lxml
log_response: true
sensor:
- unique_id: 2bcbbcc0-d33c-47a1-9848-d251fa54f27c
name: Трамвай 6 (ост. Металлург)
icon: mdi:clock
value_template: >-
{%- if 'ts_line' in value_json -%}
{%- set stops = value_json.ts_line.A -%}
{%- for stop in stops -%}
{%- if stop.st_regnum == 1989 -%} {{ stop.st_arrive }} {%- endif -%}
{%- endfor -%}
{%- endif -%}
on_error:
log: warning
value: none
Describe the bug
As a result of the request, I get an empty response (error code 200). At the same time, if I make a request to this address without the last two characters "%f2%f0" ("тр" in Russian in Win-1251 encoding), then the correct data is received in response. Previously, I used an almost identical configuration in rest requests and everything worked as it should. Therefore, I assume that these two characters are the reason for the empty response. Unfortunately, to refuse them means to receive incorrect data. This query also returns data if you use it in the browser or in the curl command. I have tried many ways to write this query, but I did not get the answer I needed.
Debug log
2024-09-10 11:05:18.600 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Трамвай 6 # New run: start (re)loading data from resource
2024-09-10 11:05:18.600 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Трамвай 6 # Deleting logging files from previous run
2024-09-10 11:05:18.602 DEBUG (MainThread) [custom_components.multiscrape.http] Трамвай 6 # Executing page-request with a GET to url: https://mu-kgt.ru/informing/wap/marsh/?action=getMarshData&m=6%f2%f0 with headers: {} and cookies: None.
2024-09-10 11:05:18.603 DEBUG (MainThread) [custom_components.multiscrape.http] Трамвай 6 # request_headers written to file: page_request_headers.txt
2024-09-10 11:05:18.670 DEBUG (MainThread) [custom_components.multiscrape.http] Трамвай 6 # Response status code received: 200
2024-09-10 11:05:18.677 DEBUG (MainThread) [custom_components.multiscrape.http] Трамвай 6 # response_headers written to file: page_response_headers.txt
2024-09-10 11:05:18.680 DEBUG (MainThread) [custom_components.multiscrape.http] Трамвай 6 # response_cookies written to file: page_response_cookies.txt
2024-09-10 11:05:18.684 DEBUG (MainThread) [custom_components.multiscrape.http] Трамвай 6 # response_body written to file: page_response_body.txt
2024-09-10 11:05:18.684 DEBUG (MainThread) [custom_components.multiscrape.scraper] Трамвай 6 # Loading the content in BeautifulSoup.
2024-09-10 11:05:18.687 DEBUG (MainThread) [custom_components.multiscrape.scraper] Трамвай 6 # page_soup written to file: page_soup.txt
2024-09-10 11:05:18.687 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Трамвай 6 # Data successfully refreshed. Sensors will now start scraping to update.
2024-09-10 11:05:18.687 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 0.087 seconds (success: True)
2024-09-10 11:05:18.687 DEBUG (MainThread) [custom_components.multiscrape.sensor] Трамвай 6 # Трамвай 6 (Test) # Start scraping to update sensor
2024-09-10 11:05:18.687 DEBUG (MainThread) [custom_components.multiscrape.scraper] Трамвай 6 # Трамвай 6 (Test) # Applying value_template only.
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.sensor] Трамвай 6 # Трамвай 6 (Test) # Selected: []
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.entity] Трамвай 6 # Трамвай 6 (Test) # Sensor updated and state written to HA
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.sensor] Трамвай 6 # Трамвай 6 (пос. Энергетиков) # Start scraping to update sensor
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.scraper] Трамвай 6 # Трамвай 6 (пос. Энергетиков) # Applying value_template only.
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.sensor] Трамвай 6 # Трамвай 6 (пос. Энергетиков) # Selected:
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.entity] Трамвай 6 # Трамвай 6 (пос. Энергетиков) # Icon template rendered and set to: mdi:clock
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.entity] Трамвай 6 # Трамвай 6 (пос. Энергетиков) # Sensor updated and state written to HA
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.sensor] Трамвай 6 # Трамвай 6 (ост. Металлург) # Start scraping to update sensor
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.scraper] Трамвай 6 # Трамвай 6 (ост. Металлург) # Applying value_template only.
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.sensor] Трамвай 6 # Трамвай 6 (ост. Металлург) # Selected:
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.entity] Трамвай 6 # Трамвай 6 (ост. Металлург) # Icon template rendered and set to: mdi:clock
2024-09-10 11:05:18.688 DEBUG (MainThread) [custom_components.multiscrape.entity] Трамвай 6 # Трамвай 6 (ост. Металлург) # Sensor updated and state written to HA
page_soup.txt
<html>
<body>
<p>
[]
</p>
</body>
</html>
Isn't the problem that the response contains a header: Content-Type: application/x-javascript; charset=utf-8, while it is not utf-8? Also my browser shows it like:
{"ts_type_id":3,"ts_type_title":"\u0422\u0440\u0430\u043c\u0432\u0430\u0439","marsh_title":"6\u0442\u0440","forecast":" ","directions":{"A":{"first":"\u041f\u0440\u0435\u0434\u043c\u043e\u0441\u0442\u043d\u0430\u044f \u043f\u043b\u043e\u0449\u0430\u0434\u044c","last":"\u043f\u043e\u0441. \u042d\u043d\u0435\u0440\u0433\u0435\u0442\u0438\u043a\u043e\u0432","length":11.576194040000001},"B":{"first":"\u043f\u043e\u0441. \u042d\u043d\u0435\u0440\u0433\u0435\u0442\u0438\u043a\u043e\u0432","last":"\u041f\u0440\u0435\u0434\u043c\u043e\u0441\u0442\u043d\u0430\u044f \u043f\u043b\u043e\u0449\u0430\u0434\u044c","length":11.757706799999999}},"ts_endstop":[],"ts_line":{"A":[{"dgt":0,"obj":"stop","st_regnum":1106,"st_title":"\u041f\u0440\u0435\u0434\u043c\u043e\u0441\u0442\u043d\u0430\u044f \u043f\u043b\u043e\u0449\u0430\u0434\u044c","st_arrive":" "},
As far as I understand, no. For example, the following request returns correct data:
- resource: https://mu-kgt.ru/informing/wap/marsh/?action=getMarshData&m=9
name: Автобус 9
scan_interval: 120
parser: lxml
# log_response: true
sensor:
- unique_id: 2a4b318e-a4b2-4216-b9ec-5f7e1d302301
name: Автобус 9 (ост. Сибтяжмаш)
icon: mdi:clock
value_template: >-
{%- if 'ts_line' in value_json -%}
{%- set stops = value_json.ts_line.A -%}
{%- for stop in stops -%}
{%- if stop.st_regnum == 333 -%} {{ stop.st_arrive }} {%- endif -%}
{%- endfor -%}
{%- endif -%}
on_error:
log: warning
value: none
For the sake of interest, I tried another request:
- resource: https://mu-kgt.ru/informing/wap/marsh/?action=getMarshData&m=5%f2
name: Троллейбус 5 (тест)
scan_interval: 120
parser: lxml
log_response: true
sensor:
- unique_id: 9791c6fb-f047-4d8a-8bcf-a11242d4888f
name: Троллейбус 5 (Test)
value_template: >-
{%- if 'ts_line' in value_json -%}
{%- set stops = value_json.ts_line.A -%}
{%- for stop in stops -%}
{%- if stop.st_regnum == 550 -%} {{ stop.st_arrive }} {%- endif -%}
{%- endfor -%}
{%- endif -%}
on_error:
log: warning
value: none
And I get an empty response. The difference in requests is apparently only in these two symbols in the link (denoting different types of transport)