[Sharepoint Online] Sharepoint REST APIs returning XML content are not handled properly

[artem-shelkovnikov]

Bug Description

In certain cases (not yet fully clear in which ones), Sharepoint REST API inside Sharepoint Online connectors return XML objects instead of JSON.

We need to either parse XML (this is a worse option) or force the server to always return JSON (this is a better one).

Googling "sharepoint rest api returns xml instead of json" shows that people also face same issue from time to time, for example:

https://github.com/SharePoint/sp-dev-docs/issues/1892 https://sharepoint.stackexchange.com/questions/263014/sharepoint-online-rest-api-is-returning-xml-instead-of-json https://sharepoint.stackexchange.com/questions/85325/return-json-rather-than-xml-in-a-sharepoint-2013-app-jsom

To Reproduce

Steps to reproduce the behavior:

1. Not fully clear at this moment, unfortunately

Expected behavior

Sharepoint Server REST API always returns JSON responses when accessing content.

[seanstory]

It looks like we're already specifying content-type: application/json when forming the client session: https://github.com/elastic/connectors/blob/01d30bbcbeb1c4151ceb44d46cc008e8df0dc2e6/connectors/sources/sharepoint_online.py#L528-L533

IDK what else we can do.

[artem-shelkovnikov]

Yeah it should work, but Sharepoint REST API is a little unpredictable and what's worse we don't have a way to reproduce the issue yet. Maybe it's the odata header that does something, maybe something else?