Ability to send HTTP headers to underlying request

[brunobraga]

Hi, I ran into SharePoint Online list throttling, and as recommended here:

https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online

It suggests sending a specific "User-Agent" to the request in order to:

• be white-listed against the Azure Identity Platform; and
• receive a "Retry-After" response header (it is not available in current code either)

First a question... is it possible to access the request to add the specific header from the high level office365.sharepoint.client_context? If not available, could you work on this? If not, would you accept a pull to your code with a suggest?

Open to any other ideas.

For anyone having throttling issues, the only way I could solve it (without the AIP white-listing for now) is to use a combination of CAML and the existing top/filter methods available in the ListItemCollection. Code snippet here:

page_size = 1000
ctx = ClientContext('site url').with_credentials(UserCredential('USERNAME', 'PASSWORD'))
spol_lists = ctx.web.lists
spol_list = spol_lists.get_by_title('list name')

def __get(filter):
    qry = CamlQuery()
    qry.ViewXml = f"""
        <View Scope='RecursiveAll'>
        <Query></Query>
        <RowLimit Paged='TRUE'>%d</RowLimit>
    </View>
    """ % page_size
    items = spol_list.get_items(qry)
    items.top(page_size).filter(filter).get().execute_query()
    ctx.load(items)
    ctx.execute_query()

# break the queries in pieces to avoid throttling.
result = ListItemCollection(self._context)
for i in [(i, i + page_size) for i in range(0, 100000, page_size)]: # dirty range, just for testing (implement something better)
    filter = "ID ge %d and ID lt %d" % (i[0], i[1])
    r = __get(filter)
    if len(r) > 0:
        for item in r:
            result.add_child(item)
    else:
        # no more records found
        break
    # random sleep
    time.sleep(random.randrange(0, 5))

# do whatever with 'result'

[vgrem]

Greetings,

First a question... is it possible to access the request to add the specific header from the high level office365.sharepoint.client_context?

surely it is supported, here is an example which demonstrates how to add our user agent information (adapted from this example:)

ctx = ClientContext(site_url).with_credentials(ClientCredential(client_id, client_secret))


def _construct_custom_request(request):
    """
    :type request: office365.runtime.http.request_options.RequestOptions
    """
    request.ensure_header("User-Agent", "NONISV|Contoso|GovernanceCheck/1.0")


# Modify/adjust underlying request headers before it gets submitted to the server
ctx.pending_request().beforeExecute += _construct_custom_request

target_web = ctx.web.get().execute_query()

If not, would you accept a pull to your code with a suggest?

would be appreciated any kind of contribution, and might be a good idea to implement ExecuteQueryWithIncrementalRetry as a built-in method.