Use the filter with Large List

[kirtast]

#Question Hello, I managed to read a large list using the paged attribute from the items in a list as:

 def load_data_from_SharePoint(list_obj,lim):

        list_items = list_obj.items.paged(True).top(lim).get().execute_query()
        fields = list_items[0].properties.keys()

        data_list = list()

        for item in list_items:
            data_list.append(item.properties)
        dict_returned = dict()
        dict_returned['data'] = data_list
        dict_returned['keys'] = fields

        return dict_returned

The others pieces of code needed are:

  def obtain_list_object(list_name,credenciales):
          from office365.sharepoint.client_context import ClientContext
          #Conexión al servidor
          ctx = ClientContext(credenciales['tenant']).with_credentials(credenciales['client_credentials'])
  
          #Obtención del objeto list
          list_obj = ctx.web.lists.get_by_title(list_name)
  
          return list_obj

  def user_credentials(config):
      from office365.runtime.auth.client_credential import ClientCredential

      credenciales = dict()
      #Datos del servidor
      client_id = config['client_id']
      client_secret = config['client_secret']
      tenant = config['url']

      #Credenciales
      client_credentials = ClientCredential(client_id,client_secret)

      credenciales['client_credentials'] = client_credentials
      credenciales['tenant'] = tenant

      return credenciales

Note that i have issues when the list is empty in "fields = list_items[0].properties.keys()" and I usually add a try, exception there.

The question comes here with this code:

def load_data_SharePoint_with_filter(list_obj,lim,id_AMIS):
        dict_returned = dict()
        querystring = "id_AMIS eq "+id_AMIS
        col = "*"
        list_items = list_obj.items.paged(True).top(lim).get().select(col).filter(querystring).execute_query()

        try:
            fields = list_items[0].properties.keys()
        except:
            fields = 'empty'

        data_list = list()
        for item in list_items:
           data_list.append(item.properties)

        dict_returned['data'] = data_list
        dict_returned['keys'] = list(fields)

        return dict_returned

ClientRequestException: ('-2147024860, Microsoft.SharePoint.SPQueryThrottledException', 'La operación que se intentó realizar está prohibida porque supera el umbral de vista de lista.', "500 Server Error: Internal Server Error for url: confidential_url/_api/Web/lists/GetByTitle('Test_AMIS_v2')/items?$select=*&$filter=id_AMIS%20eq%20379038&$top=4500")

This is the typal error when you have a large list and try to fetch the values without the paged(True) in the code above.

So what I try is to obtain a specific row, or specific rows in a 10.000+ rows List in a SharePoint. Someone knows how I could manage this case?

[RobinVds]

I'm trying to figure out the same thing, let me know if you solve it :)

[usilveiraprogen]

I also have this problem, but... On the examples section in the repository you can find an example on how to read large lists using a EventHandler property https://github.com/vgrem/Office365-REST-Python-Client/blob/master/examples/sharepoint/lists/read_large_list.py What I'm finding weird is that the getItems function doesn't properly return the itmes, as in it doesn't return the complete list, so I think you have to code something to get a returned object where you push every x items that the paged event grabs.

[kirtast]

Helo @vgrem , Did you see the issue here? I don't know if I explained correctly my question, thank you.

[enelponce]

Dear @vgrem and all may we get any solution to filter by a field in large list (5000+ items)?

Thank you