py_crunchbase icon indicating copy to clipboard operation
py_crunchbase copied to clipboard

Issue in after_id being automatically called

Open yash85763 opened this issue 1 year ago • 2 comments

Hi @FahadNoor I was trying to get the jobs data of an organization, and in the params dictionary it is automatically sending after_id, even when the limit is 2. Everytime limit 2 and updated after_id is being sent to crunchbase, this leads to continuous data pull. I tried to create custom Pagination and and updated the query builder a little bit to only send limit not after_id along with it. But could you please explain what can be a solution?

right now my custom pagination code is below:

  class CustomPaginated(Paginated):
      def __init__(self, api_client, entity_id, card_id, card_field_ids, limit):
          super().__init__()
          self.api_client = api_client
          self.entity_id = entity_id
          self.card_id = card_id
          self.card_field_ids = card_field_ids
          self.limit = limit
          self.next_id = None
          self.previous_id = None
          self.total_results = 0
  
      def set_next(self, current_list):
          pass
  
      def set_previous(self, current_list):
          pass
  
      def execute(self) -> list:
          if self.total_results >= self.limit:
              return []
  
          params = {
              'limit': self.limit,
              'card_field_ids': ','.join(self.card_field_ids)
          }
          response = self.api_client.request(self.entity_id, self.card_id, params)
          data = response.get('data', [])
          self.total_results += len(data)
  
          if self.total_results >= self.limit:
              return data[:self.limit - self.total_results]
  
          return data
  
      def reset_pagination(self):
          self.next_id = None
          self.previous_id = None
          self.total_results = 0

yash85763 avatar Jun 28 '24 16:06 yash85763