athena: pagination to listDataDirectory

[StefanLekanic-SYRMIA]

Fixes #24571

Description

@adeebshihadeh, @jnewb1, @gregjhogan, can you please take a look at this PR.

I added pagination to listDataDirectory method in athenad.py file, and created test method in test_athenad.py file.

Is this the implementation that was required by the issue? This returns just one batch (page) at the time, for example there is way it can yield all the batches, or save last file that passed etc. I didn't do it that way because i wanted input from you guys.

Verification

I run pre-commit hooks and python tests run via pytest

[github-actions[bot]]

Thanks for contributing to openpilot! In order for us to review your PR as quickly as possible, check the following:

• Convert your PR to a draft unless it's ready to review
• Read the contributing docs
• Before marking as "ready for review", ensure:
  • the goal is clearly stated in the description
  • all the tests are passing
  • the change is something we merge
  • include a route or your device' dongle ID if relevant

[StefanLekanic-SYRMIA]

Hi @gregjhogan , i saw you commented on PR #32013, can you take a look at this too, please. I need some inputs.

[adeebshihadeh]

@gregjhogan did you have an intended use case for this (smart downloader?) or was it just an idea? if not think we should just close this and the issue

[gregjhogan]

On slow network connections I have seen it be completely unusable in the current state because the request will always time out listing a large number of segments. This can cause devices to never get segments enqueued for smart downloader, but I don't think it is a common or significant issue currently.