Rate Limit Countdown for the API Module

[johnbumgarner]

Created a static method to provide a visual countdown in minutes remaining when a rate limit threshold has been reached.

UPDATED this request on 04.29.2023 with new code.

[MerleLiuKun]

Cloud you change the print to logger?

[johnbumgarner]

I can change the print statement to console log messages.

I noted that you aren't really using logging in your package. I would have to add this to api.py

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s:%(name)s:%(levelname)s: %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')

The output would look like this:

2023-05-05 09:09:16:pytwitter.api:INFO: Twitter Rate Limit threshold encountered.
2023-05-05 09:09:16:pytwitter.api:INFO: API connection sleeping for 15 minutes
2023-05-05 09:09:24:pytwitter.api:INFO: Remaining time until rate limit threshold resets is: 15 minutes

Is this what you want?

P.S. I updated the code linked to my pull request to send the information to the console using logging.

[MerleLiuKun]

logging.basicConfig(level=logging.INFO, format='%(asctime)s:%(name)s:%(levelname)s: %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

this library should not change the logging level or format, This need removed.

[johnbumgarner]

This pull request has gotten off track with trying to use logging as you requested.

According to Python 3.x official documentation for logging the best way to "Display console output for ordinary usage of a command line script or program" is using print()

These print statements would provide the user clear feedback about the rate limit threshold status and the time to reset this threshold. This is what the user would see:

Rate limit threshold encountered. Connection sleeping for 15.1333 minutes Remaining time until rate limit threshold resets is: 15 minutes Remaining time until rate limit threshold resets is: 14 minutes Remaining time until rate limit threshold resets is: 13 minutes

If we use logging then the user would get useless data, such the asctime, name of the module and the log type. This is what the user would see:

2023-05-07 10:11:09:pytwitter.api:INFO: Rate limit threshold encountered. 2023-05-07 10:11:09:pytwitter.api:INFO: Connection sleeping for 15.1333 minutes 2023-05-07 10:11:17:pytwitter.api:INFO: Remaining time until rate limit threshold resets is: 15 minutes 2023-05-07 10:12:17:pytwitter.api:INFO: Remaining time until rate limit threshold resets is: 14 minutes 2023-05-07 10:13:17:pytwitter.api:INFO: Remaining time until rate limit threshold resets is: 13 minutes

To accomplish the latter the logging code within pytwitter needs to be modified.

[MerleLiuKun]

Yes, you are right.

But the users need to control whether to display logs outside the library.

If you use print, this will always show the log.

[johnbumgarner]

Can you please provide me an example on how users are controlling logging while using pytwitter?

FYI The reason that I created this pull request was to get some visual feedback when pytwitter started sleeping. I was sitting at my desk with a countdown application running to tell me when the app would restart.

I could potentially add the switch 'notify_on_rate_limitto the API module, which could be set toFalse, which would suppress the print statements. If the user set this variable to True` then the statements would be shown.

sleep_on_rate_limit: bool = False, notify_on_rate_limit: bool = False,

:param sleep_on_rate_limit: If token reach the limit, will sleep. :param notify_on_rate_limit: Provides visual feedback on remaining rate limit sleep period.

I'm updated the code linked to my pull request to use the param notify_on_rate_limit. I have also tested the code and it works.

[MerleLiuKun]

This parameter notify_on_rate_limit is not necessary.

the logger would be controled by the logging level.

For example

....
logger.debug('Twitter rate limit threshold encountered.')
sleeping_period = int("%02d" % (seconds)) / 60
logger.debug(f"Connection sleeping for {'%g' % sleeping_period} minutes")
....

In another file example.py

import logging
from pytwitter import Api

api = Api(xxxx)

# If you want show the log, use `setLevel` to print the `debug` and above level  log.
logging.getLogger().setLevel(logging.DEBUG)


# following your code.

Then，the log will displayed.

DEBUG:pytwitter.api:Twitter rate limit threshold encountered.
DEBUG:pytwitter.api:Connection sleeping for 1 minutes

[johnbumgarner]

Personally, I would except a Python Package, such as yours to have logging enabled by default and written out to a YAML file. Also how would a user know what items that you are logging without looking at the code? And you are currently only logging twice in the entire code base. I know this because I looked at the code.

# request function in api.py
logger.debug( f"Rate limited requesting [{url}], sleeping for [{s_time}]")

The parameter notify_on_rate_limit is necessary, because it provides the user with a simple and reliable way to get feedback when a rate limit is encountered.

Api(bearer_token=token, sleep_on_rate_limit=True, notify_on_rate_limit=True)

Your logging way requires the user TO KNOW that a notification exists in the code and that they have to wrap extra code around the API to see this notification.