twintproject
[Problem] Getting tweets before "Since" date
Initial Check
If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'twint' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'twint'. Thanks.
Make sure you've checked the following:
- [] Python version is 3.6;
- [] Updated Twint with
pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint; - [] I have searched the issues and there are no duplicates of this issue/question/request.
Command Ran
replies = twint.Config() replies.Pandas = True replies.Store_pandas = True replies.Pandas_clean = True replies.Since = '2021-01-30' replies.Until = '2021-02-03' replies.To = 'snopes' twint.run.Search(replies) df = twint.storage.panda.Tweets_df
Description of Issue
When I set "since" to '2021-01-30' and "until" to '2021-02-03', the tweets I got are from 2021-01-29 (17:00:09 -0700) to 2021-02-01 (16:48:05 -0700).
When I changed "until" to '2021-02-04', the newest tweets I got are on 2021-02-02 (16:56:17 -0700).
Looks like the actual tweets I got are earlier than the "since" and "until" date by 1-2 days.
By carefully examining the output, I found the fetched tweets' GMT time (utc +0, and my time zone is utc -7) is within the "since" and "until" date (but still 1 day earlier than "until" date, even if assuming "until" date is excluded).
Is there any workaround for this issue to make the time zone match? There seems no timezone related parameters in configuration options. Currently I have to convert local time to GMT to get the "since" and "until" date I need to specify.
Environment Details
Jupyter Notebook on Mac OS X
I had similar issues a while back and adding hours-minutes-seconds helped, e.g. c.Since = 2021-01-01 00:00:00.
Perhaps this work. https://github.com/minamotorin/twint/commit/ef047337ad5daa60f98df55b14a68cb8045a928f
