webtrees
webtrees copied to clipboard
fisharebest
Website/Changes Logs Hide Everything From Past 24 Hours
I've noticed sometimes none of my edits get logged. I can't see any pattern to what shows up in the Changes log and what doesn't.
As an experiment, I just deleted the entire log, added a Census event, performed some raw GEDCOM edits, and then went back to look at the Changes log. It's still blank. So I'm definitely not imagining the randomness of all this.
The issue is possibly connected to the Website logs view. When I go in there, it shows no activity for the past 28 hours.
webtrees 2.1.7
So now I suspect webtrees is filtering the output in a very strange way. I just deleted the Website logs from the Admin page. It shows up blank now as expected. BUT....
mysql> SELECT COUNT(*) FROM wt_log;
+----------+
| COUNT(*) |
+----------+
| 125 |
+----------+
1 row in set (0.00 sec)
When I try SELECT * it dumps all of the expected activity from the past 24 hours that DIDN'T get deleted. 🤔
And guess what?
mysql> SELECT COUNT(*) FROM wt_change;
+----------+
| COUNT(*) |
+----------+
| 184 |
+----------+
1 row in set (0.01 sec)
This means the data are still in there, I just can't see them through the web viewer!
This is strange. I cannot reproduce it.
I tried changing time-zones in case that was significant, but going +/-11 hours didn't change anything.
This needs more testing, but my theory is that you're inputting a text date, passing it here...
https://github.com/fisharebest/webtrees/blob/6870fe34a1ea81f1e115ac0d22167ddcb03949fd/app/Services/PendingChangesService.php#L297
... and at some point in the timestamp conversion you're reading the UTC date instead of the text date, causing the filter to look at whichever day it is in the UTC timezone. This could cause the results to be off by up to 24 hours depending on when the query is performed. For me, that would be in the future, which doesn't make a lot of sense here. But it has to be some kind of conversion issue.
Going to wait until midnight UTC today and then check this very closely.
https://github.com/fisharebest/webtrees/blob/6870fe34a1ea81f1e115ac0d22167ddcb03949fd/app/Factories/TimestampFactory.php#L65
No timezone specified here. However, when the call to TimestampFactory::make returns a value, it specifies a different timezone. So we're taking a UTC date (::fromString) inside a PHP object and performing a conversion with Carbon::createFromTimestamp(). I'm unfamiliar with the Carbon system and will need to trace it out.
Internally we use UTC. We convert timestamps to local time for display.
The US is UTC minus 5-8 hours, so changes made in the last 5-8 hours of the day will be "tomorrow, UTC".
We query the min/max timestamps for the user interface. So, we fetch "tomorrow" from the database, and convert it to "today" for display.
Now, when we apply these filters to the search query, we need to do the reverse conversion - but this step is missing. Your filter dates are applied as UTC.
That's roughly part of the issue, yes. It doesn't quite explain how the 24 hours of logs are hidden. If it's Sep 7 here and Sep 8 UTC, I shouldn't be missing logs all the way back to 8pm on Sep 6. I will help test it more.
The best way I can describe it is "We need logs through Sep 7 UTC" (wrong) "convert this timestamp to local time => Sep 6" (wrong).
I'm going to wait until midnight UTC to continue testing. That will get the best results.
In the meantime, this is important: When displaying the Changes log and the Website logs, all of the timestamps are displayed according to the timezone set in the current user's account. I suspect this means all instances of toDateString() within the printed "From" and "To" fields are printing the wrong date between midnight UTC and midnight local for negative offsets.
Internally we use UTC. We convert timestamps to local time for display.
I think this is the confusion. With an invalid $format value, a Y-m-d string can have 2 different meanings between midnight UTC and midnight local. We're also passing this value to the SQL condition without converting it from local input to UTC input.
Example:
Timezone: America/Detroit (UTC -5) Local Clock: 2022-12-11 22:22:22 UTC Clock: 2022-12-12 03:22:22
At this point, the database contains log entries for tomorrow, because as you say, UTC is used internally. But the front end filters were designed to use local time, so we are passing "2022-12-11" into a query. By design, this will hide all log entries between midnight UTC and midnight local, even if the filter says "date < 2022-12-12". But because of the conversion bug, the PendingChangesService is querying the database with a local value "date < 2022-12-11" and hiding over 24 hours of data.
Two different bugs. One of them was semi-intentional.
More brainstorming:
When we say "To 2022-12-11" this is not equivalent to < timestampFactory()->fromString('2022-12-11', 'Y-m-d')->addDays(1)
What we really mean is "2022-12-12 00:00:00 minus offset". In other words, < '2022-12-12 05:00:00'
So rather than addDays(1) we need some proper timezone conversion logic.
Pseudocode, untested:
$timezone = new DateTimeZone( $user->getPreference(UserInterface::PREF_TIME_ZONE, Site::getPreference('TIMEZONE')) );
$todate = date_create_from_format( 'Y-m-d H:i:s', "$to 23:59:59", $timezone );
$query->where('change_time', '<=', Registry::timestampFactory()->make( $todate->getTimestamp() )->toDateTimeString();
Pseudocode was mostly accurate, except that toDateTimeString() is also not what we want; it always outputs local time.
I've re-written most of the TimestampFactory to correct the conversion problems. After I debug the string output, I will wait until midnight UTC to continue testing.
Any way to get some traction on the PR?
Real-world family issues have been taking a lot of my time recently - and will continue to do so for the next few months.
Please be patient.
Still being patient. I would like to see the pull merged unless you think it needs more work. Maybe some style issues?
Giving this another bump. The patch I wrote for the PR has been running great for over a year on my server.
PR from last year refreshed against main branch. No significant changes.