Fix: Postgres sync not working

[mavcook]

Fixes #120, where postgres wasn't syncing. Disappointingly/embarrassingly, I don't fundamentally understand what the problem was or why the solution works, so I don't have much to say (except "thanks Claude," and I hope it doesn't cause other issues).

I added in some debug logs that I think are helpful, but I can remove them if desired.

[gamingflexer]

@mavcook Does it work, i need to shift the whole pipeline bcz of no fixes on this issues

Please confirm? thanks

[mavcook]

@mavcook Does it work, i need to shift the whole pipeline bcz of no fixes on this issues

Please confirm? thanks

Yes, at least for a basic setup (small, single postgres database) the syncing issue is fixed and no apparent regressions are introduced. You can try it out yourself with this image from docker hub that includes the fix.

[gamingflexer]

This branch also crashes

[mavcook]

I can't really tell what the error is from the screenshot, but this PR fixes a problem where no logs would appear because syncing was not happening. So it is possible that this PR caused a new error that you're seeing, but it seems unlikely, especially if you were seeing that error with other branches.

I would recommend creating a new Github Issue with clear steps on how to reproduce the problem. That way you can get better help, and this PR can stay focused for the maintainer's review. If you create a Github Issue, I can take a look at it too.

[gamingflexer]

I can't really tell what the error is from the screenshot, but this PR fixes a problem where no logs would appear because syncing was not happening. So it is possible that this PR caused a new error that you're seeing, but it seems unlikely, especially if you were seeing that error with other branches.

I would recommend creating a new Github Issue with clear steps on how to reproduce the problem. That way you can get better help, and this PR can stay focused for the maintainer's review. If you create a Github Issue, I can take a look at it too.

Yup will check, thanks for the update

[Skillkiller]

With this fix, the sync now works for me. Identical configuration does not work with the original long2ice/meilisync version. I just swapped the image to mavcook/meilisync:0.1.4and then the sync worked immediately.

However, I noticed that deletions are not synchronized to Meilisearch. Can you also observe this @mavcook?

Debug logs:

2025-03-07 01:24:56 2025-03-07 00:24:56.605 | DEBUG    | meilisync.main:_:110 - progress={'start_lsn': '0/326CB68'} type=<EventType.create: 'create'> table='clip_entity' data={'twitch_id': 'TentativeNiceFoxCharlieBitMe-Cs6QloCHjJ8jZyYH', 'clip_duration': 23899999616, [...]}
2025-03-07 01:25:54 2025-03-07 00:25:54.310 | DEBUG    | meilisync.source.postgres:__handle_change:132 - Creating event event_type=<EventType.delete: 'delete'> values={'twitch_id': 'YummySplendidRaisinCclamChamp'}
2025-03-07 01:25:54 2025-03-07 00:25:54.311 | DEBUG    | meilisync.source.postgres:__aiter__:174 - Got item from queue item=Event(progress={'start_lsn': '0/3280068'}, type=<EventType.delete: 'delete'>, table='clip_entity', data={'twitch_id': 'YummySplendidRaisinCclamChamp'})
2025-03-07 01:25:54 2025-03-07 00:25:54.311 | DEBUG    | meilisync.main:_:110 - progress={'start_lsn': '0/3280068'} type=<EventType.delete: 'delete'> table='clip_entity' data={'twitch_id': 'YummySplendidRaisinCclamChamp'}

Meillisearch tasks:

{
    "results": [
        {
            "uid": 18621,
            "batchUid": 992,
            "indexUid": "clips",
            "status": "succeeded",
            "type": "documentAdditionOrUpdate",
            "canceledBy": null,
            "details": {
                "receivedDocuments": 1,
                "indexedDocuments": 1
            },
            "error": null,
            "duration": "PT0.493602209S",
            "enqueuedAt": "2025-03-07T00:24:56.610828177Z",
            "startedAt": "2025-03-07T00:24:56.742055209Z",
            "finishedAt": "2025-03-07T00:24:57.235657418Z"
        },
        {
            "uid": 18620,
            "batchUid": 992,
            "indexUid": "clips",
            "status": "succeeded",
            "type": "documentAdditionOrUpdate",
            "canceledBy": null,
            "details": {
                "receivedDocuments": 1,
                "indexedDocuments": 1
            },
            "error": null,
            "duration": "PT0.493602209S",
            "enqueuedAt": "2025-03-07T00:24:56.599420096Z",
            "startedAt": "2025-03-07T00:24:56.742055209Z",
            "finishedAt": "2025-03-07T00:24:57.235657418Z"
        }, [...]
    ]
}

It is visible that the last update was forwarded to Meilisearch as a task but the deletion was registered by Meilisync but Meilisearch has not received a deletion request. I'm not sure whether this should perhaps be a separate ticket, but I wanted to mention it here at least once.

[Skillkiller]

Oh, I was probably too quick. After 5 minutes there was a timeout error.

[mavcook]

@Skillkiller just to confirm, did you determine if you were hitting an unrelated timeout issue? I haven't observed any issues with deletions, but my testing has been very limited. I have only tested postgres:15 + meilisearch:1.10 with relatively small syncs (probably only 10 rows across 3 or so tables per minute).

[Skillkiller]

Unfortunately, I cannot say whether this is due to the change in this PR or a general problem. Since I have not been able to successfully sync with the original code. However, I do not currently assume that my error has anything to do with this PR.

[joebnb]

will some one merge this?