mc mirror stops syncing old data after a while.

[zkryakgul]

trafficstars

Expected behavior

Im using mc mirror for syncing 2 minio servers. I was expecting the old data on the source minio to be synchronized to destination minio with the new events on the source minio.

Actual behavior

After a while when i look at the mc mirror logs i was seeing only new events logs. I didnt see anything about the my old data. I checked the both minio and i confirmed that some old data does not exist in destination yet. I checked the logs but i did not see any error logs.

Steps to reproduce the behavior

Add two alias for minio deployments. And start the mirror process from standalone minio deployment to the distributed minio deployment with this command:

mc mirror -q --no-color --debug --watch --preserver minio-standalone/mybucket minio-distributed/mybucket

mc --version

mc version RELEASE.2022-03-03T21-12-24Z

System information

Source minio version: RELEASE.2021-12-10T23-03-39Z (standalone mode) Destination minio version: RELEASE.2022-02-18T01-50-10Z (distributed mode - 2 serverpool)

[harshavardhana]

Source minio version: RELEASE.2021-12-10T23-03-39Z (standalone mode) Destination minio version: RELEASE.2022-02-18T01-50-10Z (distributed mode - 2 serverpool)

Please upgrade all the setups to latest MinIO release @zkryakgul

[zkryakgul]

Is there a know bug about these versions? These are production environments. So i cant upgrade them quickly.

[harshavardhana]

Is there a know bug about these versions? These are production environments. So i cant upgrade them quickly.

No we just do not support running older releases for Community users, so you should be upgrading to latest releases.

[zkryakgul]

I think this problem is dependent on minio client regardless of minio server version. I upgraded my destination minio server to the latest version. And create a new bucket in source minio. And i can successfully mirror this bucket which contains few objects to the destination server. The bucket i actually try to mirror have almost 82 million object (around 2.2TB). When i watch log for a while i realized the mc just keep sending this GET request to the both of my servers:

GET /mybucket/?continuation-token=<token>&delimiter=&encoding-type=url&fetch-owner=true&list-type=2&metadata=true&prefix=

[klauspost]

@zkryakgul This is ListObject requests that are listing your objects to compare before copying. This is normal operation.

[zkryakgul]

I think the main problem is the comparison between two bucket. My source bucket has over 82 million object and after the mc mirror copy the 4.7 million object from the source bucket it was interrupted. When i try to mirror my source bucket to the empy new one in the destination it works as expected. But when i run mc mirror for the interrupted process it just copying new events.

[haslersn]

If the problem is the comparison between two buckets (or the listing of the source bucket), then this might be related to #4047.

[atomlab]

The similar problem. My case

mc mirror --preserve --json --watch --overwrite src-minio/mybucket dst-minio/mybucket

mc client stops show any logs about process

journalctl -fu mc-mirror.service

Releases on all sides

minio version RELEASE.2021-09-24T00-24-24Z mc version RELEASE.2022-04-26T18-00-22Z

After restart works well during few time.

UPDATE: The same problem with releases minio RELEASE.2022-05-26T05-48-41Z mc RELEASE.2022-05-09T04-08-26Z

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.