mc
mc copied to clipboard
minio
mc mirror takes forever
Expected behavior
mirroring from one server to another should take several hours
Actual behavior
The mirroring process takes a lot of days and the progress is very very little
Steps to reproduce the behavior
One of our systems uses minio version RELEASE.2020-06-18T02-23-35Z (that is the source server from which I'm trying to mirror) this server is running minio service (not on a docker container) it distributes it's data on 4 volumes (/data{1...4}). There are many small objects (5,250,000 objects) inside this server but only 113GB is in used on every volume (452 GB overall) I'm mirroring it to a server that runs on a minio docker container on 1 volume on version RELEASE.2022-07-08T00-05-23Z
I also tried cloning the server (on VMware) and run the new docker container version on the servers volumes but it messed the whole data up.
One of our systems uses minio version RELEASE.2020-06-18T02-23-35Z (that is the source server from which I'm trying to mirror)
Please upgrade to latest MinIO @roeelavi
One of our systems uses minio version RELEASE.2020-06-18T02-23-35Z (that is the source server from which I'm trying to mirror)
Please upgrade to latest MinIO @roeelavi
allright... done, still looks like its not moving. will let you know if it got better
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
I'm facing the same issues trying to use mc mirror to mirror a bucket in minio to Cloudflare R2. The speeds are in the low Kbps range. I only have 56 objects in my bucket totaling 500MB and at this rate it will take days to complete.
I'm using the latest version of minio and mc as of this date and passing these args to mc:
mirror --watch minio/cloudnative-pg cloudflare/cloudnative-pg,
It seems anytime this has been brought up in past issues the maintainer tell people to upgrade which still doesn't solve the problem. Something is really off with the mirror command as I have also tried it with backblaze b2 with the same exact results.
Related issues: https://github.com/minio/mc/issues/4660 https://github.com/minio/mc/issues/4251 https://github.com/minio/mc/issues/3895 https://github.com/minio/mc/issues/2504
mc mirror is daily used by countless users to mirror content from MinIO to AWS S3 and it has worked for years.
All these look like issues with S3 compatible implementations that are either incomplete or perhaps are low performant in their implementation.
Closing this issue since there is nothing actionable provided. We have no access to all these new vendors, neither we have the time to make it work for them.
mc's goal has been does it work with AWS S3 and it it does then our scope is covered for the project.
If you have the time we are always open to pull requests, any other debugging help and we are more than happy to take them in.