Issue with mc client specifically for armhf

[probal31]

Hello Team,

I have been working to setup a ci environment for building AppImage and push the artifacts to minio bucket. This works pretty well for amd64 and arm64 environments.

But for armhf the behavior is somewhat weird while uploading the artifacts to minio bucket. Whereas the logs are always clear and does not show any error but in the minio bucket the objects are missing.

There are basically 2 scenarios.

Scenario 1

When the bucket is not already present, then in that scenario it seems the files are not uploaded and the bucket itself behaves as an object. I mean on clicking the bucket folder in the minio console the page that opens up is the one that is actually opened while clicking an object and not a folder/bucket. Please find the related logs below.

$ du -h *.AppImage || true
70M	vvave-nightly-20211007-git+e80eaae-armhf.AppImage
$ du -h *.deb || true
du: cannot access '*.deb': No such file or directory
$ echo "Pushing Artifacts"
Pushing Artifacts
$ export BUCKET_DATE="$(date +'%Y-%m-%d')"
$ if [ $(uname -m) = 'x86_64' ]; then export MINIO_ARCH=amd64; elif [ $(uname -m) = 'aarch64' ]; then export MINIO_ARCH=arm64; elif [[ $(uname -m) = "armv7"* || $(uname -m) = "armhf" ]]; then export MINIO_ARCH=armhf; fi
$ if [[ "$BUILD_TYPE" == 'nightly' || "$BUILD_TYPE" == 'devel' || "$BUILD_TYPE" == "beta"* ]]; then mc rm --recursive --force "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc mb "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *amd64*.AppImage "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *amd64*.deb "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *arm64*.AppImage "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *arm64*.deb "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *armhf*.AppImage "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *armhf*.deb "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; echo $BUCKET_DATE | mc pipe nx/maui/$BUILD_TYPE/LATEST; fi
mc: <ERROR> No object/version found to be removed in `nx/maui/nightly/armhf/`. 
Bucket created successfully `nx/maui/nightly/armhf/`.
mc: <ERROR> Unable to validate source `*amd64*.AppImage`.
mc: <ERROR> Unable to validate source `*amd64*.deb`.
mc: <ERROR> Unable to validate source `*arm64*.AppImage`.
mc: <ERROR> Unable to validate source `*arm64*.deb`.
`vvave-nightly-20211007-git+e80eaae-armhf.AppImage` -> `nx/maui/nightly/armhf/vvave-nightly-20211007-git+e80eaae-armhf.AppImage`
Cleaning up project directory and file based variables 00:01
Job succeeded

Please see that even if nx/maui/nightly/armhf/ is the bucket but it behaves like an object when clicked in the minio console.

Scenario 2

Next is that sometimes from the log although it can be seen that multiple artifacts are pushed but those are actually missing in the bucket.

Running after script...
$ du -h *.AppImage || true
195M	buho-devel-202110070834-git+cf17a8c-armhf.AppImage
172M	index-devel-202110070834-git+c522af3-armhf.AppImage
196M	nota-devel-202110070834-git+da9c6d5-armhf.AppImage
71M	vvave-devel-202110070834-git+e80eaae-armhf.AppImage
$ du -h *.deb || true
du: cannot access '*.deb': No such file or directory
$ echo "Pushing Artifacts"
Pushing Artifacts
$ export BUCKET_DATE="$(date +'%Y-%m-%d')"
$ if [ $(uname -m) = 'x86_64' ]; then export MINIO_ARCH=amd64; elif [ $(uname -m) = 'aarch64' ]; then export MINIO_ARCH=arm64; elif [[ $(uname -m) = "armv7"* || $(uname -m) = "armhf" ]]; then export MINIO_ARCH=armhf; fi
$ if [[ "$BUILD_TYPE" == 'nightly' || "$BUILD_TYPE" == 'devel' || "$BUILD_TYPE" == "beta"* ]]; then mc rm --recursive --force "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc mb "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *amd64*.AppImage "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *amd64*.deb "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *arm64*.AppImage "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *arm64*.deb "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *armhf*.AppImage "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; mc cp *armhf*.deb "nx/maui/$BUILD_TYPE/$MINIO_ARCH/" || true; echo $BUCKET_DATE | mc pipe nx/maui/$BUILD_TYPE/LATEST; fi
Removing `nx/maui/devel/armhf/buho-devel-202110070800-git+cf17a8c-armhf.AppImage`.
Removing `nx/maui/devel/armhf/vvave-devel-202110070800-git+e80eaae-armhf.AppImage`.
Bucket created successfully `nx/maui/devel/armhf/`.
mc: <ERROR> Unable to validate source `*amd64*.AppImage`.
mc: <ERROR> Unable to validate source `*amd64*.deb`.
mc: <ERROR> Unable to validate source `*arm64*.AppImage`.
mc: <ERROR> Unable to validate source `*arm64*.deb`.
`buho-devel-202110070834-git+cf17a8c-armhf.AppImage` -> `nx/maui/devel/armhf/buho-devel-202110070834-git+cf17a8c-armhf.AppImage`
`index-devel-202110070834-git+c522af3-armhf.AppImage` -> `nx/maui/devel/armhf/index-devel-202110070834-git+c522af3-armhf.AppImage`
`nota-devel-202110070834-git+da9c6d5-armhf.AppImage` -> `nx/maui/devel/armhf/nota-devel-202110070834-git+da9c6d5-armhf.AppImage`
`vvave-devel-202110070834-git+e80eaae-armhf.AppImage` -> `nx/maui/devel/armhf/vvave-devel-202110070834-git+e80eaae-armhf.AppImage`
Cleaning up project directory and file based variables 00:01
Job succeeded

Please see that even if 4 files were uploaded but in the bucket only 3 is visible.

Re-running the build sometimes uploads the artifacts correctly without changing any config or script from ci end.

But the way, the mc client that I am using is https://dl.min.io/client/mc/release/linux-arm/mc Also please note that there is no such issue faced for the arm64 version of the client.

Any info on this would be really helpful.

Thanks.

[klauspost]

Since you do || true you are eating error codes. Please run your script manually, one step at the time and report output. Maybe add --debug if nothing important shows up.

[probal31]

@klauspost Tried removing the || true but it didn't help much.

Also ran with --debug option. Can see an error like ObjectLockConfigurationNotFoundError but this also comes while building and uploading for amd64 and there the upload to minio works fine. So assuming this is not the issue.

Attaching the debug log here.

mc_cp_debug.log

Out of 9 files, only 5 of them is uploaded to minio bucket.

Can you please help to understand where the issue is?

[klauspost]

mc: <DEBUG> POST /maui/devel/[...]?uploadId= (CompleteMultipartUpload) is indeed only present for the completed files.

For example, Searching for index-devel-2021100808 only shows parts being uploaded, but not completed.

[klauspost]

Setting to "high", even though it isn't a primary supported platform. It could be a 32 bit platform issue.

[probal31]

@klauspost Any update on this please?

[harshavardhana]

@probal31 we have no updates - armhf is not our primary platform. Unfortunately, we have no cycles to work on this.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.