budibase icon indicating copy to clipboard operation
budibase copied to clipboard
verified publisher icon Budibase

Single image - azure appservice minio issue

Open PekkaJalonen opened this issue 3 years ago • 5 comments

Hosting

  • Self
    • Method: Azure AppService docker compose
    • Budibase Version: budibase/budibase-aas:v1.2.52

Describe the bug Mounting of /data works for other components, except minio. All actions such as simple testing uploading organization logo in the beginning gives Error: Drive /data/minio, path (.minio.sys/buckets/global/.metadata.bin) returned an error (file access denied) (*fmt.wrapError). Reason probably is that the Azure File Storage mount has limited owner capabilities when mounted to App Service.

To Reproduce Steps to reproduce the behavior:

  1. Mount volume ${WEBAPP_STORAGE_HOME}:/data
  2. Run compose
  3. Sign in to Budibase, go to Organization and try to upload logo
  4. Budibase will throw error and move you back to sign in page
  5. See error from App Service Depoyment Center logs

Expected behavior Upload would be successful, or variable to disable build-in Minio and add option minio_url for single container to utilize example s3proxy to store data in Azure Storage.

PekkaJalonen avatar Aug 27 '22 17:08 PekkaJalonen

2022-08-27T16:50:59.781528370Z API: PutBucketPolicy(bucket=global) 2022-08-27T16:50:59.781533870Z Time: 16:50:59 UTC 08/27/2022 2022-08-27T16:50:59.781538170Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:50:59.781542170Z RequestID: 170F41BB2F8278BA 2022-08-27T16:50:59.781546170Z RemoteHost: 127.0.0.1 2022-08-27T16:50:59.781549970Z Host: localhost:9000 2022-08-27T16:50:59.781553870Z UserAgent: aws-sdk-nodejs/2.1030.0 linux/v16.17.0 promise 2022-08-27T16:50:59.781557871Z Error: Drive /data/minio, path (.minio.sys/buckets/global/.metadata.bin) returned an error (file access denied) (*fmt.wrapError) 2022-08-27T16:50:59.781562171Z 13: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:50:59.781566271Z 12: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf() 2022-08-27T16:50:59.781570171Z 11: internal/logger/logonce.go:135:logger.LogOnceIf() 2022-08-27T16:50:59.781574171Z 10: cmd/erasure-metadata-utils.go:156:cmd.readAllFileInfo() 2022-08-27T16:50:59.781578071Z 9: cmd/erasure-single-drive.go:751:cmd.(*erasureSingle).getObjectFileInfo() 2022-08-27T16:50:59.781582171Z 8: cmd/erasure-single-drive.go:574:cmd.(*erasureSingle).GetObjectNInfo() 2022-08-27T16:50:59.781586171Z 7: cmd/config-common.go:33:cmd.readConfigWithMetadata() 2022-08-27T16:50:59.781590071Z 6: cmd/config-common.go:55:cmd.readConfig() 2022-08-27T16:50:59.781593971Z 5: cmd/bucket-metadata.go:141:cmd.(*BucketMetadata).Load() 2022-08-27T16:50:59.781597971Z 4: cmd/bucket-metadata.go:168:cmd.loadBucketMetadata() 2022-08-27T16:50:59.781601871Z 3: cmd/bucket-metadata-sys.go:102:cmd.(*BucketMetadataSys).Update() 2022-08-27T16:50:59.781605871Z 2: cmd/bucket-policy-handlers.go:106:cmd.objectAPIHandlers.PutBucketPolicyHandler() 2022-08-27T16:50:59.781609871Z 1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP() 2022-08-27T16:51:08.605120091Z 2022-08-27T16:51:08.605273393Z API: SYSTEM() 2022-08-27T16:51:08.608079934Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.608101334Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.608107934Z Error: Drive /data/minio, path (.minio.sys/buckets/.background-heal.json) returned an error (file access denied) (*fmt.wrapError) 2022-08-27T16:51:08.608113634Z 11: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.608118934Z 10: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf() 2022-08-27T16:51:08.614057221Z 9: internal/logger/logonce.go:135:logger.LogOnceIf() 2022-08-27T16:51:08.614065721Z 8: cmd/erasure-metadata-utils.go:156:cmd.readAllFileInfo() 2022-08-27T16:51:08.614070921Z 7: cmd/erasure-single-drive.go:751:cmd.(*erasureSingle).getObjectFileInfo() 2022-08-27T16:51:08.614075821Z 6: cmd/erasure-single-drive.go:574:cmd.(*erasureSingle).GetObjectNInfo() 2022-08-27T16:51:08.614080521Z 5: cmd/config-common.go:33:cmd.readConfigWithMetadata() 2022-08-27T16:51:08.614085221Z 4: cmd/config-common.go:55:cmd.readConfig() 2022-08-27T16:51:08.614089821Z 3: cmd/data-scanner.go:122:cmd.readBackgroundHealInfo() 2022-08-27T16:51:08.614094721Z 2: cmd/data-scanner.go:200:cmd.runDataScanner() 2022-08-27T16:51:08.614178722Z 1: cmd/data-scanner.go:83:cmd.initDataScanner.func1() 2022-08-27T16:51:08.614183823Z 2022-08-27T16:51:08.614188323Z API: SYSTEM() 2022-08-27T16:51:08.614192723Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.614197223Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.614201823Z Error: Prefix access is denied: .minio.sys/buckets/.background-heal.json (cmd.PrefixAccessDenied) 2022-08-27T16:51:08.614206223Z 4: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.614210623Z 3: cmd/data-scanner.go:125:cmd.readBackgroundHealInfo() 2022-08-27T16:51:08.614215023Z 2: cmd/data-scanner.go:200:cmd.runDataScanner() 2022-08-27T16:51:08.614219623Z 1: cmd/data-scanner.go:83:cmd.initDataScanner.func1() 2022-08-27T16:51:08.614228523Z 2022-08-27T16:51:08.614233023Z API: SYSTEM() 2022-08-27T16:51:08.614243623Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.614264224Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.614268724Z Error: srcVolume: .minio.sys/tmp, srcPath: 78c4f400-2b12-4537-8f8b-f57fa20e0876, dstVolume: .minio.sys:, dstPath: buckets/.background-heal.json - error file access denied (*errors.errorString) 2022-08-27T16:51:08.614273224Z 6: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.614277524Z 5: cmd/xl-storage.go:2146:cmd.(*xlStorage).RenameData.func1() 2022-08-27T16:51:08.614281624Z 4: cmd/xl-storage.go:2228:cmd.(*xlStorage).RenameData() 2022-08-27T16:51:08.614285724Z 3: cmd/xl-storage-disk-id-check.go:373:cmd.(*xlStorageDiskIDCheck).RenameData() 2022-08-27T16:51:08.614289824Z 2: cmd/erasure-object.go:721:cmd.renameData.func1() 2022-08-27T16:51:08.614293824Z 1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() 2022-08-27T16:51:08.614297924Z 2022-08-27T16:51:08.614301824Z API: SYSTEM() 2022-08-27T16:51:08.614307924Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.614311924Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.614316024Z Error: file access denied (cmd.StorageErr) 2022-08-27T16:51:08.614319925Z 7: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.614324025Z 6: cmd/erasure-single-drive.go:1202:cmd.(*erasureSingle).putObject() 2022-08-27T16:51:08.614328125Z 5: cmd/erasure-single-drive.go:976:cmd.(*erasureSingle).PutObject() 2022-08-27T16:51:08.614332225Z 4: cmd/config-common.go:79:cmd.saveConfig() 2022-08-27T16:51:08.614336225Z 3: cmd/data-scanner.go:145:cmd.saveBackgroundHealInfo() 2022-08-27T16:51:08.614340225Z 2: cmd/data-scanner.go:209:cmd.runDataScanner() 2022-08-27T16:51:08.614344125Z 1: cmd/data-scanner.go:83:cmd.initDataScanner.func1() 2022-08-27T16:51:08.629965252Z 2022-08-27T16:51:08.630016553Z API: SYSTEM() 2022-08-27T16:51:08.630024953Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.630029853Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.630034253Z Error: Prefix access is denied: .minio.sys/buckets/.background-heal.json (cmd.PrefixAccessDenied) 2022-08-27T16:51:08.630038653Z 4: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.630042753Z 3: cmd/data-scanner.go:147:cmd.saveBackgroundHealInfo() 2022-08-27T16:51:08.630046953Z 2: cmd/data-scanner.go:209:cmd.runDataScanner() 2022-08-27T16:51:08.630051053Z 1: cmd/data-scanner.go:83:cmd.initDataScanner.func1() 2022-08-27T16:51:08.637044655Z 2022-08-27T16:51:08.637060855Z API: SYSTEM() 2022-08-27T16:51:08.637102855Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.637116056Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.637119656Z Error: Drive /data/minio, path (.minio.sys/buckets/.usage.json) returned an error (file access denied) (*fmt.wrapError) 2022-08-27T16:51:08.637132256Z 10: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.637135856Z 9: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf() 2022-08-27T16:51:08.637139256Z 8: internal/logger/logonce.go:135:logger.LogOnceIf() 2022-08-27T16:51:08.637142556Z 7: cmd/erasure-metadata-utils.go:156:cmd.readAllFileInfo() 2022-08-27T16:51:08.637146156Z 6: cmd/erasure-single-drive.go:751:cmd.(*erasureSingle).getObjectFileInfo() 2022-08-27T16:51:08.637149456Z 5: cmd/erasure-single-drive.go:574:cmd.(*erasureSingle).GetObjectNInfo() 2022-08-27T16:51:08.637152856Z 4: cmd/config-common.go:33:cmd.readConfigWithMetadata() 2022-08-27T16:51:08.637156256Z 3: cmd/config-common.go:55:cmd.readConfig() 2022-08-27T16:51:08.637159656Z 2: cmd/data-usage.go:94:cmd.loadDataUsageFromBackend() 2022-08-27T16:51:08.637163056Z 1: cmd/bucket-replication-stats.go:193:cmd.(*ReplicationStats).loadInitialReplicationMetrics() 2022-08-27T16:51:08.954505466Z 2022-08-27T16:51:08.954535067Z API: SYSTEM() 2022-08-27T16:51:08.954541567Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.954546367Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.954551067Z Error: srcVolume: .minio.sys/tmp, srcPath: f2bbc5c2-632f-45c3-86cb-a06ee3ead5d7, dstVolume: .minio.sys:, dstPath: buckets/.bloomcycle.bin - error file access denied (*errors.errorString) 2022-08-27T16:51:08.954556267Z 6: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.954560867Z 5: cmd/xl-storage.go:2146:cmd.(*xlStorage).RenameData.func1() 2022-08-27T16:51:08.954564967Z 4: cmd/xl-storage.go:2228:cmd.(*xlStorage).RenameData() 2022-08-27T16:51:08.954569467Z 3: cmd/xl-storage-disk-id-check.go:373:cmd.(*xlStorageDiskIDCheck).RenameData() 2022-08-27T16:51:08.954574067Z 2: cmd/erasure-object.go:721:cmd.renameData.func1() 2022-08-27T16:51:08.954578067Z 1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() 2022-08-27T16:51:08.954582567Z 2022-08-27T16:51:08.954586867Z API: SYSTEM() 2022-08-27T16:51:08.954591167Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.954595568Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.954599868Z Error: file access denied (cmd.StorageErr) 2022-08-27T16:51:08.954604268Z 6: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.954608568Z 5: cmd/erasure-single-drive.go:1202:cmd.(*erasureSingle).putObject() 2022-08-27T16:51:08.954613168Z 4: cmd/erasure-single-drive.go:976:cmd.(*erasureSingle).PutObject() 2022-08-27T16:51:08.954617768Z 3: cmd/config-common.go:79:cmd.saveConfig() 2022-08-27T16:51:08.954622168Z 2: cmd/data-scanner.go:233:cmd.runDataScanner() 2022-08-27T16:51:08.954637368Z 1: cmd/data-scanner.go:83:cmd.initDataScanner.func1() 2022-08-27T16:51:08.954641968Z 2022-08-27T16:51:08.954646268Z API: SYSTEM() 2022-08-27T16:51:08.954650168Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.954654368Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.954658868Z Error: srcVolume: .minio.sys/tmp, srcPath: 5a8e3f0a-1a85-4f5c-bf77-b44dec4c6b35, dstVolume: .minio.sys:, dstPath: buckets/.usage.json - error file access denied (*errors.errorString) 2022-08-27T16:51:08.954663269Z 6: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.954667669Z 5: cmd/xl-storage.go:2146:cmd.(*xlStorage).RenameData.func1() 2022-08-27T16:51:08.954671769Z 4: cmd/xl-storage.go:2228:cmd.(*xlStorage).RenameData() 2022-08-27T16:51:08.954676169Z 3: cmd/xl-storage-disk-id-check.go:373:cmd.(*xlStorageDiskIDCheck).RenameData() 2022-08-27T16:51:08.954680769Z 2: cmd/erasure-object.go:721:cmd.renameData.func1() 2022-08-27T16:51:08.954684769Z 1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1() 2022-08-27T16:51:08.954689169Z 2022-08-27T16:51:08.954693269Z API: SYSTEM() 2022-08-27T16:51:08.954697269Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.954701369Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.954705369Z Error: file access denied (cmd.StorageErr) 2022-08-27T16:51:08.954710169Z 5: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.954714369Z 4: cmd/erasure-single-drive.go:1202:cmd.(*erasureSingle).putObject() 2022-08-27T16:51:08.954718569Z 3: cmd/erasure-single-drive.go:976:cmd.(*erasureSingle).PutObject() 2022-08-27T16:51:08.954722669Z 2: cmd/config-common.go:79:cmd.saveConfig() 2022-08-27T16:51:08.954726669Z 1: cmd/data-usage.go:52:cmd.storeDataUsageInBackend() 2022-08-27T16:51:08.981532559Z 2022-08-27T16:51:08.981622360Z API: SYSTEM() 2022-08-27T16:51:08.981632960Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:08.981637760Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:08.981642060Z Error: Prefix access is denied: .minio.sys/buckets/.bloomcycle.bin (cmd.PrefixAccessDenied) 2022-08-27T16:51:08.981646260Z 3: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:08.981650161Z 2: cmd/data-scanner.go:234:cmd.runDataScanner() 2022-08-27T16:51:08.981653961Z 1: cmd/data-scanner.go:83:cmd.initDataScanner.func1() 2022-08-27T16:51:09.011854099Z 2022-08-27T16:51:09.011878200Z API: SYSTEM() 2022-08-27T16:51:09.011883400Z Time: 16:51:08 UTC 08/27/2022 2022-08-27T16:51:09.011887300Z DeploymentID: 6516cf4a-e413-4d81-8ca1-0a8bf7fbc0b4 2022-08-27T16:51:09.011910100Z Error: Prefix access is denied: .minio.sys/buckets/.usage.json (cmd.PrefixAccessDenied) 2022-08-27T16:51:09.011914500Z 2: internal/logger/logger.go:259:logger.LogIf() 2022-08-27T16:51:09.011918200Z 1: cmd/data-usage.go:53:cmd.storeDataUsageInBackend()

PekkaJalonen avatar Aug 27 '22 17:08 PekkaJalonen

@PekkaJalonen could you re-try this now pulling the latest container image. Over the last few days we have been adding the Azure App Service support and I have successfully deployed on Azure App Service. https://docs.budibase.com/docs/azure-app-service

jonnymccullagh avatar Sep 14 '22 14:09 jonnymccullagh

@PekkaJalonen I believe this is resolved so closing this ticket. If not feel free to re-open or create a new ticket.

jonnymccullagh avatar Sep 15 '22 21:09 jonnymccullagh

Hello, this issue is not resolved. Budibase-aas when deployed based on your documentation uses temporary volumes and so works perfectly when deployed. But your instructions do not have phase for enabling persistent storage in App Service. So by default, if the site is restarted you will loose all data. In App Service you need to enable data persistence by setting App Service configuration WEBSITES_ENABLE_APP_SERVICE_STORAGE=true. Then when site starts you can test logo upload which still fails due to this minio issue.

PekkaJalonen avatar Sep 16 '22 05:09 PekkaJalonen

Ah thanks for the clarification @PekkaJalonen I'll work on that now

jonnymccullagh avatar Sep 16 '22 09:09 jonnymccullagh

So are there any workarounds to this yet? If i had a s3 compatible container could i replace minio component completely? Can budibase run without minio?

aderici avatar Nov 11 '22 23:11 aderici

@aderici given the issues we have all had with AAS I think Azure Container Instances could be a better choice for hosting Budibase on Azure with persistent storage. I still need to find time to investigate ACA.

If i had a s3 compatible container could i replace minio component completely? Can budibase run without minio?

I believe you should be able to amend your environment variables to point to an S3 object storage (other than the minio within the single image) but I would need to play around and test that myself. When I do I will add to the docs.

jonnymccullagh avatar Nov 16 '22 11:11 jonnymccullagh

This issue has been automatically marked as stale because it has not had recent activity.

stale[bot] avatar Jan 16 '23 06:01 stale[bot]

This issue has been automatically marked as stale because it has not had recent activity.

stale[bot] avatar Mar 19 '23 02:03 stale[bot]

Fix for enabling custom S3 compatible storage will be tracked here: https://github.com/Budibase/budibase/issues/5636

Rory-Powell avatar Jul 04 '23 09:07 Rory-Powell