gpbackup icon indicating copy to clipboard operation
gpbackup copied to clipboard

Published 20 hours ago • verified publisher icon greenplum-db

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.

Open cobolbaby opened this issue 4 years ago • 7 comments

  • Command:
gpbackup --dbname F3_BDC --plugin-config /opt/greenplum/config/s3_backup.yml --leaf-partition-data --exclude-schema bsi_old --jobs 4 --quiet
  • Exception:
20201030:17:15:57 gpbackup:gpadmin:mdw:013535-[CRITICAL]:-ERROR: Error from segment 9: ERROR:  command error message: 2020/10/30 17:15:42 SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. (SQLSTATE 22P04)
github.com/greenplum-db/gpbackup/backup.BackupDataForAllTables
	/tmp/build/5f8239f8/go/src/github.com/greenplum-db/gpbackup/backup/data.go:167
github.com/greenplum-db/gpbackup/backup.backupData
	/tmp/build/5f8239f8/go/src/github.com/greenplum-db/gpbackup/backup/backup.go:321
github.com/greenplum-db/gpbackup/backup.DoBackup
	/tmp/build/5f8239f8/go/src/github.com/greenplum-db/gpbackup/backup/backup.go:181
main.main.func1
	/tmp/build/5f8239f8/go/src/github.com/greenplum-db/gpbackup/gpbackup.go:23
github.com/greenplum-db/gpbackup/vendor/github.com/spf13/cobra.(*Command).execute
	/tmp/build/5f8239f8/go/src/github.com/greenplum-db/gpbackup/vendor/github.com/spf13/cobra/command.go:766
github.com/greenplum-db/gpbackup/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	/tmp/build/5f8239f8/go/src/github.com/greenplum-db/gpbackup/vendor/github.com/spf13/cobra/command.go:852
github.com/greenplum-db/gpbackup/vendor/github.com/spf13/cobra.(*Command).Execute
	/tmp/build/5f8239f8/go/src/github.com/greenplum-db/gpbackup/vendor/github.com/spf13/cobra/command.go:800
main.main
	/tmp/build/5f8239f8/go/src/github.com/greenplum-db/gpbackup/gpbackup.go:27
runtime.main
	/usr/local/go/src/runtime/proc.go:198
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:2361
  • Snapshot:

2020-10-30 17-19-33屏幕截图

cobolbaby avatar Oct 30 '20 09:10 cobolbaby

The above exception occurs occasionally. I'd like to kown how to solve it.

cobolbaby avatar Oct 30 '20 10:10 cobolbaby

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

The error message is coming from an S3 API call. It seems like the issue could be related to your S3 plugin config file /opt/greenplum/config/s3_backup.yml. Could you verify that everything is correct in the config file (e.g. keys are correct, yaml formatting is correct, etc.)?

You should also see a COPY error in $MASTER_DATA_DIRECTORY/pg_log with the same message as well. It will detail how the s3 plugin is being used which could help solve the issue as well.

jimmyyih avatar Oct 30 '20 18:10 jimmyyih 

2021-05-16 20:20:33.882646 CST,"gpadmin","F6_BDC",p14595,th-1879283584,"10.13.0.65","49438",2021-05-16 20:07:07 CST,0,con1347973,cmd172,seg17,,dx23344671,,sx1,"LOG","00000","read err msg from pipe, len:366 msg:20210516:20:20:33 gpbackup_s3_plugin:gpadmin:gp6sdw5:039748-[ERROR]:-MultipartUpload: upload multipart failed
        upload id: 500b1255-637f-48ef-9639-7cc07a2d4d96
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
        status code: 403, request id: 167F8B5F9378294B, host id:
",,,,,,,0,,,,

cobolbaby avatar May 17 '21 06:05 cobolbaby

Hi,

Is this happening occasionally like you had mentioned last time you reported this? Or is it consistently failing? At the moment, I'm not sure how this error can occur if it's intermittent... maybe check your keys to see if something stands out (e.g. maybe you have a key rotation policy and the new key failed due to some weird character?). In the meantime, we'll see if we can maybe provide better logging to help debug the signature mismatch.

jimmyyih avatar May 18 '21 23:05 jimmyyih

With the default configuration the exception happens occasionally, as I increase the backup_multipart_chunksize the probability of the exception increases. I suspect it has something to do with the load of the S3 service.

cobolbaby avatar May 19 '21 04:05 cobolbaby

The next time this error occurs, post the logs from the s3_plugin logs located in $HOME/gpAdminLogs/gpbackup_s3_plugin_{backup_date}. We'll need some more details about the error to narrow down the possible issues. At the moment, it is difficult to find or reproduce the SignatureDoesNotMatch issue without additional information.

kyeap-vmware avatar Nov 30 '21 22:11 kyeap-vmware

There is nothing valuable in the $HOME/gpAdminLogs/gpbackup_s3_plugin_{backup_date} @kyeap-vmware

The root cause is read err msg from pipe.

[gpadmin@gp6mdw gpAdminLogs]$ gpbackup --version
gpbackup version 1.24.0
[gpadmin@gp6mdw gpAdminLogs]$ more gpbackup_20220809.log
...
20220809:18:25:22 gpbackup:gpadmin:gp6mdw:3041709-[CRITICAL]:-ERROR: command error message: 20220809:18:24:35 gpbackup_s3_plugin:gpadmin:gp6sdw3:2677462-[ERROR]:-MultipartUpload: upload multipart failed
        upload id: 43098ec3-42db-4ad6-9120-c36be45f5897
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
        status code: 403, request id: 1709A62F483A17DB, host id:  (seg8 10.13.0.33:40000 pid=2611103) (SQLSTATE 2F000)
github.com/greenplum-db/gpbackup/backup.backupDataForAllTables
        /tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/backup/data.go:365
github.com/greenplum-db/gpbackup/backup.backupData
        /tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/backup/backup.go:291
github.com/greenplum-db/gpbackup/backup.DoBackup
        /tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/backup/backup.go:169
main.main.func1
        /tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/gpbackup.go:23
github.com/spf13/cobra.(*Command).execute
        /tmp/build/3e49593f/go/pkg/mod/github.com/spf13/[email protected]/command.go:860
github.com/spf13/cobra.(*Command).ExecuteC
        /tmp/build/3e49593f/go/pkg/mod/github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
        /tmp/build/3e49593f/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
main.main
        /tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/gpbackup.go:27
runtime.main
        /usr/local/go/src/runtime/proc.go:255
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1581
...
[gpadmin@gp6mdw gpAdminLogs]$ cat gpbackup_s3_plugin_20220809.log
20220809:17:59:22 gpbackup_s3_plugin:gpadmin:gp6mdw:3041928-[DEBUG]:-Uploading file gpbackup_20220809175919_report with chunksize 16777216 and concurrency 6
20220809:18:25:23 gpbackup_s3_plugin:gpadmin:gp6mdw:3047591-[DEBUG]:-Uploading file gpbackup_20220809175919_config.yaml with chunksize 16777216 and concurrency 6
20220809:18:25:23 gpbackup_s3_plugin:gpadmin:gp6mdw:3047591-[INFO]:-Uploaded 50750 bytes for gpbackup_20220809175919_config.yaml in 108ms
20220809:18:25:23 gpbackup_s3_plugin:gpadmin:gp6mdw:3047602-[DEBUG]:-Uploading file gpbackup_20220809175919_report with chunksize 16777216 and concurrency 6
20220809:18:25:23 gpbackup_s3_plugin:gpadmin:gp6mdw:3047602-[INFO]:-Uploaded 2472 bytes for gpbackup_20220809175919_report in 117ms
20220809:18:25:56 gpbackup_s3_plugin:gpadmin:gp6mdw:3047983-[DEBUG]:-Uploading file gpbackup_20220809182553_report with chunksize 16777216 and concurrency 6
...
[gpadmin@gp6sdw3 pg_log]$ pwd 
/disk1/gpdata/gpsegment/primary/gpseg8/pg_log
[gpadmin@gp6sdw3 pg_log]$ vi  gpdb-2022-08-09_000000.csv
2022-08-09 18:24:35.515198 CST,"gpadmin","F3_BDC",p2611103,th-1446930304,"10.13.0.37","43344",2022-08-09 17:59:19 CST,0,con21929810,cmd278,seg8,,dx191351174,,sx1,"LOG","00000","read err msg from pipe, len:367 msg:20220809:18:24:35 gpbackup_s3_plugin:gpadmin:gp6sdw3:2677462-[ERROR]:-MultipartUpload: upload multipart failed
        upload id: 43098ec3-42db-4ad6-9120-c36be45f5897
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
        status code: 403, request id: 1709A62F483A17DB, host id:
",,,,,,,0,,,,
2022-08-09 18:24:35.515355 CST,"gpadmin","F3_BDC",p2611103,th-1446930304,"10.13.0.37","43344",2022-08-09 17:59:19 CST,0,con21929810,cmd278,seg8,,dx191351174,,sx1,"ERROR","2F000","command error message: 20220809:18:24:35 gpbackup_s3_plugin:gpadmin:gp6sdw3:2677462-[ERROR]:-MultipartUpload: upload multipart failed
        upload id: 43098ec3-42db-4ad6-9120-c36be45f5897
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
        status code: 403, request id: 1709A62F483A17DB, host id:",,,,,,"COPY inframinio.fislog_info TO PROGRAM 'gzip -c -1 | /usr/local/greenplum-db-6.19.1/bin/gpbackup_s3_plugin backup_data /tmp/20220809175919_gpbackup_s3_backup.yml <SEG_DATA_DIR>/backups/20220809/20220809175919/gpbackup_<SEGID>_20220809175919_896684.gz' WITH CSV DELIMITER ',' ON SEGMENT IGNORE EXTERNAL PARTITIONS;",0,,"copy.c",8010,
2022-08-09 18:24:35.515426 CST,"gpadmin","F3_BDC",p2611103,th-1446930304,"10.13.0.37","43344",2022-08-09 17:59:19 CST,0,con21929810,cmd278,seg8,,dx191351174,,sx1,"LOG","00000","An exception was encountered during the execution of statement: COPY inframinio.fislog_info TO PROGRAM 'gzip -c -1 | /usr/local/greenplum-db-6.19.1/bin/gpbackup_s3_plugin backup_data /tmp/20220809175919_gpbackup_s3_backup.yml <SEG_DATA_DIR>/backups/20220809/20220809175919/gpbackup_<SEGID>_20220809175919_896684.gz' WITH CSV DELIMITER ',' ON SEGMENT IGNORE EXTERNAL PARTITIONS;",,,,,,"COPY inframinio.fislog_info TO PROGRAM 'gzip -c -1 | /usr/local/greenplum-db-6.19.1/bin/gpbackup_s3_plugin backup_data /tmp/20220809175919_gpbackup_s3_backup.yml <SEG_DATA_DIR>/backups/20220809/20220809175919/gpbackup_<SEGID>_20220809175919_896684.gz' WITH CSV DELIMITER ',' ON SEGMENT IGNORE EXTERNAL PARTITIONS;",0,,"elog.c",2116,

cobolbaby avatar Aug 09 '22 11:08 cobolbaby