strato icon indicating copy to clipboard operation
strato copied to clipboard

Error while modifying a volume. Reached maximum modification rate per volume.

Open anvithks opened this issue 3 years ago • 0 comments

Issue/Feature Description: When trying to modify a volume repeatedly (3 times in 5 minutes) the API kept throwing the following error. "panic recovered: runtime error: invalid memory address or nil pointer dereference" image

The docker logs show You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volume.

The docker logs

[2020-09-18T23:59:16+05:30] [ERROR] [/root/gopath/src/github.com/sodafoundation/multi-cloud/contrib/datastore/block/aws/aws.go] [aws.(*AwsAdapter).UpdateVolume():318] [PID:1] Error in updating volume: {
  Iops: 0,
  Size: 500,
  VolumeId: "vol-045a4f6d70edb5078",
  VolumeType: "st1"
}%!(EXTRA *awserr.requestError=VolumeModificationRateExceeded: You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volume.
	status code: 400, request id: d4183d11-94fd-4268-bfdc-b7ba0e29b315)
[2020-09-18T23:59:16+05:30] [ERROR] [/root/gopath/src/github.com/sodafoundation/multi-cloud/contrib/datastore/block/aws/aws.go] [aws.(*AwsAdapter).UpdateVolume():322] [PID:1] VolumeModificationRateExceeded: You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volume.
	status code: 400, request id: d4183d11-94fd-4268-bfdc-b7ba0e29b315
[2020-09-18T23:59:16+05:30] [ERROR] [/root/gopath/src/github.com/sodafoundation/multi-cloud/block/pkg/service/service.go] [service.(*blockService).UpdateVolume():285] [PID:1] Received error in updating volumes at backend %!(EXTRA s3error.S3ErrorCode=Get object from backend failed.)
2020-09-18 23:59:16  file=grpc.go:372 level=error panic recovered: runtime error: invalid memory address or nil pointer dereference
2020-09-18 23:59:16  file=grpc.go:373 level=error goroutine 11124 [running]:
runtime/debug.Stack(0xc00000c4c0, 0xc000823f02, 0xc000823f40)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/micro/go-micro/v2/server/grpc.(*grpcServer).processRequest.func1.1(0xc00073f250)
	/root/gopath/pkg/mod/github.com/micro/go-micro/[email protected]/server/grpc/grpc.go:373 +0x2cf
panic(0x17ce720, 0x2b78200)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/opensds/multi-cloud/block/pkg/service.(*blockService).UpdateVolume(0xc000447720, 0x1e655e0, 0xc00057e030, 0xc000603a00, 0xc000497aa0, 0x498f0c, 0x18ccdc0)
	/root/gopath/src/github.com/sodafoundation/multi-cloud/block/pkg/service/service.go:286 +0x51a
github.com/opensds/multi-cloud/block/proto.(*blockHandler).UpdateVolume(0xc00053d1f0, 0x1e655e0, 0xc00057e030, 0xc000603a00, 0xc000497aa0, 0x7f1b56028008, 0x0)
	/root/gopath/src/github.com/sodafoundation/multi-cloud/block/proto/block.pb.micro.go:157 +0x5b
reflect.Value.call(0xc000452c00, 0xc0004d7418, 0x13, 0x1a9bbce, 0x4, 0xc0004331b0, 0x4, 0x4, 0x16, 0x1910980, ...)
	/usr/local/go/src/reflect/value.go:460 +0x5f6
reflect.Value.Call(0xc000452c00, 0xc0004d7418, 0x13, 0xc0004331b0, 0x4, 0x4, 0x203000, 0xc0004331b0, 0x40c1d6)
	/usr/local/go/src/reflect/value.go:321 +0xb4
github.com/micro/go-micro/v2/server/grpc.(*grpcServer).processRequest.func1(0x1e655e0, 0xc00057e030, 0x1e7b0a0, 0xc00049d900, 0x18ccdc0, 0xc000497aa0, 0x0, 0x0)
	/root/gopath/pkg/mod/github.com/micro/go-micro/[email protected]/server/grpc/grpc.go:378 +0x303
github.com/micro/go-micro/v2/util/wrapper.AuthHandler.func1.1(0x1e655e0, 0xc000497f80, 0x1e7b0a0, 0xc00049d900, 0x18ccdc0, 0xc000497aa0, 0xc000497f80, 0xc0000d4a80)
	/root/gopath/pkg/mod/github.com/micro/go-micro/[email protected]/util/wrapper/wrapper.go:282 +0x5e3
github.com/micro/go-micro/v2/util/wrapper.TraceHandler.func1.1(0x1e65520, 0xc0006039c0, 0x1e7b0a0, 0xc00049d900, 0x18ccdc0, 0xc000497aa0, 0x16, 0x18ccdc0)
	/root/gopath/pkg/mod/github.com/micro/go-micro/[email protected]/util/wrapper/wrapper.go:122 +0x1c6
github.com/micro/go-micro/v2/util/wrapper.HandlerStats.func1.1(0x1e65520, 0xc0006039c0, 0x1e7b0a0, 0xc00049d900, 0x18ccdc0, 0xc000497aa0, 0x12, 0x0)
	/root/gopath/pkg/mod/github.com/micro/go-micro/[email protected]/util/wrapper/wrapper.go:67 +0x83
github.com/micro/go-micro/v2/server/grpc.(*grpcServer).processRequest(0xc00021e780, 0x1e70c00, 0xc00011f380, 0xc000603600, 0xc00009e120, 0xc0008bd5a0, 0x16, 0x1e65520, 0xc0006039c0, 0xc000452f00, ...)
	/root/gopath/pkg/mod/github.com/micro/go-micro/[email protected]/server/grpc/grpc.go:395 +0x7ee
github.com/micro/go-micro/v2/server/grpc.(*grpcServer).handler(0xc00021e780, 0x0, 0x0, 0x1e70c00, 0xc00011f380, 0x0, 0x0)
	/root/gopath/pkg/mod/github.com/micro/go-micro/[email protected]/server/grpc/grpc.go:314 +0x77f
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0002dfc80, 0x1e7b100, 0xc000541e00, 0xc0004a2000, 0x0, 0xc000447640, 0x0, 0x0, 0x0)
	/root/gopath/pkg/mod/google.golang.org/[email protected]/server.go:1237 +0xb42
google.golang.org/grpc.(*Server).handleStream(0xc0002dfc80, 0x1e7b100, 0xc000541e00, 0xc0004a2000, 0x0)
	/root/gopath/pkg/mod/google.golang.org/[email protected]/server.go:1323 +0xc0a
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00086bfe0, 0xc0002dfc80, 0x1e7b100, 0xc000541e00, 0xc0004a2000)
	/root/gopath/pkg/mod/google.golang.org/[email protected]/server.go:722 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/root/gopath/pkg/mod/google.golang.org/[email protected]/server.go:720 +0xa1

Why this issue to fixed / feature is needed(give scenarios or use cases):

How to reproduce, in case of a bug:

Other Notes / Environment Information: (Please give the env information, log link or any useful information for this issue)

anvithks avatar Sep 18 '20 18:09 anvithks