vault icon indicating copy to clipboard operation
vault copied to clipboard
verified publisher icon hashicorp

[Vault 1.12.2] Registering Plugin Causing EOF

Open heliobmartins opened this issue 3 years ago • 10 comments

Describe the bug When trying to register a new plugin in our Vault server with vault 1.12.X CLI, we get EOF response.

To Reproduce Steps to reproduce the behavior:

  1. Run vault plugin register -sha256=$SHA_256 -command=$PLUGIN_NAME auth $PLUGIN_NAME
  2. See error: Error registering plugin slauth: Put "http://127.0.0.1:9200/v1/sys/plugins/catalog/auth/$PLUGIN_NAME": EOF
2022-12-12T22:30:28.029Z [DEBUG] core: attempting to load backend plugin: name=$PLUGIN_NAME
2022-12-12T22:30:28.029Z [DEBUG] core: spawning a new plugin process: plugin_name=$PLUGIN_NAME id=L4tEP6Xly8
2022-12-12T22:30:28.246Z [DEBUG] core: successfully dispensed v5 backend plugin: name=$PLUGIN_NAME
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x12e4c0f]

goroutine 1577 [running]:
github.com/hashicorp/vault/sdk/plugin.(*GRPCStorageServer).Get(0x50dcd80?, {0x6b602f8?, 0xc0006af650?}, 0x0?)
	/home/runner/work/vault/vault/sdk/plugin/grpc_storage.go:92 +0x2f
github.com/hashicorp/vault/sdk/plugin/pb._Storage_Get_Handler({0x5061ee0?, 0xc0012e2d90}, {0x6b602f8, 0xc0006af650}, 0xc00116d2c0, 0x0)
	/home/runner/work/vault/vault/sdk/plugin/pb/backend_grpc.pb.go:527 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0007d4000, {0x6b755c0, 0xc001394340}, 0xc00014a000, 0xc0010d2ea0, 0x99453f8, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1283 +0xcfe
google.golang.org/grpc.(*Server).handleStream(0xc0007d4000, {0x6b755c0, 0xc001394340}, 0xc00014a000, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1620 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:922 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:920 +0x28a

Expected behavior Previously, the success message would be returned Success! Registered plugin: $PLUGIN_NAME

Environment:

  • Vault Server Version (retrieve with vault status):
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.12.2
Build Date      2022-11-23T12:53:46Z
Storage Type    inmem
Cluster Name    vault-cluster-13288d5d
Cluster ID      746abeec-4043-ab30-8dde-1d735b1df0f5
HA Enabled      false
  • Vault CLI Version (retrieve with vault version):
Vault v1.12.2 (415e1fe3118eebd5df6cb60d13defdc01aa17b03), built 2022-11-23T12:53:46Z
  • Server Operating System/Architecture:
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.17.0
PRETTY_NAME="Alpine Linux v3.17"
Linux vault 5.15.49-linuxkit #1 SMP Tue Sep 13 07:51:46 UTC 2022 x86_64 Linux

Any help would be greatly appreciated.

heliobmartins avatar Dec 12 '22 22:12 heliobmartins

Hello team, I noticed the existence of this ticket: https://github.com/hashicorp/vault/issues/17878. While the cause for the error thrown is not quite the same, it looks related.

Do you have any suggestion or know if this issue is already being addressed?

Thanks in advance.

heliobmartins avatar Dec 19 '22 23:12 heliobmartins

Hello team, I noticed the existence of this ticket: #17878. While the cause for the error thrown is not quite the same, it looks related.

Do you have any suggestion or know if this issue is already being addressed?

Thanks in advance.

I'll check more in-depth soon; also, the PR that resolves the linked issue should, if I understand correctly, will be included in the next release.

heatherezell avatar Dec 20 '22 01:12 heatherezell

Thanks so much for your quick response @hsimon-hashicorp !

Yes, once I saw that merged issue, I was definitely expecting that to be released in the next version! If it turns out that fix will also be the solution for this problem, I will be more than happy to close it and wait for the next release.

Thanks once again for your help into this.

heliobmartins avatar Dec 20 '22 03:12 heliobmartins

@heliomartins would you mind providing the version of the vault/sdk that you were using when the panic occurred?

benashz avatar Dec 20 '22 20:12 benashz

😮 Hello @benashz , what a such coincidence! I'm not sure if you are going to remember, but we talked a few times at re:invent. I'm the Atlassian engineer who stopped there a couple of times. haha

Well, thanks for reaching out, and apologies for not having provided this information before. When it comes to the SDK's, we are currently running:

hashicorp/vault/api v1.8.2
hashicorp/vault/sdk v0.6.1

Thanks once again for your help into this.

heliobmartins avatar Dec 20 '22 20:12 heliobmartins

😮 Hello @benashz , what a such coincidence! I'm not sure if you are going to remember, but we talked a few times at re:invent. I'm the Atlassian engineer who stopped there a couple of times. haha

Nice! Great to hear from you again 😄

Well, thanks for reaching out, and apologies for not having provided this information before. When it comes to the SDK's, we are currently running:

hashicorp/vault/api v1.8.2
hashicorp/vault/sdk v0.6.1

Thanks! In the mean time would you mind testing against the change that might resolve your issue?

You can issue the command below from your project/module root.

 go get -u github.com/hashicorp/vault/sdk@eabc486b

If that works, we can cut a new release of vault/sdk.

benashz avatar Dec 20 '22 21:12 benashz

No worries @benashz , I will run some tests and I will get back to you as soon as possible.

heliobmartins avatar Dec 20 '22 21:12 heliobmartins

Hey @benashz , 👋

Unfortunately it looks like it didn't work as expected. :(

I ran a couple of tests locally and also in our CI and it looks like the issue still persists. After updating to the version you suggested, our go.mod used github.com/hashicorp/vault/sdk v0.6.2-0.20221202181205-eabc486b1aee, which refers to your commit I suppose.

The exception thrown was the same one:

2022-12-20T21:33:52.320Z [DEBUG] core: attempting to load backend plugin: name=slauth
2022-12-20T21:33:52.320Z [DEBUG] core: spawning a new plugin process: plugin_name=slauth id=aiyITdThTa
2022-12-20T21:33:52.546Z [DEBUG] core: successfully dispensed v5 backend plugin: name=slauth
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x12e4c0f]

goroutine 1008 [running]:
github.com/hashicorp/vault/sdk/plugin.(*GRPCStorageServer).Get(0x50dcd80?, {0x6b602f8?, 0xc000eb8990?}, 0x2027c8a5f0303fb9?)
	/home/runner/work/vault/vault/sdk/plugin/grpc_storage.go:92 +0x2f
github.com/hashicorp/vault/sdk/plugin/pb._Storage_Get_Handler({0x5061ee0?, 0xc0015fac50}, {0x6b602f8, 0xc000eb8990}, 0xc00034a120, 0x0)
	/home/runner/work/vault/vault/sdk/plugin/pb/backend_grpc.pb.go:527 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0014da540, {0x6b755c0, 0xc000bf7040}, 0xc001398240, 0xc000fdeab0, 0x99453f8, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1283 +0xcfe
google.golang.org/grpc.(*Server).handleStream(0xc0014da540, {0x6b755c0, 0xc000bf7040}, 0xc001398240, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1620 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:922 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:920 +0x28a

And it happened while trying to register the plugin with: vault plugin register -sha256=5fcb8871bdc98fc7fc0c8681c1d78102977874d960696a4b253efd07fab574e5 -command=slauth auth slauth Error registering plugin slauth: Put "http://127.0.0.1:9200/v1/sys/plugins/catalog/auth/slauth": EOF

Please, let me know if there is anything I can do to try to help you with the investigation.

heliobmartins avatar Dec 20 '22 21:12 heliobmartins

Thanks for testing that @heliomartins! I will take a closer look to see what the issue could be.

benashz avatar Dec 20 '22 21:12 benashz

No worries! Thanks for trying to help, and please, do let us know if there is any other test you wish us to perform.

heliobmartins avatar Dec 20 '22 21:12 heliobmartins

Hello @benashz , I hope you had a good break the end of the year.

I was wondering if you were actually able to find anything else. We have bumped to version 0.7.0 but the issue still persists. :(

heliobmartins avatar Jan 18 '23 00:01 heliobmartins

Hi, @heliomartins. It seems that we might have addressed this issue in https://github.com/hashicorp/vault/pull/18210. We have since backported it to the 1.12.x release branch.

ccapurso avatar Jan 23 '23 18:01 ccapurso

Hello @ccapurso, Thank you so much for your response.

That's awesome to hear that the issue was found and it is getting fixed. Do you know when are you planning on releasing the fix?

heliobmartins avatar Jan 23 '23 23:01 heliobmartins

@heliomartins, we do not have an exact date to provide but this change will be made available in the release of 1.12.3.

Edit: I am going to close this issue as it has been addressed per the comment above.

ccapurso avatar Jan 24 '23 21:01 ccapurso

Hello @benashz and @ccapurso , I just wanted to give you a heads up that I ended up having to raise another issue. https://github.com/hashicorp/vault/issues/19192

Whilst it doesn't look like the plugin initialisation it is not panicking anymore (as it was before), it looks like core is still killing it for some reason.

heliobmartins avatar Feb 15 '23 03:02 heliobmartins