weaver
weaver copied to clipboard
ServiceWeaver
panic in one component caused the app to exit?
I'm interested in the idea of "locally running components" presented by Service Weaver, and I hope it can help me build a modular monolithic service composed of many functions. In my team the application is divided into hundreds of microservices, which is difficult to maintain.
However, one advantage of microservices is that they provide physical fault isolation. For example, a panic during release will not cause a large-scale breakdown of the entire system, if the caller handles those RPC error correctly.
Since Service Weaver runs the application in multiple processes, I think of the component as an analogy to a single microservice. Problems within a component only affect itself. So I tried adding a Panicker in the Step by Step example, and then executed weaver multi deploy, called the Panicker. But this caused the app to exit. Did I misunderstand something here?
code:
// Panicker component.
type Panicker interface {
Panic(context.Context, string) (string, error)
}
// Implementation of the Panicker component.
type panicker struct {
weaver.Implements[Panicker]
}
// Panic causes panic when certain inputs are encountered
func (p *panicker) Panic(_ context.Context, s string) (string, error) {
if s == "panic" {
panic("panicker: " + s)
}
return "all good", nil
}
// app is the main component of the application. weaver.Run creates
// it and passes it to serve.
type app struct {
weaver.Implements[weaver.Main]
reverser weaver.Ref[Reverser]
panicker weaver.Ref[Panicker]
hello weaver.Listener
}
func serve(ctx context.Context, app *app) error {
// The hello listener will listen on a random port chosen by the operating
// system. This behavior can be changed in the config file.
fmt.Printf("hello listener available on %v\n", app.hello)
// Serve the /hello endpoint.
http.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
name := r.URL.Query().Get("name")
if name == "" {
name = "World"
}
reversed, err := app.reverser.Get().Reverse(ctx, name)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
panicked, err := app.panicker.Get().Panic(ctx, name)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
fmt.Fprintf(w, "Hello, %s, and %s\n", reversed, panicked)
})
return http.Serve(app.hello, nil)
}
stderr:
❯ weaver multi deploy weaver.toml
╭───────────────────────────────────────────────────╮
│ app : hello │
│ deployment : 157a2898-1aaf-4a62-bd90-cf532a7f89f3 │
╰───────────────────────────────────────────────────╯
S0101 08:00:00.000000 stdout dfb270dd │ hello listener available on 127.0.0.1:12345
S0101 08:00:00.000000 stdout 99b2877c │ hello listener available on 127.0.0.1:12345
S0101 08:00:00.000000 stderr 54f72854 │ panic: panicker panicked: panic [recovered]
S0101 08:00:00.000000 stderr 54f72854 │ panic: panicker panicked: panic
S0101 08:00:00.000000 stderr 54f72854 │
S0101 08:00:00.000000 stderr 54f72854 │ goroutine 25 [running]:
S0101 08:00:00.000000 stderr 54f72854 │ github.com/ServiceWeaver/weaver/runtime/codegen.CatchPanics({0xbf0660, 0xc00061a030})
S0101 08:00:00.000000 stderr 54f72854 │ /home/sean/go/pkg/mod/github.com/!service!weaver/[email protected]/runtime/codegen/errors.go:29 +0xee
S0101 08:00:00.000000 stderr 54f72854 │ main.panicker_server_stub.panic.func1()
S0101 08:00:00.000000 stderr 54f72854 │ /home/sean/code/hello/weaver_gen.go:350 +0x2c
S0101 08:00:00.000000 stderr 54f72854 │ panic({0xbf0660?, 0xc00061a030?})
S0101 08:00:00.000000 stderr 54f72854 │ /usr/local/go/src/runtime/panic.go:914 +0x21f
S0101 08:00:00.000000 stderr 54f72854 │ main.(*panicker).Panic(0x0?, {0xc000612035?, 0x5?}, {0xc00060e030?, 0xc00051f9e8?})
S0101 08:00:00.000000 stderr 54f72854 │ /home/sean/code/hello/panicker.go:22 +0x65
S0101 08:00:00.000000 stderr 54f72854 │ main.panicker_server_stub.panic({{0x7f300d60d010?, 0xc0005a60a0?}, 0xc000538018?}, {0xe58100, 0xc000616050}, {0xc000612031?, 0xc000538018?, 0xc0005a60a0?})
S0101 08:00:00.000000 stderr 54f72854 │ /home/sean/code/hello/weaver_gen.go:362 +0x11e
S0101 08:00:00.000000 stderr 54f72854 │ github.com/ServiceWeaver/weaver/internal/weaver.(*RemoteWeavelet).addHandlers.func1({0xe58100, 0xc000616050}, {0xc000612031, 0x9, 0x9})
S0101 08:00:00.000000 stderr 54f72854 │ /home/sean/go/pkg/mod/github.com/!service!weaver/[email protected]/internal/weaver/remoteweavelet.go:597 +0xc3
S0101 08:00:00.000000 stderr 54f72854 │ github.com/ServiceWeaver/weaver/internal/net/call.(*serverConnection).runHandler(0xc000163e60, 0xc000037620, 0x1, {0xc000612000, 0x3a, 0x3a})
S0101 08:00:00.000000 stderr 54f72854 │ /home/sean/go/pkg/mod/github.com/!service!weaver/[email protected]/internal/net/call/call.go:1001 +0x8f9
S0101 08:00:00.000000 stderr 54f72854 │ github.com/ServiceWeaver/weaver/internal/net/call.(*serverConnection).readRequests(0xc000163e60, {0xe58100?, 0xc0005b9810}, 0xc000037620, 0xc000013bd8)
S0101 08:00:00.000000 stderr 54f72854 │ /home/sean/go/pkg/mod/github.com/!service!weaver/[email protected]/internal/net/call/call.go:921 +0x1ac
S0101 08:00:00.000000 stderr 54f72854 │ created by github.com/ServiceWeaver/weaver/internal/net/call.(*serverState).serveConnection in goroutine 42
S0101 08:00:00.000000 stderr 54f72854 │ /home/sean/go/pkg/mod/github.com/!service!weaver/[email protected]/internal/net/call/call.go:281 +0x376
2023/12/20 19:40:07 http: proxy error: EOF
capture stdout: EOF
status:
❯ weaver multi status
╭────────────────────────────────────────────────────╮
│ DEPLOYMENTS │
├───────┬──────────────────────────────────────┬─────┤
│ APP │ DEPLOYMENT │ AGE │
├───────┼──────────────────────────────────────┼─────┤
│ hello │ 23e97041-3acf-4e68-bbe2-bb70e278d59d │ 7s │
╰───────┴──────────────────────────────────────┴─────╯
╭────────────────────────────────────────────────────────╮
│ COMPONENTS │
├───────┬────────────┬────────────────┬──────────────────┤
│ APP │ DEPLOYMENT │ COMPONENT │ REPLICA PIDS │
├───────┼────────────┼────────────────┼──────────────────┤
│ hello │ 23e97041 │ weaver.Main │ 1566909, 1566921 │
│ hello │ 23e97041 │ hello.Panicker │ 1566959, 1566973 │
│ hello │ 23e97041 │ hello.Reverser │ 1566934, 1566946 │
╰───────┴────────────┴────────────────┴──────────────────╯
╭─────────────────────────────────────────────────╮
│ LISTENERS │
├───────┬────────────┬──────────┬─────────────────┤
│ APP │ DEPLOYMENT │ LISTENER │ ADDRESS │
├───────┼────────────┼──────────┼─────────────────┤
│ hello │ 23e97041 │ hello │ 127.0.0.1:12345 │
╰───────┴────────────┴──────────┴─────────────────╯
Hi @soaringk, happy to hear you're interested in Service Weaver! You're 100% right that if you deploy a Service Weaver application with weaver multi deploy and a component panics, the entire app is shut down. Here's the code that's responsible for that:
https://github.com/ServiceWeaver/weaver/blob/abde5cad1004a59d716d6301b18411788ee5a721/internal/tool/multi/deployer.go#L380
I should note that this is specific to the weaver multi deployer. The weaver gke and weaver kube deployers restart failed components.
Previously, the weaver multi deployer also restarted failed components. Later, we decided that the weaver multi deployer was largely for testing your application before you deploy it with a more production-ready deployer like weaver gke and weaver kube. Because of this, if something panicked when testing locally, we wanted to escalate the error instead of swallowing it.
We're open to revisiting that decision though!
if something panicked when testing locally, we wanted to escalate the error instead of swallowing it
Hi @mwhittaker, thank you for your response. I agree with you, and this is especially true for Testing. However, waver gke and waver kube seem to become the only way of actually deploying application, while weaver multi can only be used for testing. It that true?
Would it be a good idea to add a flag to weaver multi command so that it can be used for both testing and deploying? Such as weaver multi deploy --release=true.
Hi @soaringk. That's correct, weaver gke, weaver kube and the experimental weaver ssh are intended for deployment into production. However, there's nothing preventing you from running weaver kube, weaver ssh or weaver gke on your local machine. For the latter, we have a weaver gke local variation.
Could you provide more details about your usecase? I'm curious about how you plan to use weaver multi to deploy your application into production.
As Michael mentioned, we can revisit the decisions made regarding weaver multi. Right now, we view weaver multi as a deployer that someone can use to quickly run and test on their local machine.
Hi @rgrandl. I must have missed this weaver gee local variation. Thank you for your information.
I tried running the above example with this variation and I found that the app seemed to keep "retrying". I could see the panic log in stderr and the app was never truly shut down, but the response was never received. The weaver gke-local status showed that the 2 replica of the components were unhealthy. Then, the curl command appeared to be hanging until I interrupted it. Is this expected? Because calls in RPC usually just return err to the caller if something bad happens. The caller should get some response anyway.
Speaking of my use case, my team is responsible for product development, while deployment is managed by a separate infrastructure team. As a result, it may require some additional effort to deploy the weaver to a Kubernetes cluster, as I am not familiar with this layer. In the beginning, I assumed that running the service weaver would be similar to deploying a single executable, while still having all the features of microservices. As a member of the product team, I can focus on writing code as usual and without worrying about the deployment process.
If I make the program panic with a probability of one-tenth, I would see that when the panic occurs, one instance becomes unhealthy, but I still receive the correct response, which should be from another instance.
❯ weaver gke-local status
╭───────────────────────────────────────────────────────────────╮
│ Deployments │
├───────┬──────────────────────────────────────┬───────┬────────┤
│ APP │ DEPLOYMENT │ AGE │ STATUS │
├───────┼──────────────────────────────────────┼───────┼────────┤
│ hello │ 26459c14-d58b-4d68-9c40-2aacaf2972ad │ 1m10s │ ACTIVE │
╰───────┴──────────────────────────────────────┴───────┴────────╯
╭───────────────────────────────────────────────────────────╮
│ COMPONENTS │
├───────┬────────────┬───────────┬────────────────┬─────────┤
│ APP │ DEPLOYMENT │ LOCATION │ COMPONENT │ HEALTHY │
├───────┼────────────┼───────────┼────────────────┼─────────┤
│ hello │ 26459c14 │ cn-south1 │ weaver.Main │ 2/2 │
│ hello │ 26459c14 │ cn-south1 │ hello.Panicker │ 1/1 │
│ hello │ 26459c14 │ cn-south1 │ hello.Reverser │ 2/2 │
╰───────┴────────────┴───────────┴────────────────┴─────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────╮
│ TRAFFIC │
├───────────┬────────────┬───────┬────────────┬───────────┬────────────┬──────────────────┤
│ HOST │ VISIBILITY │ APP │ DEPLOYMENT │ LOCATION │ ADDRESS │ TRAFFIC FRACTION │
├───────────┼────────────┼───────┼────────────┼───────────┼────────────┼──────────────────┤
│ hello.com │ public │ hello │ 26459c14 │ cn-south1 │ [::]:45619 │ 0.5 │
│ hello.com │ public │ hello │ 26459c14 │ cn-south1 │ [::]:41887 │ 0.5 │
╰───────────┴────────────┴───────┴────────────┴───────────┴────────────┴──────────────────╯
╭─────────────────────────────╮
│ ROLLOUT OF hello │
├─────────────────┬───────────┤
│ │ cn-south1 │
├─────────────────┼───────────┤
│ TIME │ 26459c14 │
│ Dec 25 08:14:31 │ 1.00 │
╰─────────────────┴───────────╯