Segmentation Violations on g_object_unref

[svenwltr]

We are occasionally getting segmentation violations when using bimg. It seems to happen when unreferencing objects, but I was not able to reproduce the issue locally, yet. This is probably, because it is very unlikely to happen. In production we get this roughly 49 out of 14.000.000 times.

One big problem here is that we cannot recover from this, because it is a fatal error which forces the process to exit and kill all other running processes.

Any idea how to fix this or how to provide more information about this issue?

Versions

• bimg: 1.0.19
• libvips: 8.7.4

Stack Trace

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x7f12f461a25b]

runtime stack:
runtime.throw(0xae08b4, 0x2a)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:374 +0x2f2

goroutine 33788 [syscall]:
runtime.cgocall(0x964000, 0xc0007781b0, 0xc0007781d8)
	/usr/local/go/src/runtime/cgocall.go:128 +0x5e fp=0xc000778180 sp=0xc000778148 pc=0x40739e
github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg._Cfunc_g_object_unref(0x7f129c012c80)
	_cgo_gotypes.go:421 +0x41 fp=0xc0007781b0 sp=0xc000778180 pc=0x917ce1
github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg.vipsSave.func1(0x7f129c012c80)
	/go/src/github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg/vips.go:400 +0x56 fp=0xc0007781e8 sp=0xc0007781b0 pc=0x9218c6
github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg.vipsSave(0x7f129c012c80, 0x5a, 0x6, 0x2, 0x10000, 0x0, 0x0, 0x16, 0xc00086a000, 0xd6e, ...)
	/go/src/github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg/vips.go:448 +0x256 fp=0xc000778298 sp=0xc0007781e8 pc=0x91ebf6
github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg.saveImage(0x7f129c012c80, 0x64, 0x42, 0x0, 0x0, 0x0, 0x0, 0x5a, 0x6, 0x0, ...)
	/go/src/github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg/resizer.go:169 +0x137 fp=0xc000778348 sp=0xc000778298 pc=0x91ba77
github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg.resizer(0xc000970000, 0x6dccc, 0x7fe00, 0x64, 0x42, 0x0, 0x0, 0x0, 0x0, 0x5a, ...)
	/go/src/github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg/resizer.go:124 +0x572 fp=0xc000778ab0 sp=0xc000778348 pc=0x91b0d2
github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg.Resize(0xc000970000, 0x6dccc, 0x7fe00, 0x64, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5a, ...)
	/go/src/github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg/resize.go:15 +0x105 fp=0xc000778c78 sp=0xc000778ab0 pc=0x9173e5
github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg.(*Image).Process(0xc000779040, 0x64, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5a, 0x0, 0x0, ...)
	/go/src/github.com/rebuy-de/thumbs/vendor/github.com/h2non/bimg/image.go:192 +0x78 fp=0xc000778e28 sp=0xc000778c78 pc=0x916e98
github.com/rebuy-de/thumbs/pkg.(*ImageHandler).process(0xc000195b60, 0xba03e0, 0xc000562930, 0x64, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5a, ...)
	/go/src/github.com/rebuy-de/thumbs/pkg/image_handler.go:85 +0x1f7 fp=0xc000779248 sp=0xc000778e28 pc=0x9268e7
github.com/rebuy-de/thumbs/pkg.(*ImageHandler).ServeHTTP(0xc000195b60, 0xb9fd20, 0xc0003007e0, 0xc00038d300)
	/go/src/github.com/rebuy-de/thumbs/pkg/image_handler.go:68 +0x8e2 fp=0xc000779888 sp=0xc000779248 pc=0x9265d2
github.com/rebuy-de/thumbs/pkg.(*RequestSigner).ServeHTTP(0xc000195b80, 0xb9fd20, 0xc0003007e0, 0xc00038d200)
	/go/src/github.com/rebuy-de/thumbs/pkg/request_signer.go:52 +0x631 fp=0xc000779a78 sp=0xc000779888 pc=0x928ed1
github.com/rebuy-de/thumbs/pkg.(*RequestLimiter).ServeHTTP(0xc000195ba0, 0xb9fd20, 0xc0003007e0, 0xc00038d100)
	/go/src/github.com/rebuy-de/thumbs/pkg/request_limiter.go:26 +0x520 fp=0xc000779bf8 sp=0xc000779a78 pc=0x928740
github.com/rebuy-de/thumbs/pkg.(*TracingMiddleware).ServeHTTP(0xc0001a7220, 0xb9fd20, 0xc0003007e0, 0xc00038d000)
	/go/src/github.com/rebuy-de/thumbs/pkg/tracing_middleware.go:36 +0x513 fp=0xc000779d08 sp=0xc000779bf8 pc=0x929963
net/http.(*ServeMux).ServeHTTP(0xc0003f4c90, 0xb9fd20, 0xc0003007e0, 0xc00038d000)
	/usr/local/go/src/net/http/server.go:2361 +0x127 fp=0xc000779d48 sp=0xc000779d08 pc=0x7018e7
net/http.serverHandler.ServeHTTP(0xc0003fc4e0, 0xb9fd20, 0xc0003007e0, 0xc00038d000)
	/usr/local/go/src/net/http/server.go:2741 +0xab fp=0xc000779d78 sp=0xc000779d48 pc=0x70288b
net/http.(*conn).serve(0xc000340c80, 0xba0320, 0xc0001b53c0)
	/usr/local/go/src/net/http/server.go:1847 +0x646 fp=0xc000779fc8 sp=0xc000779d78 pc=0x6fec96
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000779fd0 sp=0xc000779fc8 pc=0x45bf31
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x2f5

[svenwltr]

Hello. It looks like this is a concurrency issue. We wrapped Image.Process around a mutex and the issues seem to be disappeared. This slows everything a bit down, but we countered that by deploying more services.

func (i *ImageHandler) newImageProcess(payload []byte, options bimg.Options) ([]byte, error) {
	i.mux.Lock()
	defer i.mux.Unlock()

	return bimg.NewImage(payload).Process(options)
}

[Hpsyche]

@svenwltr @h2non I have encountered the same problem, besides add mutex, is there any better solution?

[svenwltr]

We still use the lock approach.