jackson-databind

jackson-databind copied to clipboard

I have a Microservice that contains a Jersey REST service and that accepts JSON body. I use jersey's Jackson support to deserialize the request into POJOs. This is a complex JSON (sometime upto 250kbs).

This microservice runs behind Envoy loadbalancer on a K8 cluster. So I can easily scale the number of instances.

Now the problem is, when I scale my application and Envoy starts distributing load (round-robin), my new instance has a really slow startup. For this new instance, it's like a burst of requests before JVM and service warmup. And during this time, I see that all my requests on this instance fail with a 503.

Since this new instance experienced a burst without Jackson and Jersey warmup, all threads are busy classloading. I can see this in my CPU profiling that almost 100% CPU is consumed by C1 and C2 compiler threads and loaded classes count shoots up.

My assumption here is that, all the requests are executed in parallel and all threads handling them are waiting for Jackson to load the POJO classes. Once the classloading is completed, the classes are cached, freeing up CPU and allowing threads to process the request instead.

I understand this from #1970 that this could be the problem. Is my assumption correct? Is there any thing I can do to load these classes before the first request?

Any leads would be much appreciated. Thanks

@dev-chirag a good option is switching to Scala and using jsoniter-scala. It does reflection in compile time only and is more efficient in runtime. Also it can be compiled to GraalVM native image to exclude impact of JIT compiler at all.

@plokhotnyuk Switching to Scala is typically not a good idea for performance issues. :-p

@dev-chirag You can certainly "warm up" things by eagerly calling serialize/deserialize on types that are likely used, as long as ObjectMapper being used is then shared for actual load. Beside JVM warmup aspect the first read and write of given POJO does a lot of reflection which will not be done again as long as ObjectMapper used keeps generated serializers/deserializers cached .

One note on class loading: class loading itself would be done by JVM; Jackson does not really do any of that (with the exception of 3 modules: Afterburner and Blackbird generate optimized handlers, and Mr Bean can materialize interface/abstract class implementations). But there is definitely lots of introspection and (de)serialize construction when a POJO instance is encountered for the first time. If there is concurrent access, it is possible that there is a lot of duplicated work at this point -- doing pre-emptive readValue()/writeValue() calls for warmup, before service is indicated as live, would allow avoiding this.