Add gatling for performance tests

[jreijn]

I was thinking of adding Gatling during the build cycle to do some performance tests, after which we can measure the results.

[Vest]

Hm... I was trying different options, but I couldn't make them work well:

• jmh - special type of tests, when Spring container (and all its View Resolvers) is not completely initialized. About half of view resolvers simply don't work;
• Gatling - uses newer Scala, with it Scalate adds dependencies to wrong folder, and Tomcat's Jar scanner tries to search them in the wrong places (I know it sounds weird, but you can simply change Scala from 2.11 to 2.12, and rebuild & run the project). However, Scalate still works well, but throws exceptions (weird, isn't it?!), but Gatling reports cannot product the picture similar to what we have in README.md. Either all results are aggregated, or requests are measured using average (or percentile) timings, where we need "totals" only. And instead of having a single table, Gatling produces many charts...

Well... I don't know. I was trying to play with Groovy and JMH, it collides with the Java project and tries to compile code that we don't need to the load/performance tests. I don't see anything better than using Apache Benchmark. Maybe automate it with a simple groovy script or shell, or python, or something simple.

@jreijn, here I feel helpless. Maybe you can give you a try. If you want, I can create a "gatling" branch, so you can see it, but I don't know any better solution.

[jreijn]

@Vest thanks. Sorry that I have not given this project much time. With the new year I'm going to spend some time to improve the project and perhaps we should split the project for benchmarking the template engines (just the engines) with JMH.

[Vest]

I saw one project somewhere in github, where these engines were tested without any web stack. So just raw template in, and raw HTML out. But I don't know if this is a fair comparison, between engines, because some of them might have weak points such as concurrency, caching, or I/O.

[fmcarvalho]

I have successfully configured the spring-comparing-template-engines with JMH working correctly with Spring MVC and all template engines of this project, except for JSP.

This way we are evaluating the performance of the Spring MVC stack with each template engine and discarding the overhead of the HTTP conversation.

To that end, I am using MockMvc built from class Launch and then I am performing requests in the same way as you presented in test PresentationsIT. I have distinct benchmark methods for each template engine evaluated in isolated trials with a fresh instance of SpringApplication, configured as:

    @Setup(Level.Trial)
    public synchronized void startupSpring() {
        try {
            if (context == null) {
                context = SpringApplication.run(Launch.class);
                mockMvc = MockMvcBuilders
                        .webAppContextSetup((WebApplicationContext) context)
                        .build();
            }
        } catch (Exception e) {
            //Force JMH crash
            throw new RuntimeException(e);
        }
    }

    @TearDown(Level.Trial)
    public synchronized void shutdownSpring() {
        try {
            if (context != null) {
                SpringApplication.exit(context);
                context = null;
            }
        } catch (Exception e) {
            //Force JMH crash
            throw new RuntimeException(e);
        }
    }

IMHO this is the most effective approach to benchmark the Spring MVC with different template engines. We have all the advantages of JMH with reliable isolation, measurement, warm-up, etc. The only drawback is that we are unable to test JSP.

In truth, JSP runs without errors, but the response content is empty, and we cannot effectively measure that engine because the Spring MVC Test doesn't run that servlet container (https://stackoverflow.com/a/37749746/1140754)

If you consider it useful, I can make a PR with my fork. Also, I included new unit tests to check whether the engines are returning the correct HTML. Those tests allowed me to understand that JSP was not rendering anything.

[jreijn]

Hi @fmcarvalho . Thanks for sharing the information. I'm definitely interested in your efforts. If you are willing to share or create a PR I would love to learn more.

With regards to JSP I ran into something similar when trying to write a test for it.

[fmcarvalho]

Done @jreijn https://github.com/jreijn/spring-comparing-template-engines/pull/89

[fmcarvalho]

@jreijn Any news?

[jreijn]

@fmcarvalho sorry for the late reply! I'll try to take a look at this weekend.