[jvm] Snapshots are not being written to disk when there are tests with numbered names

[george-grec]

There is an issue when creating test methods that are numbered (e.g. test1(), test2(), ...). At some point snapshots of tests are no longer written to disk. I discovered this while writing integration tests for a Spring Boot app that I generically name test1(), test2() etc. because I use @DisplayName anyway.

I cannot disclose the actual code but here is an example test file using JUnit 5, Java 21 and Selfie 2.5.3 that I could reproduce this with:

import com.diffplug.selfie.Selfie;
import org.junit.jupiter.api.Test;

class ReproduceSelfieBugTest {
  //SELFIEWRITE
  @Test
  void test() {
    Selfie.expectSelfie("0").toMatchDisk();
  }

  @Test
  void test1() {
    Selfie.expectSelfie("1").toMatchDisk();
  }

  @Test
  void test2() {
    Selfie.expectSelfie("2").toMatchDisk();
  }

  @Test
  void test3() {
    Selfie.expectSelfie("3").toMatchDisk();
  }

  @Test
  void test4() {
    Selfie.expectSelfie("4").toMatchDisk();
  }

  @Test
  void test5() {
    Selfie.expectSelfie("5").toMatchDisk();
  }

  @Test
  void test6() {
    Selfie.expectSelfie("6").toMatchDisk();
  }

  @Test
  void test7() {
    Selfie.expectSelfie("7").toMatchDisk();
  }

  @Test
  void test8() {
    Selfie.expectSelfie("8").toMatchDisk();
  }

  @Test
  void test9() {
    Selfie.expectSelfie("9").toMatchDisk();
  }

  @Test
  void test10() {
    Selfie.expectSelfie("10").toMatchDisk();
  }

  @Test
  void testFoo() {
    Selfie.expectSelfie("testFoo").toMatchDisk();
  }

  @Test
  void testFoo1() {
    Selfie.expectSelfie("testFoo1").toMatchDisk();
  }

  @Test
  void testFoo2() {
    Selfie.expectSelfie("testFoo2").toMatchDisk();
  }

  @Test
  void testFoo3() {
    Selfie.expectSelfie("testFoo3").toMatchDisk();
  }

  @Test
  void testFoo4() {
    Selfie.expectSelfie("testFoo4").toMatchDisk();
  }

  @Test
  void testFoo5() {
    Selfie.expectSelfie("testFoo5").toMatchDisk();
  }

  @Test
  void testFoo6() {
    Selfie.expectSelfie("testFoo6").toMatchDisk();
  }

  @Test
  void testFoo7() {
    Selfie.expectSelfie("testFoo7").toMatchDisk();
  }

  @Test
  void testFoo8() {
    Selfie.expectSelfie("testFoo8").toMatchDisk();
  }

  @Test
  void testFoo9() {
    Selfie.expectSelfie("testFoo9").toMatchDisk();
  }

  @Test
  void testFoo10() {
    Selfie.expectSelfie("testFoo10").toMatchDisk();
  }

  @Test
  void foo() {
    Selfie.expectSelfie("foo").toMatchDisk();
  }

  @Test
  void foo1() {
    Selfie.expectSelfie("foo1").toMatchDisk();
  }

  @Test
  void foo2() {
    Selfie.expectSelfie("foo2").toMatchDisk();
  }

  @Test
  void foo3() {
    Selfie.expectSelfie("foo3").toMatchDisk();
  }

  @Test
  void foo4() {
    Selfie.expectSelfie("foo4").toMatchDisk();
  }

  @Test
  void foo5() {
    Selfie.expectSelfie("foo5").toMatchDisk();
  }

  @Test
  void foo6() {
    Selfie.expectSelfie("foo6").toMatchDisk();
  }

  @Test
  void foo7() {
    Selfie.expectSelfie("foo7").toMatchDisk();
  }

  @Test
  void foo8() {
    Selfie.expectSelfie("foo8").toMatchDisk();
  }

  @Test
  void foo9() {
    Selfie.expectSelfie("foo9").toMatchDisk();
  }

  @Test
  void foo10() {
    Selfie.expectSelfie("foo10").toMatchDisk();
  }
}

Running this test file in IntelliJ 2024.3 produces this ReproduceSelfieBugTest.ss file:

╔═ foo ═╗
foo
╔═ foo1 ═╗
foo1
╔═ foo2 ═╗
foo2
╔═ foo3 ═╗
foo3
╔═ foo4 ═╗
foo4
╔═ foo5 ═╗
foo5
╔═ foo6 ═╗
foo6
╔═ foo7 ═╗
foo7
╔═ foo8 ═╗
foo8
╔═ foo9 ═╗
foo9
╔═ test ═╗
0
╔═ test1 ═╗
1
╔═ test2 ═╗
2
╔═ test3 ═╗
3
╔═ test4 ═╗
4
╔═ test5 ═╗
5
╔═ test6 ═╗
6
╔═ test7 ═╗
7
╔═ test8 ═╗
8
╔═ test9 ═╗
9
╔═ testFoo ═╗
testFoo
╔═ testFoo1 ═╗
testFoo1
╔═ testFoo2 ═╗
testFoo2
╔═ testFoo3 ═╗
testFoo3
╔═ testFoo4 ═╗
testFoo4
╔═ testFoo5 ═╗
testFoo5
╔═ testFoo6 ═╗
testFoo6
╔═ testFoo7 ═╗
testFoo7
╔═ testFoo8 ═╗
testFoo8
╔═ testFoo9 ═╗
testFoo9
╔═ [end of file] ═╗

You will notice the snapshots for these tests are missing: foo10(), test10(), testFoo10, i.e. all tests ending with 10.

Additional information: If you remove ALL tests except test(), test1() and test10() then the snapshot for test10() is correctly saved to disk. After that, if test2() is added back and after re-running the test file, the snapshot for test10() is missing again. Then, after removing test2() once more and re-running the test file, the snapshot for test10() is STILL missing until the test file is run one more time.

[nedtwigg]

GAH! Thanks for the report... Any chance that parallelism is involved here? Or are the tests running single-threaded?

[george-grec]

Thanks for the quick reply! No parallel test execution was configured and I am pretty sure I saw the tests running sequentially in the IDE. Anyhow, you should try to reproduce this with my example.

[nedtwigg]

Might be a couple days before I'm able to dig in, but I bet it's related to inconsistent sorting. We have a sort order in snapshot files like this:

• (good) 1, 2, ... 10, 11
• (bad) 1, 10, 11, 2, ...

My guess is that we are using this special comparator inconsistently, so that some places are sorting correctly and others aren't, and that's resulting in the GC pruning the 10s.

This is a terrible bug, but a workaround might be to name your tests 01, 02, etc.

https://github.com/diffplug/selfie/blob/75eaaec8cb097d255fde6f46addfa5146ca3a11f/jvm/selfie-lib/src/commonMain/kotlin/com/diffplug/selfie/ArrayMap.kt#L19-L69