add testgen to generate test cases

[mewmew]

Usage:

make -f MakefileVC testgen

mkdir -p testdata
wine testgen.exe

Or run the gen_tests.sh script which runs the above commands:

./gen_tests.sh

Edit: this Git history of this one is really hacky. So feel free to squash.

[AJenbo]

So how does this thing work? Do we need to run something afterward to make use of the test cases? How does a success look versus a fail?

[mewmew]

So how does this thing work? Do we need to run something afterward to make use of the test cases? How does a success look versus a fail?

The idea is we use gen_tests.sh to generate "golden" output for the DRLG algorithms. These correspond to the actual contents of the dungeon array (i.e. the tile ID map). Then, what we do is we compute the SHA1 hashsum of these outputs, and add them as expected output for the test cases.

$ sha1sum testdata/tiles_dlvl=1,quest_id=255,seed=123.bin
12a0410904ebf2507b6b7017f0ae191ae476686b  testdata/tiles_dlvl=1,quest_id=255,seed=123.bin
$ sha1sum testdata/tiles_dlvl=2,quest_id=6,seed=123.bin
659b95eec3e1c18d13b7f9932de108b88b356b9b  testdata/tiles_dlvl=2,quest_id=6,seed=123.bin

Now, we create a test case in C++ using Google test or something like that, and essentially set up the pre-condition (i.e. the state of global variables) before calling Createl5Dungeon. Then we compute the SHA1 hash of the result we got (i.e. the contents of the dungeon array). If the hashes match, we pass the test, if not we fail.

I played around with this and implemented a PoC in Go. For reference, see https://github.com/sanctuary/djavul/blob/e1ed5212bc6e67e0bdf6270f5da1bb474fe1bf97/d1/l1/l1_testxxx.go#L33, a simplified extract of which is provided below:

golden := []struct {
		dungeonName      string
		dlvl             int
		dtype            enum.DType
		questID          enum.QuestID
		seed             int32
		wantTiles        string
		wantDPieces      string
		wantArches       string
		wantTransparency string
	}{
		{
			dungeonName:      "Cathedral",
			dlvl:             1,
			dtype:            enum.DTypeCathedral,
			questID:          enum.QuestIDNone,
			seed:             123,
			wantTiles:        "12a0410904ebf2507b6b7017f0ae191ae476686b",
			wantDPieces:      "e15a7afb7505cb01b0b3d1befce5b8d4833ae1c6",
			wantArches:       "5438e3d7761025a2ee6f7fec155c840fc289f5dd",
			wantTransparency: "1269467cb381070f72bc6c8e69938e88da7e58cc",
		},
...
		{
			dungeonName:      "The Butcher",
			dlvl:             quests.QuestData[enum.QuestIDTheButcher].DLvlSingle,
			dtype:            enum.DTypeCathedral,
			questID:          enum.QuestIDTheButcher,
			seed:             123,
			wantTiles:        "659b95eec3e1c18d13b7f9932de108b88b356b9b",
			wantDPieces:      "15f2209ff5d066cfd568a1eab77e4328d08474e8",
			wantArches:       "42941df3ada356ebf87ce2987d26a06c44da711a",
			wantTransparency: "74c24e596ec57a91261bc3a559270f31d6811336",
		},
}

multi.MaxPlayers = 1 // single player.
pass := true
for _, g := range golden {
	// Establish pre-conditions.
	gendung.DLvl = g.dlvl
	gendung.DType = g.dtype
	for i := range quests.Quests {
		quests.Quests[i].QuestID = enum.QuestID(i)
		quests.Quests[i].QuestState = 0
	}
	gendung.IsSetLevel = false
	if g.questID != enum.QuestIDNone {
		quests.Quests[g.questID].QuestState = 1
		quests.Quests[g.questID].DLvl = g.dlvl
	}
	entry := int(0)
	CreateDungeon(uint32(g.seed), entry)
	if err := check(gendung.TileIDMap, "tiles", g.seed, g.wantTiles); err != nil {
		// FAIL
		pass = false
	}
}
if !pass {
	fmt.Println("test case failed")
} else {
	fmt.Println("test case passed")
}

Helper function to compute SHA1 hash:

// check validates the data against the given SHA1 hashsum.
func check(data interface{}, name string, seed int32, want string) error {
	buf := &bytes.Buffer{}
	if err := binary.Write(buf, binary.LittleEndian, data); err != nil {
		return errors.WithStack(err)
	}
	sum := sha1.Sum(buf.Bytes())
	got := fmt.Sprintf("%040x", sum[:])
	if got != want {
		return errors.Errorf("SHA1 hash mismatch for %v, seed 0x%08X; expected %q, got %q", name, seed, want, got)
	}
	return nil
}

[AJenbo]

Ok, so if I understand this correctly the goal is to provide a way for others to easily verify there important gains that of devilution, not to check devilution against vanilla?

We should take care not to use the SHA1bad implementation 😅

[mewmew]

Ok, so if I understand this correctly the goal is to provide a way for others to easily verify there important gains that of devilution, not to check devilution against vanilla?

Ideally, we would add this hook to the vanilla Diablo.exe executable, so we could get the ensured golden SHA1 hashes. Then adding test cases to Devilution would help ensure that we don't mess things up when doing future cleanups. Adding test cases to DevilutionX becomes especially important since DevilutionX seeks to do quite a few cleanups, any of which could change something subtle in how the DRLG works (e.g. signed vs. unsigned integers, 32- vs 64-bit integers, etc).

[AJenbo]

Ok, I will be looking forward for part two then :) Really exciting that this can also be used with vanilla Diablo.exe

[mewmew]

@AJenbo should we close this PR now that DevilutionX has integrated test cases?

[AJenbo]

My understanding was that this could be used to generate test data for the tests in DevilutionX. So I was thinking we keep this around until we have done so and implemented the tests for the dungeon generators (and verified it against original game exe). I'm currently seeing some issues with the Crypt levels in DevilutionX where save games appear to not match the original layout, but I haven't fully verified it yet.