Tests pass when run one test file, when running all tests, get an assertion error

[j7zAhU]

The breaking factor seems to be the last test in my file, where I serialize a dataframe :

string = df.to_string()
len(string)   # 2_000
assert string == snapshot

When I run pytest at the root of the test dir, I get an assertion error and the terminal displays the chunk of text.

When I selectively leave out tests, I can get this one to work. So it's not that particular serialization.

[noahnu]

Can you provide any more information, perhaps a repo where I can reproduce the issue? Or even a complete snippet of code I can copy and paste into a project to run?

Considering we have tests which cover multiple assertions and selectively running test files, from your description it sounds like it should work.

[j7zAhU]

Sure. I'll try to put something together

[noahnu]

I get an assertion error and the terminal displays the chunk of text

It's telling you what changed. This is likely a real test failure, not a bug of any sort. The reason you're getting a whole chunk of text, is likely due to the fact that you're converting your Dict to a string before snapshotting it. That's not necessary, and means you lose out on ambr formatting, and more useful diffs.

So instead of:

def test_case(snapshot):
    output_dir_structure = { key: "value" }
    assert json.dumps(outdir_dir_structure) == snapshot

do:

def test_case(snapshot):
    output_dir_structure = { key: "value" }
    assert outdir_dir_structure == snapshot

If the issue is not obvious after switching away from a string, you'll need to step through the 2 tests which conflict. You likely have some sort of test pollution, where one test is affecting the other, whether it's the use of an auto-incrementer, some shared state, or improper "cleanup".

I also noticed you're snapshotting some file system paths. That's "non-deterministic" and means your snapshot will fail if run on any other machine, such as for another developer or on CI. In this case, you're interested in syrupy matchers: https://github.com/tophat/syrupy#matcher

[j7zAhU]

Thank you. I have indeed been using some shared state accidentally. That is fixed now.

Regarding the matcher: I did run into the problem of non-deterministic file-paths, because pytest gives you a unique numbered directory like this: tmp_path = C:\Users\X\AppData\Local\Temp\pytest-of-X\pytest-684\test_sample0 I get around that by splitting the filepath string but your method is cleaner.

If I am building a test path such as my_path = tmp_path / "my_subdir", how exactly can I ignore everything to the left of the 684?

Regards

[noahnu]

Sorry about the delay, it's been busy on my end.

Hmm, support for a regex matcher would be useful here. I have a local branch with a proof of concept, when I can find some time, I'll polish it and get that merged in.

In the meantime, it's pretty much all or nothing with a value. You can either filter out the specific key, or use a matcher to expect any arbitrary string there. No way to match partial text in a string. As a temporary workaround, I'd recommend using a matcher that expects a string type, and then if you need to assert some content in the string, you do that outside of the snapshot assertion.

[j7zAhU]

thank you

[iamogbz]

No way to match partial text in a string

This functionality can be replicated with the path_value matcher

def test_matches_generated_string_value(snapshot, tmp_file_path):
    my_matcher = path_value(
        {".*": r"\w+://(.*/)+dir/filename.txt"}, # match all paths against the regex value
        types=(str,), # the types to check against the value regex
        replacer=lambda data, match: match[0].replace(match[1], "<tmp-file-path>/"),
        regex=True, # treat patterns as regex instead of plain string comparisons
    )

    assert tmp_file_path == snapshot(matcher=my_matcher)

In the sample some things are always constant i.e. dir/filename.txt, the rest can be swapped so the snapshot is only matching on the partial expected path.

Demo

______________________ test_matches_generated_string_value ______________________

snapshot = SnapshotAssertion(name='snapshot', num_executions=1)
tmp_file_path = 'scheme://dxoq/vraja/dir/filename.txt'

    def test_matches_generated_string_value(snapshot, tmp_file_path):
        my_matcher = path_value(
            {".*": r"\w+://(.*/)+dir/filename.txt"},
            types=(str,),
            replacer=lambda data, match: match[0].replace(match[1], "<tmp-file-path>/"),
            regex=True,
        )
    
>       assert tmp_file_path == snapshot(matcher=my_matcher)
E       AssertionError: assert [+ received] == [- snapshot]
E         Snapshot 'test_matches_generated_string_value' does not exist!
E         + 'scheme://<tmp-file-path>/dir/filename.txt'

tests/syrupy/extensions/amber/test_amber_matchers.py:124: AssertionError
---------------------------- snapshot report summary ----------------------------
1 snapshot failed.
============================ short test summary info ============================
FAILED tests/syrupy/extensions/amber/test_amber_matchers.py::test_matches_generated_string_value - AssertionError: assert [+ received] == [- snapshot]
======================= 1 failed, 258 deselected in 0.13s =======================

[noahnu]

We can probably add a simple helper for this then

[iamogbz]

Updated example with use regex enabled matcher path_value.

Feel free to reopen if this needs a follow up.