Restrict Checks for Open/MMAPed Files - CLI & Checksum

[AnorexicAtticusFinch]

This adds CLI options and the checksum file validation method in addition to the previous build-ID and file size checks as part of GSoC 2020.

CRIU uses the build-ID validation method by default whenever possible and if that cannot be done, it will try and use the checksum method with the first 1 MB of the file. If this fails as well, it will use only the file size check.

To use the checksum method first and make the build-ID method the fallback: --file-validation="chksm" --chksm-parameter=[N] (To calculate checksum with the first N bytes of the file, at most 10 MB)

--file-validation="chksm-full" (To calculate checksum with the entire file, at most 10 MB)

--file-validation="chksm-period" --chksm-parameter=[N] (To calculate checksum with every Nth byte of the file)

To explicitly use only the file size check all the time, the following command line option can be used: --file-validation="filesize" In other words, --file-validation="buildid" will not make a difference as this is the default method

Using the time command to measure the time taken for 2 tests. zdtm/transition/shmem: file size: 3.782s build-id: 4.153s chksum (Parameter = 1024): 4.465s chksum-full: 4.722s chksum-period (Parameter = 1024): 4.498s

zdtm/static/maps04: file size: 35.317s build-id: 35.720s chksum (Parameter = 1024): 35.919s chksum-full: 36.679s chksum-period (Parameter = 1024): 36.476s

Mentors: @avagin @0x7f454c46

This also adds ZDTM tests for build-ID and checksum validation methods.

Link to the previous PR: https://github.com/checkpoint-restore/criu/pull/1131

Changes from the previous PR: rebase on the current criu-dev cosmetic changes other minor changes

[AnorexicAtticusFinch]

@avagin @0x7f454c46 What if the user chooses to checksum the entire file and the files turn out to be huge? Without some kind of limit, it would take a very long time to calculate the checksum using all the bytes.

[AnorexicAtticusFinch]

Looks good. But lacks zdtm tests.

I plan on pushing zdtm tests for build-id and checksum through a separate PR since I have some doubts regarding them.

[Snorch]

Looks good. But lacks zdtm tests.

I plan on pushing zdtm tests for build-id and checksum through a separate PR since I have some doubts regarding them.

ZDTM tests are not only intended to test that you functionality will not break in future. But also to help review your functionality, so it should go along with your functionality. This way everybody can see (by running test on their own and experimenting with them) that you functionality works. We skip tests in some rare cases, but in this case I think we should have them.

@avagin Please correct me if I'm wrong.

[AnorexicAtticusFinch]

ZDTM tests are not only intended to test that you functionality will not break in future. But also to help review your functionality, so it should go along with your functionality. This way everybody can see (by running test on their own and experimenting with them) that you functionality works. We skip tests in some rare cases, but in this case I think we should have them.

Oh, in that case I've pushed the tests but they're not complete. In the tests that should fail during restore, once restore fails the test is automatically considered a failure. The crfail flag is only for tests that should fail during dumping, I couldn't find any such flag for failure during restore. I'm not sure how to proceed from here.

Since the tests aren't complete, should I mark this PR as a draft?

[Snorch]

The crfail flag is only for tests that should fail during dumping,

Maybe it's time to add new flag for fail on dump expected then.

[AnorexicAtticusFinch]

@avagin @0x7f454c46 The added tests ran and passed on Travis yet this is the result:

################### 14 TEST(S) FAILED (TOTAL 408/SKIPPED 35) ###################
 * zdtm/static/file-validation-chksm05.c(h)
 * zdtm/static/file-validation-buildid03.c(h)
 * zdtm/static/file-validation-chksm02.desc(h)
 * zdtm/static/file-validation-chksm03.c(h)
 * zdtm/static/file-validation-buildid02.c(h)
 * zdtm/static/file-validation-buildid00.c(h)
 * zdtm/static/file-validation-buildid01.c(h)
 * zdtm/static/file-validation-chksm02.c(h)
 * zdtm/static/file-validation-chksm01.c(h)
 * zdtm/static/file-validation-chksm01.desc(h)
 * zdtm/static/file-validation-chksm04.desc(h)
 * zdtm/static/file-validation-chksm04.c(h)
 * zdtm/static/file-validation-chksm03.desc(h)
 * zdtm/static/file-validation-chksm00.c(h)
##################################### FAIL #####################################

Have I missed something? Why is this happening?

[0x7f454c46]

Probably, some of the files you've added have executable bit set?

[AnorexicAtticusFinch]

Probably, some of the files you've added have executable bit set?

Yeah, that was the reason. I've re-uploaded the files. Thanks!

[avagin]

Can you move file validation helpers in a separate file? files-reg.c is too big already.

[AnorexicAtticusFinch]

Can you move file validation helpers in a separate file? files-reg.c is too big already.

Yeah, I've moved the functions to files-validation.c.

[github-actions[bot]]

A friendly reminder that this PR had no activity for 30 days.

[xiongzubiao]

If I understand correctly, this change is not merged in. Anyone knows why? CRIU's wiki has it though... https://criu.org/index.php?title=Validate_files_on_restore

[Snorch]

@xiongzubiao @avagin @checkpoint-restore/maintainers

The PR has unresolved conversation https://github.com/checkpoint-restore/criu/pull/1148#discussion_r472275419 about "how fast is the algorithm?", it seem to only work fast for files with BUILDID, checksumming is not so fast obviously.

So it seems we need to update wiki that this feature is not available.