"wpt manifest" should be able to update manifest for directories of interest only

[WeizhongX]

When run WPTs locally, we (at least at google) always update the WPT Manifest to make sure the test is run correctly (e.g. to make sure Wptrunner view the the test type correctly). Updating WPT manifest is slow at this point because "wpt manifest" does not allow a partial update. On Linux it takes about 4 seconds to update WPT manifest, and would be even slower on Windows.

Suggestions is to allow "wpt manifest" to update for the file or directories of interest. For example, the command "git ls-tree -r -z HEAD" takes about 1.5 seconds on Linux, yet "git ls-tree -r -z HEAD badging/badge-success.https.html" takes about 0.01 seconds. There would be other savings also, and we believe the total time for this can be made to be < 0.1 second for running a single test, which is the usual case.

WPT Manifest holds information about "test type", references for reftests, so updating infor for the test file will be good enough.

[WeizhongX]

Also we'd like to have a way to skip MtimeCache, or preferably remove that altogether. (MtimeCache would only be useful for running tests locally, but with the proposal above it should not be needed anymore.)

[WeizhongX]

@jgraham @gsnedders What would be your opinion on this?

[jgraham]

Note that this isn't totally trivial to get right, for example if you pass in /foo/bar/test.any.html then you have to convert that into a path, which you can't do reliably without the manifest. I think in that case the best reliable method is to take the entire directory and use that i.e. {wpt_root}/foo/bar. For reftests we may also end up looking outside the directory for references.

What's the concern with the mtime cache? Performance or correctness? Since gecko is still using Mercurial, and has some workflows that don't use VCS checkouts at all, having some caching layer that doesn't depend on the source being in a git repo continues to be important to us.

[WeizhongX]

for example if you pass in /foo/bar/test.any.html then you have to convert that into a path, which you can't do reliably without the manifest

My plan is to pass in a path relative to the tests-root, with all other parameters still kept. Do you see a risk with that?

For reftests we may also end up looking outside the directory for references.

I think this is fine.

What's the concern with the mtime cache?

I thought it is too complex and unnecessary to keep all kinds of caches in this process. But we can keep it if it is still used at gecko side.

[gsnedders]

FWIW: this was a goal when I rewrote all this code a number of years ago, it just never actually got done. This is part of why to_update is a list, along with parallelising it.

Note that this isn't totally trivial to get right, for example if you pass in /foo/bar/test.any.html then you have to convert that into a path, which you can't do reliably without the manifest. I think in that case the best reliable method is to take the entire directory and use that i.e. {wpt_root}/foo/bar.

Yes: in the case of being given a URL, you need to re-generate the entire directory, determining the appropriate manifest and directory to update based on the url_base of loaded manifests.

For reftests we may also end up looking outside the directory for references.

That doesn't have any consequence, because since we got rid of the complex reftest graph stuff in #20459. Nothing about the reference (even its existence!) affects how the test is run.