Write test cases

[Frederick888]

Maybe we could publish some dummy crates so that some proper test cases can be written.

And actually I am still a bit unsure about the behaviour of cargo update in some scenarios, e.g. how it deals with yanked dependencies.

[leoschwarz]

It would probably be better to have some dummy crates for unit tests.

But an alternative for regression tests which would also enable to keep track of future cargo additions is to simply download a large number of Cargo.toml files from crates.io and then run cargo-outdated pre and post commit (compile two binaries) on all of these files.

Then we could collect stats with a script, I would propose (A pre-commit, B post-commit):

• Sucess matrix:
  • number of: A no error, B no error (good)
  • number of: A error, B no error (great)
  • number of: A no error, B error (bad)
  • number of: A error, B error (bad)
• Output diff (only for cases where A no error, B no error):
  • number of: equal output
  • number of: different output
  • output diffs in a directory for the cases where there was a difference
• Output of B errors

I guess this could be used for other cargo tools, like cargo-geiger, cargo-tree, etc. (Maybe someone already has such a tool. I think crater is not really suitable since it has a different functionality, but maybe it could be adapted to safe time.)