Remove test vectors submodule

[Stebalien]

When cargo fetches a dependency with git:

1. It'll fetch all submodules.
2. It won't cache them.

This means we'll re-download the test vectors every time.

Unfortunately, I really don't know the right solution here. Cloning the test vectors on-demand would work, but that's still kind of annoying.

See https://github.com/rust-lang/cargo/issues/4247