DOC: Remove unneeded dependencies from RTD environment

[mgeier]

BTW: you should enable this: https://docs.readthedocs.io/en/stable/guides/autobuild-docs-for-pull-requests.html

Then we could see if this actually works.

[mgeier]

This becomes obsolete if and when #90 is merged.