clean doesn't remove all empty dirs

[gleich]

Description

clean doesn't remove some root folders such as the user folder and the folder type

Steps to reproduce

just clone a repo and try to remove it. It is hard to understand why exactly this problem is happening without taking a look at the code.

Logs/Screenshots

N/A

[gleich]

This is caused by go-git/go-git#312