d2l-en
d2l-en copied to clipboard
d2l-ai
Pin the dependencies in setup.py
The d2l library has various dependencies which are currently unpinned and install the latest (or collecting the cached) version of that dependency library. See below:
https://github.com/d2l-ai/d2l-en/blob/f742ee4b4d503187e6ced5dcc9ae54b955c7b0e4/setup.py#L4-L11
This leads to non-reproducible and unintentional bugs sometimes.
For example:
- This issue in section linear regression was not easy to debug since the code ran perfectly on CI, as can be seen in the preview plots for MXNet. CI apparently has an older version of NumPy and my environment has a newer
NumPy==1.22.2.
The same notebook with the newer version of NumPy raises the following error about np.asarray:
ValueError: setting an array element with a sequence. The requested array would exceed the maximum number of dimension of 32.
The np.asarray is used in matplotlib, see L1310) which then percolates down internally to d2l.plot function.
- Again we have a similar reason behind PR #1966 which was also not easy to debug. It works well with
numpy<=1.19.5but startingnumpy>=1.20it throws aTypeErrorwhich is actually expected if you try to cast a torch tensor on a GPU device to numpy directly without moving the tensor to CPU first.
Yes, they are easy to fix bugs but finding the reason behind the error becomes extremely tough when we have these inconsistent dependencies. If we fix the dependencies for the libraries, this will never be a problem or if the problem arises these will be easy to point out.
Pitch
I'll send two separate PRs to fix those two bugs one each in MXNet and PyTorch (#1966 is already up so it can be merged now that we know the reason) with the latest numpy version.
I'll then send a PR to pin these dependencies which we can update manually every 6 months or something like that.
All of this was caught during the current CI overhaul and actually needs fixing for updating the frameworks to their latest versions later.
cc @astonzhang @cheungdaven
