Support for DataFrames 22.0?

[rokapre]

Seems like this is not compatible with the latest DataFrames 22.0 version if you do add ShapML. I was able to download it from the github repository but it resulted in downgrading to DataFrames 21.8. There are quite a few changes from 21.8 to 22.0.

[nredell]

I just merged a pull request that addresses some (all?) of this. Doing some minor cleanup, testing, and will push a new version soon.

[rokapre]

Has it been pushed yet? I recently tried to add the package but the usual ] add ShapML didn't work as I got a bunch of dependency errors with CategoricalArrays, DataFrames etc. Then I added via the github and DataFrames still got downgraded

[casasgomezuribarri]

I have a similar problem when trying to add the package with ] add ShapML (ERROR: Unsatisfiable requirements detected for package CategoricalArrays [324d7699]:..., see below). It is a long error message mentioning several packages from the MLJ family (MLJ, MLJBase, MLJTuning, MLJMultivariateStatsInterface... etc), as well as others like Distributions, Tables and DataFrames. However, DataFrames is the only one in conflict directly with ShapML (see near the end of the error message).

I assume my problem is related, if not the same one as the poster of this issue.

ERROR: LoadError: Unsatisfiable requirements detected for package CategoricalArrays [324d7699]:

> ERROR: LoadError: Unsatisfiable requirements detected for package CategoricalArrays [324d7699]:
>  CategoricalArrays [324d7699] log:
>  ├─possible versions are: [0.3.11, 0.3.13-0.3.14, 0.4.0, 0.5.0-0.5.5, 0.6.0, 0.7.0-0.7.7, 0.8.0-0.8.3, 0.9.0] or uninstalled
>  ├─restricted by compatibility requirements with MLJ [add582a8] to versions: [0.3.11, 0.3.13-0.3.14, 0.4.0, 0.5.0-0.5.5, 0.6.0, 0.7.0-0.7.7, 0.8.0-0.8.3, 0.9.0]
>  │ └─MLJ [add582a8] log:
>  │   ├─possible versions are: [0.1.0-0.1.1, 0.2.0-0.2.5, 0.3.0, 0.4.0, 0.5.0-0.5.9, 0.6.0-0.6.1, 0.7.0, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2] or uninstalled   
>  │   ├─restricted to versions * by an explicit requirement, leaving only versions [0.1.0-0.1.1, 0.2.0-0.2.5, 0.3.0, 0.4.0, 0.5.0-0.5.9, 0.6.0-0.6.1, 0.7.0, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2]
>  │   ├─restricted by compatibility requirements with MLJBase [a7f614a8] to versions: [0.7.0, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2] or uninstalled, leaving only versions: [0.7.0, 0.8.0, 0.9.0-0.9.3, 0.10.0-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2]
>  │   │ └─MLJBase [a7f614a8] log:
>  │   │   ├─possible versions are: [0.1.0-0.1.1, 0.2.0-0.2.6, 0.3.0, 0.4.0, 0.5.0, 0.6.0, 0.7.0-0.7.5, 0.8.0-0.8.4, 0.9.0-0.9.2, 0.10.0-0.10.1, 0.11.0-0.11.10, 0.12.0-0.12.8, 0.13.0-0.13.10, 0.14.0-0.14.9, 0.15.0-0.15.7, 0.16.0-0.16.3, 0.17.0-0.17.1] or uninstalled
>  │   │   ├─restricted by compatibility requirements with MLJTuning [03970b2e] to versions: [0.10.1, 0.11.0-0.11.10, 0.12.0-0.12.8, 0.13.0-0.13.10, 0.14.0-0.14.9, 0.15.0-0.15.7, 0.16.0-0.16.3, 0.17.0-0.17.1]
>  │   │   │ └─MLJTuning [03970b2e] log:
>  │   │   │   ├─possible versions are: [0.1.0-0.1.4, 0.2.0, 0.3.0-0.3.6, 0.4.0-0.4.3, 0.5.0-0.5.5, 0.6.0-0.6.1] or uninstalled
>  │   │   │   ├─restricted to versions * by an explicit requirement, leaving only versions [0.1.0-0.1.4, 0.2.0, 0.3.0-0.3.6, 0.4.0-0.4.3, 0.5.0-0.5.5, 0.6.0-0.6.1]
>  │   │   │   └─restricted by compatibility requirements with MLJBase [a7f614a8] to versions: [0.1.0-0.1.4, 0.2.0, 0.3.0-0.3.6, 0.4.0-0.4.3, 0.5.0-0.5.5, 0.6.0] or uninstalled, leaving only versions: [0.1.0-0.1.4, 0.2.0, 0.3.0-0.3.6, 0.4.0-0.4.3, 0.5.0-0.5.5, 0.6.0]
>  │   │   │     └─MLJBase [a7f614a8] log: see above
>  │   │   ├─restricted by compatibility requirements with MLJ [add582a8] to versions: [0.1.0-0.1.1, 0.2.0-0.2.6, 
> 0.4.0, 0.5.0, 0.6.0, 0.7.0-0.7.5, 0.8.1-0.8.4, 0.9.1-0.9.2, 0.10.0-0.10.1, 0.11.0-0.11.10, 0.12.0-0.12.8, 0.13.1-0.13.10, 0.14.0-0.14.9, 0.15.1-0.15.7, 0.16.0-0.16.3], leaving only versions: [0.10.1, 0.11.0-0.11.10, 0.12.0-0.12.8, 0.13.1-0.13.10, 0.14.0-0.14.9, 0.15.1-0.15.7, 0.16.0-0.16.3]
>  │   │   │ └─MLJ [add582a8] log: see above
>  │   │   └─restricted by compatibility requirements with Distributions [31c24e10] to versions: [0.12.4-0.12.8, 0.13.0-0.13.10, 0.14.0-0.14.9, 0.15.0-0.15.7, 0.16.0-0.16.3, 0.17.0-0.17.1] or uninstalled, leaving only versions: [0.12.4-0.12.8, 0.13.1-0.13.10, 0.14.0-0.14.9, 0.15.1-0.15.7, 0.16.0-0.16.3]
>  │   │     └─Distributions [31c24e10] log:
>  │   │       ├─possible versions are: [0.16.0-0.16.4, 0.17.0, 0.18.0, 0.19.1-0.19.2, 0.20.0, 0.21.0-0.21.3, 0.21.5-0.21.12, 0.22.0-0.22.6, 0.23.0-0.23.12, 0.24.0-0.24.12] or uninstalled
>  │   │       └─restricted by compatibility requirements with MLJGLMInterface [caf8df21] to versions: [0.23.0-0.23.12, 0.24.0-0.24.12]
>  │   │         └─MLJGLMInterface [caf8df21] log:
>  │   │           ├─possible versions are: 0.1.0-0.1.3 or uninstalled
>  │   │           └─restricted to versions * by an explicit requirement, leaving only versions 0.1.0-0.1.3       
>  │   ├─restricted by compatibility requirements with Distributions [31c24e10] to versions: [0.10.2-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2] or uninstalled, leaving only versions: [0.10.2-0.10.3, 0.11.0-0.11.6, 0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2]
>  │   │ └─Distributions [31c24e10] log: see above
>  │   └─restricted by compatibility requirements with MLJModels [d491faf4] to versions: [0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2] or uninstalled, leaving only versions: [0.12.0-0.12.1, 0.13.0, 0.14.0-0.14.1, 0.15.0-0.15.2]
>  │     └─MLJModels [d491faf4] log:
>  │       ├─possible versions are: [0.1.0-0.1.1, 0.2.0-0.2.5, 0.3.0, 0.4.0, 0.5.0-0.5.9, 0.6.0-0.6.3, 0.7.0-0.7.2, 0.8.0-0.8.4, 0.9.0-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.0-0.12.11, 0.13.0-0.13.3] or uninstalled        
>  │       ├─restricted by compatibility requirements with MLJ [add582a8] to versions: [0.1.0-0.1.1, 0.2.0-0.2.5, 
> 0.3.0, 0.4.0, 0.5.0-0.5.9, 0.6.0-0.6.3, 0.7.0-0.7.2, 0.8.0-0.8.4, 0.9.0-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.1-0.12.11, 0.13.0-0.13.3]
>  │       │ └─MLJ [add582a8] log: see above
>  │       ├─restricted by compatibility requirements with MLJBase [a7f614a8] to versions: [0.7.0-0.7.2, 0.8.0-0.8.4, 0.9.0-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.0-0.12.11, 0.13.0-0.13.3] or uninstalled, leaving only versions: [0.7.0-0.7.2, 0.8.0-0.8.4, 0.9.0-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.1-0.12.11, 0.13.0-0.13.3]     
>  │       │ └─MLJBase [a7f614a8] log: see above
>  │       ├─restricted by compatibility requirements with Distributions [31c24e10] to versions: [0.9.1-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.0-0.12.11, 0.13.0-0.13.3] or uninstalled, leaving only versions: [0.9.1-0.9.12, 0.10.0-0.10.1, 0.11.0-0.11.3, 0.12.1-0.12.11, 0.13.0-0.13.3]
>  │       │ └─Distributions [31c24e10] log: see above
>  │       └─restricted by compatibility requirements with Distances [b4f34e82] to versions: [0.11.0-0.11.3, 0.12.0-0.12.11, 0.13.0-0.13.3] or uninstalled, leaving only versions: [0.11.0-0.11.3, 0.12.1-0.12.11, 0.13.0-0.13.3] 
>  │         └─Distances [b4f34e82] log:
>  │           ├─possible versions are: [0.7.0-0.7.4, 0.8.0-0.8.2, 0.9.0-0.9.2, 0.10.0-0.10.2] or uninstalled     
>  │           └─restricted by compatibility requirements with MLJMultivariateStatsInterface [1b6a4a23] to versions: [0.9.0-0.9.2, 0.10.0-0.10.2]
>  │             └─MLJMultivariateStatsInterface [1b6a4a23] log:
>  │               ├─possible versions are: 0.1.0-0.1.7 or uninstalled
>  │               └─restricted to versions * by an explicit requirement, leaving only versions 0.1.0-0.1.7       
>  ├─restricted by compatibility requirements with MLJBase [a7f614a8] to versions: [0.7.3-0.7.7, 0.8.0-0.8.3, 0.9.0]
>  │ └─MLJBase [a7f614a8] log: see above
>  ├─restricted by compatibility requirements with DataFrames [a93c6f00] to versions: 0.7.0-0.7.7, leaving only versions: 0.7.3-0.7.7
>  │ └─DataFrames [a93c6f00] log:
>  │   ├─possible versions are: [0.11.7, 0.12.0, 0.13.0-0.13.1, 0.14.0-0.14.1, 0.15.0-0.15.2, 0.16.0, 0.17.0-0.17.1, 0.18.0-0.18.4, 0.19.0-0.19.4, 0.20.0-0.20.2, 0.21.0-0.21.8, 0.22.0-0.22.5] or uninstalled
>  │   ├─restricted by compatibility requirements with ShapML [8523bd24] to versions: 0.20.0-0.20.2
>  │   │ └─ShapML [8523bd24] log:
>  │   │   ├─possible versions are: [0.1.0, 0.2.0, 0.3.0] or uninstalled
>  │   │   └─restricted to versions * by an explicit requirement, leaving only versions [0.1.0, 0.2.0, 0.3.0]     
>  │   └─restricted by compatibility requirements with Tables [bd369af6] to versions: [0.11.7, 0.12.0, 0.13.0-0.13.1, 0.20.1-0.20.2, 0.21.0-0.21.8, 0.22.0-0.22.5] or uninstalled, leaving only versions: 0.20.1-0.20.2
>  │     └─Tables [bd369af6] log:
>  │       ├─possible versions are: [0.1.0-0.1.15, 0.1.17-0.1.19, 0.2.0-0.2.11, 1.0.0-1.0.5, 1.1.0, 1.2.0-1.2.2, 1.3.0-1.3.1] or uninstalled
>  │       └─restricted by compatibility requirements with MLJGLMInterface [caf8df21] to versions: [1.1.0, 1.2.0-1.2.2, 1.3.0-1.3.1]
>  │         └─MLJGLMInterface [caf8df21] log: see above
>  └─restricted by compatibility requirements with MLJModels [d491faf4] to versions: [0.8.0-0.8.3, 0.9.0] — no versions left
>    └─MLJModels [d491faf4] log: see above

[4SAnalyticsnModelling]

Try adding ShapML as ]add "https://github.com/nredell/ShapML.jl". Add all the other packages before adding ShapML.jl so that the package can downgrade the required packages as needed. That's a work around working for me so far. I have also been able to work with it by installing ShapML in a separate environment, and all other ML packages in the base environment. In all cases I added the ShapML.jl with the git repo URL as mentioned above.

[casasgomezuribarri]

Thanks for your help. I got it working on the same environment with ]add ShapML#master, which I assume triggers the same behaviour?

[4SAnalyticsnModelling]

Yes, it does, unfortunately. It will still downgrade some of the packages including DataFrames. But that's the only work around I have for now until nredell pushes the updated version of ShapML. Thank you!

[nredell]

I'm a bit hit or miss on GitHub nowadays--maybe slowly getting back into it, though--, but now seems like a good time to actually fix this. The newest version of DataFrames has broken a lot more than the join functions.

Creating a data frame from a matrix is deprecated--which occurs in several spots--, and column names are getting lost somewhere along the way. A unit test has been failing for exactly this reason since the last pull request.

In progress.

[4SAnalyticsnModelling]

Thanks for looking into it! Yes you are right, things are breaking at several places. I have tried using DataFrames.combine and DataFrames.groupby functions in predict.jl instead of the custom _aggregate function. That passed the test with the most updated version of DataFrames.

[4SAnalyticsnModelling]

@nredell I have created 2 pull requests on proposed changes in predict.jl and aggregate.jl. Those changes gave me clean run without any deprecation warning. All tests were passed with the proposed changes. Please have a look at those proposed changes.

[4SAnalyticsnModelling]

@nredell with those proposed changes you can bump up the DataFrames.jl compatibility to its most updated version.

[pdeffebach]

Bumping this, as now we are at DataFrames 1.0. Let me know if you need help with a PR to get support 1.0.