Competences and selections, create example or a simpler API call

[eddiebergman]

Hello,

I'm using deslib as part of benchmark paper comparing classifier selection by AutoML tools vs MCS tools for which this library is hugely useful. However, I need to get the competences and selections by both DCS and DES methods in deslib. There seems to be no examples for what is exposed as public API features in estimate_competence_from_proba(query, neighbors, probabilities, distances=None) and selection(competences).

Which of these would be the better way to get the competences and selections:

• Do I use a non-listed API methods of the model to generate the neighbors, probabilities and distances. This would help ensure all values are the same as the DCS/DES model would generate but I am unsure which methods to call for these. My gut however says no if neighbors, probabilities have no defaults parameters.
• Use sklearn's knn to generate neighbors and distances, manually getting to probabilities of the models. This is a lot easier as a non-experienced user of deslib but could lead to inconsistencies with the DCS/DES model.

I understand it might be low in the priority but an example or a simplified API would be hugely helpful. If you can point me in the right direction I would be happy to write a pull request with an example or usability improvement!

[eddiebergman]

For anyone who is looking for the same:

distances, neighbors = des_model._get_region_competence(X_test)
classifiers_predicted_probabilities = des_model._predict_proba_base(X_test)

competences = des_model.estimate_competence_from_proba(query=X_test,
                                                       neighbors=neighbors,
                                                       probabilities=classifiers_predicted_probabilities,
                                                       distances=distances)
selections = des_model.selections(competences)

I haven't looked through all the different models that inherit from BaseDS but as most (if not all) model types derive from this class, and both _get_region_competence() and _predict_proba_base are methods of BaseDS, the API could be made more user friendly by calling these methods itself, rather than having the user find them. I am happy to help with pull requests if you think it could help the library.

[Menelau]

Hello,

Yes, all methods derive from the BaseDS class. I agree with you, the API needs to be improved to be easier to use and allow other users to quickly build new DS techniques by using the basic API methods (_get_region_competence, estimate_competence, etc). My idea is to in the future have those functions exposed and easier to use in a way other users can easily use them to build new techniques. If you can help with that it will be much appreciated. It will definitely help the library and I really believe that having input from users with different needs will help in improving the API.

I also agree that having an example showing how to use those API methods is necessary. But I believe that would be something that we do after refactoring the code.

[eddiebergman]

I'll try have a go at refactoring some of the API calls of the derived models to simplify them and update the documentation, later this week or early next week :)

[Menelau]

Great! This will be a good contribution to the library.

If the changes are huge it may be the case to divide the work into multiple smaller pull requests rather than a single one.