skorch
skorch copied to clipboard
skorch-dev
compatibility with ignite metrics
If possible, add a scoring callback class that accepts an ignite metric as scoring function (see here).
I completely forgot about this very old issue and I haven't kept up to date with the ignite development. I think one big advantage over sklearn metrics, which we already support, would be that ignite metrics are calculated on torch tensors, so they can for instance take advantage of GPU acceleration. Therefore, this could still be worth trying.
Maybe as a first step, you could try a simple metric like accuracy and create a prototype (no docs or tests necessary for that). If it looks like it could work, we can iterate from there without wasting too much time in case it's not feasible.
Hi @BenjaminBossan, I'm looking for contributing guide for skorch but can't find. Can you please point out?
We don't have a separate contribution guide. But honestly, there isn't much to it. In the installation instructions, there is a section on how to proceed if you want to contribute. So follow this, run pytest and pylint on your newly added code and you should be good.
Given your recent PR, I assume that you're familiar with the general approach of contributing to an open source project. I.e. you know how to fork and create a PR, you know that you should stick close to the style of the existing code, etc. If you don't, just ask.
For this specific problem, I would add that:
- API-wise, it would be convenient for users if it were similar to that of
EpochScoring, though the implementation will likely be very different. - I think it makes sense to start a new module for ignite metrics inside the
skorch/callbacks/directory. Similarly, there should be a new test module, but that's for later.
The rest we can figure out on the go, which is why it's good to start with a proof of concept.