pre-trained embedding features

[bmcfee]

Description

pumpp features currently rely on low-level librosa implementations, but we could also have wrappers for pre-trained feature extractors like openl3 and vggish (the latter as implemented by openmic).

There's some details to work out in terms of standardizing the parameters (hop size, etc).

[tomxi]

Also crema... which I think is gonna be my use case.

Why would this involve an api change?

[bmcfee]

Absolutely. Though with crema, there's a circular dependency problem: crema depends on pumpp, so we can't have pumpp depend on crema.

Probably the best option here is to add a pumpp feature extractor class to each crema model. This should be fairly easy to do. Extending foreign objects (ie from another package) is generally a bad idea, but in this case, we're in control of both packages so it shouldn't be a huge deal.

[tomxi]

Caveat about wrapping pre-trained embedding in general: If the model contains things like custom Keras layers that can’t be pickled directly, then the resulting pump object won’t be picklable, which may come as a surprise to Pumpp users... This also means things like parallelization with pump that wraps an un-deserializeable model will break with hard to parse error messages...

[beasteers]

I like the idea of making CremaModel automatically usable as a pumpp. For openl3 and vggish and other models, I feel like it might make sense to have them in pumpp and just not imported by default.

import pumpp
from pumpp.feature.integrations import vggish, openl3, keras
pump = pumpp.Pump(vggish.VGGish(...), openl3.OpenL3(...), keras.H5Model(...))

If we do that, then we don't have to worry about circular dependencies so we could add Crema as well for consistency. Just a thought.

(Or alternatively we can have them imported by default and do:

from pumpp.feature.base import FeatureExtractor

class Crema(FeatureExtractor):
    def __init__(self, name='crema', model_dir='./model', *a, **kw):
        from crema.models.base import CremaModel
        self.model = CremaModel()
        self.model._instantiate(os.path.abspath(model_dir))
        super().__init__(name, *a, **kw)

    def transform_audio(self, y):
        return self.model.outputs(y=y, sr=self.sr)

) Not sure what the protocol for using models that are outside of the crema repo is tho.

[beasteers]

Actually maybe the best option would be to do what @bmcfee suggested (crema model extending Pump) and then add vggish and openl3 as crema models.