add auto_series_param and custom_series_param

[dkrako]

The current state of the package only supports 2d image-type input. Theorywise, it shouldn't be a problem to apply the functionality to time series inputs.

This PR adds the class AutoSeriesParam, which is designed according to AutoImageParam, but supports sequences instead of images. This way 1d-support can be integrated neatly without changing any existing code.

It is therefore needed to to setup the dreamer manually this way:

    series_param = AutoSeriesParam(
        length=sequence_length,
        channels=channels,
        standard_deviation=0.01,
        batch_size=batch_size,
        device="cpu",
    )

    series_transforms = transforms.Compose(
        [
            RandomSeriesTranslate(0.1),
            RandomSeriesScale(0.5, 1.2),
        ]
    )

    dreamy_boi = Dreamer(model=model, device='cpu', quiet=False)
    dreamy_boi.set_custom_transforms(series_transforms)

    result = dreamy_boi.render(
        layers=[model.conv1],
        iters=10,
        image_parameter=series_param,
    )

Maybe the series transforms could be setup by default in the dreamer, if the image_parameter is an instance of BaseSeriesParam?

There are still some issues left where I am unsure what to do:

1. The AutoImageParam class has a normalize() method to normalize the output. It uses imagenet mean and std to apply z-score standardization. There's no such equivalent in time series data due to their heterogeneous nature so we need a way to feed in the mean and std values. The Dreamer class has a set_custom_normalization() method for this use, but the InverseTransform relies on 4d-tensors. I'm not sure how to integrate time series support into this.

2. The AutoImageParam.postprocess() method applies the function lucid_colorspace_to_rgb(). I don't really understand what it does and I'm clueless on how to treat this in the time series domain.

3. I left out the implementation of a CustomSeriesParam class for now, but that should come in a follow-up PR.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 83.09859% with 36 lines in your changes are missing coverage. Please review.

Project coverage is 83.23%. Comparing base (bdaa9e3) to head (545f28b). Report is 11 commits behind head on master.

Files	Patch %	Lines
torch_dreams/series_transforms.py	66.12%	21 Missing :warning:
torch_dreams/auto_series_param.py	76.19%	15 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #58      +/-   ##
==========================================
- Coverage   91.70%   83.23%   -8.47%     
==========================================
  Files          19       25       +6     
  Lines         747     1038     +291     
==========================================
+ Hits          685      864     +179     
- Misses         62      174     +112     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[Mayukhdeb]

Hello @dkrako, Thanks for the PR 🙂 Highly appreciate the detailed notes as well! I will take a look at this and get back to you ASAP. Thanks again!

[dkrako]

I added a CustomSeriesParam class, but I have some issues with the resulting tensor shapes after calling rfft. They are only half the required length and I'm a bit clueless what to do.

[Mayukhdeb]

Hello @dkrako,

Could you please provide more details on the exact use-case for these new components to help me better understand the PR? A Colab notebook or a reproducible snippet would greatly assist in this.

I am eager to merge it as soon as I have a clear understanding of the use-case(s), which will allow me to update the documentation in the README accordingly.

Thank you!

[Mayukhdeb]

@dkrako what is the shape of the tensor that your model expects for time series images? If you can describe the problem in-detail, I can help you debug this on your branch👀

[dkrako]

Thanks a lot!

The time series I am referring to are not really 2d-images but multi-channel 1d-series. I myself am working mostly with eye tracking data, where you can have 2 channels (horizontal and vertical component of the eye gaze) but there is a range of different applications you can imagine (EEG data, seismographic data, acceleration sensors, gyroscopes, climate readings for a specific location).

Regardless of the application, the expected data layout in pytorch is NCL (tensorflow expects NLC), where N is the batch dimension, C the channel dimension and L the length (time) dimension. A series with the shape (64, 4, 1000) has a batch size of 64, 4 channels and 1000 time steps.

Analogous to the well known 2d convolution using the pytorch Conv2d class, we use the Conv1d equivalent in the time series domain. For applying the method of feature visualization which torch-dreams is applying, we wouldn't generate any images (NCHW) but time series (NCL).

I'll provide a notebook with a general use-case ASAP.

I will use this single channel toy dataset here for the example: https://github.com/greydanus/mnist1d

[dkrako]

here is an example colab notebook I have just created: https://colab.research.google.com/drive/152dNe02_MzSzCxQwAfXY7rtvql9NfPKh?usp=sharing

It loads and plots some data, trains a simple 1d-cnn. the AutoSeriesParam renders the image successfully, but the CustomSeriesParam has some shape errors. It think the error is somewhere with using rfft wrong but I don't know how to fix this.

[dkrako]

Do you mind giving me permission to run test workflows without further approval, please? I'd like to increase test coverage but currently you need to approve every single workflow run so I have to run everything locally.

[dkrako]

The failing tests are the same error as the one in the colab notebook (CustomSeriesParam).