hls4ml
hls4ml copied to clipboard
fastmachinelearning
Add Support for ConvTranspose Layers (1D and 2D)
Description
This adds support for ConvTranspose layers. Specifically, it adds support for both io_stream and io_parallel compilation of Conv1DTranspose and Conv2DTranspose (as of now, only converted from keras).
The strategy roughly follows that of non-transposed convolution layers. We treat a conv transpose as a group of stride_width by stride_height convolutions, with their outputs interlaced. Thus, we essentially do a normal conv implementation where each kernel produces stride_width * stride_height outputs. Perhaps the most unintuitive part of how things are currently set up is that the weight matrix is transformed substantially (in the python code). This is done to split up the kernel into what amounts to the stride_width * stride_height smaller kernels.
The draft PR depends on PR #600 due to use of that PR's new implementation of io_parallel conv layers. Thus, all of the changes from that PR are currently included in this draft (will change once it is merged).
As of now, it seems both parallel and stream 1D implementations are working well, with performance matching that of the non-transposed layers. There are slight latency increases in the 2D implementations that may need to be worked out (namely the writing of data in both cases is a bit slow - in parallel it seems to have trouble writing to the output in the order the implementation wants to and in stream there are often multiple writes that get queued up which causes the implementation to take extra cycles).
Type of change
- [x] New feature (non-breaking change which adds functionality)
Tests
Still have to add tests to this PR.
Test Configuration:
Testing was done by compiling models consisting of single ConvTranspose layers, and comparing the performance of those layers to analogous Conv layers (i.e. a layer that maps the conv transpose output to its input).
Checklist
- [x] I have read the guidelines for contributing.
- [ ] I have commented my code, particularly in hard-to-understand areas.
- [ ] I have made corresponding changes to the documentation.
- [ ] My changes generate no new warnings.
- [ ] I have added tests that prove my fix is effective or that my feature works.
For the nondefault project name handling, it may be good to rebase with the current main branch. I think those things have been solved, though of course, there are no guarantees.
I was wondering about the status of this PR. We'll talk about the code status and release schedule this Friday, and the conv transpose layer is an important layer for us to support.
I was wondering about the status of this PR. We'll talk about the code status and release schedule this Friday, and the conv transpose layer is an important layer for us to support.
The PR no longer is waiting on any others. There are still slight issues in optimization for 2D transpose. I can clean it up a little bit / rebase, etc.
Hi @Jonathan-Shoemaker, I squashed your commits + rebased to main and tried to decouple the nonrelated changes on my branch, diff here: https://github.com/fastmachinelearning/hls4ml/compare/main...jmduarte:hls4ml:conv_tr_parallel
Can I push it here and we can proceed to review it?
Hi @Jonathan-Shoemaker, I squashed your commits + rebased to main and tried to decouple the nonrelated changes on my branch, diff here: main...jmduarte:hls4ml:conv_tr_parallel
Can I push it here and we can proceed to review it?
sounds good to me. I can work on adding tests
Great! sounds good. I'll also review what's here soon, have some minor comments/questions.
Also don't worry about running pre-commit yet, we can run that at the end after we're done reviewing (to not introduce large diffs).
I think we want to support this for version 0.8. I will try rebasing it on the current main.
The rebase is at https://github.com/fastmachinelearning/hls4ml/tree/conv_tr_rebase. There were lots of merge conflicts so please take a look. We can replace this PR with that one, or force push it.
#844 is the version of this PR based on my rebased attempt. I wanted to make the PR to see how the tests go.
What is the meaning of "keep_dims"?
keep_dims keeps the weight matrix from being entirely flattened, keeping the first "keep_dims" dimensions as is, flattening along all other dimensions.
The reason for this is that the ConvTranspose is computed as the interleaving of "stride" number of Conv layers. The dimensions kept are for indexing into these different Conv layers. The idea was that the weight matrix of a ConvTranspose layer can be thought of as a disjoint set of weight matrices for Conv layers and treating it as such was easier.
Hello does this mean currently Convd2DTranspose from keras is not supported in hls4ml
Good day
How do I effect the changes for Conv2DTranspose in the hls4ml .
I tried updating the version the changes still not effect.I am not familiar with github commit and branches.
Could you please share a guide.
hi @MODISENEISO. It looks like you're using a conda environment. Did you do pip install hls4ml?
You can also install any branch of hls4ml as follows:
pip install git+https://github.com/fastmachinelearning/hls4ml@conv_tr_rebase
In this case, conv_tr_rebase is the name of the branch for the updated pull request with this new feature #844.
Thanks, Javier