tensorflow
Pruning models from tf.keras.applications (efficientnet, resnet etc)
I am finetuning the tf.keras.applications model on my data for the problem of image classification. I was able to complete training and test results look good. In order to deploy the model, I wanted to perform some pruning and optimization. I tried following the Keras Pruning tutorial, but I could not figure out how to enable pruning on the efficientnet layer. If we could have a tutorial on this or a worked out example, that would be great, thank you!
Here is the model summary:
Model: "model_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_8 (InputLayer) [(None, 224, 224, 3)] 0
_________________________________________________________________
efficientnetb0 (Functional) (None, 7, 7, 1280) 4049571
_________________________________________________________________
global_average_pooling2d_3 ( (None, 1280) 0
_________________________________________________________________
dense_6 (Dense) (None, 8) 10248
_________________________________________________________________
dropout_3 (Dropout) (None, 8) 0
_________________________________________________________________
dense_7 (Dense) (None, 1) 9
=================================================================
Total params: 4,059,828
Trainable params: 10,257
Non-trainable params: 4,049,571
Can you follow the example of pruning the whole model and try prune_low_magnitude(your model)? Then post the model summary or let us know if you run into any issues. Thanks!
Thank you @liyunlu0618 , I have tried following the tutorial, here is the model.summary() and the error I get on using prune_low_magnitude() on my model:
model.summary()
Model: "model_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_8 (InputLayer) [(None, 224, 224, 3)] 0
_________________________________________________________________
efficientnetb0 (Functional) (None, 7, 7, 1280) 4049571
_________________________________________________________________
global_average_pooling2d_3 ( (None, 1280) 0
_________________________________________________________________
dense_6 (Dense) (None, 8) 10248
_________________________________________________________________
dropout_3 (Dropout) (None, 8) 0
_________________________________________________________________
dense_7 (Dense) (None, 1) 9
=================================================================
Total params: 4,059,828
Trainable params: 10,257
Non-trainable params: 4,049,571
Error
ValueError: Please initialize 'Prune' with a supported layer. Layers should either be a 'PrunableLayer' instance, or should be supported by the PruneRegistry. You passed: <class 'tensorflow.python.keras.engine.functional.Functional'>
Thanks for reporting the error. I'm working on a change to support pruning a model recursively. Will let you know once it's checked-in.
The PR is merged. https://github.com/tensorflow/model-optimization/pull/658
You can try later with a nightly build and let us know if it works. Thanks!
Thank you @liyunlu0618 , I am trying to get the tf-nightly installed and running. (never used tf-nightly before, always used stable)
Here is what I am doing:
- Uninstall current stable
tensorflow-2.4.1(pip uninstall tensorflow) - Install tf-nightly (
tf_nightly-2.6.0.dev20210407) (pip install -U tf-nightly --user)
(Please advise if I am not on the right track!)
I got an error, I'm not sure if I am missing any additional step to get the installation right. Here is what my output/error log looks like:
INFO:tensorflow:Enabling eager execution
INFO:tensorflow:Enabling v2 tensorshape
INFO:tensorflow:Enabling resource variables
INFO:tensorflow:Enabling tensor equality
INFO:tensorflow:Enabling control flow v2
---------------------------------------------------------------------------
NotFoundError Traceback (most recent call last)
<ipython-input-3-53c627d25dc5> in <module>
1 # Import packages and set TPU/GPU environment if enabled
2
----> 3 import tensorflow as tf
4
5
~/.local/lib/python3.7/site-packages/tensorflow/__init__.py in <module>
443 _main_dir = _os.path.join(_s, 'tensorflow/core/kernels')
444 if _os.path.exists(_main_dir):
--> 445 _ll.load_library(_main_dir)
446
447 # Load third party dynamic kernels.
~/.local/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py in load_library(library_location)
152
153 for lib in kernel_libraries:
--> 154 py_tf.TF_LoadLibrary(lib)
155
156 else:
NotFoundError: /opt/conda/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow6thread10ThreadPool26TransformRangeConcurrentlyExxRKSt8functionIFvxxEE
@liyunlu0618 Not sure how exactly, but now the tensorflow-nightly seems to be installed correctly. (I made sure by printing the version of tensorflow)
I am still getting the same error. It seems like there might be difference between the <class 'tensorflow.python.keras.engine.functional.Functional'> objects and Recursive sequential layers object(I peeked into the PR). Also I am wondering that, since Pruning is a part of tensorflow_model_optimization, do I have to take any additional steps to update that to nightly as well?
Here is my code snippet and error log for reference. It is unchanged from my last comment:
import tensorflow_model_optimization as tfmot
print("Tensorflow_version : ",tf.__version__)
prune_low_magnitude = tfmot.sparsity.keras.prune_low_magnitude
# Compute end step to finish pruning after 2 epochs.
batch_size = 128
epochs = 2
end_step = np.ceil(TRAIN_IMG_COUNT / batch_size).astype(np.int32) * epochs
# Define model for pruning.
pruning_params = {
'pruning_schedule': tfmot.sparsity.keras.PolynomialDecay(initial_sparsity=0.50,
final_sparsity=0.80,
begin_step=0,
end_step=end_step)
}
model_for_pruning = prune_low_magnitude(model, **pruning_params)
# `prune_low_magnitude` requires a recompile.
model_for_pruning.compile(optimizer='adam',
loss=tf.keras.losses.BinaryCrossentropy,
metrics=['accuracy'])
Output Log (clipped):
Tensorflow_version : 2.6.0-dev20210407
ValueError Traceback (most recent call last)
<ipython-input-16-cf5cd8efdcbd> in <module>
23
24
---> 25 model_for_pruning = prune_low_magnitude(model, **pruning_params)
26
27 # `prune_low_magnitude` requires a recompile.
.
.
(Trimmed)
.
.
ValueError: Please initialize `Prune` with a supported layer. Layers should either be a `PrunableLayer` instance, or should be supported by the PruneRegistry. You passed: <class 'tensorflow.python.keras.engine.functional.Functional'>
Do you mind addressing how to use/install tf-mot nightly? I could not find any documentation on that.
Currently I have this from pip freeze:
tf-nightly==2.6.0.dev20210408
tensorflow-model-optimization==0.5.0
You may have to build from source at the moment: https://www.tensorflow.org/model_optimization/guide/install#installing_from_source
Will let you know once we have a new release.
Hi, is this issue resolved? I tried pruning EfficientNetB0, and I get the error that pruning of 'Functional' layer is not supported.
