Metadata-Version: 2.1
Name: tensorflow_batchnorm_folding
Version: 1.0.9
Summary: Folds BN layers in tf keras models.
Home-page: https://github.com/pypa/sampleproject
Author: Edouard Yvinec
Author-email: Edouard Yvinec <ey@datakalab.com>
License: MIT License
        
        Copyright (c) 2022 Edouard Yvinec
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
Project-URL: documentation, https://www.ijcai.org/proceedings/2022/0223.pdf
Project-URL: repository, https://gitlab.com/ey_datakalab/batch-normalization-folding
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tensorflow>=2.13.0
Requires-Dist: scikit-network>=0.31.0

# Batch-Normalization Folding

In this repository, we propose an implementation of the batch-normalization folding algorithm from [IJCAI 2022](https://arxiv.org/pdf/2203.14646.pdf). Batch-Normalization Folding implements the batch normalization layer by folding it into a appropriate layer. The original batch-normalization layers are removed without changing the predictive function defined by the neural network. The simpliest scenario is an application for a fully-connected layer followed by a batch-normalization layer, we get
```math
x \mapsto \gamma \frac{Ax + b - \mu}{\sigma + \epsilon} + \beta = \gamma \frac{A}{\sigma +\epsilon} x + \frac{b - \mu}{\sigma + \epsilon} + \beta
```
Thus the two layers can be expressed as a single fully-connected layer at inference without any change in the predictive function.

## Use

This repository is available as a pip package (use `pip install tensorflow-batchnorm-folding`).
This implementation is compatible with tf.keras.Model instances. It was tested with the following models
- [x] ResNet 50
- [x] MobileNet V2
- [x] MobileNet V3
- [x] EfficentNet B0

To run a simple test:
```python
from batch_normalization_folding.folder import fold_batchnormalization_layers
import tensorflow as tf
mod=tf.keras.applications.efficientnet.EfficientNetB0()
folded_model,output_str=fold_batchnormalization_layers(mod,verbose=True)
```
The `output_str` is either the ratio num_layers_folded/num_layers_not_folded or 'failed' to state a failure in the process.

## Parameters

The function `fold_batchnormalization_layers` can be called with multiple parameters.

| Parameter         | Option                                                                                                                                               |
|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
| model             | Model to fold.                                                                                                                                       |
| folding_mechanism | `ban-off` uses BaN-OFF als folding mechanism (recommended) <br> `simple` uses a folding mechanism that only folds neighboring layers. <br> Default: `ban-off`  |
| verbose           | `True` prints additional information during the folding process. <br> `False` Disables the printing of additional information. <br> Default: `False`           |

## Layer folding

This Python implementation supports folding into many types of layers, supported is the folding of batch-normalization parameters into:

 - Dense
 - Conv1D
 - Conv2D
 - DepthwiseConv2D

## To Do

- [x] unit test on all keras applciations models
- [x] check package installement
- [ ] deal with Concatenate layers

## Cite

```
@inproceedings{yvinec2022fold,
  title={To Fold or Not to Fold: a Necessary and Sufficient Condition on Batch-Normalization Layers Folding},
  author={Yvinec, Edouard and Dapogny, Arnaud and Bailly, Kevin},
  journal={IJCAI},
  year={2022}
}
```

## Performance on Base Models

```
+------------------------------------+
|             ResNet 50              |
+------------------------------------+
| BN layers folded         |    53   |
| BN layers not folded     |    0    |
+------------------------------------+
|          EfficientNet B0           |
+------------------------------------+
| BN layers folded         |    49   |
| BN layers not folded     |    0    |
+------------------------------------+
|            MobileNet V2            |
+------------------------------------+
| BN layers folded         |    52   |
| BN layers not folded     |    0    |
+------------------------------------+
|            MobileNet V3            |
+------------------------------------+
| BN layers folded         |    34   |
| BN layers not folded     |    0    |
+------------------------------------+
|        Inception ResNet V2         |
+------------------------------------+
| BN layers folded         |   204   |
| BN layers not folded     |    0    |
+------------------------------------+
|            Inception V3            |
+------------------------------------+
| BN layers folded         |    94   |
| BN layers not folded     |    0    |
+------------------------------------+
|               NASNet               |
+------------------------------------+
| BN layers folded         |    28   |
| BN layers not folded     |   164   |
+------------------------------------+
|            DenseNet 121            |
+------------------------------------+
| BN layers folded         |    59   |
| BN layers not folded     |    62   |
+------------------------------------+
```
