DOLG-TensorFlow

This is an unofficial implementation of Deep Orthogonal Fusion of Local and Global Features (DOLG) in TensorFlow 2 (Keras). Paper.

It seeks to design an effective single-stage solution by integrating local and global information inside images into compact image representations. It attentively extracts representative local information with multi-atrous convolutions and self-attention at first. Components orthogonal to the global image representation are then extracted from the local information. At last, the orthogonal components are concatenated with the global representation as a complementary, and then aggregation is performed to generate the final representation.

Prerequisites: Check requirements.txt

Install

pip install dolg-tensorflow

or
git clone https://github.com/innat/DOLG-TensorFlow.git

Usage

[Option 1]

First, create a model with two output branch, one for local branch and other for global branch. It's needed for DOLG model. See the demo below.

base = applications.EfficientNetB0(...)
new_base = keras.Model(
    [base.inputs], 
    [
        base.get_layer('block5c_add').output,       # fol local branch 
        base.get_layer('block7a_project_bn').output # for global branch 
    ]
)

second, now use the above created model as follows.

from models.DOLG import DOLGNet

dolg_net = DOLGNet(new_base, num_classes=num_classe, activation='softmax')
dolg_net.build_graph().summary()

[Option 2]

Apart from the above approach, we can also integrate dolg layers with a custom model. Here is one example,

# component of DOLG model
from layers.GeM import GeneralizedMeanPooling2D
from layers.LocalBranch import DOLGLocalBranch
from layers.OrtholFusion import OrthogonalFusion

vision_input = keras.Input(shape=(img_shape, img_shape, 1), name="img")
x = Conv2D(...)(vision_input)
x = Conv2D ...
y = x = DOLGLocalBranch(IMG_SIZE=img_shape)(x)

x = MaxPooling2D(...)(x)
x = Conv2D ...
gem_pool = GeneralizedMeanPooling2D()(x)
gem_dens = Dense(1024, activation=None)(gem_pool)

vision_output = OrthogonalFusion()([y, gem_dens])
vision = keras.Model(vision_input, vision_output, name="vision")
vision.summary(expand_nested=True, line_length=110)

Code Examples

The DOLG concept can be integrated into any computer vision models i.e. NFNet, ResNeSt, or EfficietNet. Here are some end-to-end code examples.

• DenseNet DOLGNet Malaria |
• EfficientNet DOLGNet Oxford Flowers 102 |
• ResNet DOLGNet Cmaterdb |

References and Other Implementation

• Blogs.
• Official-Code
• PyTorch-Code.