need some help in customizing timm efficient net model #1280

sparshgarg23 · 2022-05-27T12:48:18Z

sparshgarg23
May 27, 2022

I was working with timm efficient net family and have the following implementations

#let's update the pretarined model:
for param in model.parameters():
  param.requires_grad=False

#orginally, it was:
#(classifier): Linear(in_features=2048, out_features=1000, bias=True)


#we are updating it as a 2-class classifier:
"""
model.classifier = nn.Sequential(
    nn.Linear(in_features=2048, out_features=625), #2048 is the orginal in_features
    nn.ReLU(), #ReLu to be the activation function
    nn.Dropout(p=0.3),
    nn.Linear(in_features=625, out_features=256),
    nn.ReLU(),
    nn.Linear(in_features=256,out_features=128),
    nn.ReLU(),
    nn.Linear(in_features=128, out_features=3),  
)
"""
model.classifier = nn.Sequential(
    
    nn.Linear(in_features=1536, out_features=625), #1792 is the orginal in_features
    nn.ReLU(), #ReLu to be the activation function
    nn.Dropout(p=0.3),
    nn.Linear(in_features=625, out_features=256),
    nn.ReLU(),
    nn.Dropout(p=0.2),
    nn.Linear(in_features=256, out_features=199)
)

I am able to train this model on my dataset and can easily make changes in it by simply removing the layers in classifier
However when I tried to put the same implementation in class as shown below

import timm
import torch
import torch.nn as nn

class AMT_Model(nn.Module):
    
    def __init__(self):
        super(AMT_Model,self).__init__()
        self.model=timm.create_model('tf_efficientnet_b3_ns',pretrained=True,num_classes=0)
        for param in self.model.parameters():
            param.requires_grad=False
        self.model.classifer=nn.Sequential(
            nn.Linear(in_features=1536,out_features=625),
            nn.ReLU(),
            nn.Dropout(p=0.3),
            nn.Linear(in_features=625,out_features=256),
            nn.ReLU(),
            nn.Dropout(p=0.2),
            nn.Linear(in_features=256,out_features=199)
            )
    
    def forward(self,x):
        out=self.model(x)
        return out

def build_model():
    model=AMT_Model()
    return model

model summary for this new model is as follows

  (conv_head): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn2): BatchNorm2d(1536, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
    (act2): SiLU(inplace=True)
    (global_pool): SelectAdaptivePool2d (pool_type=avg, flatten=Flatten(start_dim=1, end_dim=-1))
    (classifier): Linear(in_features=1536, out_features=1000, bias=True)
    (classifer): Sequential(
      (0): Linear(in_features=1536, out_features=625, bias=True)
      (1): ReLU()
      (2): Dropout(p=0.3, inplace=False)
      (3): Linear(in_features=625, out_features=256, bias=True)
      (4): ReLU()
      (5): Dropout(p=0.2, inplace=False)
      (6): Linear(in_features=256, out_features=199, bias=True)
    )
  )
)

So why are there two classifiers and what changes should I make in the class so that my custom classifier is used

I also tried just extracting the features from the model and later on passing them to classifier
For this the class implementation is as follows

class AMT_Model(nn.Module):
    
    def __init__(self):
        super(AMT_Model,self).__init__()
        self.model=timm.create_model('tf_efficientnet_b3_ns',pretrained=True)
        for param in self.model.parameters():
            param.requires_grad=False
        self.classifer=nn.Sequential(
            nn.Linear(in_features=1536,out_features=625),
            nn.ReLU(),
            nn.Dropout(p=0.3),
            nn.Linear(in_features=625,out_features=256),
            nn.ReLU(),
            nn.Dropout(p=0.2),
            nn.Linear(in_features=256,out_features=199)
            )
    
    def forward(self,x):
        features=self.model.forward_features(x)
        out=self.classifer(features)
        return out

def build_model():
    model=AMT_Model()
    return model

But this ends up giving the error


RuntimeError: mat1 and mat2 shapes cannot be multiplied (393216x16 and 1536x625)

Where model summary is

  (conv_head): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn2): BatchNorm2d(1536, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
    (act2): SiLU(inplace=True)
    (global_pool): SelectAdaptivePool2d (pool_type=avg, flatten=Flatten(start_dim=1, end_dim=-1))
    (classifier): Linear(in_features=1536, out_features=1000, bias=True)
  )
  (classifer): Sequential(
    (0): Linear(in_features=1536, out_features=625, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.3, inplace=False)
    (3): Linear(in_features=625, out_features=256, bias=True)
    (4): ReLU()
    (5): Dropout(p=0.2, inplace=False)
    (6): Linear(in_features=256, out_features=199, bias=True)
  )
)

I made one more change in the model as shown below

class AMT_Model(nn.Module):
    
    def __init__(self):
        super(AMT_Model,self).__init__()
        self.model=timm.create_model('tf_efficientnet_b3_ns',features_only=True,pretrained=True)
        for param in self.model.parameters():
            param.requires_grad=False
        self.classifer=nn.Sequential(
            nn.Linear(in_features=1536,out_features=625),
            nn.ReLU(),
            nn.Dropout(p=0.3),
            nn.Linear(in_features=625,out_features=256),
            nn.ReLU(),
            nn.Dropout(p=0.2),
            nn.Linear(in_features=256,out_features=3)
            )
    
    def forward(self,x):
        features=self.model(x)
        out=self.classifer(features)
        return out

def build_model():
    model=AMT_Model()
    return model

This ends up giving me the following error

TypeError: linear(): argument 'input' (position 1) must be Tensor, not list

This is strange because when on using the code shown in the first block i don't up getting the same error.
Also the model summary seems to be missing some stages as shown below

   (1): InvertedResidual(
          (conv_pw): Conv2d(384, 2304, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (bn1): BatchNorm2d(2304, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
          (act1): SiLU(inplace=True)
          (conv_dw): Conv2d(2304, 2304, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2304, bias=False)
          (bn2): BatchNorm2d(2304, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
          (act2): SiLU(inplace=True)
          (se): SqueezeExcite(
            (conv_reduce): Conv2d(2304, 96, kernel_size=(1, 1), stride=(1, 1))
            (act1): SiLU(inplace=True)
            (conv_expand): Conv2d(96, 2304, kernel_size=(1, 1), stride=(1, 1))
            (gate): Sigmoid()
          )
          (conv_pwl): Conv2d(2304, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (bn3): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
    )
  )
  (classifer): Sequential(
    (0): Linear(in_features=1536, out_features=625, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.3, inplace=False)
    (3): Linear(in_features=625, out_features=256, bias=True)
    (4): ReLU()
    (5): Dropout(p=0.2, inplace=False)
    (6): Linear(in_features=256, out_features=3, bias=True)
  )
)

Would appreciate it if someone could point my mistakes and give me some suggestions on correcting them.

So what changes can be made into AMT_Model to ensure that the model summary looks like the one shown in the first code block.
Basically in my class I am trying to extract the features from timm efficient net and then pass them to the classifier.

rwightman · 2022-05-27T17:19:38Z

rwightman
May 27, 2022
Maintainer

@sparshgarg23 without looking in too much detail at all of the issues, you appear to have a typo and are using 'classifer' somewhere for the 'classifier' so that you end up with two classifiers, should try to fix that first...

2 replies

sparshgarg23 May 27, 2022
Author

thanks @rwightman I will look into it and then get back to you.

sparshgarg23 May 27, 2022
Author

@rwightman
I tried changing the code as follows but it didn't make any difference.

import timm
import torch
import torch.nn as nn

class AMT_Model(nn.Module):
    
    def __init__(self):
        super(AMT_Model,self).__init__()
        self.model=timm.create_model('tf_efficientnet_b3_ns',pretrained=True)
    def forward(self,x):
        for param in self.model.parameters():
          param.requires_grad=False
        self.model.classifier=nn.Sequential(
            nn.Linear(in_features=1536,out_features=3)
        )
        out=self.model(x)
        return out

def build_model():
    model=AMT_Model()
    return model

I tried writing it as a function that works but I am not sure why its not working when I write it as a class.

def amt_model():
   model=timm.create_model('tf_efficientnet_b3_ns',pretrained=True)
   for param in model.parameters():
        param.requires_grad=False
   model.classifier=nn.Sequential(
        nn.Linear(in_features=1536,out_features=625),
        nn.ReLU(),
        nn.Dropout(p=0.3),
        nn.Linear(in_features=625,out_features=256),
        nn.ReLU(),
        nn.Dropout(p=0.2),
        nn.Linear(in_features=256,out_features=3)
        )
   return model

I admit my coding needs some practice and I would be obliged if you could give me some pointers as to where I am going wrong.

rwightman · 2022-05-27T18:13:09Z

rwightman
May 27, 2022
Maintainer

import timm
import torch
import torch.nn as nn

class AMT_Model(nn.Module):
    
    def __init__(self):
        super(AMT_Model,self).__init__()
        self.model=timm.create_model('tf_efficientnet_b3_ns',pretrained=True)
        for param in self.model.parameters():
          param.requires_grad=False

        self.model.classifier=nn.Sequential(
            nn.Linear(in_features=1536,out_features=3)
        )

    def forward(self,x):
        out=self.model(x)
        return out

def build_model():
    model=AMT_Model()
    return model

2 replies

rwightman May 27, 2022
Maintainer

this works fine

if you want to do that simple of a classifier you can just create the timm model with
timm.create_model('tf_efficientnet_b3_ns',pretrained=True, num_classes=3)

if you want a more advanced classifier you can use your approach of modifying the .classifier attribute (this is efficientnet specific)

Or you can use your AMT_model as a wrapper for any timm model

import timm
import torch
import torch.nn as nn

class AMT_Model(nn.Module):
    
    def __init__(self):
        super(AMT_Model,self).__init__()
        self.model = timm.create_model('tf_efficientnet_b3_ns',pretrained=True, num_classes=0)  # 0 = no classifier
        for param in self.model.parameters():
          param.requires_grad=False

        self.classifier = nn.Sequential(
            nn.Linear(in_features=self.model.num_features,out_features=3)
        )

    def forward(self,x):
        x = self.model(x)
        out = self.classifier(x)
        return out

def build_model():
    model=AMT_Model()
    return model

sparshgarg23 May 27, 2022
Author

I tried printing the model summary using your wrapper.It seems that there are two classifiers one is Idenityt and the other is the custom one defined.

  (conv_head): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn2): BatchNorm2d(1536, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
    (act2): SiLU(inplace=True)
    (global_pool): SelectAdaptivePool2d (pool_type=avg, flatten=Flatten(start_dim=1, end_dim=-1))
    (classifier): Identity()
  )
  (classifier): Sequential(
    (0): Linear(in_features=1536, out_features=3, bias=True)
  )
)

Wont't the two classifers be a problem.
Should I put the model declaration outside the init function?

rwightman · 2022-05-27T18:22:53Z

rwightman
May 27, 2022
Maintainer

No, nn.Identity() just passes the output through with no overhead, it's just there to avoid excessive conditionals when classifier is removed....

…

On Fri, May 27, 2022 at 11:21 AM sparshgarg23 ***@***.***> wrote: I tried printing the model summary using your wrapper.It seems that there are two classifiers one is Idenityt and the other is the custom one defined. (conv_head): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn2): BatchNorm2d(1536, eps=0.001, momentum=0.1, affine=True, track_running_stats=True) (act2): SiLU(inplace=True) (global_pool): SelectAdaptivePool2d (pool_type=avg, flatten=Flatten(start_dim=1, end_dim=-1)) (classifier): Identity() ) (classifier): Sequential( (0): Linear(in_features=1536, out_features=3, bias=True) ) ) Wont't the two classifers be a problem. — Reply to this email directly, view it on GitHub <#1280 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLQICBNWNCUXMLIKLY6YU3VMEHDDANCNFSM5XEEG2JA> . You are receiving this because you were mentioned.Message ID: <rwightman/pytorch-image-models/repo-discussions/1280/comments/2836055@ github.com>

4 replies

rwightman May 27, 2022
Maintainer

the beauty if this approach is that it can work with anything, ie here is last bit of 'resnet50'

    (global_pool): SelectAdaptivePool2d (pool_type=avg, flatten=Flatten(start_dim=1, end_dim=-1))
    (fc): Identity()
  )
  (classifier): Sequential(
    (0): Linear(in_features=2048, out_features=3, bias=True)
  )
)

you can see that there is no .classifier in the backbone model, it's .fc instead, but with this appraoch you don't need to care, you do need to make one additional change though, use self.mode.num_features as the input for your classifer channels... modifying snippet above

sparshgarg23 May 27, 2022
Author

Thanks,there is one last thing which I need to look into.Let me conduct some more experiments and if its okay with you can I ask for your suggestions.
I am planning on conducting some OOD /Domain adaption experiments using Efficient net,so that's the reason why i raised this query.

sparshgarg23 May 30, 2022
Author

Hi,if I wanted to add a bottleneck before the classifier and after the global pooling layer,how would I go about that.
currently we have

import timm
import torch
import torch.nn as nn

class AMT_Model(nn.Module):
    
    def __init__(self):
        super(AMT_Model,self).__init__()
        self.model = timm.create_model('tf_efficientnet_b3_ns',pretrained=True, num_classes=0)  # 0 = no classifier
        for param in self.model.parameters():
          param.requires_grad=False

        self.classifier = nn.Sequential(
            nn.Linear(in_features=self.model.num_features,out_features=3)
        )

    def forward(self,x):
        x = self.model(x)
        out = self.classifier(x)
        return out

def build_model():
    model=AMT_Model()
    return model

However,I would like to pass the features through a small bottleneck,and then through a customized loss function which makes the classification. as shown below

import timm
import torch
import torch.nn as nn

class AMT_Model(nn.Module):
    
    def __init__(self):
        super(AMT_Model,self).__init__()
        self.model = timm.create_model('tf_efficientnet_b3_ns',pretrained=True, num_classes=0)  # 0 = no classifier
        for param in self.model.parameters():
          param.requires_grad=False
        self.bottleneck=nn.Sequential(
                nn.Linear(self.model.num_features,out_features=625),
                Relu followed by dropout
                nn.Linear(in_features=625,out_features=N)
                Relu followed by dropout

        self.classifier = nn.Sequential(
            nn.Linear(in_features=N,out_features=3)
        )

    def forward(self,x):
        x = self.model(x)
        x=self.bottleneck(x)
        out = self.classifier(x)
        return out

def build_model():
    model=AMT_Model()
    return model

Do you think this will work?

sparshgarg23 May 30, 2022
Author

Never mind,it seems that this works.Does this mean it's possible to change and customize any layer inside the timm model.?

Uh oh!

need some help in customizing timm efficient net model #1280

Uh oh!

Uh oh!

sparshgarg23 May 27, 2022

Replies: 3 comments · 8 replies

Uh oh!

rwightman May 27, 2022 Maintainer

Uh oh!

sparshgarg23 May 27, 2022 Author

Uh oh!

Uh oh!

sparshgarg23 May 27, 2022 Author

Uh oh!

rwightman May 27, 2022 Maintainer

Uh oh!

Uh oh!

rwightman May 27, 2022 Maintainer

Uh oh!

Uh oh!

sparshgarg23 May 27, 2022 Author

Uh oh!

rwightman May 27, 2022 Maintainer

Uh oh!

Uh oh!

rwightman May 27, 2022 Maintainer

Uh oh!

sparshgarg23 May 27, 2022 Author

Uh oh!

sparshgarg23 May 30, 2022 Author

Uh oh!

sparshgarg23 May 30, 2022 Author

sparshgarg23
May 27, 2022

Replies: 3 comments 8 replies

rwightman
May 27, 2022
Maintainer

sparshgarg23 May 27, 2022
Author

sparshgarg23 May 27, 2022
Author

rwightman
May 27, 2022
Maintainer

rwightman May 27, 2022
Maintainer

sparshgarg23 May 27, 2022
Author

rwightman
May 27, 2022
Maintainer

rwightman May 27, 2022
Maintainer

sparshgarg23 May 27, 2022
Author

sparshgarg23 May 30, 2022
Author

sparshgarg23 May 30, 2022
Author