Pytorch_Tutorial : Neural Net

This post is based on the pytorch tutorial documentation and the post someone write. the address is below

pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html

justkode.kr/deep-learning/pytorch-cnn

There are some wrong figure on the pytorch tutorial documentation.(feature size of ConvNet)

 

 

A typical training procedure for a neural network is as follows:

1. Define the neural network that has some learnable parameters (or weights)

2. Iterate over a dataset of inputs

3. Process input through the network

4. Comput the loss (how far is the output from being correct)

5. Propagate  gradients back into network's parameters

6. Update the weights of the network, typically using a simple update rule (e.g. SGD etc.)

 

 

1) Define the neural net

Input :

class Net(nn.Module):
    def __init__(self):
        # 1. Define the neural network 
        
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convoluation
        # kernel
        self.conv1 = nn.Conv2d(1,6,3)
        self.conv2 = nn.Conv2d(6,16,3)
        # An affine operation : y = Wx+b
        self.fc1 = nn.Linear(16*6*6,120) # 6x6 from image dimension
        self.fc2 = nn.Linear(120,84)
        self.fc3 = nn.Linear(84,10)
        
    def forward(self,x): # feed-forwarding
        # 2. Iterate over a dataset of inputs
        # 3. Process input through the network
        
        # Max pooling over a (2,2) window
        print("Input : ",x.size())
        print("C1    : ",(self.conv1(x)).size())
        x = F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        # If the size is a square you can only specify a single number
        print("S2    : ",x.size())
        print("C3    : ",(self.conv2(x)).size())
        x = F.max_pool2d(F.relu(self.conv2(x)),(2,2))
        print("S4    : ",x.size())
        x = x.view(-1,self.num_flat_features(x))
        print("Reduct: ",x.size())
        x = F.relu(self.fc1(x))
        print("F5    : ",x.size())
        x = F.relu(self.fc2(x))
        print("F6    : ",x.size())
        x = self.fc3(x)
        print("Output: ",x.size())
        return x
    
    def num_flat_features(self,x):
        size = x.size()[1:] # All dimensions excepts the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features
 
net = Net()
print(net)

Output :

Net(
  (conv1): Conv2d(1, 6, kernel_size=(3, 3), stride=(1, 1))
  (conv2): Conv2d(6, 16, kernel_size=(3, 3), stride=(1, 1))
  (fc1): Linear(in_features=576, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
)

 

 

2) The learnable paramters of a model are returned by net.parameters()

Input :

params = list(net.parameters())
print(len(params))
print(params[0].size()) # conv1's weight

Output :

10
torch.Size([6, 1, 3, 3])

 

3) Let's try a random 32x32 input.

Input :

input = torch.randn(1,1,32,32)
out = net(input)
print(out)

Output :

Input :  torch.Size([1, 1, 32, 32])
C1    :  torch.Size([1, 6, 30, 30])
S2    :  torch.Size([1, 6, 15, 15])
C3    :  torch.Size([1, 16, 13, 13])
S4    :  torch.Size([1, 16, 6, 6])
Reduct:  torch.Size([1, 576])
F5    :  torch.Size([1, 120])
F6    :  torch.Size([1, 84])
Output:  torch.Size([1, 10])
tensor([[ 0.0267,  0.0820,  0.0325,  0.0427,  0.0956,  0.0788,  0.0231, -0.0725,
         -0.0008,  0.1146]], grad_fn=<AddmmBackward>)

 

 

The above line is the correct output. The figure for convnet on the pytorch docu has difference with the above output.

I think the network propagate as following :

Input           : 1 x 1 x 32 x 32

C1 Feature maps : 1 x 6 x 30 x 30 (Convolution with 6ch. window (3x3))

S2 Feature maps : 1 x 6 x 15 x 15 (Subsampling with 6ch. window (2x2))

C3 Feature maps : 1 x 16 x 13 x 13 (Convolution with 16ch. window (3x3))

S4 Feature maps : 1 x 16 x 6 x 6 (Subsampling with 16ch. window (2x2))

Dimension Resize: 1 x 576

F5 Layer        : 1 x 120

F6 Layer        : 1 x 84

Output Layer    : 1 x 10

 

 

The size of Kernels are seemed to be 5x5 on 1st conv. and 3x3 on 2nd conv.. the process of the neural net using the predicted window size is expected as following :

Input : 1 x 1 x 32 x 32

C1 f.maps : 1 x 6 x 28 x 28 (Conv. with 6ch 5x5)

S2 f.maps : 1 x 6 x 14 x 14 (Subsampling with 6ch 2x2)

C3 f.maps : 1 x 16 x 12 x 12 (Conv. with 6ch 3x3)

S4 f.maps : 1 x 16 x 6 x 6 (subsampling with 6ch 2x2)

Dimension Resize : 1 x 576 .....