微信客服：xiaoxionga100
微信客服：ITCS521
FIT3181-python代写-Assignment1
时间：2024-09-04
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第1/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
FIT3181: Deep Learning (2024)
CE/Lecturer (Clayton): Dr Trung Le | trunglm@monash.edu
Lecturer (Clayton): Prof Dinh Phung | dinh.phung@monash.edu
Lecturer (Malaysia): Dr Arghya Pal | arghya.pal@monash.edu
Lecturer (Malaysia): Dr Lim Chern Hong | lim.chernhong@monash.edu
Head Tutor 3181: Miss Vy Vo | [v.vo@monash.edu ]
Head Tutor 5215: Dr Van Nguyen | [van.nguyen1@monash.edu ]
Faculty of Information Technology, Monash University, Australia
Student Information
Surname: [Enter your surname here]
Firstname: [Enter your firstname here ]
Student ID: [Enter your ID here ]
Email: [Enter your email here ]
Your tutorial time: [Enter your tutorial time here ]
Deep Neural Networks
Due: 11:55pm Sunday, 8 September 2024 (Sunday)
Important note: This is an individual assignment. It contributes 25% to
your final mark. Read the assignment instructions carefully.
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第2/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
What to submit
This assignment is to be completed individually and submitted to Moodle unit site. By
the due date, you are required to submit one single zip file, named
xxx_assignment01_solution.zip where xxx is your student ID, to the
corresponding Assignment (Dropbox) in Moodle. You can use Google Colab to do
Assigmnent 1 but you need to save it to an *.ipynb file to submit to the unit
Moodle.
More importantly, if you use Google Colab to do this assignment, you need to
first make a copy of this notebook on your Google drive.
For example, if your student ID is 12356, then gather all of your assignment
solution to folder, create a zip file named 123456_assignment01_solution.zip
and submit this file.
Within this zip folder, you must submit the following files:
1. Assignment01_solution.ipynb: this is your Python notebook solution source
file.
2. Assignment01_output.html: this is the output of your Python notebook solution
exported in html format.
3. Any extra files or folder needed to complete your assignment (e.g., images
used in your answers).
Since the notebook is quite big to load and work together, one recommended option
is to split solution into three parts and work on them seperately. In that case, replace
Assignment01_solution.ipynb by three notebooks:
Assignment01_Part1_solution.ipynb, Assignment01_Part2_solution.ipynb and
Assignment01_Part3_solution.ipynb
You can run your codes on Google Colab. In this case, you have to make a copy
of your Google colab notebook including the traces and progresses of model
training before submitting.
Part 1: Theory and Knowledge Questions
[Total marks for this part: 30 points]
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第3/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
The first part of this assignment is to demonstrate your knowledge in deep learning
that you have acquired from the lectures and tutorials materials. Most of the contents
in this assignment are drawn from the lectures and tutorials from weeks 1 to 4.
Going through these materials before attempting this part is highly recommended.
**Question 1.1** Activation function plays an important role in
modern Deep NNs. For each of the activation functions below, state
its output range, find its derivative (show your steps), and plot the
activation fuction and its derivative
**(a)** Exponential linear unit (ELU):
[1.5 points]
**(b)** Gaussian Error Linear Unit (GELU): where is the
probability cummulative function of the standard Gaussian distribution or
where . In addition, the GELU activation fuction
(the link for the main paper) has been widely used in the state-of-the-art Vision for
Transformers (e.g., here is the link for the main ViT paper).
[1.5 points]
Write your answer here. You can add more cells if needed.
**Question 1.2:** Assume that we feed a data point with a ground-
truth label to the feed-forward neural network with the ReLU
activation function as shown in the following figure
ELU(x) = { 0.1 (exp(x)− 1) ifx ≤ 0
x ifx > 0
GELU(x) = xΦ(x) Φ(x)
Φ(x) = P (X ≤ x) X ∼ N (0, 1)
x
y = 2
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第4/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**(a)** What is the numerical value of the latent presentation ?
[1 point]
**(b)** What is the numerical value of the latent presentation ?
[1 point]
**(c)** What is the numerical value of the logit ?
[1 point]
**(d)** What is the corresonding prediction probabilities ?
[1 point]
**(e)** What is the predicted label ? Is it a correct and an incorect prediction?
Remind that .
[1 point]
**(f)** What is the cross-entropy loss caused by the feed-forward neural network at
? Remind that .
[1 point]
**(g)** Why is the cross-entropy loss caused by the feed-forward neural network at
(i.e., ) always non-negative? When does this loss
get the value ? Note that you need to answer this question for a general pair
and a general feed-forward neural network with, for example classes?
[1 point]
You must show both formulas and numerical results for earning full mark. Although it
is optional, it is great if you show your PyTorch code for your computation.
**Question 1.3:**
For Question 1.3, you have two options:
(1) perform the forward, backward propagation, and SGD update for one
mini-batch (10 points), or
(2) manually implement a feed-forward neural network that can work on real
tabular datasets (20 points).
You can choose either (1) or (2) to proceed.
**Option 1**
[Total marks for this option: 10 points]
h1(x)
h2(x)
h3(x)
p(x)
yˆ
y = 2
(x, y) y = 2
(x, y) CE(1y, p(x)) CE(1y, p(x))
0 (x, y)
M = 4
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第5/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Assume that we are constructing a multilayered feed-forward neural network
for a classification problem with three classes where the model parameters will
be generated randomly using your student ID. The architecture of this network
is as shown in the following figure. Note
that the ELU has the same formula as the one in Q1.1.
We feed a batch with the labels as shown in the figure. Answer the following
questions.
You need to show both formulas, numerical results, and your PyTorch code for
your computation for earning full marks.
Forward propagation
**(a)** What is the value of (the pre-activation values of )?
[0.5 point]
**(b)** What is the value of ?
[0.5 point]
3(Input)→ 5(ELU)→ 3(Output)
X Y
In [ ]: import torch
student_id = 1234 #insert your student id here for example 1234
torch.manual_seed(student_id)
Out[ ]:
In [ ]: #Code to generate random matrices and biases for W1, b1, W2, b2
h¯
1(x) h1
In [ ]: #Show your code
h1(x)
In [ ]: #Show your code
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第6/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**(c)** What is the predicted value ?
[0.5 point]
(d) Suppose that we use the cross-entropy (CE) loss. What is the value of the CE loss
incurred by the mini-batch?
[0.5 point]
Backward propagation
**(e)** What are the derivatives , and ?
[3 points]
**(f)** What are the derivatives , and ?
[3 points]
SGD update
**(g)** Assume that we use SGD with learning rate to update the model
parameters. What are the values of and after updating?
[2 points]
**Option 2**
[Total marks for this option: 20 points]
yˆ
In [ ]: #Show your code
l
In [ ]: #Show your code
,∂l
∂h2
∂l
∂W 2
∂l
∂b2
In [ ]: #Show your code
, ,∂l
∂h1
∂l
∂h¯
1
∂l
∂W 1
∂l
∂b1
In [ ]: #Show your code
η = 0.01
W 2, b2 W 1, b1
In [ ]: #Show your code
In [ ]: import torch
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第7/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
In Option 2, you need to implement a feed-forward NN manually using PyTorch
and auto-differentiation of PyTorch. We then manually train the model on the
MNIST dataset.
We first download the MNIST dataset and preprocess it.
Each data point has dimension [28,28] . We need to flatten it to a vector to input to
our FFN.
Develop the feed-forward neural networks
(a) You need to develop the class MyLinear with the following skeleton. You need
to declare the weight matrix and bias of this linear layer.
[3 points]
In [ ]: transform = transforms.Compose([
transforms.ToTensor(), # Convert the image to a tensor with shape [C, H, W]
transforms.Normalize((0.5,), (0.5,)), # Normalize to [-1, 1]
transforms.Lambda(lambda x: x.view(28*28)) # Flatten the tensor to shape [-1,HW]
])
# Load the MNIST dataset
train_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
test_dataset = datasets.MNIST(root='./data', train=False, download=True, transform=transform)
train_data, train_labels = train_dataset.data, train_dataset.targets
test_data, test_labels = test_dataset.data, test_dataset.targets
print(train_data.shape, train_labels.shape)
print(test_data.shape, test_labels.shape)
In [ ]: train_dataset.data = train_data.data.view(-1, 28*28)
test_dataset.data = test_data.data.view(-1, 28*28)
train_data, train_labels = train_dataset.data, train_dataset.targets
test_data, test_labels = test_dataset.data, test_dataset.targets
print(train_data.shape, train_labels.shape)
print(test_data.shape, test_labels.shape)
In [ ]: train_loader = DataLoader(dataset=train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(dataset=test_dataset, batch_size=64, shuffle=False)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第8/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
(b) You need to develop the class MyFFN with the following skeleton
[7 points]
In [ ]: class MyLinear(torch.nn.Module):
def __init__(self, input_size, output_size):
"""
input_size: the size of the input
output_size: the size of the output
"""
super().__init__()
#Your code here
self.W =
self.b =
#forward propagation
def forward(self, x): #x is a mini-batch
#Your code here
In [ ]: class MyFFN(torch.nn.Module):
def __init__(self, input_size, num_classes, hidden_sizes, act = torch.nn.ReLU()):
"""
input_size: the size of the input
num_classes: the number of classes
act is the activation function
hidden_sizes is the list of hidden sizes
for example input_size = 3, hidden_sizes = [5, 7], num_classes = 4, and act = torch.nn.ReLU()
means that we are building up a FFN with the confirguration
(3 (Input) -> 5 (ReLU) -> 7 (ReLU) -> 4 (Output))
"""
super(MyFFN, self).__init__()
self.input_size = input_size
self.num_classes = num_classes
self.act = act
self.hidden_sizes = hidden_sizes
self.num_layers = len(hidden_sizes) + 1
def create_FFN(self):
"""
This function creates the feed-forward neural network
We stack many MyLinear layers
"""
hidden_sizes = [self.input_size] + self.hidden_sizes + [self.num_classes]
self.layers = []
#Your code here
def forward(self,x):
"""
This implements the forward propagation of the batch x
This needs to return the prediction probabilities of x
"""
#Your code here
def compute_loss(self, x, y):
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第9/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
(c) Write the code to evaluate the accuracy of the current myFFN model on a data
loader (e.g., train_loader or test_loader).
[2.5 points]
(c) Write the code to evaluate the loss of the current myFFN model on a data loader
(e.g., train_loader or test_loader).
[2.5 points]
"""
This function computes the cross-entropy loss
You can use the built-in CE loss of PyTorch
"""
#Your code here
def update_SGD(self, x, y, learning_rate = 0.01):
"""
This function updates the model parameters using SGD using the batch (x,y)
You need to implement the update rule manually and cannot rely on the built-in optimizer
"""
#Your code here
def update_SGDwithMomentum(self, x, y, learning_rate = 0.01, momentum = 0.9):
"""
This function updates the model parameters using SGD with momentum using the batch (x,y)
You need to implement the update rule manually and cannot rely on the built-in optimizer
"""
#Your code here
def update_AdaGrad(self, x, y, learning_rate = 0.01):
"""
This function updates the model parameters using AdaGrad using the batch (x,y)
You need to implement the update rule manually and cannot rely on the built-in optimizer
"""
#Your code here
In [ ]: myFFN = MyFFN(input_size = 28*28, num_classes = 10, hidden_sizes = [100, 100], act = torch.nn.ReLU)
myFFN.create_FFN()
print(myFFN)
In [ ]: def compute_acc(model, data_loader):
"""
This function computes the accuracy of the model on a data loader
"""
#Your code here
In [ ]: def compute_loss(model, data_loader):
"""
This function computes the loss of the model on a data loader
"""
#Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第10/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Train on the MNIST data with 50 epochs using updateSGD .
(d) Implement the function updateSGDMomentum in the class and train the model
with this optimizer in 50 epochs. You can update the corresponding function in the
MyFNN class.
[2.5 points]
(e) Implement the function updateAdagrad in the class and train the model with
this optimizer in 50 epochs. You can update the corresponding function in the
MyFNN class.
[2.5 points]
Part 2: Deep Neural Networks (DNN)
[Total marks for this part: 25 points]
The second part of this assignment is to demonstrate your basis knowledge in deep
learning that you have acquired from the lectures and tutorials materials. Most of the
contents in this assignment are drawn from the tutorials covered from weeks 1 to
2. Going through these materials before attempting this assignment is highly
recommended.
In the second part of this assignment, you are going to work with the FashionMNIST
dataset for image recognition task. It has the exact same format as MNIST (70,000
grayscale images of 28 × 28 pixels each with 10 classes), but the images represent
fashion items rather than handwritten digits, so each class is more diverse, and the
problem is significantly more challenging than MNIST.
In [ ]: num_epochs = 50
for epoch in range(num_epochs):
for i, (x, y) in enumerate(train_loader):
myFFN.update_SGD(x, y, learning_rate = 0.01)
train_acc = compute_acc(myFFN, train_loader)
train_loss = compute_loss(myFFN, train_loader)
test_acc = compute_acc(myFFN, test_loader)
test_loss = compute_loss(myFFN, test_loader)
print(f"Epoch {epoch+1}/{num_epochs}, Train Loss: {train_loss:.4f}, Train Acc: {train_acc*100:.2f}%, Test Loss: {test_loss:.4f}, Test Acc: {test_acc*100:.2f}%")
In [ ]: #Your code here
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第11/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Load the Fashion MNIST using torchvision
torch.Size([60000, 28, 28]) torch.Size([60000])
torch.Size([10000, 28, 28]) torch.Size([10000])
torch.Size([60000, 784]) torch.Size([60000])
torch.Size([10000, 784]) torch.Size([10000])
Number of training samples: 18827
Number of training samples: 16944
Number of validation samples: 1883
**Question 2.1:** Write the code to visualize a mini-batch in
train_loader including its images and labels.
[5 points]
In [ ]: import torch
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
torch.manual_seed(1234)
In [ ]: transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
train_dataset_orgin = datasets.FashionMNIST(root='./data', train=True, download=True, transform=transform)
test_dataset = datasets.FashionMNIST(root='./data', train=False, download=True, transform=transform)
print(train_dataset_orgin.data.shape, train_dataset_orgin.targets.shape)
print(test_dataset.data.shape, test_dataset.targets.shape)
train_dataset_orgin.data = train_dataset_orgin.data.view(-1, 28*28)
test_dataset.data = test_dataset.data.view(-1, 28*28)
print(train_dataset_orgin.data.shape, train_dataset_orgin.targets.shape)
print(test_dataset.data.shape, test_dataset.targets.shape)
N = len(train_dataset_orgin)
print(f"Number of training samples: {N}")
N_train = int(0.9*N)
N_val = N - N_train
print(f"Number of training samples: {N_train}")
print(f"Number of validation samples: {N_val}")
train_dataset, val_dataset = torch.utils.data.random_split(train_dataset_orgin, [N_train, N_val])
train_loader = DataLoader(dataset=train_dataset_orgin, batch_size=64, shuffle=True)
val_loader = DataLoader(dataset=val_dataset, batch_size=64, shuffle=False)
test_loader = DataLoader(dataset=test_dataset, batch_size=1000, shuffle=False)
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第12/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 2.2:** Write the code for the feed-forward neural net using
PyTorch
[5 points]
We now develop a feed-forward neural network with the architecture
. You can choose your own way
to implement your network and an optimizer of interest. You should train model in
epochs and evaluate the trained model on the test set.
**Question 2.3:** Tuning hyper-parameters with grid search
[5 points]
Assume that you need to tune the number of neurons on the first and second hidden
layers , and the used activation function
. The network has the architecture pattern
where , and are in their
grides. Write the code to tune the hyper-parameters , and . Note that you
can freely choose the optimizer and learning rate of interest for this task.
**Question 2.4:** Implement the loss with the form:
where is
the entropy of , is the prediction probabilities of a data point with
the ground-truth label , is an one-hot label, and is a trade-
off parameter. Set to train a model.
[5 points]
784→ 40(ReLU)→ 30(ReLU)→ 10(softmax)
50
In [ ]: #Your code here
n1 ∈ {20, 40} n2 ∈ {20, 40}
act ∈ {sigmoid, tanh, relu}
784→ n1(act)→ n2(act)→ 10(softmax) n1,n2 act
n1,n2 act
In [ ]: #Your code here
loss(p, y) = CE(1y, p) + λH(p) H(p) = −∑
M
i=1 pi log pi
p p x
y 1y λ > 0
λ = 0.1
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第13/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 2.5:** Experimenting with sharpness-aware minimization
technique
[5 points]
Sharpness-aware minimization (SAM) (i.e., link for main paper from Google
Deepmind) is a simple yet but efficient technique to improve the generalization ability
of deep learning models on unseen data examples. In your research or your work, you
might potentially use this idea. Your task is to read the paper and implement
Sharpness-aware minimization (SAM). Finally, you need to apply SAM to the best
architecture found in Question 2.3.
Part 3: Convolutional Neural Networks and Image
Classification
[Total marks for this part: 45 points]
The third part of this assignment is to demonstrate your basis knowledge in deep
learning that you have acquired from the lectures and tutorials materials. Most of the
contents in this assignment are drawn from the tutorials covered from weeks 3 to
6. Going through these materials before attempting this assignment is highly
recommended.
The dataset used for this part is a specific dataset for this unit consisting of
approximately images of classes of Animals, each of which has
approximately 500 images. You can download the dataset at download here if
you want to do your assignment on your machine.
In [ ]: #Your code here
10, 000 20
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第14/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
CUDA is not available. Training on CPU ...
Download the dataset to the folder of this Google Colab.
Downloading...
From (original): https://drive.google.com/uc?id=1aEkxNWaD02Z8ZNvZzeMefUoY
97C-3wTG
From (redirected): https://drive.google.com/uc?id=1aEkxNWaD02Z8ZNvZzeMefU
oY97C-3wTG&confirm=t&uuid=c6914688-3441-4198-b7a3-ee55f3c61869
To: /content/Animals_Dataset.zip
100% 643M/643M [00:13<00:00, 48.0MB/s]
We unzip the dataset to the folder.
In [ ]: import os
import requests
import tarfile
import time
from torchvision import datasets, transforms
from torch.utils.data import DataLoader, random_split
import torchvision.models as models
import torch.nn as nn
import torch
import PIL.Image
import pathlib
from torchsummary import summary
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
# check if CUDA is available
train_on_gpu = torch.cuda.is_available()
if not train_on_gpu:
print('CUDA is not available. Training on CPU ...')
else:
print('CUDA is available! Training on GPU ...')
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.manual_seed(1234)
Out[ ]:
In [ ]: !gdown --fuzzy https://drive.google.com/file/d/1aEkxNWaD02Z8ZNvZzeMefUoY97C-3wTG/view?usp=drive_link
# !gdown --fuzzy https://drive.google.com/file/d/1qdElRqDS4TitXfv_iG_TFQSi9QIfy4uM/view?usp=drive_link # backup url
In [ ]: !unzip -q Animals_Dataset.zip
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第15/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Number of instance in train_set: 8519
Number of instance in val_set: 947
In [ ]: data_dir = "./FIT5215_Dataset"
# We resize the images to [3,64,64]
transform = transforms.Compose([transforms.Resize((64,64)), #resises the image so it can be perfect for our model.
transforms.RandomHorizontalFlip(), # FLips the image w.r.t horizontal axis
#transforms.RandomRotation(4), #Rotates the image to a specified angel
#transforms.RandomAffine(0, shear=10, scale=(0.8,1.2)), #Performs actions like zooms, change shear angles.
transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2), # Set the color params
transforms.ToTensor(), # convert the image to tensor so that it can work with torch
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), # Normalize the images, each R,G,B value is normalized with mean=0.5 and std=0.5
])
# Load the dataset using torchvision.datasets.ImageFolder and apply transformations
dataset = datasets.ImageFolder(data_dir, transform=transform)
# Split the dataset into training and validation sets
train_size = int(0.9 * len(dataset))
valid_size = len(dataset) - train_size
train_dataset, val_dataset = random_split(dataset, [train_size, valid_size])
# Example of DataLoader creation for training and validation
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False)
print("Number of instance in train_set: %s" % len(train_dataset))
print("Number of instance in val_set: %s" % len(val_dataset))
In [ ]: class_names = ['bird', 'bottle', 'bread', 'butterfly', 'cake', 'cat', 'chicken', 'cow', 'dog', 'duck',
'elephant', 'fish', 'handgun', 'horse', 'lion', 'lipstick', 'seal', 'snake', 'spider', 'vase']
In [ ]: # obtain one batch of training images
dataiter = iter(train_loader)
images, labels = next(dataiter)
images = images.numpy() # convert images to numpy for display
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第16/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
For questions 3.1 to 3.7, you'll need to write your own model in a way that makes
it easy for you to experiment with different architectures and parameters. The
goal is to be able to pass the parameters to initialize a new instance of
YourModel to build different network architectures with different parameters.
Below are descriptions of some parameters for YourModel :
In [ ]: import math
def imshow(img):
img = img / 2 + 0.5 # unnormalize
plt.imshow(np.transpose(img, (1, 2, 0))) # convert from Tensor image
def visualize_data(images, categories, images_per_row = 8):
class_names = ['bird', 'bottle', 'bread', 'butterfly', 'cake', 'cat', 'chicken', 'cow', 'dog', 'duck',
'elephant', 'fish', 'handgun', 'horse', 'lion', 'lipstick', 'seal', 'snake', 'spider', 'vase']
n_images = len(images)
n_rows = math.ceil(float(n_images)/images_per_row)
fig = plt.figure(figsize=(1.5*images_per_row, 1.5*n_rows))
fig.patch.set_facecolor('white')
for i in range(n_images):
plt.subplot(n_rows, images_per_row, i+1)
plt.xticks([])
plt.yticks([])
imshow(images[i])
class_index = categories[i]
plt.xlabel(class_names[class_index])
plt.show()
In [ ]: visualize_data(images, labels)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第17/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
1. Block confirguration : Our network consists of many blocks. Each block
has the pattern [conv, batch norm, activation, conv, batch norm,
activation, max pool, dropout] . All convolutional layers have filter size
, strides and padding = 1, and all max pool layers have strides ,
kernel size , and padding = 0. The network will consists of a few blocks before
applying a linear layer to output the logits for the softmax layer.
2. list_feature_maps : the number of feature maps in the blocks of the
network. For example, if list_feature_maps = [16, 32, 64] , our network
has two blocks with the input_channels or number of feature maps are 16, 32 ,
and 64 respectively.
3. drop_rate : the keep probability for dropout. Setting drop_rate to
means not using dropout.
4. batch_norm : the batch normalization function is used or not. Setting
batch_norm to false means not using batch normalization.
5. use_skip : the skip connection is used in the blocks or not. Setting this to
true means that we use 1x1 Conv2D with strides=2 for the skip
connection.
6. At the end, you need to apply global average pooling (GAP)
( AdaptiveAvgPool2d((1, 1)) ) to flatten the 3D output tensor before
defining the output linear layer for predicting the labels.
Here is the model confirguration of YourCNN if the list_feature_maps = [16,
32, 64] and batch_norm = true .
(3, 3) (1, 1) (2, 2)
2
0.0
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第18/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 3.1:** You need to implement the aforementioned CNN.
First, you need to implement the block of our CNN in the class YourBlock . You
can ignore use_skip and skip connection for simplicity. However, you
cannot earn full marks for this question.
[6 points]
In [ ]: #Your code here
class YourBlock(nn.Module):
def __init__(self, in_feature_maps, out_feature_maps, drop_rate = 0.2, batch_norm = True, use_skip = True):
super(YourBlock, self).__init__()
self.use_skip = use_skip
#Your code here
def forward(self, x):
#Write your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第19/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Second, you need to use the above YourBlock to implement the class
YourCNN .
[6 points]
We declare my_cnn from YourCNN as follows.
In [ ]: class YourCNN(nn.Module):
def __init__(self, list_feature_maps = [16, 32, 64], drop_rate = 0.2, batch_norm= True, use_skip = True):
super(YourCNN, self).__init__()
layers = []
#Write your code here
def forward(self, x):
#Write your code here
In [ ]: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
my_cnn = YourCNN(list_feature_maps = [16, 32, 64], use_skip = True)
my_cnn = my_cnn.to(device)
print(my_cnn)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第20/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
YourCNN(
(block): ModuleList(
(0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_runn
ing_stats=True)
(2): ReLU(inplace=True)
(3): YourBlock(
(block): ModuleList(
(0): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(
1, 1))
(1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_
running_stats=True)
(2): ReLU(inplace=True)
(3): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(
1, 1))
(4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_
running_stats=True)
(5): ReLU(inplace=True)
(6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ce
il_mode=False)
(7): Dropout(p=0.2, inplace=False)
)
(skip_conv): Conv2d(16, 32, kernel_size=(1, 1), stride=(2, 2))
)
(4): YourBlock(
(block): ModuleList(
(0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(
1, 1))
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_
running_stats=True)
(2): ReLU(inplace=True)
(3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(
1, 1))
(4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_
running_stats=True)
(5): ReLU(inplace=True)
(6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ce
il_mode=False)
(7): Dropout(p=0.2, inplace=False)
)
(skip_conv): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2))
)
(5): AdaptiveAvgPool2d(output_size=(1, 1))
(6): Flatten(start_dim=1, end_dim=-1)
(7): Linear(in_features=64, out_features=20, bias=True)
)
)
We declare the optimizer and the loss function.
Here are the codes to compute the loss and accuracy.
In [ ]: # Loss and optimizer
learning_rate = 0.001
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(my_cnn.parameters(), lr=learning_rate)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第21/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Here is the code to train our model.
In [ ]: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
def compute_loss(model, loss_fn, loader):
loss = 0
# Set model to eval mode for inference
model.eval()
with torch.no_grad(): # No need to track gradients for validation
for (batchX, batchY) in loader:
# Move data to the same device as the model
batchX, batchY = batchX.to(device).type(torch.float32), batchY.to(device).type(torch.long)
loss += loss_fn(model(batchX), batchY)
# Set model back to train mode
model.train()
return float(loss)/len(loader)
In [ ]: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
def compute_acc(model, loader):
correct = 0
totals = 0
# Set model to eval mode for inference
model.eval()
for (batchX, batchY) in loader:
# Move batchX and batchY to the same device as the model
batchX, batchY = batchX.to(device).type(torch.float32), batchY.to(device)
outputs = model(batchX) # feed batch to the model
totals += batchY.size(0) # accumulate totals with the current batch size
predicted = torch.argmax(outputs.data, 1) # get the predicted class
# Move batchY to the same device as predicted for comparison
correct += (predicted == batchY).sum().item()
return correct / totals
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第22/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
In [ ]: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
def fit(model= None, train_loader = None, valid_loader= None, optimizer = None,
num_epochs = 50, verbose = True, seed= 1234):
torch.manual_seed(seed)
# Move the model to the device before initializing the optimizer
model.to(device) # Move the model to the GPU
if optimizer == None:
optim = torch.optim.Adam(model.parameters(), lr = 0.001) # Now initialize optimizer with model on GPU
else:
optim = optimizer
history = dict()
history['val_loss'] = list()
history['val_acc'] = list()
history['train_loss'] = list()
history['train_acc'] = list()
for epoch in range(num_epochs):
model.train()
for (X, y) in train_loader:
# Move input data to the same device as the model
X,y = X.to(device), y.to(device)
# Forward pass
outputs = model(X.type(torch.float32)) # X is already on the correct device
loss = loss_fn(outputs, y.type(torch.long))
# Backward and optimize
optim.zero_grad()
loss.backward()
optim.step()
#losses and accuracies for epoch
val_loss = compute_loss(model, loss_fn, valid_loader)
val_acc = compute_acc(model, valid_loader)
train_loss = compute_loss(model, loss_fn, train_loader)
train_acc = compute_acc(model, train_loader)
history['val_loss'].append(val_loss)
history['val_acc'].append(val_acc)
history['train_loss'].append(train_loss)
history['train_acc'].append(train_acc)
if not verbose: #verbose = True means we do show the training information during training
print(f"Epoch {epoch+1}/{num_epochs}")
print(f"train loss= {train_loss:.4f} - train acc= {train_acc*100:.2f}% - valid loss= {val_loss:.4f} - valid acc= {val_acc*100:.2f}%")
return history
In [ ]: history = fit(model= my_cnn, train_loader=train_loader, valid_loader = val_loader, optimizer = optimizer, num_epochs= 10, verbose = False)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第23/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Epoch 1/10
train loss= 2.2247 - train acc= 29.49% - valid loss= 2.3166 - valid acc=
29.88%
Epoch 2/10
train loss= 1.9459 - train acc= 38.62% - valid loss= 2.2254 - valid acc=
34.42%
Epoch 3/10
train loss= 1.7757 - train acc= 42.18% - valid loss= 1.8783 - valid acc=
42.13%
Epoch 4/10
train loss= 1.6908 - train acc= 44.52% - valid loss= 1.7666 - valid acc=
40.55%
Epoch 5/10
train loss= 1.6232 - train acc= 47.31% - valid loss= 1.8377 - valid acc=
44.56%
Epoch 6/10
train loss= 1.5740 - train acc= 48.37% - valid loss= 1.7418 - valid acc=
47.41%
Epoch 7/10
train loss= 1.4314 - train acc= 53.27% - valid loss= 1.7969 - valid acc=
52.06%
Epoch 8/10
train loss= 1.3645 - train acc= 54.60% - valid loss= 1.4301 - valid acc=
52.16%
Epoch 9/10
train loss= 1.3372 - train acc= 55.76% - valid loss= 1.4144 - valid acc=
53.64%
Epoch 10/10
train loss= 1.2766 - train acc= 57.69% - valid loss= 1.5284 - valid acc=
54.28%
**Question 3.2:** Now, let us tune the number of blocks,
and . Write your
code for this tuning and report the result of the best model on the testing set.
Note that you need to show your code for tuning and evaluating on the test set
to earn the full marks. During tuning, you can set the instance variable
verbose of your model to True for not showing the training details of each
epoch.
Note that for this question, depending on your computational resource, you can
choose list_feature_maps= [32, 64] or list_feature_maps= [16, 32,
64] .
[3 points]
Please note that you struggle in implementing the aforementioned CNN. You can
use the MiniVGG network in our labs for doing the following questions. However,
you cannot earn any mark for 3.1 and 3.2.
use_skip ∈ {true, false} learning_rate ∈ {0.001, 0.0005}
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第24/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 3.3:** Exploring Data Mixup Technique for Improving Generalization
Ability.
[4 points]
Data mixup is another super-simple technique used to boost the generalization ability
of deep learning models. You need to incoroporate data mixup technique to the
above deep learning model and experiment its performance. There are some papers
and documents for data mixup as follows:
Main paper for data mixup link for main paper and a good article article link.
You need to extend your model developed above, train a model using data mixup, and
write your observations and comments about the result.
**Question 3.4:** Exploring CutMix Technique for Improving Generalization
Ability.
[4 points]
CutMix is another super-simple technique used to boost the generalization ability of
deep learning models. You need to incoroporate data CutMix technique to the above
deep learning model and experiment its performance. There are some papers and
documents for data mixup as follows:
Main paper for Cutmix link for main paper and a good article article link.
You need to extend your model developed above, train a model using data CutMix,
and write your observations and comments about the result.
In [ ]: #Your code here
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第25/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 3.5:** Implement the one-versus-all (OVA) loss. The details are as
follows:
You need to apply the sigmoid activation function to logits
instead of the softmax activation function as usual
to obtain , meaning that .
Note that is the number of classes.
Given a data example with the ground-truth label , the idea is to maximize the
likelihood and to minimize the likelihoods . Therefore, the objective
function is to find the model parameters to
or equivalently
.
For example, if and , you need to minimize
.
Compare the model trained with the OVA loss and the same model trained with the
standard cross-entropy loss.
[4 points]
**Question 3.6:** Attack your best obtained model with PGD attacks with
on the testing set. Write the code for the attacks
and report the robust accuracies. Also choose a random set of 20 clean images
in the testing set and visualize the original and attacked images.
[4 points]
**Question 3.7:** Train a robust model using adversarial training with PGD
. Write the code for the adversarial training and
report the robust accuracies. After finishing the training, you need to store your
best robust model in the folder ./models and load the model to evaluate the
robust accuracies for PGD and FGSM attacks with
on the testing set.
[4 points]
h = [h1,h2, . . . ,hM ]
p = [p1, p2, . . . , pM ] pi = sigmoid(hi), i = 1, . . . ,M
M
x y
py pi, i ≠ y
max{log py +∑i≠y log(1− pi)}
min{− log py −∑i≠y log(1− pi)}
M = 3 y = 2
min {− log(1− p1)− log p2 − log(1− p3)}
In [ ]: #Your code here
ϵ = 0.0313, k = 20, η = 0.002
In [ ]: #Your code here
ϵ = 0.0313, k = 10, η = 0.002
ϵ = 0.0313, k = 20, η = 0.002
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第26/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 3.8 (Kaggle competition)**
[10 points] You can reuse the best model obtained in this assignment or develop new models to
evaluate on the testing set of the FIT3181/5215 Kaggle competion. However, to
gain any points for this question, your testing accuracy must exceed the accuracy
threshold from a base model developed by us as shown in the leader board of the
competition.
The marks for this question are as follows:
If you are in top 10% of your cohort, you gain 10 points.
If you are in top 20% of your cohort, you gain 8 points.
If you are in top 30% of your cohort, you gain 6 points.
If you beat our base model, you gain 4 points.
END OF ASSIGNMENT
GOOD LUCK WITH YOUR ASSIGNMENT 1!
第1/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
FIT3181: Deep Learning (2024)
CE/Lecturer (Clayton): Dr Trung Le | trunglm@monash.edu
Lecturer (Clayton): Prof Dinh Phung | dinh.phung@monash.edu
Lecturer (Malaysia): Dr Arghya Pal | arghya.pal@monash.edu
Lecturer (Malaysia): Dr Lim Chern Hong | lim.chernhong@monash.edu
Head Tutor 3181: Miss Vy Vo | [v.vo@monash.edu ]
Head Tutor 5215: Dr Van Nguyen | [van.nguyen1@monash.edu ]
Faculty of Information Technology, Monash University, Australia
Student Information
Surname: [Enter your surname here]
Firstname: [Enter your firstname here ]
Student ID: [Enter your ID here ]
Email: [Enter your email here ]
Your tutorial time: [Enter your tutorial time here ]
Deep Neural Networks
Due: 11:55pm Sunday, 8 September 2024 (Sunday)
Important note: This is an individual assignment. It contributes 25% to
your final mark. Read the assignment instructions carefully.
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第2/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
What to submit
This assignment is to be completed individually and submitted to Moodle unit site. By
the due date, you are required to submit one single zip file, named
xxx_assignment01_solution.zip where xxx is your student ID, to the
corresponding Assignment (Dropbox) in Moodle. You can use Google Colab to do
Assigmnent 1 but you need to save it to an *.ipynb file to submit to the unit
Moodle.
More importantly, if you use Google Colab to do this assignment, you need to
first make a copy of this notebook on your Google drive.
For example, if your student ID is 12356, then gather all of your assignment
solution to folder, create a zip file named 123456_assignment01_solution.zip
and submit this file.
Within this zip folder, you must submit the following files:
1. Assignment01_solution.ipynb: this is your Python notebook solution source
file.
2. Assignment01_output.html: this is the output of your Python notebook solution
exported in html format.
3. Any extra files or folder needed to complete your assignment (e.g., images
used in your answers).
Since the notebook is quite big to load and work together, one recommended option
is to split solution into three parts and work on them seperately. In that case, replace
Assignment01_solution.ipynb by three notebooks:
Assignment01_Part1_solution.ipynb, Assignment01_Part2_solution.ipynb and
Assignment01_Part3_solution.ipynb
You can run your codes on Google Colab. In this case, you have to make a copy
of your Google colab notebook including the traces and progresses of model
training before submitting.
Part 1: Theory and Knowledge Questions
[Total marks for this part: 30 points]
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第3/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
The first part of this assignment is to demonstrate your knowledge in deep learning
that you have acquired from the lectures and tutorials materials. Most of the contents
in this assignment are drawn from the lectures and tutorials from weeks 1 to 4.
Going through these materials before attempting this part is highly recommended.
**Question 1.1** Activation function plays an important role in
modern Deep NNs. For each of the activation functions below, state
its output range, find its derivative (show your steps), and plot the
activation fuction and its derivative
**(a)** Exponential linear unit (ELU):
[1.5 points]
**(b)** Gaussian Error Linear Unit (GELU): where is the
probability cummulative function of the standard Gaussian distribution or
where . In addition, the GELU activation fuction
(the link for the main paper) has been widely used in the state-of-the-art Vision for
Transformers (e.g., here is the link for the main ViT paper).
[1.5 points]
Write your answer here. You can add more cells if needed.
**Question 1.2:** Assume that we feed a data point with a ground-
truth label to the feed-forward neural network with the ReLU
activation function as shown in the following figure
ELU(x) = { 0.1 (exp(x)− 1) ifx ≤ 0
x ifx > 0
GELU(x) = xΦ(x) Φ(x)
Φ(x) = P (X ≤ x) X ∼ N (0, 1)
x
y = 2
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第4/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**(a)** What is the numerical value of the latent presentation ?
[1 point]
**(b)** What is the numerical value of the latent presentation ?
[1 point]
**(c)** What is the numerical value of the logit ?
[1 point]
**(d)** What is the corresonding prediction probabilities ?
[1 point]
**(e)** What is the predicted label ? Is it a correct and an incorect prediction?
Remind that .
[1 point]
**(f)** What is the cross-entropy loss caused by the feed-forward neural network at
? Remind that .
[1 point]
**(g)** Why is the cross-entropy loss caused by the feed-forward neural network at
(i.e., ) always non-negative? When does this loss
get the value ? Note that you need to answer this question for a general pair
and a general feed-forward neural network with, for example classes?
[1 point]
You must show both formulas and numerical results for earning full mark. Although it
is optional, it is great if you show your PyTorch code for your computation.
**Question 1.3:**
For Question 1.3, you have two options:
(1) perform the forward, backward propagation, and SGD update for one
mini-batch (10 points), or
(2) manually implement a feed-forward neural network that can work on real
tabular datasets (20 points).
You can choose either (1) or (2) to proceed.
**Option 1**
[Total marks for this option: 10 points]
h1(x)
h2(x)
h3(x)
p(x)
yˆ
y = 2
(x, y) y = 2
(x, y) CE(1y, p(x)) CE(1y, p(x))
0 (x, y)
M = 4
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第5/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Assume that we are constructing a multilayered feed-forward neural network
for a classification problem with three classes where the model parameters will
be generated randomly using your student ID. The architecture of this network
is as shown in the following figure. Note
that the ELU has the same formula as the one in Q1.1.
We feed a batch with the labels as shown in the figure. Answer the following
questions.
You need to show both formulas, numerical results, and your PyTorch code for
your computation for earning full marks.
Forward propagation
**(a)** What is the value of (the pre-activation values of )?
[0.5 point]
**(b)** What is the value of ?
[0.5 point]
3(Input)→ 5(ELU)→ 3(Output)
X Y
In [ ]: import torch
student_id = 1234 #insert your student id here for example 1234
torch.manual_seed(student_id)
Out[ ]:
In [ ]: #Code to generate random matrices and biases for W1, b1, W2, b2
h¯
1(x) h1
In [ ]: #Show your code
h1(x)
In [ ]: #Show your code
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第6/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**(c)** What is the predicted value ?
[0.5 point]
(d) Suppose that we use the cross-entropy (CE) loss. What is the value of the CE loss
incurred by the mini-batch?
[0.5 point]
Backward propagation
**(e)** What are the derivatives , and ?
[3 points]
**(f)** What are the derivatives , and ?
[3 points]
SGD update
**(g)** Assume that we use SGD with learning rate to update the model
parameters. What are the values of and after updating?
[2 points]
**Option 2**
[Total marks for this option: 20 points]
yˆ
In [ ]: #Show your code
l
In [ ]: #Show your code
,∂l
∂h2
∂l
∂W 2
∂l
∂b2
In [ ]: #Show your code
, ,∂l
∂h1
∂l
∂h¯
1
∂l
∂W 1
∂l
∂b1
In [ ]: #Show your code
η = 0.01
W 2, b2 W 1, b1
In [ ]: #Show your code
In [ ]: import torch
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第7/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
In Option 2, you need to implement a feed-forward NN manually using PyTorch
and auto-differentiation of PyTorch. We then manually train the model on the
MNIST dataset.
We first download the MNIST dataset and preprocess it.
Each data point has dimension [28,28] . We need to flatten it to a vector to input to
our FFN.
Develop the feed-forward neural networks
(a) You need to develop the class MyLinear with the following skeleton. You need
to declare the weight matrix and bias of this linear layer.
[3 points]
In [ ]: transform = transforms.Compose([
transforms.ToTensor(), # Convert the image to a tensor with shape [C, H, W]
transforms.Normalize((0.5,), (0.5,)), # Normalize to [-1, 1]
transforms.Lambda(lambda x: x.view(28*28)) # Flatten the tensor to shape [-1,HW]
])
# Load the MNIST dataset
train_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
test_dataset = datasets.MNIST(root='./data', train=False, download=True, transform=transform)
train_data, train_labels = train_dataset.data, train_dataset.targets
test_data, test_labels = test_dataset.data, test_dataset.targets
print(train_data.shape, train_labels.shape)
print(test_data.shape, test_labels.shape)
In [ ]: train_dataset.data = train_data.data.view(-1, 28*28)
test_dataset.data = test_data.data.view(-1, 28*28)
train_data, train_labels = train_dataset.data, train_dataset.targets
test_data, test_labels = test_dataset.data, test_dataset.targets
print(train_data.shape, train_labels.shape)
print(test_data.shape, test_labels.shape)
In [ ]: train_loader = DataLoader(dataset=train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(dataset=test_dataset, batch_size=64, shuffle=False)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第8/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
(b) You need to develop the class MyFFN with the following skeleton
[7 points]
In [ ]: class MyLinear(torch.nn.Module):
def __init__(self, input_size, output_size):
"""
input_size: the size of the input
output_size: the size of the output
"""
super().__init__()
#Your code here
self.W =
self.b =
#forward propagation
def forward(self, x): #x is a mini-batch
#Your code here
In [ ]: class MyFFN(torch.nn.Module):
def __init__(self, input_size, num_classes, hidden_sizes, act = torch.nn.ReLU()):
"""
input_size: the size of the input
num_classes: the number of classes
act is the activation function
hidden_sizes is the list of hidden sizes
for example input_size = 3, hidden_sizes = [5, 7], num_classes = 4, and act = torch.nn.ReLU()
means that we are building up a FFN with the confirguration
(3 (Input) -> 5 (ReLU) -> 7 (ReLU) -> 4 (Output))
"""
super(MyFFN, self).__init__()
self.input_size = input_size
self.num_classes = num_classes
self.act = act
self.hidden_sizes = hidden_sizes
self.num_layers = len(hidden_sizes) + 1
def create_FFN(self):
"""
This function creates the feed-forward neural network
We stack many MyLinear layers
"""
hidden_sizes = [self.input_size] + self.hidden_sizes + [self.num_classes]
self.layers = []
#Your code here
def forward(self,x):
"""
This implements the forward propagation of the batch x
This needs to return the prediction probabilities of x
"""
#Your code here
def compute_loss(self, x, y):
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第9/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
(c) Write the code to evaluate the accuracy of the current myFFN model on a data
loader (e.g., train_loader or test_loader).
[2.5 points]
(c) Write the code to evaluate the loss of the current myFFN model on a data loader
(e.g., train_loader or test_loader).
[2.5 points]
"""
This function computes the cross-entropy loss
You can use the built-in CE loss of PyTorch
"""
#Your code here
def update_SGD(self, x, y, learning_rate = 0.01):
"""
This function updates the model parameters using SGD using the batch (x,y)
You need to implement the update rule manually and cannot rely on the built-in optimizer
"""
#Your code here
def update_SGDwithMomentum(self, x, y, learning_rate = 0.01, momentum = 0.9):
"""
This function updates the model parameters using SGD with momentum using the batch (x,y)
You need to implement the update rule manually and cannot rely on the built-in optimizer
"""
#Your code here
def update_AdaGrad(self, x, y, learning_rate = 0.01):
"""
This function updates the model parameters using AdaGrad using the batch (x,y)
You need to implement the update rule manually and cannot rely on the built-in optimizer
"""
#Your code here
In [ ]: myFFN = MyFFN(input_size = 28*28, num_classes = 10, hidden_sizes = [100, 100], act = torch.nn.ReLU)
myFFN.create_FFN()
print(myFFN)
In [ ]: def compute_acc(model, data_loader):
"""
This function computes the accuracy of the model on a data loader
"""
#Your code here
In [ ]: def compute_loss(model, data_loader):
"""
This function computes the loss of the model on a data loader
"""
#Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第10/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Train on the MNIST data with 50 epochs using updateSGD .
(d) Implement the function updateSGDMomentum in the class and train the model
with this optimizer in 50 epochs. You can update the corresponding function in the
MyFNN class.
[2.5 points]
(e) Implement the function updateAdagrad in the class and train the model with
this optimizer in 50 epochs. You can update the corresponding function in the
MyFNN class.
[2.5 points]
Part 2: Deep Neural Networks (DNN)
[Total marks for this part: 25 points]
The second part of this assignment is to demonstrate your basis knowledge in deep
learning that you have acquired from the lectures and tutorials materials. Most of the
contents in this assignment are drawn from the tutorials covered from weeks 1 to
2. Going through these materials before attempting this assignment is highly
recommended.
In the second part of this assignment, you are going to work with the FashionMNIST
dataset for image recognition task. It has the exact same format as MNIST (70,000
grayscale images of 28 × 28 pixels each with 10 classes), but the images represent
fashion items rather than handwritten digits, so each class is more diverse, and the
problem is significantly more challenging than MNIST.
In [ ]: num_epochs = 50
for epoch in range(num_epochs):
for i, (x, y) in enumerate(train_loader):
myFFN.update_SGD(x, y, learning_rate = 0.01)
train_acc = compute_acc(myFFN, train_loader)
train_loss = compute_loss(myFFN, train_loader)
test_acc = compute_acc(myFFN, test_loader)
test_loss = compute_loss(myFFN, test_loader)
print(f"Epoch {epoch+1}/{num_epochs}, Train Loss: {train_loss:.4f}, Train Acc: {train_acc*100:.2f}%, Test Loss: {test_loss:.4f}, Test Acc: {test_acc*100:.2f}%")
In [ ]: #Your code here
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第11/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Load the Fashion MNIST using torchvision
torch.Size([60000, 28, 28]) torch.Size([60000])
torch.Size([10000, 28, 28]) torch.Size([10000])
torch.Size([60000, 784]) torch.Size([60000])
torch.Size([10000, 784]) torch.Size([10000])
Number of training samples: 18827
Number of training samples: 16944
Number of validation samples: 1883
**Question 2.1:** Write the code to visualize a mini-batch in
train_loader including its images and labels.
[5 points]
In [ ]: import torch
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
torch.manual_seed(1234)
In [ ]: transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
train_dataset_orgin = datasets.FashionMNIST(root='./data', train=True, download=True, transform=transform)
test_dataset = datasets.FashionMNIST(root='./data', train=False, download=True, transform=transform)
print(train_dataset_orgin.data.shape, train_dataset_orgin.targets.shape)
print(test_dataset.data.shape, test_dataset.targets.shape)
train_dataset_orgin.data = train_dataset_orgin.data.view(-1, 28*28)
test_dataset.data = test_dataset.data.view(-1, 28*28)
print(train_dataset_orgin.data.shape, train_dataset_orgin.targets.shape)
print(test_dataset.data.shape, test_dataset.targets.shape)
N = len(train_dataset_orgin)
print(f"Number of training samples: {N}")
N_train = int(0.9*N)
N_val = N - N_train
print(f"Number of training samples: {N_train}")
print(f"Number of validation samples: {N_val}")
train_dataset, val_dataset = torch.utils.data.random_split(train_dataset_orgin, [N_train, N_val])
train_loader = DataLoader(dataset=train_dataset_orgin, batch_size=64, shuffle=True)
val_loader = DataLoader(dataset=val_dataset, batch_size=64, shuffle=False)
test_loader = DataLoader(dataset=test_dataset, batch_size=1000, shuffle=False)
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第12/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 2.2:** Write the code for the feed-forward neural net using
PyTorch
[5 points]
We now develop a feed-forward neural network with the architecture
. You can choose your own way
to implement your network and an optimizer of interest. You should train model in
epochs and evaluate the trained model on the test set.
**Question 2.3:** Tuning hyper-parameters with grid search
[5 points]
Assume that you need to tune the number of neurons on the first and second hidden
layers , and the used activation function
. The network has the architecture pattern
where , and are in their
grides. Write the code to tune the hyper-parameters , and . Note that you
can freely choose the optimizer and learning rate of interest for this task.
**Question 2.4:** Implement the loss with the form:
where is
the entropy of , is the prediction probabilities of a data point with
the ground-truth label , is an one-hot label, and is a trade-
off parameter. Set to train a model.
[5 points]
784→ 40(ReLU)→ 30(ReLU)→ 10(softmax)
50
In [ ]: #Your code here
n1 ∈ {20, 40} n2 ∈ {20, 40}
act ∈ {sigmoid, tanh, relu}
784→ n1(act)→ n2(act)→ 10(softmax) n1,n2 act
n1,n2 act
In [ ]: #Your code here
loss(p, y) = CE(1y, p) + λH(p) H(p) = −∑
M
i=1 pi log pi
p p x
y 1y λ > 0
λ = 0.1
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第13/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 2.5:** Experimenting with sharpness-aware minimization
technique
[5 points]
Sharpness-aware minimization (SAM) (i.e., link for main paper from Google
Deepmind) is a simple yet but efficient technique to improve the generalization ability
of deep learning models on unseen data examples. In your research or your work, you
might potentially use this idea. Your task is to read the paper and implement
Sharpness-aware minimization (SAM). Finally, you need to apply SAM to the best
architecture found in Question 2.3.
Part 3: Convolutional Neural Networks and Image
Classification
[Total marks for this part: 45 points]
The third part of this assignment is to demonstrate your basis knowledge in deep
learning that you have acquired from the lectures and tutorials materials. Most of the
contents in this assignment are drawn from the tutorials covered from weeks 3 to
6. Going through these materials before attempting this assignment is highly
recommended.
The dataset used for this part is a specific dataset for this unit consisting of
approximately images of classes of Animals, each of which has
approximately 500 images. You can download the dataset at download here if
you want to do your assignment on your machine.
In [ ]: #Your code here
10, 000 20
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第14/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
CUDA is not available. Training on CPU ...
Download the dataset to the folder of this Google Colab.
Downloading...
From (original): https://drive.google.com/uc?id=1aEkxNWaD02Z8ZNvZzeMefUoY
97C-3wTG
From (redirected): https://drive.google.com/uc?id=1aEkxNWaD02Z8ZNvZzeMefU
oY97C-3wTG&confirm=t&uuid=c6914688-3441-4198-b7a3-ee55f3c61869
To: /content/Animals_Dataset.zip
100% 643M/643M [00:13<00:00, 48.0MB/s]
We unzip the dataset to the folder.
In [ ]: import os
import requests
import tarfile
import time
from torchvision import datasets, transforms
from torch.utils.data import DataLoader, random_split
import torchvision.models as models
import torch.nn as nn
import torch
import PIL.Image
import pathlib
from torchsummary import summary
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
# check if CUDA is available
train_on_gpu = torch.cuda.is_available()
if not train_on_gpu:
print('CUDA is not available. Training on CPU ...')
else:
print('CUDA is available! Training on GPU ...')
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.manual_seed(1234)
Out[ ]:
In [ ]: !gdown --fuzzy https://drive.google.com/file/d/1aEkxNWaD02Z8ZNvZzeMefUoY97C-3wTG/view?usp=drive_link
# !gdown --fuzzy https://drive.google.com/file/d/1qdElRqDS4TitXfv_iG_TFQSi9QIfy4uM/view?usp=drive_link # backup url
In [ ]: !unzip -q Animals_Dataset.zip
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第15/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Number of instance in train_set: 8519
Number of instance in val_set: 947
In [ ]: data_dir = "./FIT5215_Dataset"
# We resize the images to [3,64,64]
transform = transforms.Compose([transforms.Resize((64,64)), #resises the image so it can be perfect for our model.
transforms.RandomHorizontalFlip(), # FLips the image w.r.t horizontal axis
#transforms.RandomRotation(4), #Rotates the image to a specified angel
#transforms.RandomAffine(0, shear=10, scale=(0.8,1.2)), #Performs actions like zooms, change shear angles.
transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2), # Set the color params
transforms.ToTensor(), # convert the image to tensor so that it can work with torch
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), # Normalize the images, each R,G,B value is normalized with mean=0.5 and std=0.5
])
# Load the dataset using torchvision.datasets.ImageFolder and apply transformations
dataset = datasets.ImageFolder(data_dir, transform=transform)
# Split the dataset into training and validation sets
train_size = int(0.9 * len(dataset))
valid_size = len(dataset) - train_size
train_dataset, val_dataset = random_split(dataset, [train_size, valid_size])
# Example of DataLoader creation for training and validation
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False)
print("Number of instance in train_set: %s" % len(train_dataset))
print("Number of instance in val_set: %s" % len(val_dataset))
In [ ]: class_names = ['bird', 'bottle', 'bread', 'butterfly', 'cake', 'cat', 'chicken', 'cow', 'dog', 'duck',
'elephant', 'fish', 'handgun', 'horse', 'lion', 'lipstick', 'seal', 'snake', 'spider', 'vase']
In [ ]: # obtain one batch of training images
dataiter = iter(train_loader)
images, labels = next(dataiter)
images = images.numpy() # convert images to numpy for display
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第16/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
For questions 3.1 to 3.7, you'll need to write your own model in a way that makes
it easy for you to experiment with different architectures and parameters. The
goal is to be able to pass the parameters to initialize a new instance of
YourModel to build different network architectures with different parameters.
Below are descriptions of some parameters for YourModel :
In [ ]: import math
def imshow(img):
img = img / 2 + 0.5 # unnormalize
plt.imshow(np.transpose(img, (1, 2, 0))) # convert from Tensor image
def visualize_data(images, categories, images_per_row = 8):
class_names = ['bird', 'bottle', 'bread', 'butterfly', 'cake', 'cat', 'chicken', 'cow', 'dog', 'duck',
'elephant', 'fish', 'handgun', 'horse', 'lion', 'lipstick', 'seal', 'snake', 'spider', 'vase']
n_images = len(images)
n_rows = math.ceil(float(n_images)/images_per_row)
fig = plt.figure(figsize=(1.5*images_per_row, 1.5*n_rows))
fig.patch.set_facecolor('white')
for i in range(n_images):
plt.subplot(n_rows, images_per_row, i+1)
plt.xticks([])
plt.yticks([])
imshow(images[i])
class_index = categories[i]
plt.xlabel(class_names[class_index])
plt.show()
In [ ]: visualize_data(images, labels)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第17/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
1. Block confirguration : Our network consists of many blocks. Each block
has the pattern [conv, batch norm, activation, conv, batch norm,
activation, max pool, dropout] . All convolutional layers have filter size
, strides and padding = 1, and all max pool layers have strides ,
kernel size , and padding = 0. The network will consists of a few blocks before
applying a linear layer to output the logits for the softmax layer.
2. list_feature_maps : the number of feature maps in the blocks of the
network. For example, if list_feature_maps = [16, 32, 64] , our network
has two blocks with the input_channels or number of feature maps are 16, 32 ,
and 64 respectively.
3. drop_rate : the keep probability for dropout. Setting drop_rate to
means not using dropout.
4. batch_norm : the batch normalization function is used or not. Setting
batch_norm to false means not using batch normalization.
5. use_skip : the skip connection is used in the blocks or not. Setting this to
true means that we use 1x1 Conv2D with strides=2 for the skip
connection.
6. At the end, you need to apply global average pooling (GAP)
( AdaptiveAvgPool2d((1, 1)) ) to flatten the 3D output tensor before
defining the output linear layer for predicting the labels.
Here is the model confirguration of YourCNN if the list_feature_maps = [16,
32, 64] and batch_norm = true .
(3, 3) (1, 1) (2, 2)
2
0.0
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第18/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 3.1:** You need to implement the aforementioned CNN.
First, you need to implement the block of our CNN in the class YourBlock . You
can ignore use_skip and skip connection for simplicity. However, you
cannot earn full marks for this question.
[6 points]
In [ ]: #Your code here
class YourBlock(nn.Module):
def __init__(self, in_feature_maps, out_feature_maps, drop_rate = 0.2, batch_norm = True, use_skip = True):
super(YourBlock, self).__init__()
self.use_skip = use_skip
#Your code here
def forward(self, x):
#Write your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第19/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Second, you need to use the above YourBlock to implement the class
YourCNN .
[6 points]
We declare my_cnn from YourCNN as follows.
In [ ]: class YourCNN(nn.Module):
def __init__(self, list_feature_maps = [16, 32, 64], drop_rate = 0.2, batch_norm= True, use_skip = True):
super(YourCNN, self).__init__()
layers = []
#Write your code here
def forward(self, x):
#Write your code here
In [ ]: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
my_cnn = YourCNN(list_feature_maps = [16, 32, 64], use_skip = True)
my_cnn = my_cnn.to(device)
print(my_cnn)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第20/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
YourCNN(
(block): ModuleList(
(0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_runn
ing_stats=True)
(2): ReLU(inplace=True)
(3): YourBlock(
(block): ModuleList(
(0): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(
1, 1))
(1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_
running_stats=True)
(2): ReLU(inplace=True)
(3): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(
1, 1))
(4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_
running_stats=True)
(5): ReLU(inplace=True)
(6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ce
il_mode=False)
(7): Dropout(p=0.2, inplace=False)
)
(skip_conv): Conv2d(16, 32, kernel_size=(1, 1), stride=(2, 2))
)
(4): YourBlock(
(block): ModuleList(
(0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(
1, 1))
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_
running_stats=True)
(2): ReLU(inplace=True)
(3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(
1, 1))
(4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_
running_stats=True)
(5): ReLU(inplace=True)
(6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ce
il_mode=False)
(7): Dropout(p=0.2, inplace=False)
)
(skip_conv): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2))
)
(5): AdaptiveAvgPool2d(output_size=(1, 1))
(6): Flatten(start_dim=1, end_dim=-1)
(7): Linear(in_features=64, out_features=20, bias=True)
)
)
We declare the optimizer and the loss function.
Here are the codes to compute the loss and accuracy.
In [ ]: # Loss and optimizer
learning_rate = 0.001
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(my_cnn.parameters(), lr=learning_rate)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第21/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Here is the code to train our model.
In [ ]: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
def compute_loss(model, loss_fn, loader):
loss = 0
# Set model to eval mode for inference
model.eval()
with torch.no_grad(): # No need to track gradients for validation
for (batchX, batchY) in loader:
# Move data to the same device as the model
batchX, batchY = batchX.to(device).type(torch.float32), batchY.to(device).type(torch.long)
loss += loss_fn(model(batchX), batchY)
# Set model back to train mode
model.train()
return float(loss)/len(loader)
In [ ]: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
def compute_acc(model, loader):
correct = 0
totals = 0
# Set model to eval mode for inference
model.eval()
for (batchX, batchY) in loader:
# Move batchX and batchY to the same device as the model
batchX, batchY = batchX.to(device).type(torch.float32), batchY.to(device)
outputs = model(batchX) # feed batch to the model
totals += batchY.size(0) # accumulate totals with the current batch size
predicted = torch.argmax(outputs.data, 1) # get the predicted class
# Move batchY to the same device as predicted for comparison
correct += (predicted == batchY).sum().item()
return correct / totals
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第22/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
In [ ]: device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
def fit(model= None, train_loader = None, valid_loader= None, optimizer = None,
num_epochs = 50, verbose = True, seed= 1234):
torch.manual_seed(seed)
# Move the model to the device before initializing the optimizer
model.to(device) # Move the model to the GPU
if optimizer == None:
optim = torch.optim.Adam(model.parameters(), lr = 0.001) # Now initialize optimizer with model on GPU
else:
optim = optimizer
history = dict()
history['val_loss'] = list()
history['val_acc'] = list()
history['train_loss'] = list()
history['train_acc'] = list()
for epoch in range(num_epochs):
model.train()
for (X, y) in train_loader:
# Move input data to the same device as the model
X,y = X.to(device), y.to(device)
# Forward pass
outputs = model(X.type(torch.float32)) # X is already on the correct device
loss = loss_fn(outputs, y.type(torch.long))
# Backward and optimize
optim.zero_grad()
loss.backward()
optim.step()
#losses and accuracies for epoch
val_loss = compute_loss(model, loss_fn, valid_loader)
val_acc = compute_acc(model, valid_loader)
train_loss = compute_loss(model, loss_fn, train_loader)
train_acc = compute_acc(model, train_loader)
history['val_loss'].append(val_loss)
history['val_acc'].append(val_acc)
history['train_loss'].append(train_loss)
history['train_acc'].append(train_acc)
if not verbose: #verbose = True means we do show the training information during training
print(f"Epoch {epoch+1}/{num_epochs}")
print(f"train loss= {train_loss:.4f} - train acc= {train_acc*100:.2f}% - valid loss= {val_loss:.4f} - valid acc= {val_acc*100:.2f}%")
return history
In [ ]: history = fit(model= my_cnn, train_loader=train_loader, valid_loader = val_loader, optimizer = optimizer, num_epochs= 10, verbose = False)
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第23/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
Epoch 1/10
train loss= 2.2247 - train acc= 29.49% - valid loss= 2.3166 - valid acc=
29.88%
Epoch 2/10
train loss= 1.9459 - train acc= 38.62% - valid loss= 2.2254 - valid acc=
34.42%
Epoch 3/10
train loss= 1.7757 - train acc= 42.18% - valid loss= 1.8783 - valid acc=
42.13%
Epoch 4/10
train loss= 1.6908 - train acc= 44.52% - valid loss= 1.7666 - valid acc=
40.55%
Epoch 5/10
train loss= 1.6232 - train acc= 47.31% - valid loss= 1.8377 - valid acc=
44.56%
Epoch 6/10
train loss= 1.5740 - train acc= 48.37% - valid loss= 1.7418 - valid acc=
47.41%
Epoch 7/10
train loss= 1.4314 - train acc= 53.27% - valid loss= 1.7969 - valid acc=
52.06%
Epoch 8/10
train loss= 1.3645 - train acc= 54.60% - valid loss= 1.4301 - valid acc=
52.16%
Epoch 9/10
train loss= 1.3372 - train acc= 55.76% - valid loss= 1.4144 - valid acc=
53.64%
Epoch 10/10
train loss= 1.2766 - train acc= 57.69% - valid loss= 1.5284 - valid acc=
54.28%
**Question 3.2:** Now, let us tune the number of blocks,
and . Write your
code for this tuning and report the result of the best model on the testing set.
Note that you need to show your code for tuning and evaluating on the test set
to earn the full marks. During tuning, you can set the instance variable
verbose of your model to True for not showing the training details of each
epoch.
Note that for this question, depending on your computational resource, you can
choose list_feature_maps= [32, 64] or list_feature_maps= [16, 32,
64] .
[3 points]
Please note that you struggle in implementing the aforementioned CNN. You can
use the MiniVGG network in our labs for doing the following questions. However,
you cannot earn any mark for 3.1 and 3.2.
use_skip ∈ {true, false} learning_rate ∈ {0.001, 0.0005}
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第24/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 3.3:** Exploring Data Mixup Technique for Improving Generalization
Ability.
[4 points]
Data mixup is another super-simple technique used to boost the generalization ability
of deep learning models. You need to incoroporate data mixup technique to the
above deep learning model and experiment its performance. There are some papers
and documents for data mixup as follows:
Main paper for data mixup link for main paper and a good article article link.
You need to extend your model developed above, train a model using data mixup, and
write your observations and comments about the result.
**Question 3.4:** Exploring CutMix Technique for Improving Generalization
Ability.
[4 points]
CutMix is another super-simple technique used to boost the generalization ability of
deep learning models. You need to incoroporate data CutMix technique to the above
deep learning model and experiment its performance. There are some papers and
documents for data mixup as follows:
Main paper for Cutmix link for main paper and a good article article link.
You need to extend your model developed above, train a model using data CutMix,
and write your observations and comments about the result.
In [ ]: #Your code here
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第25/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 3.5:** Implement the one-versus-all (OVA) loss. The details are as
follows:
You need to apply the sigmoid activation function to logits
instead of the softmax activation function as usual
to obtain , meaning that .
Note that is the number of classes.
Given a data example with the ground-truth label , the idea is to maximize the
likelihood and to minimize the likelihoods . Therefore, the objective
function is to find the model parameters to
or equivalently
.
For example, if and , you need to minimize
.
Compare the model trained with the OVA loss and the same model trained with the
standard cross-entropy loss.
[4 points]
**Question 3.6:** Attack your best obtained model with PGD attacks with
on the testing set. Write the code for the attacks
and report the robust accuracies. Also choose a random set of 20 clean images
in the testing set and visualize the original and attacked images.
[4 points]
**Question 3.7:** Train a robust model using adversarial training with PGD
. Write the code for the adversarial training and
report the robust accuracies. After finishing the training, you need to store your
best robust model in the folder ./models and load the model to evaluate the
robust accuracies for PGD and FGSM attacks with
on the testing set.
[4 points]
h = [h1,h2, . . . ,hM ]
p = [p1, p2, . . . , pM ] pi = sigmoid(hi), i = 1, . . . ,M
M
x y
py pi, i ≠ y
max{log py +∑i≠y log(1− pi)}
min{− log py −∑i≠y log(1− pi)}
M = 3 y = 2
min {− log(1− p1)− log p2 − log(1− p3)}
In [ ]: #Your code here
ϵ = 0.0313, k = 20, η = 0.002
In [ ]: #Your code here
ϵ = 0.0313, k = 10, η = 0.002
ϵ = 0.0313, k = 20, η = 0.002
In [ ]: #Your code here
4/9/2024 21:03FIT3181_DeepLearningAssignment1_Official
第26/26⻚file:///Users/yijiaxue/Downloads/FIT3181_DeepLearningAssignment1_Official.html
**Question 3.8 (Kaggle competition)**
[10 points] You can reuse the best model obtained in this assignment or develop new models to
evaluate on the testing set of the FIT3181/5215 Kaggle competion. However, to
gain any points for this question, your testing accuracy must exceed the accuracy
threshold from a base model developed by us as shown in the leader board of the
competition.
The marks for this question are as follows:
If you are in top 10% of your cohort, you gain 10 points.
If you are in top 20% of your cohort, you gain 8 points.
If you are in top 30% of your cohort, you gain 6 points.
If you beat our base model, you gain 4 points.
END OF ASSIGNMENT
GOOD LUCK WITH YOUR ASSIGNMENT 1!
