MLP 2021/22: Coursework 2 Due: 3 December 2021
Machine Learning Practical 2021/22: Coursework 2
Released: Monday 22 November 2021
Submission due: 16:00 Friday 3 December 2021
1 Introduction
The aim of this coursework is to explore the classification of images using convolutional neural networks (CNNs)
on a different dataset, CIFAR100 (pronounced as “see far 100”). CIFAR100 consists of 60,000 32 × 32 colour
images in 100 classes, with 600 images per class. The first part of the coursework concerns with debugging a
“broken” neural network and quantitatively analyzing the problem behind this broken network. The second part
involves exploring two solutions in literature for fixing this “broken” neural network, and then subsequently
implementing them to improve the performance and training of this network.
In order to support your experiments, we have acquired Google Cloud Platform credits which allow the use of
the Google Compute Engine infrastructure. You will need this to run all the tasks of the coursework (which will
be carried out in PyTorch). Each student enrolled on the MLP course will receive a $50 Google Cloud credit
coupon which is enough to carry out the experiments required for this coursework. You will receive an email
which will give you the URL you will need to access in order to request your Google Cloud Platform coupon.
Note that there is no ability to provide you with further credits, and you need to be careful to not unnecessarily
leave processes running on the Cloud that will unproductively consume your allowance.
As in the previous coursework, you will need to submit your python code and a report. The detailed submission
instructions are given in Section 8.2 – please follow these instructions carefully.
2 Github branch mlp2021-22/coursework2
The provided code and setup information for this coursework is available on the course Github repository on a
branch mlp2021-22/coursework2. To create a local working copy of this branch in your local repository you
need to do the following.
1. Make sure all modified files on the branch you are currently have been committed (see notes/getting-
started-in-a-lab.md if you are unsure how to do this).
2. Fetch changes to the upstream origin repository by running
git fetch origin
3. Checkout a new local branch from the fetched branch using
git checkout -b coursework2 origin/mlp2021-22/coursework2
You will now have a new branch in your local repository with everything you need to carry out the coursework.
Before continuing, remember to run bash install.sh to install some additional dependencies re-
quired. This assumes that you have already installed your environment as explained in Lab1, under
notes/environment-set-up.md.
This branch includes the following additions to your setup:
• For the PyTorch Framework:
– A Jupyter notebook, Coursework_2_Pytorch_Introduction.ipynb, which introduces Pytorch,
and provides further resources on tutorials, documentation and debugging.
1
MLP 2021/22: Coursework 2 Due: 3 December 2021
– A directory pytorch_mlp_framework/, which includes tooling and ready to run scripts
to enable straightforward experimentation on GPU. Documentation on this is included in
notes/pytorch-experiment-framework.md
– A note on how to use the Google Cloud Platform (using your student credits),
notes/google_cloud_setup.md.
• For the report:
– A directory called report which contains the LaTeX template and style files for your report. You
should copy all these files into the directory which will contain your report.
3 Tasks
This coursework has three tasks and objectives:
• Diagnose a deep CNN that cannot be properly trained due to optimization issues. Discuss what the
problem is, why it happens in a deep CNN and quantitatively analyze the problem.
• Describe the solutions, Batch Normalization (BN) and Residual Connection in (RC) detail and explain
how these methods address the aforementioned problem.
• Implement BN and BN+RC, run the respective experiments and then present and discuss these and other
results provided in the Template.
Carrying out larger CNN experiments using the MLP numpy framework is computationally inefficient because
(1) it runs on CPU and not on GPU, and (2) a default implementation of a convolutional layer is unlikely to be
computationally efficient. For these reasons in this coursework, which involves training CNNs, experiments we
will use GPU computing (on the Google Compute Engine) and the efficient PyTorch framework.
4 Task 1: Identifying training problems of a deep CNN
Time budget1: This section should take about 20% of the time you have allocated for your coursework.
This part of the coursework involves debugging and tuning deep CNNs using PyTorch and the Google Compute
Engine.
Identifying and resolving any optimization-related problems that might prevent your model from fitting the
training set are critical skills when working with deep neural networks. Being able to work around this problem
is key to building high performance deep neural networks.
4.1 Introducing our broken CNN
Note: This section’s objective is to test your knowledge, understanding and research skill. It’s not meant
to be an implementation-oriented section. The solutions only require you to add at most 4-5 lines of code.
Using the Pytorch-based research framework that can be found in the pytorch_mlp_framework folder, we
have built, trained and evaluated 2 deep CNNs. One consisting of a total of 7 convolutional layers + one fully
connected, and another consisting of 37 convolutional layers + one fully connected layer. Figure 1 illustrates the
training/val loss performance of the two models. One can clearly see that the 37 layer CNN (VGG_38) was
1 These are simply suggested guidelines, feel free to adjust them based on your own preferences
2
MLP 2021/22: Coursework 2 Due: 3 December 2021
0 20 40 60 80 100
Epoch number
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Lo
ss
VGG_38_train_loss
VGG_38_val_loss
VGG_08_train_loss
VGG_08_val_loss
0 20 40 60 80 100
Epoch number
0.0
0.1
0.2
0.3
0.4
0.5
Ac
cu
ra
cy VGG_38_train_acc
VGG_38_val_acc
VGG_08_train_acc
VGG_08_val_acc
Figure 1: Training and validation plots of Healthy (VGG_08) vs Broken (VGG_38) in terms of error and
accuracy.
unable to minimize its loss, unlike the healthy 7 layer CNN (VGG_08) which converges to a low error. Given
that we know that extra layers means more abstraction power, parameters and capacity, one would expect the
deeper model to be doing better at learning than the shallow one, however this is simply not the case.
Identifying the problem. One of our two networks suffers from the Vanishing Gradients problem as you can
see in Figure 1. You can reproduce the figures by running the notebooks/Plot_Results.ipynb notebook.
This file takes as input the collected metrics in folders VGG_08 and VGG_38.
Quantitative analysis of the problem. Question 1 on the template asks you to identify the problem and support
your position using arguments based on quantitative observations. For instance, Figure 2 shows how the gradient
flows in the healthy network, VGG_08 across layers. These are the gradients with respect to the weights of the
model. Visualize and discuss how gradient flows for the broken network affects/does not affect the loss curves,
training and convergence of the VGG_38.
Implementation details. The curves shown in Figure 1 can be reproduced by running the bash scripts
to train each model from scratch with the default settings given in run_vgg_08_default.sh and
run_vgg_38_default.sh.
For reproducing Figure 2 and visualizing the gradient flows for the broken network, implement the function
plot_grad_flows() within pytorch_mlp_framework/experiment_builder.py. This function takes as
input the model parameters during training, accumulates the absolute mean of the gradients in all_grads and
the layer names in layers. The matplotlib function plt plots gradient values for each layer and the function
plot_grad_flows() returns this final plot.
Template Questions related to Task 1 are:
• Question Figure 3
• Question 1
(20 Marks)
5 Task 2: Proposed Solutions
Time budget1: This section should take about 20% of the time you have allocated for your coursework.
There exist several methods that can improve the training performance of the broken 37 layer CNN introduced
in Section 4.1. We ask that you explain two of those as presented in the Template, namely Batch Normalization
3
MLP 2021/22: Coursework 2 Due: 3 December 2021
in
pu
t_
co
nv
_c
on
v_
0
in
pu
t_
co
nv
_b
n_
0
re
du
ct
io
n_
bl
oc
k_
0_
co
nv
_0
re
du
ct
io
n_
bl
oc
k_
0_
co
nv
_1
re
du
ct
io
n_
bl
oc
k_
1_
co
nv
_0
re
du
ct
io
n_
bl
oc
k_
1_
co
nv
_1
re
du
ct
io
n_
bl
oc
k_
2_
co
nv
_0
re
du
ct
io
n_
bl
oc
k_
2_
co
nv
_1
we
ig
ht
_lo
gi
t_
lin
ea
r_
la
ye
r
Layers
0.00
0.05
0.10
0.15
0.20
av
er
ag
e
gr
ad
ie
nt
Gradient flow
Figure 2: Gradient Flow in each layer for the VGG_08 network.
[Ioffe and Szegedy, 2015] and Residual Connections [He et al., 2016] in detail, using relevant equations, while
also explaining how they can help mitigate the Vanishing Gradient problem.
Template Questions related to Task 2 are:
• Question 2
• Question 3
(20 Marks)
6 Task 3: Solution and Experiments
Time budget1: This section should take about 50% of the time you have allocated for your coursework.
Solution Overview. You will need to incorporate Batch Normalization, and Batch Normalization with Residual
Connections to VGG_38 model to improve its training performance. You will then run one experiment instance
for each setup and, on the Template, fill in the respective Table, and generate and add Figures. Finally, you will
present and discuss all relevant results (including those already filled in on the Template), while also making
suggestions for further experiments you could have run.
Experiments. The techniques used in these solutions are described in Ioffe and Szegedy [2015], He et al.
[2016]. We have written the MLP Pytorch framework in a way that allows you to implement said solutions with
minimal effort. Each solution will require at approximately 8-10 lines of code.
4
MLP 2021/22: Coursework 2 Due: 3 December 2021
Important Components to Note
1. The classes ConvolutionalProcessingBlock and ConvolutionalDimensionalityReductionBlock
located within pytorch_mlp_framework/model_architectures.py. The former class implements a
basic cascade of 2 convolutional layers, each followed by a Leaky ReLU activation function, while the
latter implements a basic cascade of 2 convolutional layers, with an average pooling layer in the middle,
which effectively halves the height and width dimensions of the tensor volume passing through it. The
former is used as the basic building block of the model (repeated num_blocks_per_stage times),
while the latter is used only as a dimensionality reduction layer, usually once after each network stage. A
network stage is considered a cascade of convolutional layers, followed by a dimensionality reduction
function such as average pooling.
2. Lines 43-50 in pytorch_mlp_framework/train_evaluate_image_classification_system.py
showcase how one can create an if-else structure that can read arguments from the command line in order
to choose which processing and dim_reduction blocks will be used.
3. Lines 17-27 in the same file showcase how some simple data-augmentation strategies can be applied to
the system (useful in improving the generalization of the model.)
Implementing a proposed solution
1. Copy the classes ConvolutionalProcessingBlock and ConvolutionalDimensionalityReductionBlock
located within pytorch_mlp_framework/model_architectures.py, paste them at the end of the file
and provide a name referring to your proposed solution.
2. Modify the two blocks to implement your proposed solution. Make sure your code is clear and easily
readable as you will be marked on it.
3. To create a new block ensure that you rewrite the build_module() and forward() methods.
4. Add a new clause in the if-else structure in lines 43-50 of pytorch_mlp_framework/train_evaluate
_image_classification_system.py to add a choice for a configuration using your new blocks.
5. Write small unit-tests for your blocks to ensure that the layers can fprop some data without throwing errors
related to Pytorch.
6. Once tested, you can use the argument parser to easily run experiments with your new modules. Hint:
Have a look at the block_type argument under pytorch_mlp_framework/arg_extractor.py, to
get an idea on how to easily pass module names as an argument. Additionally, we ask you to run an
experiment with a different learning rate. The parser can come in handy for this.
Using Google Compute Engine
1. You will receive an email containing the URL you will need to access in order to request a Google Cloud
Platform coupon, and information about how to do this.
2. In the coursework2 branch of the GitHub, notes/google_cloud_setup.md gives the instructions you
should follow to set up a Google Compute Engine instance to carry out this coursework.
3. The PyTorch experimental framework that is used for this coursework is described in
notes/pytorch-experiment-framework.md and in
notebooks/Coursework_2_Pytorch_experiment_framework.ipynb
Template Questions related to Task 3 are:
5
MLP 2021/22: Coursework 2 Due: 3 December 2021
• Question Figure 4
• Question Figure 5
• Question Table 1
• Question 4
Your implementations of VGG38 BN and VGG38 BN+RC will also be marked as part of this Task.
(50 Marks)
7 Report
Time budget1: This section should take about 10% of the time you have allocated for your coursework.
The report template is divided into sections which roughly corresponds to each task in the coursework specs. In
addition to the mark distribution for tasks, remaining sections of the report will be graded as follows:
Conclusion: 10 Marks
The directory coursework2/report contains a template for your report (mlp-cw2-template.tex) and
a file for filling in your answers to the questions (mlp-cw2-questions.tex); the generated pdf file
(mlp-cw2-template.pdf) is also provided, and you should read this file carefully as it contains some useful
information on the relevant theory, and about the required structure and content. The template is written in
LaTeX, and uses the file (mlp-cw2-questions.tex) to fill in the answers to the questions on the template.
You should fill in your answers in (mlp-cw2-questions.tex) and compile (mlp-cw2-template.tex) to
produce (mlp-cw2-template.pdf) which you will need to submit .
You should copy the files in the report directory to the directory containing the LaTeX file of your report, as
pdflatex will need to access these files when building the pdf document from the LaTeX source file. The
coursework 1 spec outlines how to create a pdf file from a LaTeX source file.
As discussed in the coursework 1 spec, all figures should ideally be included in your report file as vector graphics
files, rather than raster files as this will make sure all detail in the plot is visible.
If you make use of any any books, articles, web pages or other resources you should appropriately cite these in
your report. You do not need to cite material from the course lecture slides or lab notebooks.
Template Questions related to the Report are:
• Question 5
(10 Marks)
8 Mechanics
Marks: This assignment will be assessed out of 100 marks and forms 40% of your final grade for the course.
Academic conduct: Assessed work is subject to University regulations on academic conduct:
http://web.inf.ed.ac.uk/infweb/admin/policies/academic-misconduct
Extension requests: For additional information about late penalties and extension requests, see the School web
page below. Do not email any course staff directly about extension requests as these are handled by the
ITO; you must follow the instructions on the web page.
6
MLP 2021/22: Coursework 2 Due: 3 December 2021
Submission: You can submit more than once up until the submission deadline. All submissions are timestamped
automatically. Identically named files will overwrite earlier submitted versions, so we will mark the latest
submission that comes in before the deadline.
If you submit anything before the deadline, you may not resubmit after the deadline. (This policy allows us to
begin marking submissions immediately after the deadline, without having to worry that some may need to be
re-marked).
If you do not submit anything before the deadline, you may submit exactly once after the deadline, and a late
penalty will be applied to this submission unless you have received an approved extension. Please be aware
that late submissions may receive lower priority for marking, and marks may not be returned within the same
timeframe as for on-time submissions.
Warning: Unfortunately the submission system on Learn will technically allow you to submit late even if you
submitted before the deadline (i.e. it does not enforce the above policy). Don’t do this! We will mark the version
that we retrieve just after the deadline.
http://web.inf.ed.ac.uk/infweb/student-services/ito/admin/coursework-projects/late-coursework-extension-requests
Late submission penalty: Following the University guidelines, late coursework submitted without an authorised
extension will be recorded as late and the following penalties will apply: 5 percentage points will be deducted
for every calendar day or part thereof it is late, up to a maximum of 7 calendar days. After this time a mark of
zero will be recorded.
8.1 Backing up your work
It is strongly recommended you use some method for backing up your work. Those working in their AFS
homespace on DICE will have their work automatically backed up as part of the routine backup of all user
homespaces. If you are working on a personal computer you should have your own backup method in place
(e.g. saving additional copies to an external drive, syncing to a cloud service or pushing commits to your local
Git repository to a private repository on Github). Loss of work through failure to back up does not constitute
a good reason for late submission.
You may additionally wish to keep your coursework under version control in your local Git repository on the
coursework2 branch.
If you make regular commits of your work on the coursework this will allow you to better keep track of the
changes you have made and if necessary revert to previous versions of files and/or restore accidentally deleted
work. This is not however required and you should note that keeping your work under version control is a
distinct issue from backing up to guard against hard drive failure. If you are working on a personal computer
you should still keep an additional back up of your work as described above.
8.2 Submission
Your coursework submission should be done online on the Learn course webpage.
Your submission should include one zip file sxxxxxxx.zip that should contain a folder named sxxxxxxx. This
folder should have -
• your completed report as a PDF file renamed as sxxxxxxx_report.pdf, using the provided template
• your local version of the Pytorch experiment framework (mlp/pytorch_experiment_scripts), includ-
ing any changes you’ve made to existing files and any newly created files. In particular, this should contain
the model_architectures.py python script with your implementation of BN and RC.
• a copy of your Pytorch experiment directories (mlp/results), including only the .csv files for your
training, validation and test statistics. Please do not include model weights.
7
MLP 2021/22: Coursework 2 Due: 3 December 2021
Please do not submit anything else (e.g. log files).
You should copy all of the files to a single directory, sxxxxxxx, e.g.
Your folder directory structure should look like this -
sxxxxxxx
sxxxxxxx_report.pdf
mlp
pytorch_experiment_scripts/
results/
You can use this command on Linux machines to zip all the files together -
zip -r sxxxxxxx.zip sxxxxxxx
Replace sxxxxxxx with your student id.
Once you have successfully created the .zip file, you need to login to your Learn Machine Learning
Practical (2021-2022)[YR] webpage and submit the file.
• Migrate to the section Coursework on the left column on the course page.
• Click on Coursework 2.
• A page will appear where you will need to browse and upload your .zip file that you created previously in
Attach Files and then click Submit.
You can amend an existing submission by attaching a different .zip file using the Attach Files option and then
Submit again.
Note that we will only mark the last uploaded coursework in case you amend your files. Thus it is your
responsibility to make sure that correct files are uploaded.
9 Marking Guidelines
This document (Section 3 in particular) and the template report (mlp-cw2-template.pdf) provide a description
of what you are expected to do in this assignment, and how the report should be written and structured.
Assignments will be marked using the scale defined by the University Common Marking Scheme:
Numeric mark Equivalent letter grade Approximate meaning
< 40 F fail
40-49 D poor
50-59 C acceptable
60-69 B good
70-79 A3 very good/distinction
80-100 A1, A2 excellent/outstanding/high distinction
Please note the University specifications for marks above 50:
A1 90-100 Often faultless. The work is well beyond what is expected for the level of study.
A2 80-89 A truly professional piece of scholarship, often with an absence of errors.
As ‘A3’ but shows (depending upon the item of assessment): significant personal insight / creativity / originality
and / or extra depth and academic maturity in the elements of assessment.
A3 70-79
Knowledge: Comprehensive range of up-to-date material handled in a professional way.
8
MLP 2021/22: Coursework 2 Due: 3 December 2021
Understanding/handling of key concepts: Shows a command of the subject and current theory.
Focus on the subject: Clear and analytical; fully explores the subject.
Critical analysis and discussion: Shows evidence of serious thought in critically evaluating and integrating the
evidenced and ideas. Deals confidently with the complexities and subtleties of the arguments. Shows elements
of personal insight / creativity / originality.
Structure: Clear and coherent showing logical, ordered thought.
Presentation: Clear and professional with few, relatively minor flaws. Accurate referencing. Figures and tables
well constructed and accurate. Good standard of spelling and grammar.
B 60-69
Knowledge: Very good range of up-to-date material, perhaps with some gaps, handled in a competent way.
Understanding and handling of key concepts: Shows a firm grasp of the subject and current theory but there may
be gaps.
Focus on the subject: Clear focus on the subject with no or only trivial deviation.
Critical analysis and discussion: Shows initiative, the ability to think clearly, critically evaluate ideas, to bring
different ideas together, and to draw sound conclusions.
Structure: Clear and coherent showing logical, ordered thought. Additionally for code: re-usability may be
somewhat limited. No unused variables or dead code.
Presentation: Clear and well presented with few, relatively minor flaws. For writing: Accurate referencing;
using the correct referencing system. Figures and tables well-constructed and accurate. Good standard of
spelling and grammar. Alternatively for code: well-documented, readable code.
C 50-59
Knowledge: Sound but limited. Inaccuracies, if any, are minor.
Understanding and handling of key concepts: Understands the subject but does not have a firm grasp and depth
of understanding of all the key concepts.
Focus on the subject: Addresses the subject with relatively little irrelevant material.
Critical analysis and discussion: Limited critical analysis and evaluation of sources of evidence.
Structure: Reasonably clear and coherent, generally structuring ideas and information or code in a logical way.
Additionally for code: Few or no unused variables or dead code.
Presentation: Generally well presented but there may be some flaws, for example in figures, tables, referencing
technique and standard of English. Alternatively for code: generally well-documented, readable code, but with
some weaknesses.
References
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing
internal covariate shift. In ICML, pages 448–456, 2015. URL http://proceedings.mlr.press/v37/ioffe15.html.