CP322B Machine Learning - Assignment 3
Due Date: Mar 23, 2022 at 11:59 PM
About Submission
When writing and submitting your assignments follow these requirements:
• You are expected to submit a single IPython Notebook file for this assignment. Answers to the conceptual questions can
be embedded to the notebook file as markdown cells, and you may use heading cells to further organize your document.
Do not submit the dataset.
• Late assignment submissions will not be accepted and will be marked with 0.
• Your assignment should be submitted online through the MyLearningSpace website. Email submission is not accepted.
• Please document your program carefully. Analysis of your observed result is required.
1 Concept Question
1.1 Neural Nets: 3 points
Consider a neural net for a binary classification which has one hidden layer as shown in Figure 1. We use a linear activation
function h(z) = cz at hidden units and a sigmoid activation function g(z) = 11+e−z at the output unit to learn the function
for P (y = 1|x,w), where x = (x1, x2) and w = (w1, w2, . . . , w9).
Figure 1: Neural Nets.
1. What is the output P (y = 1|x,w) from the above neural net? Express it in terms of xi, c and weights wi (2 points).
2. Is it true that any multi-layered neural net with linear activation functions at hidden layers can be represented as a
neural net without any hidden layer? Briefly explain your answer (1 point).
1.2 Support Vector Machines: 4 points
As shown in Figure 2, there are 4 training samples in a 2-dimensional space. x1 = (0, 0) and x2 = (2, 2) are being positive,
while x3 = (h, 1) and x4 = (0, 3) are being negative. h is a parameter whose value falls in the range of [0, 3].
1. How large can h ≥ 0 be so the training examples are still linearly separable (1 point)?
2. Will the direction of the maximum margin decision boundary change as a function of h when the samples are separable?
Explain your answer with one sentence (1 point).
3. What will be the margin obtained by the maximum margin boundary as a function of h? Note that the margin as a
function of h is actually a linear function (2 points).
1
Figure 2: Support Vector Machines
2 Programming Question
The 2019 novel coronavirus (COVID-19) presents several unique features. While the diagnosis is confirmed using polymerase
chain reaction (PCR), infected patients with pneumonia may present on chest X-ray and computed tomography (CT) images
with a pattern that is only moderately characteristic for the human eye. In late January 2020, a research team published
a paper detailing the clinical and paraclinical features of COVID-19. They reported that patients present abnormalities in
chest CT images with most having bilateral involvement.
Convolutional neural networks (CNNs) have been one of the most influential innovations in the field of computer vision.
They have performed a lot better than traditional computer vision and have produced state-of-the-art results. These neural
networks have proven to be successful in many different real-life case studies and applications like image classification. Our
goal in this assignment is to use these X-ray images to develop a CNN in Keras to predict and understand the infection. This
would give physicians an edge and allow them to act with more confidence while they wait for the analysis of a radiologist
by having a digital second opinion to confirm their assessment of a patient’s condition.
The given dataset in MyLS consists of 188 chest X-ray images of patients suffering from COVID-19 and those of healthy
patients. The dataset has been split into training and testing part, with each containing two folders including “NORMAL”
and “PNEUMONIA”, and can be used for model construction and model evaluation respectively.
To implement the predictor, the following steps are required in this assignment.
1. Image pre-processing (2 points). Whenever we deal with image data, image pre-processing is the very first step and
also the most crucial step to be performed. Here, we must reshape all the images to the desired size (64*64) and divide
them by 255 as a step of normalization. Depending on the power of the computer, we can represent the image with
three RGB channels or only 1 channel indicating black/white. It is optional to perform additional data augmentation
that improves the quality of the data. Some useful functions in the Keras ImageDataGenerator1 module include:
• ImageDataGenerator
• flow from directory
We are free to explore different parameters but only need to report the best result in the submission. A sample setting
of converting the image is shown below.
from tensorflow.keras.preprocessing.image import ImageDataGenerator datagen=ImageDataGenerator(
zoom_range=0.2, # Zooming rate of the image
horizontal_flip=True, # Make a horizontal copy
rescale=1.0/255.0, # Normalize the new images
width_shift_range=0.10, # Percentage of width shifting
1https://www.tensorflow.org/api docs/python/tf/keras/preprocessing/image/ImageDataGenerator
2
height_shift_range=0.10, # Percentage of height shifting)
2. Model building (3 points). We are free to design our own convolution layers. For example, we can build a network with
a structure shown in Figure 3. In this example, we first construct two convolutional layers followed by two max-pooling
layers. After flattening the output, we then add one fully connected dense layer and one output layer. The Relu
activation function is applied at each layer except for the last output layer which adopts the sigmod function.
Figure 3: CNN structure
To further improve the predictor’s performance, we can also optionally add a tensorflow.keras.callbacks module.
Callback is a strategy we use to reduce over fitting and save time. For example, we can use EarlyStopping to stop
the training process if the accuracy does not improve for certain iterations. This can be simply implemented using a
sample code snippet shown below.
from tensorflow.keras.callbacks import EarlyStopping
earlystop=EarlyStopping(patience=6)
We can also optionally add dropout layers to further reduce the problem of overfitting2, and introduce Adam to
optimize the SGD process in learning the system parameters3.
3. Model evaluation (3 points). To examine the effectiveness of the predictor, we use evaluation metrics including Accuracy
and Losses (Binary Cross Entropy) and plot the change over 20 epochs. In addition, to better understand the learning
process, we should demonstrate the performance on both training and testing data. For example, plots of Accuracy and
Losses similar to Figure 4 should be included in the submission. Finally, we provide analysis based on the experimental
results and discuss whether overfitting occurs.
Figure 4: Performance evaluation
2https://machinelearningmastery.com/how-to-reduce-overfitting-with-dropout-regularization-in-keras/
3https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/
3