Python代写|机器学习代写 - COMP5328 - Advanced Machine Learning

Assignment 2
Due: 19 November 2020, 23:59PM
This assignment is to be completed in groups of 2 to 3 students. It is worth
Introduction
The objective of this assignment is to build an transition matrix estimator and
two classification algorithms that are robust to label noise.
Three input datasets are given. For each dataset, the training and validation
data contains class-conditional random label noise, whereas the test data is clean.
You need to build at least two different classifiers trained and validated on
the noisy data, that can have a good classification accuracy on the clean test data.
You are required to compare the robustness of the two algorithms to label noise.
For the first two datasets, the transition matrices are provided. You can directly
use the given transition matrices for designing classifiers that are robust to label
noise.
For the last dataset, the transition matrix is not provided. You are required
to build an transition matrix estimator to estimate the transition matrix.
transition matrix must be included in your final report. Note that to validate the
effectiveness of your transition matrix estimator, you could use your estimator on
the first two datasets and compare your estimation to the given transition matri-
ces. The code contained in tutorial 9 could be a good starting point.
Data prepossessing is allowed, but please remember to clarify and justify it in
the report carefully.
1
1 A Guide to Using the Datasets
Three image datasets with .npz format are provided. You can download them via
canvas.
1.1 Attributes Contained in a Dataset
The following code is used to load a dataset and check the shape of its attributes.
import numpy as np
# Remember to r ep l a c e the \$FILE PATH
datase t = np . load (\$FILE PATH)
Xtr va l = datase t [ ’ Xtr ’ ]
S t r v a l = datase t [ ’ Str ’ ]
Xts = datase t [ ’ Xts ’ ]
Yts = datase t [ ’ Yts ’ ]
print ( Xtr va l . shape )
print ( S t r v a l . shape )
print ( Xts . shape )
print ( Yts . shape )
1.1.1 Training and validation data
The variable Xtr val contains the features of the training and validation data.
The shape is (n, image shape) where n represents the total number of the in-
stances.
The variable Str val contains the noisy labels of the n instances. The shape
is (n, ). For all datasets, the class set of the noisy labels is {0, 1, 2}.
Note that do not use all the n examples to train your models. You are re-
quired to independently and randomly sample 80% of the n examples to train a
model and use the rest 20% examples to validate the model.
1.1.2 Test data
The variable Xts contains features of the test data. The shape is (m, image shape),
where m represents the total number of the test instances.
The variable Yts contains the clean labels of the m instances. The class set
of the clean labels is also {0, 1, 2}.
2
1.2 Dateset Description
1.2.1 FashionMINIST0.5.npz
Number of the training and validation examples n = 18000.
Number of the test examples m = 3000.
The shape of each example image shape = (28× 28).
The transition matrix T =
0.5 0.2 0.30.3 0.5 0.2
0.2 0.3 0.5
 .
1.2.2 FashionMINIST0.6.npz
Number of the training and validation examples n = 18000.
Number of the test examples m = 3000.
The shape of each example image shape = (28× 28).
The transition matrix T =
0.4 0.3 0.30.3 0.4 0.3
0.3 0.3 0.4
 .
1.2.3 CIFAR.npz
Number of the training and validation examples n = 15000.
Number of the test examples m = 3000.
The shape of each example image shape = (32× 32× 3).
The transition matrix T is unknown.
3
2 Performance Evaluation
The performance of each classifier will be evaluated with the top-1 accuracy metric,
that is,
top-1 accuracy =
number of correctly classified examples
total number of test examples
∗ 100%.
To have a rigorous performance evaluation, you need to train each classifier
at least 10 times with the different training and validation sets gener-
ated by random sampling. Then report both the mean and the standard
derivation of the test accuracy.
You need to implement at least two label noise robustness classifiers with at
least one not taught in this course, and test their performance on the three datasets.
You need to implement an estimator to estimate the transition matrix. The
code must be written in Python 3. You are allowed to use external libraries for
optimisation and linear algebraic calculation. If you have any ambiguity whether
you can use a particular library or a function, please post your question on canvas
or Ed.
3.1 Image Classification with Known Flip Rates
For the first two datasets, the transition matrices are provided. You can directly
use the given transition matrices for designing classifiers that are robust to label
noise. As mentioned in the section 2, for each classifier, you should report the
mean and the standard derivation of the test accuracy.
3.2 Image Classification with Unknown Flip Rates
For the last dataset, Since the transition matrix is not provided, you need to imple-
ment an estimator to estimate the transition matrix. Then use the estimated
transition matrix to build a noise robust classifier. Note that you can use the
provided transition matrices of the first two datasets to validate the effectiveness
of your transition matrix estimator. You need to include your estimated transition
matrix in the final report. You also need to report the mean and the standard
derivation of the test accuracy for each of your designed noise robustness classifier.
Both estimation accuracy of the transition matrix and the test accuracy on the
last dataset contribute to the final mark.
4
3.3 Report
The report should be organized similar to research papers, and should contain the
following sections:
• In abstract, you should briefly introduce the topic of this assignment, your
methods, and describe the organization of your report.
• In introduction, you should first introduce the problem of learning with
label noise, and then its significance and applications. You should give an
overview of the methods you want to use.
• In related work, you are expected to review the main idea of related label
• In methods, you should describe the details of your classification models,
including the formulation of the cost functions, the theoretical foundations
or views (if any) of the cost functions, and the optimization methods. You
should describe the details of the transition matrix estimation methods, the-
oretical foundations (if any), and optimization algorithms.
• In experiments, you should introduce your experimental setup (e.g., datasets,
algorithms, evaluation metric, etc.). Then, you should show the experimen-
tal results, compare, and analyze your results. If possible, give your personal
reflection or thoughts on these results.
• In conclusion, you should summarize your methods, results, and your in-
sights for the future work.
• In references, you should list all references cited in your report and format-
ted all references in a consistent way.
• In appendix, you should provide instructions on how to run your code.
4 Submission guidelines
1. Go to Canvas and upload the following files/folders compressed together as
a zip file.
(a) report (a pdf file)
The report should include all member’s details (student IDs and names).
(b) code (a folder)
5
i. algorithm (a sub-folder)
Your code (could be multiple files or a project)
ii. input data (a sub-folder)
Empty
Please do NOT include the dataset in the zip file as they are large.
We will copy the dataset to the input folder when we test the code.
Only one student needs to submit the zip file which must be named as
student ID numbers of all group members separated by underscores. E.g.
“xxxxxxxx xxxxxxxx xxxxxxxx.zip”.
2. A plagiarism checker will be used.
3. A penalty of minus 5% marks per each day after the due date (email late
submissions to the teaching assistant and confirm late submission dates with
him). Maximum delay is 5 (five) days, after that assignments will not be
accepted.
4. Remember, the submission deadline is 19 November 2020, 23:59PM.
6
5 Marking scheme
Report 
Abstract 
•problem, methods, and organization
Introduction 
•the problem you intend to solve
•the importance of the problem
Previous work 
•previous relevant methods used in literature
Label noise methods with known flip
rates 
•pre-processing (if any)
•label noise methods’ formulation
•cross-validation method for model selection
or avoiding overfitting (if any)
•experiments
•discussions
Noise rate estimation method 
•noise rate estimation method’s formulation
•experiments
•discussions
Label noise methods with unknown flip
rates 
•pre-processing (if any)
•label noise methods’ formulation (if different
from above)
•cross-validation method for model selection
or avoiding overfitting (if any)
•experiments
•discussions
Conclusions and future work 
•meaningful conclusions based on the results
•meaningful future work suggested
7
Presentation 
spelling mistakes
•good structure and layout, consistent format-
ting
•appropriate citation and referencing
•use graphs and tables to summarize data
Other 
•at the discretion of the assessor: illustrate
outstanding comprehensive theoretical analy-
sis, demonstrate the insightful and compre-
hensive assessment of the significance of their
results, provide descriptions and explanations
that have depth but clarity, and are concisely
worded
Code 
•reasonable code running time
•well organized, commented and documented
Note: Marks for each category is indicated in square brackets. The minimum mark for the assignment will be 0 (zero).
8  