This assignment is for students enrolled both in the PG and UG versions of this course. Note that this is a group assignment, with groups of maximum three students. Objective To understand the performance characteristics of a distributed consensus protocol and to communicate these characteristics in writing, through a performance report. Description In this assignment, you will be evaluating the performance of the Paxos implementation you implemented in assignment 3. Alternatively, you can also use an existing Paxos implementation, either found by yourself or from the list below. You will be designing your own experimental plan and reporting on your findings through a written report. The experimental plan will aim to answer the following questions: What is the performance in terms of runtime (and other metrics) of your system given specific configurations, in a functioning mode without failures? What is the performance in terms of runtime (and other metrics) of your system given specific configurations, in a functioning mode with failures? In your experimental plan, you will be using at least two metrics. The first is the runtime (measured in seconds or milliseconds as appropriate): you will have to define this runtime, and you will have to think very carefully about what you are measuring. Regardless of the functioning mode (with/without failures), there will be parameters that will influence this runtime. One could be the number of nodes in the system, whereas another might be the number of simultaneous proposers. You are expected to present at least one plot showing how the runtime varies with one of these parameters. Since the conditions of your experiments might vary slightly, even if your scenario is not stochastic, you will have to present an average of at least ten runs, as well as the standard deviation, with your results. You are responsible for defining the second metric, based on what you know about what the protocol does and also what you’ve learned about other protocols, such as the voting protocols discussed in the lectures Similarly to the first metric since the conditions of your experiments might vary you are 2020/11/11 Assignment 4 https://myuni.adelaide.edu.au/courses/58025/assignments/190380 2/5 the lectures. Similarly to the first metric, since the conditions of your experiments might vary, you are expected to present average metrics of at least ten runs (and the standard deviation). The structure of the report is as follows: 1. Introduction (worth 5% of your mark) 1. In this section, you will be describing the protocol and the implementation you choose to use 2. Make sure you cover the purpose of Paxos as well as key points of the implementation that might influence you results 2. Experimental Setup (worth 20% of your mark) 1. In this section, you will be describing your experimental setup, including how you calculate the metrics and the experimental framework you use to run your experiments (your scripts, how you save and interpret your data, etc.) 3. No Failure Mode (worth 30% of your mark) 1. In this section, you will be presenting the results of your experiments when there are no failures in your system 2. Make sure that all your plots have axes and all axes have labels and units 3. Make sure you include a discussion subsection where you discuss your results 4. With Failure Mode (worth 30% of your mark) 1. In this section, you will be presenting the results of your experiments when there are failures in your system 2. Make sure that all your plots have axes and all axes have labels and units 3. Make sure you include a discussion subsection where you discuss your results 5. Conclusion (worth 10% of your mark) 1. This section briefly concludes your report, using the discussions from sections 3 and 4 as starting points 2. Discuss what you have learned about Paxos and its performance Your report should have a minimum of 3 pages and a maximum of 4 pages, Arial 11 with 1.5 cm margins. The quality and clarity of your writing is worth 5% of your mark. Rubric Excellent (85-100) Very Good (75-84) Good (65-74) Fair (50-64) ( 2020/11/11 Assignment 4 https://myuni.adelaide.edu.au/courses/58025/assignments/190380 3/5 Introduction In-depth coverage of Paxos and critical implementation characteristics. Good coverage of Paxos and critical implementation characteristics. Some coverage of Paxos and some implementation characteristics. Some coverage of Paxos. Missing coverag and its impleme Experimental Setup Excellent overview of metrics and how they are calculated. Excellent coverage of how the research questions translate into experiments. Good overview of how the code is run to produce the results, including a link to SVN where the experimental framework code and results are committed. Good overview of metrics and how they are calculated. Good coverage of how the research questions translate into experiments. Good overview of how the code is run to produce the results, including a link to SVN where the experimental framework code and results are committed. A discussion of metrics and how they related to the experiments. Some coverage of how the research questions translate into experiments. Good overview of how the code is run to produce the results, including a link to SVN where the experimental framework code and results are committed. An overview of the experiments and how metrics are calculated, without any in￾depth insight. Good overview of how the code is run to produce the results, including a link to SVN where the experimental framework code and results are committed. Missing overview or expe plan. No Failures In-depth analysis of a number of parameters and metrics. At least five points on each plot/table. Results clearly and extensively explained. In-depth analysis of a number of parameters and metrics. At least three points on each plot/table. Results clearly and extensively explained. Analysis of two parameters and metrics. At least three points on each plot/table. Results are followed by some logical explanations. At least three points on each plot/table. Results are followed by some logical explanations. At least points o plot/tab are not any logi explana 2020/11/11 Assignment 4 https://myuni.adelaide.edu.au/courses/58025/assignments/190380 4/5 With Failures In-depth analysis of a number of parameters and metrics. At least five points on each plot/table. Results clearly and extensively explained. In-depth analysis of a number of parameters and metrics. At least three points on each plot/table. Results clearly and extensively explained. Analysis of two parameters and metrics. At least three points on each plot/table. Results are followed by some logical explanations. At least three points on each plot/table. Results are followed by some logical explanations. At least points o plot/tab are not any logi explana Conclusion Logical conclusions backed by results and other research, with insight into Paxos functionality. Some but not all conclusions backed by results and other research, with some insight into Paxos functionality. Some but not all conclusions backed by results and other research, with no insight into Paxos functionality. Some conclusions backed by results. Missing conclus backed or other No insig Paxos f Presentation Excellent and clear writing style, no grammar errors or typos. Good writing style, sometimes lacking in clarity, but with no grammar errors or typos. Good writing style, sometimes lacking in clarity, with some grammar errors or typos. Unclear writing style, with grammar errors or typos. Copious errors o unclear illogical How to submit your assignment Your submission will contain a PDF (make sure you include your names and IDs). Make sure you commit your experimental framework code and raw results and include a link to a SVN folder. You will submit your assignment in myuni, within this assignment. Paxos implementations 2020/11/11 Assignment 4 https://myuni.adelaide.edu.au/courses/58025/assignments/190380 5/5 For this assignment, you can use your own implementation from A3 or you could use any Paxos implementation available online. Some implementations are listed below: http://libpaxos.sourceforge.net/ (http://libpaxos.sourceforge.net/) https://github.com/Tencent/phxpaxos (https://github.com/Tencent/phxpaxos) https://github.com/search?q=paxos (https://github.com/search?q=paxos) https://awesomeopensource.com/projects/paxos (https://awesomeopensource.com/projects/paxos)