微信客服：xiaoxionga100
微信客服：ITCS521
JAVA代写-COMP2207
时间：2021-04-21
1
Distributed file system coursework
Leonardo Aniello and Kirk Martinez
Course: COMP2207
Document version: 1.0 Mar 26, 2021
1 Introduction
In this coursework you will build a distributed storage system. This will involve knowledge
of Java, networking and distributed systems. The system has one Controller and N Data
Stores (Dstores). It supports multiple concurrent clients sending store, load, list, remove
requests. Each file is replicated R times over different Dstores. Files are stored by
the Dstores, the Controller orchestrates client requests and maintains an index with the
allocation of files to Dstores, as well as the size of each stored file. The client actually
gets the files directly from Dstores – which makes it very scalable. For simplicity all these
processes will be on the same machine but the principles are similar to a system
distributed over several servers. Files in the distributed storage are not organised in
folders and sub-folders. Filenames do not contain spaces.
The Controller is started first, with R as an argument. It waits for Dstores to join the
datastore (see Rebalance operation). The Controller does not serve any client request
until at least R Dstores have joined the system.
As Dstores may fail and be restarted, and new Dstores can join the datastore at runtime,
rebalance operations are required to make sure each file is replicated R times and files
are distributed evenly over the Dstores.
2
2 Networking
The modules will communicate with each other via TCP connections. Because they could
be on the same machine for testing, the datastores will listen on different ports.
3 Code development
Only use Java openjdk-14-jdk-headless, on Linux/Unix. Do not use Windows. The code
must be testable and not depend on any IDE directory structure/config files.
Command line parameters to start up the system:
Controller: java Controller cport R timeout rebalance_period
A Dstore: java Dstore port cport timeout file_folder
A client: java Client cport timeout
Where the Controller is given a port to listen on (cport), a replication factor (R), a
timeout in milliseconds (timeout) and how long to wait (in milliseconds) to start the next
rebalance operation (rebalance_period).
The Dstore is started with the port to listen on (port) and the controller’s port to talk to
(cport), timeout in milliseconds (timeout) and where to store the data locally
(file_folder). Each Dstore should use a different path and port so they don’t clash.
The client is started with the controller port to communicate with it (cport) and a timeout
in milliseconds (timeout).
3
Store operation
• Client -> Controller: STORE filename
• Controller
o updates index, “store in progress”
o Selects R Dstores, their endpoints are port1, port2, …, portR
o Controller -> Client: STORE_TO port1 port2 … portR
• For each Dstore i
o Client->Dstore i: STORE filename filesize
o Dstore i -> Client: ACK
o Client->Dstore i: file_content
o Once Dstore i finishes storing the file,
Dstore i -> Controller: STORE_ACK filename
• Once Controller received all acks
o updates index, “store complete”
o Controller -> Client: STORE_COMPLETE
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Client
o Log the error and continue
• Malformed message received by Dstore
o Log the error and continue
• If filename already exists in the index
o Controller->Client: ERROR ALREADY_EXISTS filename
• Client cannot connect or send data to all R Dstores, or the Controller does not
receive R acks within a timeout
o Log the error and continue; future rebalances will sort things out
4
Load operation
• Client -> Controller: LOAD filename
• Controller selects one the R Dstores that stores that file, let port be its endpoint
• Controller->Client: LOAD_FROM port filesize
• Client -> Dstore: LOAD_DATA filename
• Dstore -> Client: file_content
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Client
o Log the error and continue
• Malformed message received by Dstore
o Log the error and continue
• If file does not exist in the index
o Controller -> Client: ERROR DOES_NOT_EXIST
• If Client cannot connect to or receive data from Dstore
o Client -> Controller: RELOAD filename
o Controller selects a different Dstore with endpoint port’
o Controller->Client: LOAD_FROM port’ filesize
o If Client cannot connect to or receive data from any of the R Dstores
Controller->Client: ERROR LOAD
• If Dstore does not have the requested file
o Dstore -> Client: ERROR DOES_NOT_EXIST
5
Remove operation
• Client -> Controller: REMOVE filename
• Controller updates index, “remove in progress”
• For each Dstore i storing filename
o Controller->Dstore i: REMOVE filename
o Once Dstore i finishes removing the file,
Dstore i -> Controller: REMOVE_ACK filename
• Once Controller received all acks
o updates index, “remove complete”
o Controller -> Client: REMOVE_COMPLETE
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Client
o Log the error and continue
• Malformed message received by Dstore
o Log the error and continue
• If filename does not exist in the index
o Controller->Client: ERROR DOES_NOT_EXIST
• Controller cannot connect to some Dstore
o Controller logs error and the protocol continues
• If Dstore does not have the requested file
o Dstore -> Controller: ERROR DOES_NOT_EXIST filename
List operation
• Client->Controller: LIST
• Controller->Client: file_list
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Client
o Log the error and continue
6
Storage Rebalance operation
This operation is started periodically by the Controller (e.g. based on the
rebalance_period argument) and when a new Dstore joins the data store. In the latter
case, this is the message (where port is the endpoint of the new Dstore)
Dstore -> Controller: JOIN port
• For each Dstore i
o Controller -> Dstore i: LIST
o Dstore i -> Controller: file_list
• Controller revises file allocation to ensure (i) each file is replicated over R Dstores,
and (ii) files are evenly stored among Dstores1
• Controller produces for each Dstore i a pair
(files_to_send, files_to_remove), where
o files_to_send is a list of entries, each entry in the form
(filename, dstores), where dstores is the list of endpoints of Dstores to
send filename to
o files_to_remove is the list of filenames to remove
• For each Dstore i
o Controller->Dstore i: REBALANCE files_to_send files_to_remove
Example
Assume that
• file f1 needs to be sent to Dstores p1 and p2
• file f2 needs to be sent to Dstore p3
• file f2 needs to be removed
• file f3 needs to be removed
REBALANCE 2 f1 2 p1 p2 f2 1 p3 2 f2 f3
o Dstore i will send required files to other Dstores, e.g., to send a file
to Dstore j
Dstore i -> Dstore j: STORE filename filesize
Dstore j -> Dstore i: ACK
Dstore i -> Dstore j: file_content
o Dstore i will remove specified files
o When rebalance is completed
Dstore i -> Controller: REBALANCE COMPLETE
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Dstore
o Log the error and continue
• Controller does not receive REBALANCE COMPLETE from a Dstore within a
timeout
o Storage rebalance executed again from the beginning
7
4 Logging
Each process (i.e., Controller, Dstore, Client) appends the network messages and log
events mentioned above to a textual log file. Each time a process is started, a new log
file is created. Java classes will be provided to manage logging.
5 Submission Requirements
• Your submission should include the following files:
– Controller.java Controller.class
– Dstore.java Dstore.class
As well as all the additional .java files you developed (and corresponding .class
files)
• These files should be contained in a single zip file called username.zip
• There should be no package structure to your java code
• When extracted from the zip file, the files should be located in the current directory.
• These will be executed at the Linux command line by us for automatic testing
6 Marking Scheme
You are asked to implement a Controller and Dstores. You will be given the client, as an
obfuscated jar. The client allows the execution of operations via a terminal. You will also
be given some Java classes for the logging (to file).
• >= 40%
o The datastore works in compliance with the protocol and correctly
serves sequential requests from a single client
• >= 50%
o In addition to the criteria above
o Each file is replicated R times
o Files are evenly spread over the Dstores (only when stored, not
when Dstores fail or new Dstores join the datastore)
• >= 60%
o In addition to the criteria above
o The datastore correctly serves concurrent requests from more clients
• >= 70%
o In addition to the criteria above
o The datastore correctly tolerates the failure of one Dstore
• >= 80%
o In addition to the criteria above
o The datastore correctly tolerates the failure of up to N-R Dstores
• >= 90%
o In addition to the criteria above
o Files are evenly spread over the Dstores despite Dstores failing
and new Dstores joining the datastore
8
7 Code development suggestions
There are various things to develop step-by-step. This includes making TCP connections
and passing data to/from, implementing timeouts for when the communication is broken,
and so on. This is a good place to start.
• Draw an outline of your system to keep track of the functionality/code structure
• Use techniques you tested from the Java sockets worksheet.
• for the Controller you can start by making it accept connections
• Avoid multithreading until you are ready for it
• Write a simple log(message) function which prints the timestamp and message into
a logfile.
• Parse incoming commands from the client one at a time, with error codes for
failures.
• Send commands to a simplified Dstore (e.g. LIST)
• Work with just the Dstore to be able to save and read files
• Progressively add the features such as delete and allocating files to Dstores
• Test progressively so you know each area works and can return errors.
• Finally write the rebalance operations
学霸联盟
Distributed file system coursework
Leonardo Aniello and Kirk Martinez
Course: COMP2207
Document version: 1.0 Mar 26, 2021
1 Introduction
In this coursework you will build a distributed storage system. This will involve knowledge
of Java, networking and distributed systems. The system has one Controller and N Data
Stores (Dstores). It supports multiple concurrent clients sending store, load, list, remove
requests. Each file is replicated R times over different Dstores. Files are stored by
the Dstores, the Controller orchestrates client requests and maintains an index with the
allocation of files to Dstores, as well as the size of each stored file. The client actually
gets the files directly from Dstores – which makes it very scalable. For simplicity all these
processes will be on the same machine but the principles are similar to a system
distributed over several servers. Files in the distributed storage are not organised in
folders and sub-folders. Filenames do not contain spaces.
The Controller is started first, with R as an argument. It waits for Dstores to join the
datastore (see Rebalance operation). The Controller does not serve any client request
until at least R Dstores have joined the system.
As Dstores may fail and be restarted, and new Dstores can join the datastore at runtime,
rebalance operations are required to make sure each file is replicated R times and files
are distributed evenly over the Dstores.
2
2 Networking
The modules will communicate with each other via TCP connections. Because they could
be on the same machine for testing, the datastores will listen on different ports.
3 Code development
Only use Java openjdk-14-jdk-headless, on Linux/Unix. Do not use Windows. The code
must be testable and not depend on any IDE directory structure/config files.
Command line parameters to start up the system:
Controller: java Controller cport R timeout rebalance_period
A Dstore: java Dstore port cport timeout file_folder
A client: java Client cport timeout
Where the Controller is given a port to listen on (cport), a replication factor (R), a
timeout in milliseconds (timeout) and how long to wait (in milliseconds) to start the next
rebalance operation (rebalance_period).
The Dstore is started with the port to listen on (port) and the controller’s port to talk to
(cport), timeout in milliseconds (timeout) and where to store the data locally
(file_folder). Each Dstore should use a different path and port so they don’t clash.
The client is started with the controller port to communicate with it (cport) and a timeout
in milliseconds (timeout).
3
Store operation
• Client -> Controller: STORE filename
• Controller
o updates index, “store in progress”
o Selects R Dstores, their endpoints are port1, port2, …, portR
o Controller -> Client: STORE_TO port1 port2 … portR
• For each Dstore i
o Client->Dstore i: STORE filename filesize
o Dstore i -> Client: ACK
o Client->Dstore i: file_content
o Once Dstore i finishes storing the file,
Dstore i -> Controller: STORE_ACK filename
• Once Controller received all acks
o updates index, “store complete”
o Controller -> Client: STORE_COMPLETE
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Client
o Log the error and continue
• Malformed message received by Dstore
o Log the error and continue
• If filename already exists in the index
o Controller->Client: ERROR ALREADY_EXISTS filename
• Client cannot connect or send data to all R Dstores, or the Controller does not
receive R acks within a timeout
o Log the error and continue; future rebalances will sort things out
4
Load operation
• Client -> Controller: LOAD filename
• Controller selects one the R Dstores that stores that file, let port be its endpoint
• Controller->Client: LOAD_FROM port filesize
• Client -> Dstore: LOAD_DATA filename
• Dstore -> Client: file_content
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Client
o Log the error and continue
• Malformed message received by Dstore
o Log the error and continue
• If file does not exist in the index
o Controller -> Client: ERROR DOES_NOT_EXIST
• If Client cannot connect to or receive data from Dstore
o Client -> Controller: RELOAD filename
o Controller selects a different Dstore with endpoint port’
o Controller->Client: LOAD_FROM port’ filesize
o If Client cannot connect to or receive data from any of the R Dstores
Controller->Client: ERROR LOAD
• If Dstore does not have the requested file
o Dstore -> Client: ERROR DOES_NOT_EXIST
5
Remove operation
• Client -> Controller: REMOVE filename
• Controller updates index, “remove in progress”
• For each Dstore i storing filename
o Controller->Dstore i: REMOVE filename
o Once Dstore i finishes removing the file,
Dstore i -> Controller: REMOVE_ACK filename
• Once Controller received all acks
o updates index, “remove complete”
o Controller -> Client: REMOVE_COMPLETE
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Client
o Log the error and continue
• Malformed message received by Dstore
o Log the error and continue
• If filename does not exist in the index
o Controller->Client: ERROR DOES_NOT_EXIST
• Controller cannot connect to some Dstore
o Controller logs error and the protocol continues
• If Dstore does not have the requested file
o Dstore -> Controller: ERROR DOES_NOT_EXIST filename
List operation
• Client->Controller: LIST
• Controller->Client: file_list
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Client
o Log the error and continue
6
Storage Rebalance operation
This operation is started periodically by the Controller (e.g. based on the
rebalance_period argument) and when a new Dstore joins the data store. In the latter
case, this is the message (where port is the endpoint of the new Dstore)
Dstore -> Controller: JOIN port
• For each Dstore i
o Controller -> Dstore i: LIST
o Dstore i -> Controller: file_list
• Controller revises file allocation to ensure (i) each file is replicated over R Dstores,
and (ii) files are evenly stored among Dstores1
• Controller produces for each Dstore i a pair
(files_to_send, files_to_remove), where
o files_to_send is a list of entries, each entry in the form
(filename, dstores), where dstores is the list of endpoints of Dstores to
send filename to
o files_to_remove is the list of filenames to remove
• For each Dstore i
o Controller->Dstore i: REBALANCE files_to_send files_to_remove
Example
Assume that
• file f1 needs to be sent to Dstores p1 and p2
• file f2 needs to be sent to Dstore p3
• file f2 needs to be removed
• file f3 needs to be removed
REBALANCE 2 f1 2 p1 p2 f2 1 p3 2 f2 f3
o Dstore i will send required files to other Dstores, e.g., to send a file
to Dstore j
Dstore i -> Dstore j: STORE filename filesize
Dstore j -> Dstore i: ACK
Dstore i -> Dstore j: file_content
o Dstore i will remove specified files
o When rebalance is completed
Dstore i -> Controller: REBALANCE COMPLETE
Failure Handling
• Malformed message received by Controller
o Log the error and continue
• Malformed message received by Dstore
o Log the error and continue
• Controller does not receive REBALANCE COMPLETE from a Dstore within a
timeout
o Storage rebalance executed again from the beginning
7
4 Logging
Each process (i.e., Controller, Dstore, Client) appends the network messages and log
events mentioned above to a textual log file. Each time a process is started, a new log
file is created. Java classes will be provided to manage logging.
5 Submission Requirements
• Your submission should include the following files:
– Controller.java Controller.class
– Dstore.java Dstore.class
As well as all the additional .java files you developed (and corresponding .class
files)
• These files should be contained in a single zip file called username.zip
• There should be no package structure to your java code
• When extracted from the zip file, the files should be located in the current directory.
• These will be executed at the Linux command line by us for automatic testing
6 Marking Scheme
You are asked to implement a Controller and Dstores. You will be given the client, as an
obfuscated jar. The client allows the execution of operations via a terminal. You will also
be given some Java classes for the logging (to file).
• >= 40%
o The datastore works in compliance with the protocol and correctly
serves sequential requests from a single client
• >= 50%
o In addition to the criteria above
o Each file is replicated R times
o Files are evenly spread over the Dstores (only when stored, not
when Dstores fail or new Dstores join the datastore)
• >= 60%
o In addition to the criteria above
o The datastore correctly serves concurrent requests from more clients
• >= 70%
o In addition to the criteria above
o The datastore correctly tolerates the failure of one Dstore
• >= 80%
o In addition to the criteria above
o The datastore correctly tolerates the failure of up to N-R Dstores
• >= 90%
o In addition to the criteria above
o Files are evenly spread over the Dstores despite Dstores failing
and new Dstores joining the datastore
8
7 Code development suggestions
There are various things to develop step-by-step. This includes making TCP connections
and passing data to/from, implementing timeouts for when the communication is broken,
and so on. This is a good place to start.
• Draw an outline of your system to keep track of the functionality/code structure
• Use techniques you tested from the Java sockets worksheet.
• for the Controller you can start by making it accept connections
• Avoid multithreading until you are ready for it
• Write a simple log(message) function which prints the timestamp and message into
a logfile.
• Parse incoming commands from the client one at a time, with error codes for
failures.
• Send commands to a simplified Dstore (e.g. LIST)
• Work with just the Dstore to be able to save and read files
• Progressively add the features such as delete and allocating files to Dstores
• Test progressively so you know each area works and can return errors.
• Finally write the rebalance operations
学霸联盟
