PASSiNg thE MESSAgE
Assignment for all students of Systems, Networks and Concurrency
This is the second, carefully marked assignment of the course. Extra care is expected on all levels. A good solu- tion will have a surprisingly small amount of code. Make sure that this code is excellently written – in whichever language you prefer.
Overview
Routers are basic ingredients of most distributed systems. they commonly have no a-priori knowledge about the wider network topology in which they are embedded. Nevertheless they are expected to send messages via a network port which will move those messages closer to their destinations. Many constraints can be taken into consideration for a practical router, yet this assignment focuses on sending messages on the shortest paths to their destinations.
What an individual router needs to provide to clients:
• Acceptamessagefromaclient(inanpredefinedformat).themessagealsocontainsades-
tination id. the expectation is that this message will eventually end at the router of this id.
• Store and provide received messages for a client to pick up (in an predefined format).
What is accessible to an individual router:
• the local network ports which are connected to other routers.
• the Id’s of all neighbouring routers. in the physical world this would need to be established by means of initial communications first, yet we assume here that this already happened.
• the total number of routers as well as the fact that they are numbered consecutively from 1 to the total number of routers. in a real router you would use hash tables for the router id’s.
What the routers need to agree upon before they are deployed:
• One or multiple message formats which are exchanged between routers (in addi- tion to the already given formats used to send to and received from clients).
• the semantics for those messages. What a router can do:
• Send messages in any of the globally defined inter-router formats over any of its ports to any of its directly connected neighbours.
• Receive messages on any local port and process their contents.
What a router cannot do is to sneak behind the scenes and extract the global topology of the network from a central place in a direct way. Routers can still organize themselves in whichever form they see fit. thus a central place could for instance be one router which by election, or other means nominated itself as a central place. Communication with this router would need to go through the normal network channels though – no secret direct wires to a central coordina- tor.Yet there does not need to be any central instance at all – distributed solutions are usually also more robust.
1 | ANU College of Engineering and Computer Science October 2020
How to implement a router
You can implement this router in any language or environment, as long as it roughly follows the frame described above. One essential feature for your implementation needs to be kept in mind: the connections between routers are “wires” – not software buffers. A close software equivalent to a wire is synchronous message passing. Any buffer which might be needed,
will need to be implemented as part of the router.You can emulate synchronous (or otherwise “immediate”) communication by means of multiple asynchronous messages or by means of asynchronous messages combined with signals/interrupts.
there is a framework available to you in Ada which generates a number of classical network topologies and places router tasks inside a chosen topology. it also runs a simple evaluation which sends messages from each router to every other one and takes notes about the number of hops the message passed through.
if you decide to write it in another language than Ada, then you will need to program this simple infrastructure as well (details below). Simply contact us which your specific plans and we will see what is achievable with reasonable effort in your chosen environment.
The Ada framework
the program test_routers can be fully controlled via command line parameters: uwe@Paddington-i9 Executables % ./test_routers
accepted options: [-t {Topology
by Size : by Degree, Depths : by Dimension, Size :
: String }] -> CUBE_CONNECTED_CYCLES Line, Ring, Star, Fully_Connected
Tree
Mesh, Torus
by Dimension
:
Hypercube, Cube_Connected_Cycles, Butterfly, Wrap_Around_Butterfly
[-s {Size
[-g {Degree
[-p {Depths
[-d {Dimension
[-c {Print connections [-i {Print distances [-w {Routers settle time [-o {Comms timeout
: Positive }] -> 20
: Positive }] -> 3
: Positive }] -> 4
: Positive }] -> 3
: Boolean }] -> TRUE
: Boolean }] -> TRUE
: Seconds }] -> 0.10
: Seconds }] -> 0.10
: String }] -> ONE_TO_ALL
[-m {Test mode
Available modes: One_to_All, All_to_One
[-x {Dropouts : Natural }] -> 0 [-r {Repeats : Positive }] -> 100
Available network topologiTime for routers to establish their strategies : 0.10 second(s) —————————- Measurements ————————————
parameters which are required for a given topology. thus you can test your routers in a 4-d
fortwork. After
e during which no client messages will be sent through the
[-g {Degree : Positive }] -> 3
nelbesen
sett g t
–
i{
Pr
i
nt
d
is
t
a
nc
es
:
Bo
ol
e
]-
>
TRU
E
this
lin
e
————— Information about the selected network topology —————- Topology : CUBE_CONNECTED_CYCLES
|
message was not delivered or whether the destination router is not responsive, as
Dimension : 3 Number of nodes in topology : 24 Constant connection degree : 3
15 | 2 2
3 4 4 4 3 4 5 5
. 2 3 3 3 2 3 4 4 5| 2 2 . 4 3 4 3 2 3|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
2 | ANU College of Engineering and Computer Science
October 2020
+————————————————————————+
1 | . <-><-><-> 2 |<-> . <->
<->
. <-><-><->
| | | | |
<-> | |
3 |<-><-> . 4 |<->
5 |
6 |
. <-><-> <-> . <-> <-><-> .
<->
7 |
<->
Figure 1: Command line options
———————– Instantiating router tasks —————————–
=> Routers up and running
——————————– Waiting —————————————
–es are denoted by name (out of the list above) together with the Number of runs : 100
Minimal hops :
1
Maximal hops : Average hops : 3.22
hypercube by: ./test_routers -t Hypercube -d 4
6
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 You can allow your routers s+-o—————-t-o——-t———-s–e———o————e–d——-d—s–p+
uwe@Paddington-i9 Executables % ./test_routers 1 2 3 4 5
accepted options: 1 | me time ge them lves rganiz an ecify for instance: . 2 2 3 2 3 4 3 4 3 3 2 4 4 3 5 4 4 6 5 5|
./te
ters
-w 0
2 3 4 4 3 3 2 4 5 4 4 4 3 5 6 5| 4 3 2 3 4 4 3 3 3 2 6 5 5 5 4 4|
[-t {Topology : String }] -> CUBE_CONNECTED_CYCLES
bySize :Line,Ring,Sta2r,|Fully._Conne2cted3 3 2
bsytDe_greoe,uDepths :-Trtee Hyperc3u|be -d. 42 3 3 3.223 4 4 5 2 2 3 4 4 4 3 4 5 5 6| by Dimension, Size : Mesh, Torus
b
on
be
nn
d_
y
Di
me
n
si
:
Hy
perc
u
be
,
Cu
|o 4_C
ect2e
2Cyc
.les,
[-as{2Si0ze0 :rPaostiitoivne}p]6-h>|22033 .44532334422 556434|
ms prepa
as
4
3
Butterfly, Wrap5_A|rou2nd_3But3terfly. 3 4 4 2 2 4 5 4 3 3 2 5 6 5 4 4 3|
7 | 3 2 3 4 3 4 . 2 2 5 4 4 6 5 5 3 3 2 4 4 3|
[-
[-d{Dimension :Positive}]8->|23 2344 . 233443565332454| [-c{Printconnections :Boolean }]9->|TR3UE23445 .23343455622 344|
[
p{
De
pt
hs
:
P
os
i
t
iv an
e} }
]-
>
4
im
for the routers, the evaluation phase starts and a messages
10 | 4 3 4 3 2 3 2 2 . 6 5 5 5 4 4 4 4 3 3 3 2|
[-w {Routers settle time : Seconds }] -> 0.10
[-o{Commstimeout :Seconds }]11->|03.10442 2233 . 565443454332|
-m
{
Tes
t
mo
de
:
St
ri
ng
}
]
O
_A
t via eve
ou
5LL ry
th
rr
uter.t
12- wil Available modes: One_to_All, All_to_One
T4O
2
5
6
sa
es
ll b
N4E_
ryr tertoeve o e o
sem g wi e
sent in groups, i.e. all
[
[-x {Dropouts : Natural }] -> 0
[-r{Repeats :Positive}]14->|1030 3 2 4 5 4 4 4 3 5 6 5
4 3
3
4
4 2
2 3
2 4
| 4|
>|
3
3
2
3
3
.
5
4 2
3
2 2 3 2 3 4 3 4|
13 | 3 3 2 4 4 3 5 4 4 6 5 5 .
msa
g from a specific rout-
ho
es
.
3
2
in
tin
es
sa
ge
orig
6
4
3233 . 3442 2|
23
4
4
2
2
2
3
3
es
|
——
–
er will be sent in bulk (“One to all”). this
17 | 4 5 4 3 3 2 5 6 5 4 4 ——————————– Waiting 1-8–|—3—4—4—2—-2——5—5—6—4—3–
=> Routers up and running
Time for routers to establish their strategies : 0.10 second(s)
—————————— Measurements ————————————
Number of runs : 100
20 | 4 4 3 5 6 5 3 3 2 4 5 21 | 4 3 4 5 5 6 2 2 3 4 22 | 6 5 5 5 4 4 4 4 3 3 3 23 | 5 6 5 4 4 3 4 5 4 3 3
Minimal hops Maximal hops Average hops
: 1
: 6
: 3.22
2 3 4 4 2 3 4 4 5 3 4 3 2 3 4 4 2 2 4 5 3 2 3 3 2 4 4 3 3 2 4 5 4 2 3 4 4 4 3 3 3 2 5 4 3 3 2
. 2 3 3 Co4nst4ant3 co5nne6ct5ion3 de3gre2e 4: 5 34| . 2 3 3 4 3 4 5 5 6 2 2 3 4 4| 2 2 . 655544443332|
16 | 4 4 3 3 3 2 6 5 5 5 4
———————– Instantiating router tasks —————————–
19 | 5 4 4 6 5 5 3 3 2 4 4
1 2 3 4 5 6 7 8 9 10 11 122413|145155166174183194203214224232242 +—————————————+———————————+—
1 |.
2 |
3 |
4 |
2 2 3 2 3 4 3 4 3 3 2 4 4 3 5 4 4 6 5 5|
5 | 2 6| 2 7| 3 8| 2 9| 3
2 2 .
3 3 .
3 3 . 2 3 4 3 4
3 4 4 2 2454332565443|
. 2 3 3 . 2 3 3
2 2 3 4 4 3 3 2 4 5 4 4 4 3 5 6 5|
3 2 3 4 4 -5—2—2——3–4 In4for4mat3ion4 ab5out5 th6e| se 4 3 4 3 2 3 To4pol4ogy3 3 3 2 6 5 5 5: C4UBE4_|CON
Dimension : 3
4 4 5 3 2 334422 556434|
. 2 2 Nu5mbe4r o4f n6ode5s 5in 3top3olo2gy 4: 4243|
10 | 4
11 | 3
12 | 4
13 | 3
14 | 3
15 | 2
16 | 4
17 | 4
18 | 3 4 4 2 2 19|54465533244353|234<-3>4.<.-> 2<2-|> 20|44356533245462| 23<-4><4->.. 233|
2 3 3 . 5 16 25 34 4 35 46 57 48 39 130 121| 12 2 3 3 . 5+–5—6—4—3–4—3—4—4—2—2—-|— 5 4 4 6 5 5 . 2 2 3 2 3 4 3 4|
1 | . <-><-><->
4 4 3 5 6 5 . 2332 4 3 4 5 5 6 2 |<-> . <-2> 3 3 3 6 5 5 5 4 4 3 |<-2><-2> . 4
23 2<3->4 3 4 3 442 453
4 4| 4 5| 2 3|
5 6 5 4 4 3 233.3
2| 2 3|
22 | 6 5 5 5 4 4 4 4 3 3 3 2 4 3 4 3 2 3 2 2 . | 23 | 5 6 5 4 4 3 4 5 4 3 3 2 3 4 4 2 2 2 3 3 . | 24 | 5 5 6 4 3 4 3 4 4 2 2 4 4 5 3 2 3 2 3 3 .|
+————————————————————————+
4 |<-> . <-><->
556434233 .4
21 | 4 3 4 5 5 6 2 2 3 4 4 3 2 3 4 4 5 . 2 3 3|
7 | . <-><-><->
will be changed via the command line
4 2 3 3 . 4 4 5 3 2 3|
3 3 2 3 4 3 4 . 2 2|
parameter -m All_To_One to a schema,
42 2344 . 233|
4 3 2 3 4 4 5 24w3h4323e2s2.|
—
—
er
which are destined to
—
e al
—
—
ifi
–
–
–
—
lm
sag
+ 4 a4 s5p3ec2 3c2ro3ut3er wil.l|
be send in bulk. the
—
—
default time-out for any network activity is
100ms (but can also be adjusted via the command line interface). if a router does
————————————-+
13 14n1o5 t16re17s1p8 o19n2d0 2a1 f2t2e2r3 2t4his time-out then the
Figure 2: Measurements
lected network topology —————-
NECTED_CYCLES
| communication test|
<-> | | |
—
. 2 3 3|
—
—
.
—
is noted as unsuc- cessful. From the testing environment
it cannot be distinguished whether the
<-> |
.
the internal states of the routers are not accessed. Next, the results of the communication tests
are displayed (Figure 2). in the provided framework the routers do not exist yet, thus this sec-
tion will not appear up on your screen until you provide a router implementation.
the minimum number of hops for a message should be ‘1’, and the largest measured number
of hops should represents the diameter1 of the chosen topology. the average number of hops is characteristic for a specific topology. By using the command line option -i True you will also see the full table of individual measurements between all nodes. As we only consider bidirectional connections, the number of hops should be identical in both directions between any two routers. if the measurements are different, warnings will be displayed. 1-hop connections are left blank to make the matrix somewhat more readable.
the last section (Figure 3) which will appear
on screen gives general parameters of the
current topology, like the minimum and maxi-
mal connection degree and the total number
of nodes.You can also print the connec-
tions inside the topology as a matrix – which
should be the mirror image of the empty
spots in the measurement matrix. Double arrows denote bidirectional connections.
the individual routers need to respond to the following four pre-defined entries:
task type Router_Task (Task_Id : Router_Range := Draw_Id) is
entry Configure (Links : Ids_To_Links);
entry Shutdown;
entry Send_Message (Message : Messages_Client);
entry Receive_Message (Message : out Messages_Mailbox);
Configure is only called once right after the task has been created. the existing links to neigh- bouring routers are delivered here in the form of an array over the router id’s in which connect- ed routers will appear while all other entries are null.
type Ids_To_Links is array (Router_Range) of Router_Task_P;
type Router_Task_P is access all Router_Task;
two message formats are already defined, yet you need to add at least one more message format and entry which you will be using for communications between routers.
type Messages_Client is record
Destination : Router_Range;
The_Message : The_Core_Message;
end record;
type Messages_Mailbox is record
Sender : Router_Range;
The_Message : The_Core_Message;
Hop_Counter : Natural := 0;
end record;
the purpose of Shutdown (which will be called upon once after all tests have been performed) is
to allow for a graceful way to stop operations. if your router accepts this call, it is assumed that
[1] The diameter of a given topology is the greatest distance between any two nodes.
3 | ANU College of Engineering and Computer Science October 2020
2| . 2 2
3| . 2 2 3
4 | 2 2 .
5 | 2 2 .
6 | 2 2 3 .
7 | 3 2 2 3 3 2
8 | 2 2 3 3 3
9 | 2 2 2 3 3
10 | 3 3 2 3 2 2
11 | 3 3 3 2 2
12 | 2 3 3 2 2
13 | 2 2 3 3 3
14 | 2 3 2 3 3 2
2 2 3 3 3 2
2 2 2 3 3
3 3 2 3 2 2 3
3 3 3 2 2 3 2 3 3 2 2 3 . 2 2 3
. 2 2 2
. 2 2 3 3
2 2 . 3 2 2 . 4
2 2 3 . 4
3 2 3 3 4 4 .
3 2 3
2 2 3
3 3 2
3 2 2 2 3
2 3 3
3 3 4
3 3 4
4 4 3
3 4 2
4 3 3
3 2 2 3 3 2 3 3 3 3
2 3 4 4
3 2 4 3 4
2 2 4 4 3 4 4 2 2
4 34| 4 43| 3 23| 2 33| 3 33| 3 33| 3 32| 3 23| 2 2| 2 32|
2 3 3 4 3 4
3 33| 2 2 33| 16 | 3 3 3 2 2 3 4 4 3 2 3 2 2 . 3 3 2 3 2 2|
15 |
2 2 3
2 3
3 3 3 4 4 3
. 2
2 3
2
1
7
|
3
3
2
2
3
2
4
3
4
2
3
3
2
2
.
3
3
3
2
2
|
19
18 |3 23 22443333223 .2332 2|
3
2
3
3
4
4
2
2
3
3
3
3
2
2
3
2
.
2
2|
|
20 | 2 3 3 4 3 4 2 3 2 3 3 2 2 2 3 3 3 . 2 2|
3
21 | 3 3 3 4 4 3 2 2 3 2 3 2 2 2 3 3 . 2 2 3|
22 | 3 4432333322 3323222 2.
| 23 | 4 3 4 2 3 3 3 3 2 2 3 2 3 3 3 2 2 2 2 . |
24 | 4 4 3 3 3 3 3 2 3 2 2 2 3 3 2 2 2 2 3 .| +————————————————————————+
————— Information about the selected network topology —————-
Topology : Dimension : Number of nodes in topology : Constant connection degree :
1 2 3 4 5 6 7 8 9 +————————————————————————+
1 | . <-><-> <->
2 |<-> . <-><->
3 |<-><-> .
4 | <->
<-> <->
<-> | | <-> | <-> | | <-> | <-> | | <-> | <->| | <-> | | | | | <->| <-> | <-> | <-> . <-><-> |
5 |<->
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
. <-><-> <-> . <-> <-><-> .
<-> <->
15 |<->
16 |
17 |
18 |
19 |
20 |
21 |
22 |
23 |
24 |
. <-><-> <-> . <-><-> <-><-> .
<-> <->
. <-><->
<-> <->
<->
<-> <-><->
<->
. <-><-> <-> . <-> <-><-> .
<->
. <-><-> <-> . <-><-> <-><-> .
<->
<->
<-> <->
. <-><-> <-> . <-> <-><-> .
<->
WRAP_AROUND_BUTTERFLY 3
24 4
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
<->
| . <-><->| <-> . <->| <-><-> . | +————————————————————————+
<->
<-><-> . <->
<->
localhost:Executables uwe$ Figure 3: Topology
<-> <->
<->
3 4 2 3 2 4 3 2 2
2 3 3 3 3
3 3 3 3 2
3 3 3 2 3 22|
2 2 2 2
3 2 2 2 2
3 32|
<-><-> <->
the router task (and all tasks or resources initiated by it) will terminate eventually. if the call to Shutdown is not accepted then an abort of this task is attempted by the environment.
All this sounds complicated but it really isn’t. the core of the assignment is just to decide in which direction to forward a specific message and to use your knowledge from the course to implement your strategy without locking up, unnecessary busy-waiting loops or losing informa- tion. in short: your solution should be implementing a router which is responsive on all ports and directs traffic onto the optimal routes.
Implementation alternatives
the basic requirements for an implementation in another language are:
• Option to select between and parametrize topologies as provided in the framework.
• Make sure that you routers do not access any information other than the initial connection list to neighbouring routers and the information which they gain by exchanging messages.
• instantiate the number of routers required to populate the topology.
• the routers need to be instantiated as processes, threads, tasks, or what- ever your operating system or language will provide as concurrent entities.
• Provide each router entity with communication links to neighbouring routers. those links shall resemble a “wire” semantic, i.e. synchronous forms of communication.
• Run a simple test by sending messages from every router to every oth- er router and measure the actual number of hops the messages took.
• Summarize the results as maximal as well as average number of measured hops.
if this sounds too demanding for your chosen environment, then contact us and we will work out a plan which can be done – a concurrent implementation of the router tasks is obviously mandatory though.
Candidates include Erlang, go (basic templates provided for both), Rust, Smalltalk, LabViEW, Occam, OpenMP… just as long as you can rely on message passing (synchronous message passing might be emulated by asynchronous message passing) and support for concurrency.
if your environment cannot be reproduced by us with reasonable effort in order to run your pro- gram then we will potentially ask you to provide more detailed documentation of your tests.
Extension
As you have been attentive you will have noticed a command line parameter (-x) which controls the number of routers dropping out. After the initial configuration phase (and before normal traffic starts) a specific number of random routers can be powered down purposefully. Many network topologies offer multiple routes to the same node, so the resulting network may still be fully operational (minus any direct communication to the powered down nodes obviously).
Your extension assignment is to design routers which are robust with respect to such losses in the network and will still deliver the packets to the destination, even if the original/optimal path became unavailable. in the provided Ada framework routers will detect the loss of an im- mediate neighbour by receiving a Tasking_Error exception when trying to communicate to an already terminated router task. handle this exception and provide an alternative route.
Reflect first for a moment on the following issues (all of which will be related to specific topolo- gies):
• how many dropped out routers can you handle and still guarantee delivery?
• how does the number of dropped out routers affect the performance?You may want to dis- tinguish the situation when you first detect the failure from the following routing operations.
• Can you respond locally or will you need to propagate the detection of the failure?
this extension is meant for the student who wants to dig deeper and find out more about real- world routing.
4 | ANU College of Engineering and Computer Science October 2020
Deliverables
You do not need to provide a rationale or any report of sorts if you believe that your design is obvious and a few comments inside your code will make the idea sufficiently clear.
Yet you need to provide two items:
a. the sources for a working router (and an environment to test them in for the stu- dents not using the provided Ada framework). Zip up the Router directory (which should hold all sources which you manipulated or added) for submission.
b. A graphical representation of the workings inside your router (as a pdf file named Diagram.pdf). Add this file to your Router directory before you zip it up.
in any case, we will read your sources carefully. the core part of your router will be small – take your time to structure this part well.
Different entities in your graphical representation will have different meanings: Use different lines / arrows / colours / shadings / etc. to indicate what is what and provide a legend. Before you start drawing: decide what the main aspect of our diagram should be. Will you primarily provide a call-graph / data-flow-graph / synchronization-graph / dependency-graph / etc. or a mixture of multiple aspects. if you need multiple diagrams for multiple aspect, then please provide multiple diagrams (in the same pdf file).Your diagram will likely contain text labels, yet should not include full sentences or paragraphs.
there is no predefined template for your diagram as the idea is that you communicate your concept with it – whichever format supports this best is for you to decide.
Tutor-eyesight-health-warning: Make sure that your graphics will appear technically in vector (scalable) format inside your pdf file – never as an image made out of pixels!
Criteria
When you design your system, you might want to keep the following criteria in mind:
• Messages shall arrive at the destination router eventually (this is the minimal requirement).
• the number of hops should be minimal.
• the setup time should be proportional to the number of nodes or even bet- ter to the diameter of the network. Beware of combinatorial explosions.
• Performance of the solution. Did you arrange concurrent entities inside your router to guarantee minimal delivery delays? Assume that your rout- er can be provided with any number of physical concurrent entities.
• Elegance of the solution. Elegance is often a measure related to compactness and sim- plicity but not always.You will see whether your design is elegant once you reevalu- ate/scrutinize your sources a few days after you achieved a running system.
• Clarity and expressiveness of the provided graph. A meaningful graph will use a clear legend of graphical elements to distinguish different issues.
• Robustness of the routing algorithm. Could you design an algorithm which han- dles dropped out routers gracefully and with minimal effect to performance? if you do not target a high distinction in this assignment, you may consider drop- ping this criterion and focus on the remaining ones (this one is hard).
Final words
implementing this basic router will give you a good starting point to look further and deeper into the world of distributed systems. the assignment combines what you now know about concurrent systems and expands your practical abilities towards problems in distributed envi- ronments. getting closer to the end of the course, try to recollect the new concepts and tools which you have now at your disposal. While you should not expect that we made an experi- enced designer of concurrent and distributed systems out of you quite yet, you do have now a
5 | ANU College of Engineering and Computer Science October 2020
substantial amount of knowledge to already join in at discussions among professionals in the field.
Outlook
Students in Real-time and Embedded Systems (final year course) will be working with actual hardware (below) and concurrent and distributed algorithms under strict real-time constraints. there is strong demand for outstanding professionals who can handle high integrity systems interacting with the physical world.Your course provides many foundations for any such career – academic or industrial.
6 | ANU College of Engineering and Computer Science October 2020