2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
CS 3640: Introduction to Networks and Their Applications [Fall 2018]
Instructor: Rishab Nithyanand | Office hours: Wednesday 9-10 am or by appointment Teaching assistant: Md. Kowsar Hossain | Office hours: Monday 1:30-2:30 pm
Assignment 1: Crafting a picture of the Internet Released on: August 30th, 2018 | Due by: September 13th, 2018 (11:59:59 pm)
Maximum score: 100 | Value towards final grade: 13%
I. Groups
As stated in class, on the website, and on Piazza, this assignment will be done in groups of 4 or 5. Here are the groups for this assignment.
Group ID
1 2 3 4 5 6 7 8 9
10 11 12 13
Hawk IDs of group members
‘kdzhou’, ‘okueter’, ‘lburden’, ‘tnlowry’, ‘xxing2’ ‘mcagley’, ‘yzheng19’, ‘tsimonson’, ‘mfmrphy’ ‘dstutz’, ‘msmith3’, ‘sklemm’, ‘ymann’ ‘jmagri’, ‘zzhang103’, ‘uupadhyay’, ‘atran4’ ‘jglowacki’, ‘ppeterschmidt’, ‘kzhang24’, ‘tgoodmn’ ‘zhenbsong’, ‘bzhang22’, ‘ywang391’, ‘cweiske’ ‘xiaosong’, ‘bchoskins’, ‘jpflint’, ‘hpen’ ‘apatrck’, ‘xchen117’, ‘rdong6’, ‘weigui’ ‘jdhatch’, ‘susmerano’, ‘jpthiede’, ‘yitzhou’ ‘zluo1’, ‘godkin’, ‘nsonalkar’, ‘nicgoh’ ‘jstoltz’, ‘lye1’, ‘shangwchen’, ‘ywang455’ ‘hrunning’, ‘apizzimenti’, ‘yonghfan’, ‘jblue’ ‘gongyzhou’, ‘trjns’, ‘awestemeier’, ‘gmich’
Getting in touch with your team
Use the HawkIDs listed to email your team-mates or make a post on Piazza. If you’re unable to get in touch with any member of your team, let me know ASAP!
Congeniality and healthy collaboration
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 1/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
I will take complaints of discrimination and harassment extremely seriously and these will be reported to the appropriate authorities. You are expected to treat your classmates with respect and kindness.
II. Submission
Each group is to submit this notebook after filling in the blank cells as instructed. The submissions are due on ICON at 23:59:59 on September 13th, 2018. The last submission submitted by a team member before midnight on the due date will be the one graded unless ALL team members let the TA and me know that they want another submission to be graded (the late penalty if a submission made past the due date is chosen).
Late submissions
I am being generous in the amount of time allotted to this assignment to account for difficulties in scheduling meetings, etc. There will be no extensions of the due date under any circumstances. If a submission is received past the due date, the late policy detailed on the course webpage will apply.
Team-mate feedback
Each team member may also send me an email (rishab-nithyanand@uiowa.edu (mailto:rishab- nithyanand@uiowa.edu)) with subject “Feedback: Assignment 1, Group N” detailing their experience working with each of their team-mates. For each team member, tell me at least one good thing and one thing they could improve. These will be anonymized and released to each individual at the end of the term. It’s important to know how to work well in a team and early feedback before you move on to bigger and better things is always helpful. Sending feedback for all 4 assignments will fetch you a 4% bonus at the end of the term. Note: Sending with an incorrect subject line means that the email will not get forwarded to the right inbox.
III. Learning goals
My goal is to give you assignments that will help you build a nice starter pack of computer networking skills that will help you get a better picture of how the Internet works in addition to helping you in the real-world. Each of these tasks will be useful if you ever need to develop Internet app, manage a network, or do research. By the end of this assignment you should:
Know how to sniff packets leaving your computer (Task 1)
Here are some scenarios where I’ve needed to programmatically sniff packets on the network:
During Web and mobile app development. Being able to capture specific packets leaving your computer will allow you to inspect the packets generated by your app to see if they’re being created as expected. It helps with understanding if you’re getting errors because of something weird in the application/network layer of your code.
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 2/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
Identifying anomalous network events. Knowing how to sniff packets on the network will allow you to programmatically detect anomalies in network traffic which may be caused by a rogue end-host or a network attack. It can help you pin-point the victim and adversary causing the anomalous behaviour.
Know how to programmatically craft a packet in Python (Task 2)
Here are some scenarios where I’ve needed to craft a packet myself:
Identify badly configured or malicious routers and violations of the end-to-end principle.
Being able to craft your own packet will allow you to test if there are specific headers (control information) that cause a router to behave in an unexpected way — either because they are malicious or because they are poorly configured. We use this all the time in research to identify censors and proxies. If you want to know more, just ask me how.
Know how to programmatically trace the route taken by your packet and understand the functions of ICMP (Task 3)
Here are some scenarios where I’ve needed to know how my packets are being routed:
Identify pain-points in the network. Being able to observe the routers handling your traffic will allow you to figure out which routers in the network are experiencing heavy load. Knowing this will allow you to act before they fail — e.g., by installing a load-balancer.
Identify which entities are able to observe your Internet traffic. Besides just being something cool to know, this will come in handy if you ever decide to get involved in privacy and anonymity research.
IV. Team Information
Fill in your team information in this cell.
Group ID: XX
Group members: AAA, BBB, CCC, DDD, EEE
V. Task 1: Programmatically sniffing packets using Python and Scapy
Maximum score: 30 points
Step 1: 5 points Step 2: 20 points Step 3: 5 points
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 3/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
These are all the functions that should be sufficient for you to write a packet sniffer using Scapy. You do not have use just these, however. If you use other functions, do all the imports in this cell. You may not use functions not provided by Scapy.
You can read about what these functions do and generally find more Scapy documentation here:
https://scapy.readthedocs.io/en/latest/index.html (https://scapy.readthedocs.io/en/latest/index.html)
You may use the Internet to find more resources. Just make sure to cite them in the “credit reel cell” at the end of this notebook.
If you’re using a system on which Scapy is not already installed, you can install it using “pip install scapy” on the Linux terminal or the Anaconda command prompt on Windows. [some systems might require you to use “sudo”]
In [ ]:
In [ ]:
from scapy.all import sniff, ifaces
[5 points] Step 1: Print all the interfaces available on your device.
Background: An “interface” is basically the link between your computer and the network you are connected to. It generally operates at the link layer. Typically you should find an ethernet and a wireless interface interface listed in the output if you’re using a laptop.
Sample output:
INDEX IFACE IP MAC
14 Marvell AVASTAR Wireless-AC Network Controller 192.168.0.194 C4:9D:ED:27:F1:CA
16 Surface Ethernet Adapter 128.255.45.21 1 28:16:A8:07:DB:C1 8 Bluetooth Device (Personal Area Network)
C4:9D:ED:27:F1:CB
[15 points] Step 2: Write a function that takes an interface, a list of host IPs, and a maximum packet count as input and sniffs packets leaving the input interface and prints all packets to/from any of the host IPs. The function should stop after max_count packets have been sniffed. Return the packets to/from any of the host IPs.
Hint: You will have to look at the source and destination IP address fields to match the host IP address. There are two versions of IP: IPv4 and IPv6. Most packets are IPv4, but you also will need to check source and destination fields of IPv6 packets.
TL;DR: Make sure to look at IPv6 headers in addition to IP headers.
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 4/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
In [ ]:
[10 points] Step 3
a: Call the function you just wrote and supply it with the interface being used by your computer,
max_count=10**3 , host_ips=[“2620:0:e50:6810::80ff:6044”, “128.255.96.68”] .
b: Open your browser and navigate to “http://homepage.divms.uiowa.edu/~rnithyanand/cs3640- f18/assignment-1/nggyu-ra.mp4″ (http://homepage.divms.uiowa.edu/~rnithyanand/cs3640- f18/assignment-1/nggyu-ra.mp4”) and then to “http://128.255.96.68/” (http://128.255.96.68/”) (in another tab) and then to “https://www.youtube.com” (https://www.youtube.com”) (in another tab). Enjoy the music for a while. If everything is working correctly, you should have a bunch of packets going to and from my UIowa homepage. Print the following information from the headers of each layer:
link layer: dst , src . [source and destination]
network layer: dst , src . [source and destination IP addresses] transport layer: dport , sport . [source and destination ports]
Sample output (a snippet from the full output):
In [ ]:
def find_packets_to_hosts(interface, host_ips, max_count):
Packet #40 Link layer: 28:16:a8:07:db:c1 (src link address) -> e4:c7:22:67:58:c 2 (dst link address) Network layer: 2620:0:e50:1018:f083:c947:5041:a54c (src IPv6 addres s) -> 2620:0:e50:6810::80ff:6044 (dst IPv6 address) Transport layer: 55595 (src port) -> 80 (dst port) Packet #41 Link layer: e4:c7:22:67:58:c2 (src link address) -> 28:16:a8:07:db:c 1 (dst link address) Network layer: 128.255.96.68 (src IPv4 address) -> 128.255.45.211 (d st IPv4 address) Transport layer: 80 (src port) -> 55593 (dst port)
VI. Task 2: Programmatically crafting a packet using Scapy Maximum score: 30 points
Step 1: 5 points Step 2: 5 points
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 5/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
Step 3: 10 points Step 4: 10 points
These are all the functions that should be sufficient for you to craft a packet using Scapy. You do not have use just these, however. If you use other functions, do all the imports in this cell. You may not use functions not provided by Scapy.
You can read about what these functions do and generally find more Scapy documentation here:
https://scapy.readthedocs.io/en/latest/index.html (https://scapy.readthedocs.io/en/latest/index.html)
You may use the Internet to find more resources. Just make sure to cite them in the “credit reel cell” at the end of this notebook.
In [ ]:
from scapy.all import IP, ICMP, hexdump
[5 points] Step 1: From the set of packets you captured in Task 1, identify the first packet carrying application layer data (“HTTP”), used IPv4, and went from your computer to 128.255.96.68 . Print the entire contents of this packet — including all headers and application layer data. Save it in a variable named target_packet .
Sample output:
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 6/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
Packet #6 ###[ Ethernet ]###
dst src type
###[ IP ]### version
= e4:c7:22:67:58:c2 = 28:16:a8:07:db:c1 = 0x800
=4
=5
= 0x0
= 456
= 17862
= DF
=0
= 128
= tcp
= 0x0
= 128.255.45.211 = 128.255.96.68
ihl tos len id flags frag ttl proto chksum src dst \options \ ###[ TCP ]###
sport dport seq ack dataofs reserved = 0
= 55959 = http = 3801435958L = 1506251082 = 5
flags window chksum urgptr options
ip, deflate\r\nAccept-Language: en-US,en;q=0.9\r\n\r\n’ Hint: Application layer data is in the transport layer payload .
In [ ]:
= PA = 2053 = 0x91d0 = 0 = []
= 'GET / HTTP/1.1\r\nHost: 128.255.96.68\r\nCon nection: keep-alive\r\nCache-Control: max-age=0\r\nUpgrade-Insecure- Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/ 537.36\r\nDNT: 1\r\nAccept: text/html,application/xhtml+xml,applicat ion/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\r\nAccept-Encoding: gz
###[ Raw ]### load
[5 points] Step 2: Modify the application layer payload to show a different user-agent. The user- agent of the modified packet should be CS3640|Assignment-1|GroupID:NN (replace NN with your team ID). Print the contents of the modified packet.
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 7/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
Sample output:
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 8/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
0000 474554202F20485454502F312E310D0A GET / HTTP/1.1.. 0010 486F73743A203132382E3235352E3936 Host: 128.255.96 0020 2E36380D0A436F6E6E656374696F6E3A .68..Connection: 0030 206B6565702D616C6976650D0A436163 keep-alive..Cac 0040 68652D436F6E74726F6C3A206D61782D he-Control: max- 0050 6167653D300D0A557067726164652D49 age=0..Upgrade-I 0060 6E7365637572652D5265717565737473 nsecure-Requests 0070 3A20310D0A557365722D4167656E743A : 1..User-Agent: 0080 204353333634307C41737369676E6D65 CS3640|Assignme 0090 6E742D317C47726F757049443A4E4E0D nt-1|GroupID:NN. 00a0 0A444E543A20310D0A4163636570743A .DNT: 1..Accept: 00b0 20746578742F68746D6C2C6170706C69 text/html,appli 00c0 636174696F6E2F7868746D6C2B786D6C cation/xhtml+xml 00d0 2C6170706C69636174696F6E2F786D6C ,application/xml 00e0 3B713D302E392C696D6167652F776562 ;q=0.9,image/web 00f0 702C696D6167652F61706E672C2A2F2A p,image/apng,*/* 0100 3B713D302E380D0A4163636570742D45 ;q=0.8..Accept-E 0110 6E636F64696E673A20677A69702C2064 ncoding: gzip, d 0120 65666C6174650D0A4163636570742D4C eflate..Accept-L 0130 616E67756167653A20656E2D55532C65 anguage: en-US,e
0140 6E3B713D302E390D0A0D0A ###[ Ethernet ]###
n;q=0.9….
dst src type
###[ IP ]### version
= e4:c7:22:67:58:c2 = 28:16:a8:07:db:c1 = 0x800
=4
=5
= 0x0
= 447
= 19401
= DF
=0
= 128
= tcp
= 0x0
= 128.255.45.211 = 128.255.96.68
ihl tos len id flags frag ttl proto chksum src dst \options \ ###[ TCP ]###
sport dport seq ack dataofs reserved = 0
flags window chksum
= PA = 2053 = 0x91c7
= 56643 = http = 3015529313L = 3280833796L = 5
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 9/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
urgptr = 0 options = []
Hint: You will need to use “hexdump” to modify the application layer data. Look for more information here: https://scapy.readthedocs.io/en/latest/usage.html (https://scapy.readthedocs.io/en/latest/usage.html)
In [ ]:
[10 points] Step 3: Write a function that creates ICMP packets with the IP destination set to the address passed via an argument and a TTL value set to a value passed via another argument. Return the newly created icmp packet.
Background: ICMP (Internet Control Message Protocol) is a network layer protocol specifically built for the purposes of troubleshooting and logging. Think of it as an extension of the IP protocol. ICMP packets are encapsulated within an IP packet. The packet contains some information to aid with debugging. You can find more information about ICMP and its echo here: https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol (https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol) and https://tools.ietf.org/html/rfc792 (https://tools.ietf.org/html/rfc792)
Hint: You can learn how to compose packets utilizing different protocols in each layer of the stack here: https://scapy.readthedocs.io/en/latest/usage.html#stacking-layers (https://scapy.readthedocs.io/en/latest/usage.html#stacking-layers)
In [ ]:
def generate_icmp_packet(destination_ip, ttl):
[10 points] Step 4: Using the function you wrote in the previous step, generate three ICMP packets. The first one with a destination IP address of 128.255.96.68 (UIowa), the next with a destination IP address of 8.8.8.8 (Google), and the last with a destination IP address of 129.94.124.115 (UNSW – Sydney). Print the contents of these packets. Set the TTL to 100 for each of these packets.
Sample output (a snippet from the full output):
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 10/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
In [ ]:
google ###[ IP ]###
version = 4 ihl = None tos = 0x0 len = None id = 1 flags = frag = 0 ttl = 100 proto = icmp chksum = None src = 128.255.45.211 dst = 8.8.8.8 \options \
###[ ICMP ]### type code
chksum id
seq
###[ Raw ]### load
= echo-request = 0 = None = 0x0
= 0x0
= ‘xxxx’
VII. Task 3: Programmatically trace the route taken by your packet Maximum score: 30 points
Step 1: 10 points Step 2: 5 points Step 3: 15 points
These are all the functions that should be sufficient for you to trace your packets using Scapy. You do not have use just these, however. If you use other functions, do all the imports in this cell. You may not use functions not provided by Scapy.
You can read about what these functions do and generally find more Scapy documentation here:
https://scapy.readthedocs.io/en/latest/index.html (https://scapy.readthedocs.io/en/latest/index.html)
You may use the Internet to find more resources. Just make sure to cite them in the “credit reel cell” at the end of this notebook.
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 11/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
In [ ]:
In [ ]:
In [ ]:
from scapy.all import sr
[10 points] Step 1: Write a function which sends packets passed to it and collects responses to these packets. Return the responses received.
Hint: Use sr() from Scapy, not sr1() . sr1() returns only the first response. Use the timeout argument of sr() to set the maximum time you’re willing to wait for a response. 10 is
usually a good value.
def send_and_receive_packets(packets_to_send):
[5 points] Step 2: Use the function you created in the previous step to send the three ICMP packets you crafted in task 2. Print the responses you get for each of them.
Sample output (a snippet from the full output):
Begin emission: Finished sending 1 packets.
Received 2 packets, got 1 answers, remaining 0 packets google (<IP frag=0 ttl=100 proto=icmp dst=8.8.8.8 |<ICMP |<Raw
load='xxxx' |>>>, <IP version=4 ihl=5 tos=0x0 len=32 id=2849 flags = frag=0 ttl=123 proto=icmp chksum=0x75da src=8.8.8.8 dst=128.255.4 5.211 options=[] |<ICMP type=echo-reply code=0 chksum=0xf0f id=0x0
seq=0x0 |<Raw load='xxxx' |<Padding load='\x00\x00\x00\x00\x00\x0 0\x00\x00\x00\x00\x00\x00\x00\x00' |>>>>)
[15 points] Step 3: Write a function which takes an IP address as an input argument and calls generate_icmp_packet() in a loop. In each iteration of the loop, the TTL argument passed to generate_icmp_packet() should be reduced by 1 (start at 10) and the destination_ip
argument passed should be set to the same IP address passed to the function you’re writing. Send these packets and record the responses using send_and_receive_packets() . Print the source IP address of the response for each TTL value. Call this function with the destination_ip set to
8.8.8.8 .
Basically, answer this question: Where did the response come from when I sent an ICMP packet with TTL=n?
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 12/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
In [ ]:
Hint: You may need to use try and except to handle timeouts and other errors that may arise from non-responsive routers.
Sample output (a snippet from the full output):
TTL value: 10 | Source IP in response: 8.8.8.8 TTL value: 9 | Source IP in response: 8.8.8.8 TTL value: 8 | Source IP in response: 8.8.8.8 TTL value: 7 | Source IP in response: 8.8.8.8 TTL value: 6 | Source IP in response: 74.125.37.43 TTL value: 5 | Source IP in response: 108.170.243.174 TTL value: 4 | Source IP in response: 205.213.119.62 TTL value: 2 | Source IP in response: 128.255.2.129 TTL value: 1 | Source IP in response: 128.255.44.1
def get_all_hops(destination_ip):
The IP addresses you see in your output are from all the network devices (routers, load-balancers, etc.) which handled your packets every time you tried to communicate with 8.8.8.8 ! They give you an insight into how your packets traverse the Internet and the globe. You basically implemented the traceroute function! This is something that is used every day millions of times by people configuring their networks, doing research, etc.
Important: Be able to explain how traceroute works! You can read more about its working in Section 4.2 here: https://www2.eecs.berkeley.edu/Pubs/TechRpts/1997/CSD-97-945.pdf (https://www2.eecs.berkeley.edu/Pubs/TechRpts/1997/CSD-97-945.pdf)
You can use the code you just wrote anytime you want to see who gets to look at your packets later in the term when we’ll explore how to identify the owners of these IP addresses and their real-world locations.
Task 4: Credit reel!
You get 10 points just for submitting this assignment with a well written and formatted credit reel! Use the following cell to:
tell us what references you used to complete this assignment — e.g., online documentation, stackoverflow posts, etc.
who helped you IRL or on Piazza
which team-mates did which parts of the assignment
An example is filled in.
References used:
1. URL 1 to figure out how to do blah blah blah in Task 1.
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 13/14
2018/9/10 [CS3640, Assignment 1] Building a picture of the Internet
2. URL 2 to figure out how to do blah blah blah in Task 2.
Who helped us:
1. XYZ helped us with the functions to use to do Task 3 (Piazza). 2. ABC helped us with the functions to use to do Task 3 (IRL).
Team credits:
- AAA thought of the idea of blah blah for Task 2. She also wrote the complete implementation of Task 2, step 3.
- BBB did the implementation of Task 3, step 1.
http://localhost:8888/notebooks/%5BCS3640%2C%20Assignment%201%5D%20Building%20a%20picture%20of%20the%20Internet.ipynb 14/14