Python Network Programming
http://www.dabeaz.com
Copyright By PowCoder代写 加微信 powcoder
Edition: Thu Jun 17 19:49:58 2010
Copyright (C) 2010
Rights Reserved
Python Network Programming : Table of Contents
! 1. Network Fundamentals !! ! ! ! ! ! 4
! 2. Client Programming! ! ! ! ! ! ! 32
! 3. Internet Data Handling! ! ! ! ! ! ! 49
! 4. Web Programming Basics! ! ! ! ! ! 65
! 5. Advanced Networks! ! ! ! ! ! ! 93
Edition: Thu Jun 17 19:49:58 2010
Introduction
Copyright (C) 2010, http://www.dabeaz.com 1-
Support Files
• Course exercises:
http://www.dabeaz.com/python/pythonnetwork.zip
• This zip file should be downloaded and extracted
someplace on your machine
• All of your work will take place in the the
“PythonNetwork” folder
Copyright (C) 2010, http://www.dabeaz.com 1-
Python Networking
• Network programming is a major use of Python
• Python standard library has wide support for
network protocols, data encoding/decoding, and
other things you need to make it work
• Writing network programs in Python tends to be
substantially easier than in C/C++
Copyright (C) 2010, http://www.dabeaz.com 1-
This Course
• This course focuses on the essential details of
network programming that all Python
programmers should probably know
• Low-level programming with sockets
• High-level client modules
• How to deal with common data encodings
• Simple web programming (HTTP)
• Simple distributed computing
Copyright (C) 2010, http://www.dabeaz.com 1-
Standard Library
• We will only cover modules supported by the
Python standard library
• These come with Python by default
• Keep in mind, much more functionality can be
found in third-party modules
• Will give links to notable third-party libraries as
appropriate
Copyright (C) 2010, http://www.dabeaz.com 1-
Prerequisites
• You should already know Python basics
• However, you don’t need to be an expert on all
of its advanced features (in fact, none of the code
to be written is highly sophisticated)
• You should have some prior knowledge of
systems programming and network concepts
Network Fundamentals
Copyright (C) 2010, http://www.dabeaz.com 1-
The Problem
• Communication between computers
• It’s just sending/receiving bits
Copyright (C) 2010, http://www.dabeaz.com 1-
Two Main Issues
• Addressing
• Specifying a remote computer and service
• Data transport
• Moving bits back and forth
Copyright (C) 2010, http://www.dabeaz.com 1-
Network Addressing
• Machines have a hostname and IP address
• Programs/services have port numbers
foo.bar.com
205.172.13.4
www.python.org
82.94.237.218
Copyright (C) 2010, http://www.dabeaz.com 1-
Standard Ports
• Ports for common services are preassigned
21 FTP
22 SSH
23 Telnet
25 SMTP (Mail)
80 HTTP (Web)
110 POP3 (Mail)
119 NNTP (News)
443 HTTPS (web)
• Other port numbers may just be randomly
assigned to programs by the operating system
Copyright (C) 2010, http://www.dabeaz.com 1-
Using netstat
• Use ‘netstat’ to view active network connections
shell % netstat -a
Active Internet connections (servers and established)
-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:imaps *:* LISTEN
tcp 0 0 *:pop3s *:* LISTEN
tcp 0 0 localhost:mysql *:* LISTEN
tcp 0 0 *:pop3 *:* LISTEN
tcp 0 0 *:imap2 *:* LISTEN
tcp 0 0 *:8880 *:* LISTEN
tcp 0 0 *:www *:* LISTEN
tcp 0 0 192.168.119.139:domain *:* LISTEN
tcp 0 0 localhost:domain *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
• Note: Must execute from the command shell on
both Unix and Windows
Copyright (C) 2010, http://www.dabeaz.com 1-
Connections
• Each endpoint of a network connection is always
represented by a host and port #
• In Python you write it out as a tuple (host,port)
(“www.python.org”,80)
(“205.172.13.4”,443)
• In almost all of the network programs you’ll
write, you use this convention to specify a
network address
Copyright (C) 2010, http://www.dabeaz.com 1-
Client/Server Concept
• Each endpoint is a running program
• Servers wait for incoming connections and
provide a service (e.g., web, mail, etc.)
• Clients make connections to servers
www.bar.com
205.172.13.4
web Port 80browser
Client Server
Copyright (C) 2010, http://www.dabeaz.com 1-
Request/Response Cycle
• Most network programs use a request/
response model based on messages
• Client sends a request message (e.g., HTTP)
GET /index.html HTTP/1.0
• Server sends back a response message
HTTP/1.0 200 OK
Content-type: text/html
Content-length: 48823
• The exact format depends on the application
Copyright (C) 2010, http://www.dabeaz.com 1-
Using Telnet
• As a debugging aid, telnet can be used to
directly communicate with many services
telnet hostname portnum
• Example:
shell % telnet www.python.org 80
Trying 82.94.237.218…
Connected to www.python.org.
Escape character is ‘^]’.
GET /index.html HTTP/1.0
HTTP/1.1 200 OK
Date: Mon, 31 Mar 2008 13:34:03 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2
mod_ssl/2.2.3 OpenSSL/0.9.8c
return a few
Try info.cern.ch instead
GET / HTTP/1.1
Host: info.cern.ch\n\n
Copyright (C) 2010, http://www.dabeaz.com 1-
Data Transport
• There are two basic types of communication
• Streams (TCP): Computers establish a
connection with each other and read/write data
in a continuous stream of bytes—like a file. This
is the most common.
• Datagrams (UDP): Computers send discrete
packets (or messages) to each other. Each
packet contains a collection of bytes, but each
packet is separate and self-contained.
Copyright (C) 2010, http://www.dabeaz.com 1-
• Programming abstraction for network code
• Socket: A communication endpoint
socket socket
• Supported by socket library module
• Allows connections to be made and data to be
transmitted in either direction
Copyright (C) 2010, http://www.dabeaz.com 1-
Socket Basics
• Address families
import socket
s = socket.socket(addr_family, type)
• Example:
• To create a socket
socket.AF_INET Internet protocol (IPv4)
socket.AF_INET6 Internet protocol (IPv6)
• Socket types
socket.SOCK_STREAM Connection based stream (TCP)
socket.SOCK_DGRAM Datagrams (UDP)
from socket import *
s = socket(AF_INET,SOCK_STREAM)
Copyright (C) 2010, http://www.dabeaz.com 1-
Socket Types
• Most common case: TCP connection
from socket import *
s = socket(AF_INET, SOCK_STREAM)
s = socket(AF_INET, SOCK_DGRAM)
• Almost all code will use one of following
s = socket(AF_INET, SOCK_STREAM)
Copyright (C) 2010, http://www.dabeaz.com 1-
Using a Socket
• Creating a socket is only the first step
s = socket(AF_INET, SOCK_STREAM)
• Further use depends on application
• Listen for incoming connections
• Make an outgoing connection
Copyright (C) 2010, http://www.dabeaz.com 1-
TCP Client
• How to make an outgoing connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.connect((“www.python.org”,80)) # Connect
s.send(“GET /index.html HTTP/1.0\n\n”) # Send request
data = s.recv(10000) # Get response
• s.connect(addr) makes a connection
s.connect((“www.python.org”,80))
• Once connected, use send(),recv() to
transmit and receive data
• close() shuts down the connection
use info.cern.ch instead
addr must be a tuple (host,port)
Copyright (C) 2010, http://www.dabeaz.com 1-
Exercise 1.1
Time : 10 Minutes
Copyright (C) 2010, http://www.dabeaz.com 1-
Server Implementation
• Network servers are a bit more tricky
• Must listen for incoming connections on a
well-known port number
• Typically run forever in a server-loop
• May have to service multiple clients
Try the above example using Python, but don’t try it interactively
using Python shell. Create a file “client.py”, then run it with
“python client.py”.
Otherwise, you may get a socket error 54!
Copyright (C) 2010, http://www.dabeaz.com 1-
TCP Server
• A simple server
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
• Send a message back to a client
% telnet localhost 9000
Connected to localhost.
Escape character is ‘^]’.
Hello 127.0.0.1
Connection closed by foreign host.
Server message
Copyright (C) 2010, http://www.dabeaz.com 1-
TCP Server
• Address binding
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
• Addressing
s.bind((“”,9000))
s.bind((“localhost”,9000))
s.bind((“192.168.2.1”,9000))
s.bind((“104.21.4.2”,9000))
binds the socket to
a specific address
If system has multiple
IP addresses, can bind
to a specific address
binds to localhost
Copyright (C) 2010, http://www.dabeaz.com 1-
TCP Server
• Start listening for connections
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
• s.listen(backlog)
• backlog is # of pending connections to allow
• Note: not related to max number of clients
Tells operating system to
start listening for
connections on the socket
Copyright (C) 2010, http://www.dabeaz.com 1-
TCP Server
• Accepting a new connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
• s.accept() blocks until connection received
• Server sleeps if nothing is happening
Accept a new client connection
Copyright (C) 2010, http://www.dabeaz.com 1-
TCP Server
• Client socket and address
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
Accept returns a pair (client_socket,addr)
(“104.23.11.4”,27743)
This is the network/port
address of the client that
This is a new socket
that’s used for data
Copyright (C) 2010, http://www.dabeaz.com 1-
TCP Server
• Sending data
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
Send data to client
Note: Use the client socket for
transmitting data. The server
socket is only used for
accepting new connections.
Copyright (C) 2010, http://www.dabeaz.com 1-
TCP Server
• Closing the connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
c.close() Close client connection
• Note: Server can keep client connection alive
as long as it wants
• Can repeatedly receive/send data
Copyright (C) 2010, http://www.dabeaz.com 1-
TCP Server
• Waiting for the next connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
Wait for next connection
• Original server socket is reused to listen for
more connections
• Server runs forever in a loop like this
Copyright (C) 2010, http://www.dabeaz.com 1-
Exercise 1.2
Time : 20 Minutes
Copyright (C) 2010, http://www.dabeaz.com 1-
Advanced Sockets
• Socket programming is often a mess
• Huge number of options
• Many corner cases
• Many failure modes/reliability issues
• Will briefly cover a few critical issues
Copyright (C) 2010, http://www.dabeaz.com 1-
Partial Reads/Writes
• Be aware that reading/writing to a socket
may involve partial data transfer
• send() returns actual bytes sent
• recv() length is only a maximum limit
>>> len(data)
>>> s.send(data)
>>> data = s.recv(10000)
>>> len(data)
Sent partial data
Received less than max
Copyright (C) 2010, http://www.dabeaz.com 1-
Partial Reads/Writes
• Be aware that for TCP, the data stream is
continuous—no concept of records, etc.
s.send(data)
s.send(moredata)
data = s.recv(maxsize)
This recv() may return data
from both of the sends
combined or less data than
even the first send
• A lot depends on OS buffers, network
bandwidth, congestion, etc.
#send’s not equal to #recv’s
but, len(#bytes sent) = len(#bytes recv’d)
Copyright (C) 2010, http://www.dabeaz.com 1-
Sending All Data
• To wait until all data is sent, use sendall()
s.sendall(data)
• Blocks until all data is transmitted
• For most normal applications, this is what
you should use
• Exception : You don’t use this if networking is
mixed in with other kinds of processing
(e.g., screen updates, multitasking, etc.)
Copyright (C) 2010, http://www.dabeaz.com 1-
End of Data
• How to tell if there is no more data?
• recv() will return empty string
>>> s.recv(1000)
• This means that the other end of the
connection has been closed (no more sends)
Copyright (C) 2010, http://www.dabeaz.com 1-
Data Reassembly
• Receivers often need to reassemble
messages from a series of small chunks
• Here is a programming template for that
fragments = [] # List of chunks
while not done:
chunk = s.recv(maxsize) # Get a chunk
if not chunk:
break # EOF. No more data
fragments.append(chunk)
# Reassemble the message
message = “”.join(fragments)
• Don’t use string concat (+=). It’s slow.
Copyright (C) 2010, http://www.dabeaz.com 1-
• Most socket operations block indefinitely
• Can set an optional timeout
s = socket(AF_INET, SOCK_STREAM)
s.settimeout(5.0) # Timeout of 5 seconds
• Will get a timeout exception
>>> s.recv(1000)
Traceback (most recent call last):
File “
socket.timeout: timed out
• Disabling timeouts
s.settimeout(None)
Copyright (C) 2010, http://www.dabeaz.com 1-
Exercise 1.3
Time : 15 Minutes
Copyright (C) 2010, http://www.dabeaz.com 1-
Odds and Ends
• Other supported socket types
• Datagram (UDP) sockets
• Unix domain sockets
• Raw sockets/Packets
• Sockets and concurrency
• Useful utility functions
Copyright (C) 2010, http://www.dabeaz.com 1-
UDP : Datagrams
• Data sent in discrete packets (Datagrams)
• No concept of a “connection”
• No reliability, no ordering of data
• Datagrams may be lost, arrive in any order
• Higher performance (used in games, etc.)
DATA DATA DATA
Copyright (C) 2010, http://www.dabeaz.com 1-
UDP Server
• A simple datagram server
from socket import *
s = socket(AF_INET,SOCK_DGRAM)
s.bind((“”,10000))
while True:
data, addr = s.recvfrom(maxsize)
resp = “Get off my lawn!”
s.sendto(resp,addr)
Create datagram socket
• No “connection” is established
• It just sends and receives packets
Bind to a specific port
Wait for a message
Send response
(optional)
Copyright (C) 2010, http://www.dabeaz.com 1-
UDP Client
• Sending a datagram to a server
from socket import *
s = socket(AF_INET,SOCK_DGRAM)
msg = “Hello World”
s.sendto(msg,(“server.com”,10000))
data, addr = s.recvfrom(maxsize)
Create datagram socket
• Key concept: No “connection”
• You just send a data packet
Send a message
Wait for a response
(optional)
returned data remote address
Copyright (C) 2010, http://www.dabeaz.com 1-
Unix Domain Sockets
• Available on Unix based systems. Sometimes
used for fast IPC or pipes between processes
• Creation:
s = socket(AF_UNIX, SOCK_STREAM)
s = socket(AF_UNIX, SOCK_DGRAM)
• Rest of the programming interface is the same
• Address is just a “filename”
s.bind(“/tmp/foo”) # Server binding
s.connect(“/tmp/foo”) # Client connection
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com