Python Network Programming . Beazley
http://www.dabeaz.com
Edition: Thu Jun 17 19:49:58 2010
Copyright (C) 2010 Rights Reserved
Copyright By PowCoder代写 加微信 powcoder
Python Network Programming : Table of Contents
! 1. Network Fundamentals !
! 2. Client Programming! !
! 3. Internet Data Handling! !
! 4. Web Programming Basics!
! 5. Advanced Networks! !
Edition: Thu Jun 17 19:49:58 2010
! ! ! ! ! ! ! ! ! ! ! ! ! ! !
! ! 32 ! ! 49 ! ! 65 ! ! 93
Slide Title Index 0. Introduction
Introduction
Support Files
Python Networking
This Course
Standard Library
Prerequisites
1. Network Fundamentals
Network Fundamentals
The Problem
Two Main Issues
Network Addressing
Standard Ports
Using netstat
Connections
Client/Server Concept
Request/Response Cycle
Using Telnet
Data Transport
Socket Basics
Socket Types
Using a Socket
TCP Client
Exercise 1.1
Server Implementation
TCP Server
Exercise 1.2
Advanced Sockets
Partial Reads/Writes
Sending All Data
End of Data
Data Reassembly
Non-blocking Sockets
Socket Options
Sockets as Files
Exercise 1.3
Odds and Ends
UDP : Datagrams
UDP Server
UDP Client
Unix Domain Sockets
Raw Sockets
Sockets and Concurrency
Threaded Server 1-50
Forking Server (Unix) 1-51
Asynchronous Server 1-52
Utility Functions 1-53
Omissions 1-54
Discussion 1-55
0-5 Client Programming 2-1
0-6 Overview 2-2
urllib Module 2-3
urllib protocols 2-5
HTML Forms 2-6
Web Services 2-8
1-1 Parameter Encoding 2-9
1-2 Sending Parameters 2-10
1-3 Response Data 2-12
1-4 Response Headers 2-13
1-5 Response Status 2-14
1-6 Exercise 2.1 2-15
1-7 urllib Limitations 2-16
1-8 urllib2 Module 2-17
1-9 urllib2 Example 2-18
1-10 urllib2 Requests 2-19
1-11 Requests with Data 2-20
1-12 Request Headers 2-21
1-13 urllib2 Error Handling 2-22
1-14 urllib2 Openers 2-23
1-15 urllib2 build_opener() 2-24
1-16 Example : Login Cookies 2-25
1-17 Discussion 2-26
1-18 Exercise 2.2 2-27
1-19 Limitations 2-28
1-27 ftplib 2-29
1-28 Upload to a FTP Server 2-30
1-29 httplib 2-31
1-31 smtplib 2-32
1-32 Exercise 2.3 2-33
2. Client Programming
3. Internet Data Handling
1-37 Internet Data Handling 3-1
1-39 Overview 3-2
1-40 CSV Files 3-3
1-41 Parsing HTML 3-4
1-42 Running a Parser 3-6
1-43 HTML Example 3-7
1-44 XML Parsing with SAX 3-9
1-45 Brief XML Refresher 3-10
1-46 SAX Parsing 3-11
Exercise 3.1
XML and ElementTree
etree Parsing Basics
Obtaining Elements
Iterating over Elements
Element Attributes
Search Wildcards
cElementTree
Tree Modification
Tree Output
Iterative Parsing
Exercise 3.2
Sample JSON File
Processing JSON Data
Exercise 3.3
4. Web Programming
Web Programming Basics
Introduction
Disclaimer
HTTP Explained
HTTP Client Requests
HTTP Responses
HTTP Protocol
Content Encoding
Payload Packaging
Exercise 4.1
Role of Python
Typical Python Tasks
Content Generation
Example : Page Templates
Commentary
Exercise 4.2
HTTP Servers
A Simple Web Server
Exercise 4.3
A Web Server with CGI
CGI Scripting
CGI Example
CGI Mechanics
Classic CGI Interface
CGI Query Variables
cgi Module
CGI Responses
Note on Status Codes
CGI Commentary
Exercise 4.4
WSGI Interface
3-13 WSGI Example 4-37
3-14 WSGI Applications 4-38
3-15 WSGI Environment 4-39
3-17 Processing WSGI Inputs 4-41
3-18 WSGI Responses 4-42
3-19 WSGI Content 4-44
3-20 WSGI Content Encoding 4-45
3-22 WSGI Deployment 4-46
3-23 WSGI and CGI 4-48
3-24 Exercise 4.5 4-49
3-25 Customized HTTP 4-50
3-28 Exercise 4.6 4-53
3-29 Web Frameworks 4-54
3-30 Commentary 4-56
5. Advanced Networking
Advanced Networking 5-1
Overview 5-2
4-1 Problem with Sockets 5-3
4-2 SocketServer 5-4
4-3 SocketServer Example 5-5
4-4 Execution Model 5-11
4-5 Exercise 5.1 5-12
4-6 Big Picture 5-13
4-7 Concurrent Servers 5-14
4-8 Server Mixin Classes 5-15
4-9 Server Subclassing 5-16
4-10 Exercise 5.2 5-17
4-11 Distributed Computing 5-18
4-12 Discussion 5-19
4-13 XML-RPC 5-20
4-14 Simple XML-RPC 5-21
4-15 XML-RPC Commentary 5-23
4-17 XML-RPC and Binary 5-24
4-18 Exercise 5.3 5-25
4-19 Serializing Python Objects 5-26
4-20 pickle Module 5-27
4-21 Pickling to Strings 5-28
4-22 Example 5-29
4-23 Miscellaneous Comments 5-31
4-24 Exercise 5.4 5-32
4-27 multiprocessing 5-33
4-28 Connections 5-34
4-29 Connection Use 5-35
4-30 Example 5-36
4-31 Commentary 5-38
4-32 What about… 5-40
4-33 Network Wrap-up 5-41
4-34 Exercise 5.5 5-42
Section 0 Introduction
Support Files
• Course exercises:
http://www.dabeaz.com/python/pythonnetwork.zip
• This zip file should be downloaded and extracted
someplace on your machine
• All of your work will take place in the the “PythonNetwork” folder
Copyright (C) 2010, http://www.dabeaz.com 1- 2
Python Networking
• Network programming is a major use of Python
• Python standard library has wide support for network protocols, data encoding/decoding, and other things you need to make it work
• Writing network programs in Python tends to be substantially easier than in C/C++
Copyright (C) 2010, http://www.dabeaz.com 1- 3
This Course
• This course focuses on the essential details of network programming that all Python programmers should probably know
• Low-level programming with sockets
• High-level client modules
• How to deal with common data encodings • Simple web programming (HTTP)
• Simple distributed computing
Copyright (C) 2010, http://www.dabeaz.com
Standard Library
• We will only cover modules supported by the Python standard library
• These come with Python by default
• Keep in mind, much more functionality can be found in third-party modules
• Will give links to notable third-party libraries as appropriate
Copyright (C) 2010, http://www.dabeaz.com
Prerequisites
• You should already know Python basics
• However, you don’t need to be an expert on all of its advanced features (in fact, none of the code to be written is highly sophisticated)
• You should have some prior knowledge of systems programming and network concepts
Copyright (C) 2010, http://www.dabeaz.com 1- 6
Section 1 Network Fundamentals
The Problem
• Communication between computers
• It’s just sending/receiving bits
Copyright (C) 2010, http://www.dabeaz.com 1- 2
Two Main Issues
• Addressing
• Specifying a remote computer and service
• Data transport
• Moving bits back and forth
Copyright (C) 2010, http://www.dabeaz.com 1- 3
Network Addressing
• Machines have a hostname and IP address • Programs/services have port numbers
foo.bar.com
205.172.13.4
www.python.org
82.94.237.218
Copyright (C) 2010, http://www.dabeaz.com
Standard Ports
• Ports for common services are preassigned
25 SMTP (Mail)
80 HTTP (Web)
110 POP3 (Mail)
119 NNTP (News)
443 HTTPS (web)
• Other port numbers may just be randomly assigned to programs by the operating system
Copyright (C) 2010, http://www.dabeaz.com
Using netstat
• Use ‘netstat’ to view active network connections
shell % netstat -a
Active Internet connections (servers and established)
-Q Send-Q Local Address
Foreign Address
tcp 0
tcp 0
tcp 0
tcp 0
tcp 0
tcp 0
tcp 0
tcp 0
tcp 0
tcp 0
0 localhost:mysql
0 192.168.119.139:domain *:*
0 localhost:domain *:*
0 *:ssh *:*
Note: Must execute from the command shell on both Unix and Windows
Copyright (C) 2010, http://www.dabeaz.com
Connections
• Each endpoint of a network connection is always represented by a host and port #
• In Python you write it out as a tuple (host,port)
(“www.python.org”,80)
(“205.172.13.4”,443)
• In almost all of the network programs you’ll write, you use this convention to specify a network address
Copyright (C) 2010, http://www.dabeaz.com
Client/Server Concept
• Each endpoint is a running program
• Servers wait for incoming connections and provide a service (e.g., web, mail, etc.)
• Clients make connections to servers
Client Server
www.bar.com
205.172.13.4
web Port 80
Copyright (C) 2010, http://www.dabeaz.com
Request/Response Cycle
• Most network programs use a request/ response model based on messages
• Client sends a request message (e.g., HTTP)
GET /index.html HTTP/1.0
• Server sends back a response message
HTTP/1.0 200 OK
Content-type: text/html
Content-length: 48823
…
• The exact format depends on the application Copyright (C) 2010, http://www.dabeaz.com
type this and press return a few times
UsingTelnet
• As a debugging aid, telnet can be used to directly communicate with many services
telnet hostname portnum • Example:
shell % telnet www.python.org 80 Trying 82.94.237.218… Connected to www.python.org. Escape character is ‘^]’.
GET /index.html HTTP/1.0
HTTP/1.1 200 OK
Date: Mon, 31 Mar 2008 13:34:03 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2
mod_ssl/2.2.3 OpenSSL/0.9.8c
Copyright (C) 2010, http://www.dabeaz.com
DataTransport
• There are two basic types of communication
• Streams (TCP): Computers establish a connection with each other and read/write data in a continuous stream of bytes—like a file. This is the most common.
• Datagrams (UDP): Computers send discrete packets (or messages) to each other. Each packet contains a collection of bytes, but each packet is separate and self-contained.
Copyright (C) 2010, http://www.dabeaz.com
• Programming abstraction for network code • Socket: A communication endpoint
• Supported by socket library module
• Allows connections to be made and data to be transmitted in either direction
Copyright (C) 2010, http://www.dabeaz.com 1- 12
Socket Basics • To create a socket
import socket
s = socket.socket(addr_family, type)
• Address families
socket.AF_INET Internet protocol (IPv4)
socket.AF_INET6 Internet protocol (IPv6)
• Socket types
socket.SOCK_STREAM Connection based stream (TCP)
socket.SOCK_DGRAM Datagrams (UDP)
• Example:
from socket import *
s = socket(AF_INET,SOCK_STREAM)
Copyright (C) 2010, http://www.dabeaz.com
Socket Types
• Almost all code will use one of following
from socket import *
s = socket(AF_INET, SOCK_STREAM)
s = socket(AF_INET, SOCK_DGRAM)
• Most common case:TCP connection
s = socket(AF_INET, SOCK_STREAM)
Copyright (C) 2010, http://www.dabeaz.com
Using a Socket
• Creating a socket is only the first step
s = socket(AF_INET, SOCK_STREAM)
• Further use depends on application • Server
Copyright (C) 2010, http://www.dabeaz.com
• Listen for incoming connections Client
• Make an outgoing connection
TCP Client
• How to make an outgoing connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.connect((“www.python.org”,80)) # Connect
s.send(“GET /index.html HTTP/1.0\n\n”) # Send request
data = s.recv(10000) # Get response
• s.close()
s.connect(addr) makes a connection
s.connect((“www.python.org”,80))
• Once connected, use send(),recv() to transmit and receive data
• close() shuts down the connection Copyright (C) 2010, http://www.dabeaz.com
Copyright (C) 2010, http://www.dabeaz.com
Exercise 1.1 Time : 10 Minutes
Server Implementation
• Network servers are a bit more tricky
• Must listen for incoming connections on a well-known port number
• Typically run forever in a server-loop
• May have to service multiple clients
Copyright (C) 2010, http://www.dabeaz.com
TCP Server • A simple server
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
• Send a message back to a client
% telnet localhost 9000
Connected to localhost.
Escape character is ‘^]’.
Hello 127.0.0.1
Connection closed by foreign host. %
Copyright (C) 2010, http://www.dabeaz.com
Server message
TCP Server • Address binding
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
from socket import *
s = socket(AF_INET,SOCK_STREAM)
binds the socket to a specific address
• Addressing
binds to localhost
Copyright (C) 2010, http://www.dabeaz.com
s.bind((“”,9000))
s.bind((“localhost”,9000))
s.bind((“192.168.2.1”,9000))
s.bind((“104.21.4.2”,9000))
If system has multiple IP addresses, can bind to a specific address
TCP Server Start listening for connections
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
Tells operating system to start listening for connections on the socket
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
• s.listen(backlog)
• backlog is # of pending connections to allow • Note: not related to max number of clients
Copyright (C) 2010, http://www.dabeaz.com
TCP Server Accepting a new connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
Accept a new client connection
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
• s.accept() blocks until connection received • Server sleeps if nothing is happening
Copyright (C) 2010, http://www.dabeaz.com
TCP Server • Client socket and address
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
Accept returns a pair (client_socket,addr)
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
This is a new socket that’s used for data
Copyright (C) 2010, http://www.dabeaz.com
(“104.23.11.4”,27743)
This is the network/port address of the client that connected
TCP Server • Sending data
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
Note: Use the client socket for transmitting data.The server socket is only used for accepting new connections.
Send data to client
Copyright (C) 2010, http://www.dabeaz.com
TCP Server Closing the connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
print “Received connection from”, a c.send(“Hello %s\n” % a[0]) c.close()
• Note: Server can keep client connection alive as long as it wants
• Can repeatedly receive/send data
Copyright (C) 2010, http://www.dabeaz.com
Close client connection
TCP Server Waiting for the next connection
from socket import *
s = socket(AF_INET,SOCK_STREAM) s.bind((“”,9000))
s.listen(5)
while True:
c,a = s.accept()
Wait for next connection
print “Received connection from”, a
c.send(“Hello %s\n” % a[0])
• Original server socket is reused to listen for more connections
• Server runs forever in a loop like this
Copyright (C) 2010, http://www.dabeaz.com
Copyright (C) 2010, http://www.dabeaz.com
Exercise 1.2 Time : 20 Minutes
Advanced Sockets
• Socket programming is often a mess • Huge number of options
• Many corner cases
• Many failure modes/reliability issues • Will briefly cover a few critical issues
Copyright (C) 2010, http://www.dabeaz.com
Partial Reads/Writes
• Be aware that reading/writing to a socket may involve partial data transfer
• send() returns actual bytes sent
• recv() length is only a maximum limit
>>> len(data)
>>> s.send(data)
37722 Sent partial data >>>
>>> data = s.recv(10000)
>>> len(data)
6420 Received less than max >>>
Copyright (C) 2010, http://www.dabeaz.com
Partial Reads/Writes
• Be aware that for TCP, the data stream is continuous—no concept of records, etc.
s.send(data)
s.send(moredata)
data = s.recv(maxsize)
This recv() may return data from both of the sends combined or less data than even the first send
• A lot depends on OS buffers, network
bandwidth, congestion, etc.
Copyright (C) 2010, http://www.dabeaz.com
Sending All Data
• To wait until all data is sent, use sendall()
s.sendall(data)
• Blocks until all data is transmitted
• For most normal applications, this is what you should use
• Exception :You don’t use this if networking is mixed in with other kinds of processing
(e.g., screen updates, multitasking, etc.)
Copyright (C) 2010, http://www.dabeaz.com
End of Data
• How to tell if there is no more data? • recv() will return empty string
>>> s.recv(1000) ”
• This means that the other end of the connection has been closed (no more sends)
Copyright (C) 2010, http://www.dabeaz.com
Data Reassembly
• Receivers often need to reassemble messages from a series of small chunks
• Here is a programming template for that
fragments = [] # List of chunks
while not done:
chunk = s.recv(maxsize) # Get a chunk
if not chunk:
break # EOF. No more data
fragments.append(chunk)
# Reassemble the message
message = “”.join(fragments)
• Don’t use string concat (+=). It’s slow. Copyright (C) 2010, htt
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com