HTTP Programming

Transcript HTTP Programming

HTTP Programming
Copyright (C) 2009 Dr.Yong Uk Song
Yonsei University Wonju Campus
What is HTTP?
 Hypertext Transfer Protocol (HTTP) is an application-level
(application-layer) protocol for distributed, collaborative,
hypermedia information systems, i.e. the World Wide Web.
 HTTP development was coordinated by the World Wide Web
Consortium and the Internet Engineering Task Force (IETF),
culminating in the publication of a series of Requests for Comments
(RFCs), most notably RFC 2616 (June 1999), which defines HTTP/1.1,
the version of HTTP in common use.
 HTTP is a request/response standard of a client and a server.
 A client is the end-user, the server is the web site.
 The client making a HTTP request - using a web browser, spider, or other
end-user tool - is referred to as the user agent.
 The responding server - which stores or creates resources such as HTML
files and images - is called the origin server.
TCP/IP & OSI 7 Layer
OSI 7 Layer
Application
Presentation
Session
Internet Protocol Suite
BGP · DHCP · DNS · FTP · GTP · HTTP · IMAP ·
IRC · Megaco · MGCP · NNTP · NTP · POP · RIP Application
· RPC · RTP · RTSP · SDP · SIP · SMTP · SNMP ·
SOAP · SSH · Telnet · TLS/SSL · XMPP · (more)
Transport
Network
TCP
ICMP
Transport
UDP
IP
ARP
RARP
Internet
Data Link
H/W interface: Ethernet, Token-Ring …
Physical
Link
Architecture of WWW
user
input
GUI
user
output
content
File Input
request with
arguments
Web
Browser
HTTP
response
Web Server
HTML
file
CGI input
CGI
CGI output
CGI
program
History of HTTP
 HTTP/0.9 : 1991
 HTTP/1.0 : 1992, RFC1945
 HTTP/1.1 : Jan. 1997, RFC2068
 HTTP/1.1 : Jun. 1999, RFC2616
What is in HTTP?
 Dialogue-layer protocol
 Presentation-layer protocol
 Other protocols
HTTP: Dialogue-layer Protocol (1)
 It is based on the Request/Response paradigm.




A connection is established.
A client sends a request to a server.
The server sends a response to the client.
The connection is ended.
 In terms of TCP/IP programming:








After socket connection,
A client sends (prints) a message to a server.
The server receives (reads) the message from the client.
The server makes a response message based on the message from the client.
The server sends (prints) the response message to the client.
The client receives (reads) the message from the server.
The socket connection is ended.
The client processes the message: prints out on window, or, ….
HTTP: Dialogue-layer Protocol (2)
* Client
* Server
…
String request = "…";
output.println(request);
…
input.readLine();
…
input.readLine();
String response = "…";
output.println(response);
…
HTTP: Presentation-layer protocol (1)
 It defines the format of request and response.
 Definitions
 HTTP-message
 Request
=
=
 Response
=
Request | Response
Request-Line
* (General-Header
| Request-Header
| Entity-Header)
crlf
[Entity-Body]
Status-Line
* (General-Header
| Response-Header
| Entity-Header)
crlf
[Entity-Body]
Legend:
| - or
* - multiple
[ ] - optional
HTTP: Presentation-layer protocol (2)
 Example : Request (1)
GET http://www.miami.muohio.edu/ HTTP/1.1
Host: www.example.com
crlf
 Example : Request (2)
POST /HTTP2Server/login.jsp HTTP/1.1
Host: www.example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 30
crlf
username=yusong&password=10000
HTTP: Presentation-layer protocol (3)
 Example : Response
HTTP/1.1 200 OK
Date: Wed, 08 Jul 2009 21:39:26 GMT
Server: Apache/2.0.46 (Red Hat)
Content-Language: en-US
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
crlf
<!DOCTYPE html>
<html>
<head>
<title>Hello HTML</title>
</head>
<body>
<p>Hello World!</p>
</body>
</html>
Digression
URL
URL (1)
 A Uniform Resource Locator (URL) is a type of Uniform
Resource Identifier (URI) that specifies where an identified
resource is available and the mechanism for retrieving it.
 In its current strict technical meaning, a URL is a URI that,
in addition to identifying a resource, provides a means of
locating the resource by describing its primary access
mechanism (e.g., its network 'location').
 In popular language, a URL is also referred to as a Web
address.
URL (2)
 Format
 resource_type://domain:port/filepathname?query_string#anchor
e.g.
http://dragon.yonsei.ac.kr/~yusong/index.html
http://yonsei.ac.kr:8000/~yusong/a.cgi?username=yusong&b=b
ftp://dragon.yonsei.ac.kr/~yusong/sample.file
telnet://dragon.yonsei.ac.kr
rlogin://[email protected]
 Tim Berners-Lee regrets:
http://www.serverroute.com/path/to/file.html
http://com/serverroute/www/path/to/file.html
HTTP (1) : Client using GET Method
HTTP (1) : Client using GET Method
HTTP1Client
import java.io.*;
import java.net.*;
try {
Socket socket = new Socket("www.miami.muohio.edu", 80);
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
output.println("GET http://www.miami.muohio.edu/ HTTP/1.1");
output.println("Host: www.example.com");
output.println("");
String response;
for (; (response = input.readLine()) != null;) {
Host
Port
System.out.println(response);
}
output.close();
input.close();
socket.close();
Request
} catch (Exception e) {
System.err.println(e);
}
Response
HTTP (1) : Client using GET Method
 Request Message
GET http://www.miami.muohio.edu/ HTTP/1.1
Host: www.example.com
crlf
HTTP (1) : Client using GET Method
 Response Message
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2009 18:03:36 GMT
Server: Apache/2.0.46 (Red Hat)
Set-Cookie: SaneID=134.53.7.196-1248285816897469; path=/; expires=Sat, 26-Jul-14
18:03:36 GMT; domain=.muohio.edu
Content-Language: en-US
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
crlf
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="keywords" content="miami,university,ohio,college,public" />
<meta name="description" content="Located in Oxford, Ohio, Miami University is a public
university of 16,000 students offering strong academics, a vibrant residence life program, and
faculty who love to teach and mentor students." />
<title>Miami University</title>
…
</html>
HTTP (1) : Client using GET Method
 Exercise
 Make a Java program such that:
 Send a request to the Web page of "http://www.muohio.edu:8080/",
and print out the response from the Web server.
 Project name
 HTTP1X1Client
Digression
Website (1) : Login
Website (1) : Login HTTP2Server/login.html
<html>
<body>
<h1>Login</h1>
<form method="post" action="login.jsp">
<table>
<tr>
<td>User name:</td>
<td><input type="text" name="username"></td>
</tr>
<tr>
<td>Password:</td>
<td><input type="password" name="password"></td>
</tr>
<tr>
<td colspan="2"><input type="submit" value="Login"></td>
</tr>
</table>
</form>
</body>
</html>
Website (1) : Login HTTP2Server/login.jsp
<%
String username = request.getParameter("username");
String password = request.getParameter("password");
%>
<html>
<head>
<title>Login</title>
</head>
<body>
<h1>Hello "<%= username%>"</h1>
<h3>You typed the password as "<%= password%>".</h3>
</body>
</html>
HTTP (2) : Client using POST Method
HTTP (2) : Client using POST Method
HTTP2Client
import java.io.*;
import java.net.*;
try {
Socket socket = new Socket("localhost", 8080);
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
output.println("POST /HTTP2Server/login.jsp HTTP/1.1");
output.println("Host: www.example.com");
output.println("Content-Type: application/x-www-form-urlencoded");
output.println("Content-Length: 30");
output.println("");
output.println("username=yusong&password=10000");
Host
Port
String response;
for (; (response = input.readLine()) != null;) {
System.out.println(response);
}
output.close();
Request
input.close();
socket.close();
} catch (Exception e) {
System.err.println(e);
}
Response
HTTP (2) : Client using POST Method
HTTP2Client
import java.io.*;
import java.net.*;
try {
Socket socket = new Socket("localhost", 8080);
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
output.println("POST /HTTP2Server/login.jsp HTTP/1.1");
output.println("Host: www.example.com");
output.println("Content-Type: application/x-www-form-urlencoded");
output.println("Content-Length: " + "username=yusong&password=10000".length());
output.println("");
output.println("username=yusong&password=10000");
String response;
for (; (response = input.readLine()) != null;) {
System.out.println(response);
}
output.close();
input.close();
socket.close();
} catch (Exception e) {
System.err.println(e);
}
HTTP (2) : Client using POST Method
 Request Message
POST /HTTP2Server/login.jsp HTTP/1.1
Host: www.example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 30
crlf
username=yusong&password=10000
HTTP (2) : Client using POST Method
 Response Message
HTTP/1.1 200 OK
X-Powered-By: JSP/2.1
Server: Sun GlassFish Enterprise Server v2.1
Set-Cookie: JSESSIONID=39d808dbc6c8600ab0b0786f6fbb; Path=/HTTP2Server
Content-Type: text/html;charset=UTF-8
Content-Length: 282
Date: Wed, 22 Jul 2009 17:58:44 GMT
crlf
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Login</title>
</head>
<body>
<h1>Hello "yusong"</h1>
<h3>You typed the password as "10000".</h3>
</body>
</html>
HTTP (2) : Client using POST Method
 Exercise
 Make a Java program such that:
 Send a request to the Web page at
"http://localhost:8080/HTTP2X1Server/register.jsp", with an entity
body of
"name=X&username=X&password=X&password2=X&address=X&zip
=X".
 Print out the response from the Web server.
 Project names
 HTTP2X1Server/register.html (see next slide)
 HTTP2X1Server/register.jsp (see next slide)
 HTTP2X1Client
Website (2) : Register
HTTP2X1Server/register.html
<html>
<body>
<h1>Register</h1>
<form method="post" action="login.jsp">
<table>
<tr> <td>Name:</td> <td><input type="text" name="name"></td> </tr>
<tr> <td>User name:</td> <td><input type="text" name="username"></td> </tr>
<tr> <td>Password:</td> <td><input type="password" name="password"></td>
</tr>
<tr> <td>Password (retype):</td> <td><input type="password" name="password2">
</td> </tr>
<tr> <td>Address:</td> <td><input type="text" name="address"></td> </tr>
<tr> <td>ZIP code:</td> <td><input type="text" name="zip"></td> </tr>
<tr> <td colspan="2"><input type="submit" value="Register"></td> </tr>
</table>
</form>
</body>
</html>
Website (2) : Register
HTTP2X1Server/register.jsp
<%
String name = request.getParameter("name");
String username = request.getParameter("username");
String password = request.getParameter("password");
String password2 = request.getParameter("password2");
String address = request.getParameter("address");
String zip = request.getParameter("zip");
%>
<html>
<head>
<title>Login</title>
</head>
<body>
<h1>Hello "<%= name %>"!</h1>
<h3>Your information (<%= username %>, <%= password %>, <%= password2
%>, <%= address %>, <%= zip %>) will be processed later.</h3>
</body>
</html>
HTTP (3) : Web Server
HTTP (3) : Web Server HTTP3Server
import java.io.*;
import java.net.*;
try {
ServerSocket serverSocket = new ServerSocket(8000);
Socket clientConn = serverSocket.accept();
BufferedReader input = new BufferedReader(new InputStreamReader(clientConn.getInputStream()));
PrintWriter output = new PrintWriter(clientConn.getOutputStream(), true);
String request;
for (; (request = input.readLine()).isEmpty() == false;) {
System.out.println(request);
}
output.println("HTTP/1.1 200 OK");
output.println("");
output.println("<HTML>");
output.println("<BODY><H1>This is a stupid Web server.</H1></BODY>");
output.println("</HTML>");
output.close();
input.close();
Request
clientConn.close();
serverSocket.close();
} catch (Exception e) {
System.err.println(e);
}
Response
Port
HTTP (3) : Web Server
 Response Message
HTTP/1.1 200 OK
crlf
<HTML>
<BODY><H1>This is a stupid Web server.</H1></BODY>
</HTML>
HTTP (3) : Web Server for Redirect
 Exercise
 Make a Java program such that:
 Respond to any request from any Web browser.
 Print out the request header from the Web browser.
 Send a response like below:
HTTP/1.1 302 Found
Location: http://www.miami.muohio.edu/
crlf
We moved.
 Project name
 HTTP3X1Server
HTTP (3) : Web Server for Redirect
HTTP3X1Server
import java.io.*;
import java.net.*;
try {
ServerSocket serverSocket = new ServerSocket(8000);
Socket clientConn = serverSocket.accept();
BufferedReader input = new BufferedReader(new InputStreamReader(clientConn.getInputStream()));
PrintWriter output = new PrintWriter(clientConn.getOutputStream(), true);
String request;
for (; (request = input.readLine()).isEmpty() == false;) {
System.out.println(request);
}
output.println("HTTP/1.1 302 Found");
output.println("Location: http://www.miami.muohio.edu/");
output.println("");
output.println("We moved.");
output.close();
input.close();
clientConn.close();
Request
serverSocket.close();
} catch (Exception e) {
System.err.println(e);
}
Response
Port
HTTP (3) : Web Server for Redirect
 Response Message
HTTP/1.1 302 Found
Location: http://www.miami.muohio.edu/
crlf
We moved.
Encoding
Note on Encoding
 Encoding is the process of transforming information from one format into another.
 ASCII : alphabet → integer (e.g. A → 65, …)
 Presentation-layer protocols are also encoding standards.
 Request/response message format in HTTP
 HTML
 …
 Encoding is used everywhere.
 What was the first encoding method in human history?
 ?
 Numbers
 Arabic numeral : 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
 Roman numeral : I, II, III, IV, V, VI, VII, VIII, IX, X, XI, XI, …
 Letters
 Meaning : Chinese
 Syllable : Japanese
 Phoneme : English, Korean, … (Alphabet, consonant, vowel)
Digression
HTML
HTML
 HTML, an initialism of HyperText Markup Language, is the
predominant markup language for web pages.
 A markup language is a set of annotations to text that
describe how it is to be structured, laid out, or formatted.
 Hypertext refers to text on a computer that will lead the user
to other, related information on demand.
 Rather than remaining static like traditional text, hypertext
makes possible a dynamic organization of information through
links and connections (called hyperlinks).
HTML
 HTML provides a means to describe the structure of text-
based information in a document — by denoting certain text
as links, headings, paragraphs, lists, and so on — and
to supplement that text with interactive forms, embedded
images, and other objects.
 HTML is written in the form of tags, surrounded by angle
brackets.
Simple Example
<HTML>
<HEAD>
<TITLE>Simple HTML</TITLE>
</HEAD>
<BODY>
<H1>Head</H1>
First paragraph<P>
Second paragraph<P>
</BODY>
</HTML>
Tags
Angle brackets
Heading
Paragraphs
HTML: Markup Tag
 Formats
 <tag-name>
 e.g. <br>
 <tag-name> text </tag-name>
 e.g. <b>Bold</b>, <TITLE> … </TITLE>, <BODY> … </BODY>
 <tag-name attribute=argument>
 e.g. <img src="a.jpg">
 <tag-name attribute=argument> text </tag-name>
 e.g. <a href="http://www.muohio.edu">Miami University</a>
 Note
 Tag-names are case-insensitive.
 <HTML>, <html>, <Html>, <hTmL>
 Multiple consecutive whitespaces (space, tab, EOLN) are regarded as
one space.
HTML: Text
 paragraph : <P> ... [</P>]
 line break : <BR>
 centering : <CENTER> ... </CENTER>
 bold : <B> ... </B>
 italic : <I> ... </I>
 headline : <Hn> ... </Hn>
 Many tags are deprecated now. (no more standard)
HTML: Special Characters
 Format
 &character-name;
 Special characters
<
>
"
&
→
→
→
→
&LT;
&GT;
&QUOTE;
&AMP;
HTML: Hyperlink
 Anchor Tag
<A HREF = "address"> ... </A>
e.g.
<A HREF="http://www.apple.com">Apple</A>
 Relative Path vs. Absolute Path
 HREF = "abc.html"
 HREF = "../ddd/abc.html"
 HREF = "http://www.joongang.co.kr"
HTML: List
 Unnumbered Lists
 <UL> <LI> ... <LI> ... </UL>
 Numbered Lists
 <OL> <LI> ... <LI> ... </OL>
 Definition Lists
 <DL> [<DT> ...] <DD> ... [<DT> ...] <DD> ... </DL>
HTML: Image
 <IMG SRC = "address/... .gif" ALIGN =
TOP/MIDDLE/BOTTOM/LEFT/RIGHT>
 <IMG SRC = "address/... .jpg">
e.g.
<IMG SRC = "picture.gif">
HTML: Table
<TABLE BORDER[ = n]>
<CAPTION> ...
</CAPTION>
<TR>
<TH> ... </TH>
<TD> ... </TD>
</TR>
<TR>
<TD> ... </TD>
<TD> ... </TD>
</TR>
</TABLE>
e.g.
<TABLE BORDER>
<TR>
<TD> A11 </TD>
<TD> A12 </TD>
</TR>
<TR>
<TH> B11 </TH>
<TD> B12 </TD>
</TR>
</TABLE>
HTML: Form
 Will be addressed later in JSP
Exercise 1: Text and Hyperlink
<a href="…">…</a>
Exercise 2: Numbered List
<ol><li>…<li>…</ol>
Exercise 3: Numbered List and
Hyperlink <ol><li><a href="…">…</a><li>…</ol>
Exercise 4: Table
<table><tr><td>…</td>...</tr>…</table>
Exercise 5: Table and Hyperlink
<table><tr><td><a href="…">…</a></td>...</tr>…</table>
Exercise 6: Table and Height
<td height="50">
Exercise 7: Table and Width
<td width="200">
Exercise 8: Table and Size
<td height="50" width="200">
Exercise 9: Image <img src="a.jpg">
Exercise 10: Image and Hyperlink
<a href="…"><img src="…"></a>
Exercise 11: Table, Image and
Hyperlink
<table><tr><td>…</td><td><a href="…"><img src="…"></a></td></tr></table>
HTML
 Exercise
 Make your own homepage.
 Project name
 HTML1X1
Demo:
A Web Server Supporting HTML Files
Demo (1)
 Project name
 HTTPD1Server
 Testing Scenarios
 Prepare an HTML file to browse.
 Access the Demo Web server using a Web browser.
Demo (1) : Architecture
user
input
GUI
user
output
content
File Input
request with
arguments
Web
Browser
HTTP
response
Web Server
HTML
file
CGI input
CGI
CGI output
CGI
program
Demo (1)
import java.io.*;
import java.net.*;
import java.util.*;
public class Main {
private static String getFilename(String request) {
StringTokenizer t = new StringTokenizer(request, " ");
t.nextToken();
return ("." + t.nextToken());
}
public static void main(String[] args) {
try {
ServerSocket serverSocket = new ServerSocket(8080);
Socket clientConn = serverSocket.accept();
BufferedReader ninput = new BufferedReader(new InputStreamReader(clientConn.getInputStream()));
PrintWriter noutput = new PrintWriter(clientConn.getOutputStream(), true);
String request = ninput.readLine();
String filename = getFilename(request);
System.out.println(request);
System.out.println("Filename - " + filename);
BufferedReader finput = new BufferedReader(new InputStreamReader(new FileInputStream(filename)));
noutput.println("HTTP/1.1 200 OK\n");
String line;
while ((line = finput.readLine()) != null) {
noutput.println(line);
}
finput.close();
noutput.close();
ninput.close();
clientConn.close();
serverSocket.close();
} catch (Exception e) {
System.err.println(e);
}
}
}
Java URL & HttpURLConnection Class
Java URL & HttpURLConnection Class
 As Internet sockets were provided for TCP/IP programming, Java
provides URL class and HttpURLConnection class for HTTP
programming.
 The URL class is used for the requests using GET method.
 Simple to use, but limited in functionality
 The HttpURLConnection class is used for a general HTTP
programming.
 Complicated to use, but has full functionalities
 The action of sending and receiving messages to and from Web
servers is called Web Scrapping, and URL class and
HttpURLConnection class are the basic tools for Web Scrapping.
Java URL & HttpURLConnection Class
 import java.net.*;
 import java.io.*;
 public URL(String fullURL) throws MalformedURLException
 public InputStream openStream( ) throws IOException
 public URLConnection openConnection( ) throws IOException





public abstract HttpURLConnection(URL url)
public void setDoOutput(boolean dooutput)
public OutputStream getOutputStream( ) throws IOException
public String getHeaderField(String name)
public InputStream getInputStream( ) throws IOException
HTTP (4) : URL Class HTTP4Client
import java.io.*;
import java.net.*;
try {
URL url = new URL("http://www.miami.muohio.edu/");
BufferedReader input = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
for (; (line = input.readLine()) != null;) {
System.out.println(line);
URL
}
} catch (Exception e) {
System.err.println(e);
}
Response Entity Body
HTTP (4) : URL Class
 Request Message (automatically generated)
GET http://www.miami.muohio.edu/ HTTP/1.1
Host: www.example.com
crlf
HTTP (4) : URL Class
 Response Message
input.readLine()
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2009 18:03:36 GMT
Server: Apache/2.0.46 (Red Hat)
Set-Cookie: SaneID=134.53.7.196-1248285816897469; path=/; expires=Sat, 26-Jul-14
18:03:36 GMT; domain=.muohio.edu
Content-Language: en-US
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
crlf
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="keywords" content="miami,university,ohio,college,public" />
<meta name="description" content="Located in Oxford, Ohio, Miami University is a public
university of 16,000 students offering strong academics, a vibrant residence life program, and
faculty who love to teach and mentor students." />
<title>Miami University</title>
…
</html>
HTTP (1) : Client using GET Method
import java.io.*;
import java.net.*;
try {
Socket socket = new Socket("www.miami.muohio.edu", 80);
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
output.println("GET http://www.miami.muohio.edu/ HTTP/1.1");
output.println("Host: www.example.com");
output.println("");
String response;
for (; (response = input.readLine()) != null;) {
Host
Port
System.out.println(response);
}
output.close();
input.close();
socket.close();
Request
} catch (Exception e) {
System.err.println(e);
}
Response
HTTP (4) : URL Class
 Exercise
 Make a Java program such that:
 Send a request to the Web page of "http://www.muohio.edu:8080/",
and print out the response from the Web server.
 Project name
 HTTP4X1Client
HTTP (5) : HttpURLConnection Class
HTTP5Client
import java.io.*;
import java.net.*;
try {
URL url = new URL("http://localhost:8080/HTTP2Server/login.jsp");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
// Request
urlConnection.setRequestMethod("POST");
urlConnection.setFollowRedirects(true);
URL
urlConnection.setDoOutput(true);
PrintWriter output = new PrintWriter(urlConnection.getOutputStream(), true);
output.print("username=yusong&password=10000");
output.flush();
// Response
String res = urlConnection.getResponseMessage();
System.out.println("Response message - " + res);
// OK
Request
String enc = urlConnection.getContentType();
System.out.println("Content Type - " + enc);
// text/html;charset=UTF-8
BufferedReader input = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
for (; (line = input.readLine()) != null;) {
System.out.println(line);
}
urlConnection.disconnect();
} catch (Exception e) {
Response Entity Body
System.err.println(e);
}
HTTP (5) : HttpURLConnection Class
 Request Message (automatically generated)
POST /HTTP2Server/login.jsp HTTP/1.1
Host: www.example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 30
crlf
username=yusong&password=10000
HTTP (5) : HttpURLConnection Class
 Response Message
input.readLine()
HTTP/1.1 200 OK
X-Powered-By: JSP/2.1
Server: Sun GlassFish Enterprise Server v2.1
Set-Cookie: JSESSIONID=39d808dbc6c8600ab0b0786f6fbb; Path=/HTTP2Server
Content-Type: text/html;charset=UTF-8
Content-Length: 282
Date: Wed, 22 Jul 2009 17:58:44 GMT
crlf
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Login</title>
</head>
<body>
<h1>Hello "yusong"</h1>
<h3>You typed the password as "10000".</h3>
</body>
</html>
HTTP (2) : Client using POST Method
import java.io.*;
import java.net.*;
try {
Socket socket = new Socket("localhost", 8080);
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
output.println("POST /HTTP2Server/login.jsp HTTP/1.1");
output.println("Host: www.example.com");
output.println("Content-Type: application/x-www-form-urlencoded");
output.println("Content-Length: 30");
output.println("");
output.println("username=yusong&password=10000");
Host
Port
String response;
for (; (response = input.readLine()) != null;) {
System.out.println(response);
}
output.close();
Request
input.close();
socket.close();
} catch (Exception e) {
System.err.println(e);
}
Response
HTTP (5) : HttpURLConnection Class
 Exercise
 Make a Java program such that:
 Send a request to the Web page at
"http://localhost:8080/HTTP2X1Server/register.jsp", with an entity
body of
"name=X&username=X&password=X&password2=X&address=X&zip
=X".
 Print out the response from the Web server.
 Project names
 HTTP2X1Server
 HTTP2X1Server
 HTTP5X1Client
Application 1:
Search Engine
Application 1
 Exercise
 Make a search engine such that:
 Step 1) Set a seed URL as CurrentURL.
 Step 2) Get the document at CurrentURL by an HTTP request.
 Step 3) Find all hyperlinked documents' URLs from the document, and
put them in URLPool.
 Step 4) Take out a URL from URLPool and set the URL as CurrentURL.
Go to Step 2.
 Project name
 HTTPA1Client
Application 1 : Amazon.com
http://www.amazon.com/access
http://www.amazon.com/
http://www.amazon.com/gp/yourstore?ie=UTF8&signIn=1
http://www.amazon.com/gp/flex/signout.html?ie=UTF8&path=%2Fgp%2Fyourstore&signIn=1&useRedirectOnSuccess=1&action=sign-out
http://www.amazon.com/gp/product/B00154JDAI
http://www.amazon.com/gp/site-directory
http://www.amazon.com/gp/cart/view.html
http://www.amazon.com/wishlist
http://www.amazon.com/books-used-books-textbooks/b?ie=UTF8&node=283155
http://www.amazon.com/Kindle-Books/b?ie=UTF8&node=1286228011
http://www.amazon.com/New-Used-Textbooks-Books/b?ie=UTF8&node=465600
http://www.amazon.com/magazines/b?ie=UTF8&node=599858
http://www.amazon.com/dvds-used-hd-action-comedy-oscar/b?ie=UTF8&node=130
http://www.amazon.com/Blu-ray-movies-discs-store-deals/b?ie=UTF8&node=193640011
http://www.amazon.com/Video-On-Demand/b?ie=UTF8&node=16261631
http://www.amazon.com/music-rock-classical-pop-jazz/b?ie=UTF8&node=5174
http://www.amazon.com/MP3-Music-Download/b?ie=UTF8&node=163856011
http://www.amazon.com/musical-instruments-accessories-sound-recording/b?ie=UTF8&node=11091801
http://www.amazon.com/computer-video-games-hardware-accessories/b?ie=UTF8&node=468642
http://www.amazon.com/Game-Downloads/b?ie=UTF8&node=979455011
http://www.amazon.com/kindle-store-ebooks-newspapers-blogs/b?ie=UTF8&node=133141011
http://www.amazon.com/amazon-shorts-digital-shorts/b?ie=UTF8&node=13993911
http://www.amazon.com/Kindle-Amazons-Wireless-Reading-Generation/dp/B00154JDAI
http://www.amazon.com/gp/product/B0015TCML0
http://www.amazon.com/Kindle-Accessories/b?ie=UTF8&node=1268192011
http://www.amazon.com/Kindle-Newspapers/b?ie=UTF8&node=1263068011
http://www.amazon.com/Kindle-Magazines/b?ie=UTF8&node=1263069011
…
Application 2:
Automatic Registration
Application 2
 Exercise
 Make client and server programs such that:
 The server is a JSP Web site for registration.
 The client is a registration software such that:
 It generates user names like a, b, …, z, aa, ab, …, zz.
 It sends registration requests for each of the generated user names.
 Project names
 Client - HTTPA2Client
 Server - HTTPA2Server
 Testing Scenarios
 Run both of client and server program on your own computer.
 Make a group of two students and assign a role of client and that of server to
each student respectively. After testing, switch the roles and do the same
things again.
 Something wrong?
 Try to use the debugger.
Application 2 : Web Server Setup
Application 2 : Web Server Setup
Application 2 : Web Server Setup
Application 2 : Web Server Setup
Application 2 : Web Server Setup
Application 2 : Web Server Setup
Application 2 : Web Server Setup
Application 2 : Web Server Setup
Application 2 : New Project
Application 2 : New Project
Application 2 : New Project
Application 2 : New Project
Application 2 : Source Code
Application 2 : DSN Setup
Application 2 : DSN Setup
Application 2 : DSN Setup
Application 2 : DSN Setup
Application 2 : DSN Setup
Application 2 : DSN Setup
Application 2 : DSN Setup
Application 2 : DSN Setup
Application 2 : Run HTTPA2Server
Application 2 : New Project
Application 2 : New Project
Application 2 : Source Code
Application 2 : Run HTTPA2Client
Application 2 : Run HTTPA2Client
a -> Internal Server Error
aa -> Internal Server Error
ab -> OK
ac -> OK
ad -> OK
ae -> OK
af -> OK
ag -> OK
ah -> OK
ai -> OK
aj -> OK
ak -> OK
al -> OK
am -> OK
an -> OK
ao -> OK
ap -> OK
aq -> OK
ar -> OK
as -> OK
at -> OK
au -> OK
av -> OK
aw -> OK
ax -> OK
ay -> OK
az -> OK
b -> OK
ba -> OK
bb -> OK
…
Application 2 : Simple Version
import java.io.*;
import java.net.*;
public class Main {
private static void register(String username) {
try {
URL url = new URL("http://localhost:8080/EF01/register.jsp");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
// Request
urlConnection.setRequestMethod("POST");
urlConnection.setFollowRedirects(true);
urlConnection.setDoOutput(true);
PrintWriter output = new PrintWriter(urlConnection.getOutputStream(), true);
output.print("username=" + username +
"&password=1234&f_name=Song&l_name=Song&address=Wonju&zip=12345&gender=M&agree=N");
output.flush();
// Response
String res = urlConnection.getResponseMessage();
System.out.println(username + " - " + res);
urlConnection.disconnect();
} catch (Exception e) {
System.err.println(e);
}
}
public static void main(String[] args) {
char c;
for (c = 'a'; c <= 'z'; c++) {
register("" + c);
// "" + c : to convert char to String
}
}
}
Business model:
Agents
Agents
 A software agent is a piece of software that acts for a user or other program in a
relationship of agency.
 "Action on behalf of" implies the authority to decide which (and if) action is
appropriate.
 Related and derived concepts include Intelligent agents (in particular exhibiting
some aspect of Artificial Intelligence, such as learning and reasoning),
autonomous agents (capable of modifying the way in which they achieve their
objectives), distributed agents (being executed on physically distinct
computers), multi-agent systems (distributed agents that do not have the
capabilities to achieve an objective alone and thus must communicate), and
mobile agents (agents that can relocate their execution onto different
processors).
 Agents in Electronic Commerce
 Search Engines
 Product Search Engines (Price Comparison Engines)
 …
Search Engines

Directory service


Know-how → Know-where
Yahoo directory service



Early search engines



To demonstrate its speed by DEC (Digital Corp.)
Google



To improve quality
Send requests to several search engines, and merge the responses.
Altavista


See Architecture next page
Automatic, but low speed, low quality
Meta-search engines



Good quality
Labor-intensive
Good quality, good speed
Word of mouth by counting the number of hyperlinks
Note: Search engines in Korea - www.naver.com, www.daum.net, …


Provide human-made results for the most popular 100 keywords. (labor-intensive)
Provide "Keyword Hot 100" (like Billboard Hot 100)
Search Engines: Architecture
Keyword
Web
Server
Web
Server
…
Web
Server
Search
Agent
Web
Site
Web
Scrapping
Web
Browser
DB
Keyword
Web
Browser
URL
miami
http://www.muohio.edu
miami
http://www.miami.edu/
song
http://web.yonsei.ac.kr/yusong
song
http://en.wikipedia.org/wiki/Song
…
…
…
Web
Browser
AltaVista (Late 1990s)
Yahoo (Late 1990s)
Yahoo (07/21/2009)
Google (07/21/2009)
Product Search Engines
 BargainFinder
 Andersen Consulting (now Accenture)
 Compare CD album prices among album shops like CDnow, …
 Failed due to blocking by album shops
 Jango
 To overcome the blocking by album shops
 A client software to send and receive messages to and from album shops
 Inconvenient for users
 Product Search Engines (nowadays)
 Data gathering
 Web scraping
 DB sharing
 Danawa (http://www.danawa.com)
 Web-hosting service for shops in electronics mall in Seoul
 Use the data in Web-hosting DB for price comparison
CDnow (Late 1990s)
Product Search Engines : Danawa
Building Blocks for HTTP
Programming
Building Block: URL
 Format
 resource_type://domain:port/filepathname?query_string#anc
hor
e.g.
http://dragon.yonsei.ac.kr/~yusong/index.html
http://yonsei.ac.kr:8000/~yusong/a.cgi?user=yusong&b=b
ftp://dragon.yonsei.ac.kr/~yusong/sample.file
telnet://dragon.yonsei.ac.kr
rlogin://[email protected]
HTTP (1) : Client using GET Method
 Request Message
GET http://www.miami.muohio.edu/ HTTP/1.1
Host: www.example.com
crlf
HTTP (1) : Client using GET Method
HTTP1Client
import java.io.*;
import java.net.*;
try {
Socket socket = new Socket("www.miami.muohio.edu", 80);
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
output.println("GET http://www.miami.muohio.edu/ HTTP/1.1");
output.println("Host: www.example.com");
output.println("");
String response;
for (; (response = input.readLine()) != null;) {
Host
Port
System.out.println(response);
}
output.close();
input.close();
socket.close();
Request
} catch (Exception e) {
System.err.println(e);
}
Response
HTTP (2) : Client using POST Method
 Request Message
POST /HTTP2Server/login.jsp HTTP/1.1
Host: www.example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 30
crlf
username=yusong&password=10000
HTTP (2) : Client using POST Method
HTTP2Client
import java.io.*;
import java.net.*;
try {
Socket socket = new Socket("localhost", 8080);
BufferedReader input = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
output.println("POST /HTTP2Server/login.jsp HTTP/1.1");
output.println("Host: www.example.com");
output.println("Content-Type: application/x-www-form-urlencoded");
output.println("Content-Length: 30");
output.println("");
output.println("username=yusong&password=10000");
Host
Port
String response;
for (; (response = input.readLine()) != null;) {
System.out.println(response);
}
output.close();
Request
input.close();
socket.close();
} catch (Exception e) {
System.err.println(e);
}
Response
HTTP (3) : Web Server
 Response Message
HTTP/1.1 200 OK
crlf
<HTML>
<BODY><H1>This is a stupid Web server.</H1></BODY>
</HTML>
HTTP (3) : Web Server HTTP3Server
import java.io.*;
import java.net.*;
try {
ServerSocket serverSocket = new ServerSocket(8000);
Socket clientConn = serverSocket.accept();
BufferedReader input = new BufferedReader(new InputStreamReader(clientConn.getInputStream()));
PrintWriter output = new PrintWriter(clientConn.getOutputStream(), true);
String request;
for (; (request = input.readLine()).isEmpty() == false;) {
System.out.println(request);
}
output.println("HTTP/1.1 200 OK");
output.println("");
output.println("<HTML>");
output.println("<BODY><H1>This is a stupid Web server.</H1></BODY>");
output.println("</HTML>");
output.close();
input.close();
Response
clientConn.close();
serverSocket.close();
} catch (Exception e) {
System.err.println(e);
}
Request
Port
HTTP (3) : Web Server for Redirect
 Response Message
HTTP/1.1 302 Found
Location: http://www.miami.muohio.edu/
crlf
We moved.
HTTP (3) : Web Server for Redirect
HTTP3X1Server
import java.io.*;
import java.net.*;
try {
ServerSocket serverSocket = new ServerSocket(8000);
Socket clientConn = serverSocket.accept();
BufferedReader input = new BufferedReader(new InputStreamReader(clientConn.getInputStream()));
PrintWriter output = new PrintWriter(clientConn.getOutputStream(), true);
String request;
for (; (request = input.readLine()).isEmpty() == false;) {
System.out.println(request);
}
output.println("HTTP/1.1 302 Found");
output.println("Location: http://www.miami.muohio.edu/");
output.println("");
output.println("We moved.");
output.close();
input.close();
clientConn.close();
Request
serverSocket.close();
} catch (Exception e) {
System.err.println(e);
}
Response
Port
HTTP (4) : URL Class HTTP4Client
import java.io.*;
import java.net.*;
try {
URL url = new URL("http://www.miami.muohio.edu/");
BufferedReader input = new BufferedReader(new
InputStreamReader(url.openStream()));
String line;
for (; (line = input.readLine()) != null;) {
System.out.println(line);
URL
}
} catch (Exception e) {
System.err.println(e);
}
Response
HTTP (5) : HttpURLConnection Class
HTTP5Client
import java.io.*;
import java.net.*;
try {
URL url = new URL("http://localhost:8080/HTTP2Server/login.jsp");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
// Request
urlConnection.setRequestMethod("POST");
urlConnection.setFollowRedirects(true);
URL
urlConnection.setDoOutput(true);
PrintWriter output = new PrintWriter(urlConnection.getOutputStream(), true);
output.print("username=yusong&password=10000");
output.flush();
// Response
String res = urlConnection.getResponseMessage();
System.out.println("Response message - " + res);
// OK
Request
String enc = urlConnection.getContentType();
System.out.println("Content Type - " + enc);
// text/html;charset=UTF-8
BufferedReader input = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
for (; (line = input.readLine()) != null;) {
System.out.println(line);
}
urlConnection.disconnect();
} catch (Exception e) {
Response
System.err.println(e);
}
Exercises
Exercise (1)
 Exercise
 Make a Java program such that:
 The client sends a GET request to and receives the response from the
URL "http://www.muohio.edu".
 Use the Socket-related classes only. (Do not use URL or
HttpURLConnection classes.)
 Project name
 HTTPX1Client
Exercise (2)
 Exercise
 Make a Java program such that:
 The client sends a POST request to and receives the response from the
URL "http://localhost:8080/HTTPX2Server/login.jsp".
 The "login.jsp" corresponds to the "login.html" on the next slide.
 Use "yusong" as the username and "12345" as the password.
 Use the Socket-related classes only. (Do not use URL or
HttpURLConnection classes.)
 Project name
 HTTPX2Client
Exercise (2) HTTPX2Server/login.html
<html>
<body>
<h1>Login</h1>
<form method="post" action="login.jsp">
<table>
<tr>
<td>User name:</td>
<td><input type="text" name="UN"></td>
</tr>
<tr>
<td>Password:</td>
<td><input type="password" name="PW"></td>
</tr>
<tr>
<td colspan="2"><input type="submit" value="Login"></td>
</tr>
</table>
</form>
</body>
</html>
Exercise (3)
 Exercise
 Make a Java program such that:
 The client sends a POST request to and receives the response from the
URL "http://localhost:8080/HTTPX3Server/login.jsp".
 The "login.jsp" corresponds to the "login.html" on the next slide.
 Use "Yong Uk" as the username and "1 2$3" as the password.
 Note that you have to encode the above values by URL encoding rule.
 ASCII code for '$' is 24 in hexadecimal.
 Use the Socket-related classes only. (Do not use URL or
HttpURLConnection classes.)
 Project name
 HTTPX3Client
Exercise (3) HTTPX3Server/login.html
<html>
<body>
<h1>Login</h1>
<form method="post" action="login.jsp">
<table>
<tr>
<td>User name:</td>
<td><input type="text" name="UN"></td>
</tr>
<tr>
<td>Password:</td>
<td><input type="password" name="PW"></td>
</tr>
<tr>
<td colspan="2"><input type="submit" value="Login"></td>
</tr>
</table>
</form>
</body>
</html>
Exercise (4)
 Exercise
 Make a Web server which always responds with the HTML
document on the next slide.
 Use a "for" statement for the Web server to run permanently.
 Project name
 HTTPX4Server
Exercise (4)
<HTML>
<HEAD>
<TITLE> Exercise</TITLE>
</HEAD>
<BODY>
Welcome to the world of HTTP.
</BODY>
</HTML>
Exercise (5)
 Exercise
 Make a Web server which always responds with a redirect
message of which HTML document appears on the next slide.
 The new address is "http://www.muohio.edu".
 Use a "for" statement for the Web server to run permanently.
 Project name
 HTTPX5Server
Exercise (5)
<HTML>
<HEAD>
<TITLE> Exercise</TITLE>
</HEAD>
<BODY>
We moved to <A HREF="http://www.muohio.edu"> a new
address</A>
</BODY>
</HTML>
Exercise (6)
 Exercise
 Make a Java program such that:
 The client sends a GET request to and receives the response from the
URL "http://www.muohio.edu".
 Use the URL classes.
 Project name
 HTTPX6Client
Exercise (7)
 Exercise
 Make a Java program such that:
 The client sends a POST request to and receives the response from the
URL "http://localhost:8080/HTTPX7Server/login.jsp".
 The "login.jsp" corresponds to the "login.html" on the next slide.
 Use "yusong" as the username and "12345" as the password.
 Use the HttpURLConnection class.
 Project name
 HTTPX7Client
Exercise (7) HTTPX7Server/login.html
<html>
<body>
<h1>Login</h1>
<form method="post" action="login.jsp">
<table>
<tr>
<td>User name:</td>
<td><input type="text" name="UN"></td>
</tr>
<tr>
<td>Password:</td>
<td><input type="password" name="PW"></td>
</tr>
<tr>
<td colspan="2"><input type="submit" value="Login"></td>
</tr>
</table>
</form>
</body>
</html>
Exercise (8)
 Exercise
 Make a Java program such that:
 The client sends a POST request to and receives the response from the
URL "http://localhost:8080/HTTPX3Server/login.jsp".
 The "login.jsp" corresponds to the "login.html" on the next slide.
 Use "Yong Uk" as the username and "1 2$3" as the password.
 Note that you have to encode the above values by URL encoding rule.
 ASCII code for $ - 24 (in hexadecimal)
 Use the HttpURLConnection class.
 Project name
 HTTPX8Client
Exercise (8) HTTPX8Server/login.html
<html>
<body>
<h1>Login</h1>
<form method="post" action="login.jsp">
<table>
<tr>
<td>User name:</td>
<td><input type="text" name="UN"></td>
</tr>
<tr>
<td>Password:</td>
<td><input type="password" name="PW"></td>
</tr>
<tr>
<td colspan="2"><input type="submit" value="Login"></td>
</tr>
</table>
</form>
</body>
</html>
Examinations
Examination (1)
 Examination
 Make a Java program such that:
 The client sends a GET request to and receives the response from the
URL "http://web.yonsei.ac.kr/yusong/index.htm".
 Use the Socket-related classes only. (Do not use URL or
HttpURLConnection classes.)
 Project name
 EM05
Examination (1) EM05.Main
import java.io.*;
import java.net.*;
try {
Socket socket = new Socket("web.yonsei.ac.kr", 80);
BufferedReader input = new BufferedReader(new
InputStreamReader(socket.getInputStream()));
PrintWriter output = new PrintWriter(socket.getOutputStream(), true);
output.println("GET /yusong/index.htm HTTP/1.1");
output.println("Host: www.example.com");
output.println("");
String response;
for (; (response = input.readLine()) != null;) {
System.out.println(response);
}
output.close();
input.close();
socket.close();
} catch (Exception e) {
System.err.println(e);
}