---

Some technical information about the Internet

Writing programs that can communicate via the Internet requires some knowledge of how the Internet works. The Internet is a complex, many-layered combination of hardware and software, but you actually need to know surprisingly little about the underlying technology in order to write programs that use it. Still, there are issues that you will need to be aware of, especially if you want to do some or all of your work on your own computer.

Loads of useful information is available online about all of the topics summarized below; few of the problems I discuss are insurmountable. (Of course, the challenge, as always, is to separate the useful information from the noise.) But all of these issues will have at least some effect on whether you can get your programs to connect to programs running on other machines, even if your programs are completely correct, so it's best to be aware of these issues before you get started.

IP addresses

In general, every machine connected to the Internet has an IP address. An IP address is akin to a telephone number; by specifying that a message is to be sent to a particular IP address, the network will be able to determine who should receive the message and how the message should get there, hiding these details from the machines on either end.

An IP address is generally displayed as a sequence of four numbers separated by dots; each of the numbers has a value in the range 0-255 (a range chosen because values in this range can be stored in eight bits, or one byte). For example, as of this writing, the IP address of one of the machines that act as a "server" for the ICS web site has the address 128.195.1.76.

If you want a program of yours to connect to another program running on another machine, you'll have to know the IP address where that other program is running. Note, too, that many machines have a different IP address as often as every time they connect to the Internet, so this isn't the kind of information you can necessarily bookmark and reuse forever.

(For completeness, I should point out that there is more than one addressing scheme supported on the Internet, and the one I've described above is just one of them.)

The "loopback" address

There is a special range of IP addresses that can always be used to connect a computer to itself, regardless of what its IP address is. These are called "loopback" addresses, the most common of which is 127.0.0.1. So if you want to test connecting two programs on your own machine, you can use 127.0.0.1 to do that. (This also explains T-shirts and bumper stickers that you may have seen that say "There's no place like 127.0.0.1.")

Ports

When you want to connect a program to another program running on another machine, it's not enough to know the IP address of the other machine. Multiple programs on the same machine are likely to be connected to the Internet at any given time. So there needs to be a way to identify not only what machine you'd like to connect to, but also which program on that machine you'd like to connect to.

The mechanism used for this on the Internet is called a port. A program acting as a server will register its interest in a particular port by binding to it; the operating system will generally only allow one program to be bound to a particular port at a time. Once a program binds to a port on a machine, when a connection is made to that port on that machine, the connection is routed to the program that bound to the port.

Ports have numbers that range from 0-65535 (a range chosen because values in this range can be stored in sixteen bits, or two bytes). It's generally a good idea not to use ports with numbers below 1024, becuase these tend to be reserved for common uses (e.g., web traffic, FTP traffic, email traffic). Beyond that, you may discover some ports at or above 1024 in use — depending on what programs are running on your machine — but most should be available.

The important thing to realize here is this: In order for a client to initiate a connection to a server, the client will need to use not only the IP address of the machine where the server is running, but also the port that the server is listening on.

The Domain Name System and DNS lookups

Though every machine connected to the Internet has an IP address, users don't typically use IP addresses on an everyday basis. Just as IP addresses are akin to telephone numbers, there is an Internet service called the Domain Name System (DNS) that acts as a kind of phone book; given the name of a computer, DNS can tell you its IP address, so long as that name has been registered with DNS previously.

So, for example, when you brought up this web page in your browser, your browser first had to know the address of www.ics.uci.edu; it found this out by doing a DNS lookup, by sending a message to a Domain Name Server and asking "What is the IP address of www.ics.uci.edu?" In return, the browser received a message that said "It's 128.195.1.76," at which time your browser could connect to that address (on port 80, since that's the port used for web traffic when not otherwise specified) and download this web page.

DNS is unlikely to affect your connections to programs on other machines significantly in this course, other than the fact that you may want to use a hostname instead of an IP address when you know one. But we'll certainly use it later this quarter.

Note that the "loopback" address has its own name: localhost. The name localhost always resolves to a "loopback" address.

Firewalls

The Internet offers a certain amount of anonymity — it's hard to know who's contacting you or what their motives are if all you can see is their IP address and what port they're connecting to. This kind of anonymity has its benefits, though it also has its serious downsides; when you can't know who's contacting you and can't know what they're trying to accomplish, and when you can't always trust your operating system and other software not to provide outsiders access to information they shouldn't have, the wise solution is to restrict incoming traffic. The theory is that if no one can connect to you, no one can take advantage of you (without you having "asked for it," in some sense, by connecting to them). This is the theory behind firewalls, which are software or hardware that restrict other computers' access to computers behind them.

It was once the case that firewalls were mostly used in businesses, as they were the primary targets of online crime and mischief. Nowadays, though, most computers come with some kind of firewall software built into them. This may make it more difficult for programs on other machines to connect to yours when you want them to, because your computer may be configured to disallow incoming connections. Some firewall software also allows you to disallow certain kinds of outgoing connections, which might also affect your ability to connect to programs running on other machines. There are usually ways to "open a port," which means that you've told our firewall to allow traffic bound for a certain port to move into or out of your machine, while traffic on other ports will still be forbidden. Details of how to do this vary from one context to another, but there is a fair amount of documentation online if you want to learn how to open ports using your particular combination of hardware and/or software firewalls.

I should point out, also, that some Internet service provides have their own firewalls and traffic limitations in place, so if you're working from home, your experience — especially in terms of being able to have others connect to you — may vary considerably depending on your provider. This will be of little consequence to your work in this course, as this problem affects servers much moreso than clients, but it's something to be aware of if you want to take your work further than what is assigned.

Routers and network address translation (NAT)

In general, every machine connected to the Internet has an IP address. However, many of us are not connected directly to the Internet at all. For example, I have several computers in my home, but I have only one Internet connection: a cable modem. In order to use more than one of my computers at a time online, it's necessary for me to have some way of sharing that connection.

In order to do that, I do what most people do in this situation: I use a device called a router. The router is connected to my one Internet connection. Whenever it's connected, it has an IP address. (Most home Internet users, me included, don't get the same IP address every time they connect, though, which is one reason why it's often hard to run a server from your home.) My computers don't connect directly to the Internet; instead, they connect to the router. The router's job is to forward outgoing traffic from each computer to the single Internet connection, and to take the incoming traffic and route it to the appropriate computer.

The router and my computers form their own local-area network, or LAN. The router assigns a "fake" IP address to each of my computers, using a range of addresses that is never assigned to computers on the Internet. As traffic flows into and out of the router, it performs a task called network address translation, or NAT, which means that it converts the internal, "fake" IP addresses used by my computers to its own IP address for traffic going out, and converts its own IP address back to the "fake" IP addresses on the way back in. As far as the outside world is concerned, I don't have many computers; I just have one: the router.

Many routers also act as firewalls, disallowing incoming traffic in most cases unless you specifically configure them to allow it. Most home computer users only ever initiate connections, so this is a safe and relatively painless restriction for most people.

Why this will affect you when you work from home is that your router will make it much more difficult for programs on other machines to connect to yours. They'll need to have your router's IP address, not your computer's IP address. (That's not hard to give them, since Googling What is my IP address? will show you the router's IP address, since that's the only address the outside world ever sees.) You'll also need to configure your router to allow incoming traffic on at least one port, and to send incoming traffic on those ports to a particular one of your computers. Details of how you set this up vary considerably from one router to another, but are generally available online if you know what model of router you have.

Wait... I'm getting overwhelmed!

If most of these details are new to you, you might be feeling overwhelmed. I present these details as useful background information, though they're not the focus of our work here. (They are details that are worth learning more about if you want to be in the technology field, but it's fine to acquire them gradually.) Other than the use of IP addresses and ports, none of these details is likely to affect your work all that much when you're working on the machines in the ICS labs, but you'll need to be aware of some of these issues if you want to use your own machine.

---

Connecting to a server without using Python

If you want to write a client program in Python that will talk to a server, it can often be useful to first experiment with the server a bit manually, so you completely understand the behavior of the program you'll be talking to. To do that, what you'll need is a program that can let you type text and send it to the server, while displaying whatever the server sends back. Depending on what operating system you prefer, you'll either have a program like this already, or you'll need to get one.

Using PuTTY on Windows

If you run Windows, a good tool to use for this purpose is PuTTY, which you can download here. What you want to download from that page is putty.exe. It's not an installer; it's a program you can drop pretty much anywhere, including a USB stick, if you prefer.

To connect to a server using PuTTY, you would launch it and ask it to connect. Suppose that you wanted to connect to a program running on www.ics.uci.edu that was listening on port 1234. (Note: There isn't likely to be a program on that machine listening on that port; this is just an example.) Launch PuTTY and do the following:

• Under Host Name (or IP address), type www.ics.uci.edu
• Under Connection type, select Raw (which means that we don't want to use encryption or anything else fancy; we just want to send text back and forth)
• Under Port, type 1234
• Under Close window on exit, select Never (which will allow you to see what happened, even after the connection has been closed)

A number of other settings — such as the window size and whether the window closes automatically when you log out — are available. To avoid setting these things up repeatedly, when you have them the way you want them, type a name (e.g., MyServer) under Saved Sessions and click the Save button; this will allow you to load that session and reuse it later, so you don't have to change these settings every time you connect, which is especially useful if you'll be frequently connecting to the same server. (On Windows 7 and above, you'll find that frequently-used sessions are easily accessible from the taskbar if you pin PuTTY to it and then right-click the icon on the taskbar. Very handy!)

Using "telnet" or "nc" on Mac OS X, Linux, or other Unix-based operating systems

If you're using Mac OS X, Linux, or other Unix-based operating systems, you should already have one of two programs able to play this same role: one called telnet or the other called nc. Mac OS X, in recent years, has stopped including telnet in its operating system, though it does have nc; most flavors of Linux and Unix have both.

To use one of these programs, you'll need to open up a Terminal window — that's what Mac OS X calls it; if you're using something else that's Unix-based, you'll need whatever the equivalent is on your operating system. A Terminal window displays a shell prompt and lets you type commands, then executes those commands and displays the results.

Determining which of these programs you have is a simple matter of typing these two commands at a Terminal prompt.

which telnet
which nc

In each case, you'll either see something that looks like a filesystem path, showing where the program is installed, or some kind of error message indicating that the program isn't there. Make a note of which one is there and use that; if you have both, it doesn't make much difference which one you use.

Suppose that you wanted to connect to a program running on www.ics.uci.edu that was listening on port 1234. (Note: There isn't likely to be a program on that machine listening on that port; this is just an example.)

To use telnet for this, in a Terminal window, you would type the command telnet www.ics.uci.edu 1234 and press the Return (Enter) key. The telnet program will then try to connect to port 1234 on www.ics.uci.edu. If that fails, you'll see some kind of error message; if it succeeds, though, you'll see a prompt that looks something like this (it varies from one operating system to another):

Trying 128.195.1.94...
Connected to www.ics.uci.edu.
Escape character is '^]'.

and, from then on, telnet is connected to another program on another machine via a socket. Anything you type will be sent to the other program; anything the other program sends back will be displayed to you. This will continue until the connection is closed, either because the other program "hangs up" on you, or because you hang up on it. If you want to be the one to close the connection, hold down the Ctrl key and press ] (right bracket). This will display a prompt that looks like this:

telnet>

At that prompt, type close and press the Return (Enter) key. Your connection will now be closed.

Alternatively, to use nc to connect to the same machine, you'd type the command nc -c www.ics.uci.edu 1234. (The -c part turns out to be important; that's how you tell it to end each line you type with the same '\r\n' characters that all of our protocols will require.) At that point, anything you type will be sent to the other program on the other machine; anything it sends back will be displayed to you. This will continue until you hold down the Ctrl key and press C, which is a common way to terminate programs running in a Terminal window.

---

Writing a simple Python client program

Now that we know some of the basic terminology, it's time to write a Python program that can act as a client, connecting to a server program running somewhere else and conversing with it. First, we need the most basic tool that we use for this task in Python: a socket.

The socket module

The Python Standard Library includes a module called socket, which is a very good entry point into the world of Internet-connected Python programs. We'll explore that module some here; while there is lots more to be said about it, our exploration here will demonstrate everything we'll need to write Python client programs that communicate with a server by exchanging text, which is our present (and only) goal.

The first thing to know about the socket module is that it includes a type called socket, which is the kind of object we'll need to interact with. Recall, from the Background section above, that a socket object represents the endpoint of a connection between a Python program and another program; the other program need not be running on the same computer (though it can be) and need not be written in Python (though, again, it can be).

Before we can proceed, we need to know how to get one of these socket objects. This is simple: we call its constructor, a special function that constructs a socket object for us. To do that, we'll first need to import the socket module.

>>> import socket

Then we can say this to create our socket object and assign it into a variable example_socket:

>>> example_socket = socket.socket()

Sockets can be used, initially, for two things: listening and connecting. Listening means that we want to wait for another program to contact us; connecting means that we want to connect to another program. In this example, since we're writing a client, we'll connect. Once our socket has established a connection — either by receiving a connection when listening, or successfully initiating one when connecting — then we can use it to transfer information between programs, by writing to its output stream and reading from its input stream.

Connecting our socket to a server

Connecting requires that we ask the socket to connect to a particular port on a particular host — we identify the host either by an IP address, such as 128.195.1.83, or by a name, such as www.ics.uci.edu. The socket module represents host/port commbinations as two-element Python tuples, with the first element of the tuple being a string identifying the host and the second element of the tuple being an integer specifying the port number.

Once we have our address tuple, we can connect the socket to it:

>>> connect_address = ('128.195.1.83', 5151)
>>> example_socket.connect(connect_address)

If the call to the connect method succeeds, our socket is connected successfully; if the attempt to connect fails, connect will raise an exception instead. The keys to success are that our Internet connection is up and running, and also that there really is a server program running on the host and listening on the port we specified.

Once connected successfully, we can use our socket to send and receive data between our client program and the server that we're connected to. The one tricky part is that the data that is sent and received is made up of what are called bytes, which are most directly represented by a Python object with the type bytes. If what we intend to send back and forth is actually text, we would need to perform conversion between the bytes and str types.

However, managing these kinds of details is often more trouble than it's worth. If we know that we're exchanging text with the server, then we can use a better tool for the job.

Pseudo-file objects

It seems a shame that we can't just treat a socket the way we do a text file. Files contain bytes, too, but when we read from text files in Python, the file object converts between bytes and strings and finds where one line ends and another begins automatically; we just have to call readline() and the right things happen.

As is often the case in a programming language library, when you dig a little bit further, you find the tool you were looking for. If you want to treat a socket similarly to how you treat a text file — reading lines and getting back strings, writing strings — you can do it. You just have to ask for a "middleman" of sorts, an object that behaves outwardly the way a file object does, but that reads and writes via a socket rather than a file. (Whenever you read from one, it reads from the underlying socket; whenever you write to one, it writes to the underlying socket.)

Sockets in Python provide a makefile() method that can create you such a "middleman." The argument you pass to this method is the same as the "mode" argument you pass to the built-in open() function that opens files, and the object returned to you appears a lot like a file object — it supports methods like readline() (if it was created for reading) or write() (if it was created for writing). We'll call these objects pseudo-file objects, because they are, in essence, "fake" files; they'll feel like the file objects we've seen before when we use them, but the reality is that they are connected to the socket instead of a file on our hard drive.

Most often, we'll actually want to do both reading and writing, so we'll need two separate pseudo-file objects: one that reads and another that writes.

>>> input_stream = example_socket.makefile('r')
>>> output_stream = example_socket.makefile('w')

We can write a line of text to our socket by calling write() on output_stream, just like when we write to a text file. Also, as when we write to a text file, we'll need to add an end-of-line sequence — a special sequence of characters indicating the end of a line — wherever we want one; this will not be automatic. All of the servers we'll use in ICS 32 will use the same end-of-line sequence, which is made up of two characters called a carriage return and a linefeed; they are denoted in Python string literals as '\r' and '\n'. (This is a fairly common end-of-line sequence in text-based Internet protocols.)

>>> output_stream.write('Hello there\r\n')
>>> output_stream.flush()

Notice that we also called the flush() method after writing. The reason for that is a small, but totally vital, detail. File objects — including the ones we get when we call makefile() on a socket — do something called buffering. When you write to them, they store data in memory temporarily and then write it once in a while when the buffer runs out of space. This is mainly done because the act of actually writing the data — especially when you're talking about files on a hard disk, but even when you're using a socket — is much slower than storing it in memory, so only writing data once in a while can dramatically reduce the total cost of a long sequence of small writes. But when you're talking to another program via a socket, it's usually important that a message is sent immediately, because the other program won't know what to do until it is received. For this reason, it becomes important to tell the file object "Take whatever is in your buffer and send it now! Don't wait until an opportune time!" That's what flush() does.

When we want to read a line of text from our socket, we can call readline(), just like when we want to read a line of text from a text file instead. Note that readline() will wait until a complete line of text has arrived in the input stream, so we have to be sure that we're calling it at a point in time when we expect the server will send one. (More on that later.) What we get back from readline() is just like what we get back when we read a line of text from a file: We get the entire line of text, along with a single newline character '\n' on the end of it.

>>> input_stream.readline()
'Hi!\n'

The two programs can exchange lines of text in this manner, which makes for a good starting point for our exploration into networks and network protocols.

Closing the socket

Finally, when we're done with the socket, we'll want to close both the pseudo-file objects and the socket itself, to let the other program know that we've finished our conversation, and also to release any operating system resources associated with the open connection. The right order is to first close the two pseudo-file objects, then to close the socket.

>>> input_stream.close()
>>> output_stream.close()
>>> s.close()

---

The code

The final code example from lecture is below.

• The echo_client module

---

Trying out the example client

An echo server like the one we connected to during the lecture is now running, on the same machine where all of the ICS 32 servers are running.

Information about where the server is running will be distributed via email; I'll also keep you posted about planned downtime (e.g., if I need to fix a problem, if I know that the machine where it's running will be down, etc.). It may be necessary to move the server from time to time; when I do that, I will let everyone know via email.

The echo server is listening on port 5151. There are two other servers running on the same machine — on ports 5150 and 5152 instead — that work slightly differently. Try each of those and see if you can figure out why they respond the way they do to each line of text sent to them.

Please note that the echo server — like all of the ICS 32 servers — is running on a machine on the ICS network that is not exposed to the open Internet. In order to connect to it, you will need to be connected to the campus network, which means you'll either need to be on campus or you'll need to be connected to something called the campus VPN, which allows you to access the campus' network from off-campus. Note, also, that certain residential areas are not connected to a part of the campus network that will allow you direct access to the ICS 32 servers, so you'll need to use the campus VPN in those cases, too. In general, if you're not able to connect to the ICS 32 servers, the first thing you should try is using the campus VPN.

Connecting to the campus VPN requires that you install some software, which you can obtain from UCI's Office of Information Technology at the following link:

• Campus VPN software

Open the link above, find the section titled VPN Software (as opposed to WebVPN, which won't work for this purpose), and click the link titled Download the VPN Software. This will require you to log in with your UCInetID and password, at which point you'll be offered links to Windows, Mac OS X, or Linux versions of the VPN software. Download the one that's appropriate for your operating system and install it; instructions for setting it up are available from that page, as well.