ICS 32 Winter 2022
Notes and Examples: Third-Party Libraries
When Python's standard library falls short
We've already seen that Python's standard library is an extensive, varied set of tools that are provided to us automatically (and free!) when we install Python. If you peruse the table of contents of the Python standard library documentation — particularly starting with the links to libraries like string, re, difflib, and so on — you'll see a listing of modules for everything from string handling and searching, dates and calendars, complex data structures, numeric and mathematical tools, operating system and file handling, compression, networking, common Internet protocols, images and sounds, and on and on. It's an astounding body of work, and it's all available right out of the box; if you have Python, you have all of these tools.
The goal of a standard library in a programming language is to provide the kinds of functionality that are not inherent in the language, but that nonetheless address common needs that a broad group of users of that language will have. Some standard libraries are slimmer than others, but even the large ones like Python's or Java's tend to be focused on the things that are common and broadly-applicable. The narrower the applicability, the less likely you'll find it in a standard library.
But what you can do with a programming language without writing it yourself isn't limited only to what's in its standard library. Increasingly, third-party libraries have emerged to fill the gaps, where a library is written to serve a narrower purpose, but is then shared with the community under various terms. Most of these libraries are open source, which simply means that the libraries' source code (i.e., the code that implements the library) is made available, in addition to "pre-packaged" versions of it, which means that you can not only use them, but you can also learn about how they work internally, or even suggest or make changes and fixes that might benefit the whole community of users.
It should be noted that not all third-party libraries are free, and not even all open-source libraries come without strings attached. If you're going to use a third-party library in a project of yours, you're going to want to be aware of the terms of its license. It isn't necessarily the case that the agreement is "Do what you want with this!", even when you have access to the source code. Particularly when you're interested in using it as part of a business, but even when you're not, you'll be well-advised to take a look at what the license terms of your chosen libraries are, before you commit to spending time learning and using them. For example, some licenses require that any changes that you make to the library must be given back to the community; but if those changes are the "secret sauce" for a company you're building, you'll be loathe to give them up, so best not to use the library. Other licenses don't permit commercial use without payment, which can lead to a thorny situation later if you begin making profitable use of a library in a business.
For our work in this course, we'll respect license terms, but that will be easier for us, as we're not intending to use our work to create a business; we're mainly interested in experimentation and learning. One way that open source software has dramatically changed the landscape in the last few decades is by providing a depth and breadth of work that is available for experimentation, free of charge; that's a wonderful environment to learn in.
Installing a third-party library in Python
When we want to use a third-party library in our work, we'll first need to install it; since it doesn't come with Python, we'll need to find a way to make it available in our Python installation. Of course, the simplest way to do this is just to obtain a .py file and include it in the same directory as our own code, but this approach has its limitations:
So we should tend to want our third-party libraries to be more like the ones that are built into Python. When we want to use the socket library, we don't have to go find a file called socket.py somewhere and copy it into the same directory as our code; instead, we just say import socket and Python is able to find the library for us.
Third-party libraries can be set up in the same way, so that Python can find them for us in a more automated way. If we want to use the third-party library pygame, for example, we'd like to just be able to say import pygame and be done with it. Fortunately, the tools for setting that kind of thing up have improved a lot in recent years, with everything we need built into our Python installation. The only trick is that we'll need to become familiar with using the command line (such as a Command Prompt on Windows or a Terminal window on a Mac) to make it work, but this is valuable knowledge going forward, anyway, so well worth our time.
Virtual environments
The first problem we face is that installing third-party libraries entails a certain amount of risk. I don't mean that libraries are "out to get you," necessarily, or that something bad will necessarily happen because you installed one. However, as you install more of them, you may discover that one library is incompatible with another, or that you want to use more than one version of the same library — an older version in an older project of yours that you don't want to update, but a newer version in a newer one. For this reason, installing third-party libraries directly into our Python installation is a generally bad idea; it works fine initially, but can spiral out of control in a hurry.
As an alternative, Python supports the ability to set up virtual environments, which are completely separate, self-contained Python installations that you place into a chosen directory. Once you activate one of them, you'll be working within that environment instead of your "global" Python installation, which means you can work with relative impunity; the worst that can happen is you have to create a new one and start over if something goes wrong.
Setting up a virtual environment requires a module called venv, which is part of Python's standard library. It's best done using command-line tools, and how you do it is slightly different on one operating system as opposed to another. Below are instructions on setting up a virtual environment, along with a third-party library called pygame, for both Windows and macOS.
Setting up a virtual environment on Windows
Your first order of business is to launch a Command Prompt window. The easiest way to do this is to press Win+R (where Win is the "Windows" key on your keyboard), which is how you ask Windows to run a program for you. In the ensuing dialog, type cmd and click OK. This should pop up a window titled Command Prompt, though this detail can vary a bit from one version of Windows to another.
What you should see in that window is literally a command prompt, which is to say a prompt at which you can enter commands. A command prompt is somewhat like the Python shell, except that instead of interacting with Python one expression or statement at a time, you're instead interacting with your operating system one command at a time. On Windows 10, what I see when I launch a Command Prompt window looks like this; yours may be slightly different.
Microsoft Windows [Version 10.0.19042.1348] (c) Microsoft Corporation. All rights reserved. C:\Users\thornton>
A cursor will be blinking just after the > character on the last line. The C:\Users\thornton> part is the prompt, the way that >>> is a prompt within the Python shell. One thing to notice about a command prompt, however, is that it contains a path to a directory — in this case, I've landed in my "user" directory, C:\Users\thornton, though this can be configured in other ways if you prefer to begin somewhere else. That a command prompt includes a directory tells you something important: The commands you execute are executed relative to a directory, which means that what happens when you do certain things depends very much on where you are.
You can move from one directory to another by using the command cd. You give cd either an absolute path (which is the complete path to a directory on the same drive) or a relative path (which is the name of a directory you expect to find within the directory you're already in) as a parameter. Note that parameters are separated by spaces. (Note, too, that spaces within directory and file names can be problematic, so we're going to be fastidious about avoiding them from here on out.)
C:\Users\thornton> cd Desktop C:\Users\thornton\Desktop> cd .. C:\Users\thornton>
The directory .. is special; it is the parent directory of wherever you are (i.e., the one that contains the directory you're currently in).
To create our virtual environment, we're going to need a directory to put it in. Decide where you want that directory to be, then change yourself into the directory that will contain it. For example, if I wanted it on my desktop, I might next do this.
C:\Users\thornton> cd Desktop C:\Users\thornton\Desktop>
Now, we can create our new directory using the command mkdir, giving it the name of the directory we want to create as a parameter; then, we can change into it.
C:\Users\thornton\Desktop> mkdir 32Example C:\Users\thornton\Desktop> cd 32Example C:\Users\thornton\Desktop\32Example>
Next, we're ready to create our virtual environment, which we can do from within the directory where we want it stored by executing the following command. Note that the space and the dot at the end are both important. The special directory . generally always means "The directory I'm in now," so what I'm really saying here is "Create me a new virtual environment in the directory where I am now."
C:\Users\thornton\Desktop\32Example> python -m venv .
It may take a little while to set this up; don't worry if it takes a minute or two. (If you see an error message saying something like 'python' is not recognized as an internal or external command, then this means you don't have Python set up to run from the command line properly. See the Python installation instructions for Windows from Project #0, and pay particular attention to Step 3, which explains how to set that up.)
Once it's done, let's take a look at what we ended up with. We can do that using the command dir, which means to show us what's in the current directory.
C:\Users\thornton\Desktop\32Example> dir
Volume in drive C is Win10
Volume Serial Number is ABCD-1234
Directory of C:\Users\thornton\Desktop\32Example
01/01/2022 11:07 PM <DIR> .
01/01/2022 11:07 PM <DIR> ..
01/01/2022 11:07 PM <DIR> Include
01/01/2022 11:07 PM <DIR> Lib
01/01/2022 11:07 PM 75 pyvenv.cfg
01/01/2022 11:07 PM <DIR> Scripts
1 File(s) 75 bytes
5 Dir(s) 318,214,103,040 bytes free
So what do we see here?
Let's make an additional directory for our own source code, which we'll call src. We won't use it right off, but we'll want it available to us a little later.
C:\Users\thornton\Desktop\32Example> mkdir src C:\Users\thornton\Desktop\32Example>
The Scripts directory turns out to be an important one, because it contains a script called activate that lets us "turn on" our virtual environment. Until we've done that, the things we do will have an effect on our "global" Python installation, rather than the virtual environment that we've just created.
C:\Users\thornton\Desktop\32Example> cd Scripts C:\Users\thornton\Desktop\32Example\Scripts> activate (32Example) C:\Users\thornton\Desktop\32Example\Scripts>
Notice that our prompt changed after running the activate script. In addition to showing us the path to a directory, it's also showing us the name of our virtual environment; this is a handy way of helping us to remember that we've activated our virtual environment.
Next, let's install a third-party library called pygame, which we can do using a tool built into Python called pip. (Once we've activated our virtual environment, anything we install using pip will be installed into our virtual environment, rather than our global Python installation.) What pip does, in short, is download a library from a community-run server on the Internet and then installs it for us. There are all kinds of details that aren't especially important here, but when what you want is simple, it's easy to get.
(32Example) C:\Users\thornton\Desktop\32Example\Scripts> pip install pygame Collecting pygame Using cached pygame-2.1.0-cp310-cp310-win_amd64.whl (5.1MB) Installing collected packages: pygame Successfully installed pygame-2.1.0
Specifically what you'll see may vary a bit, but the general idea is that you'll be apprised of the installation progress. (In my case, installation was a snap because I had downloaded this once already; you may see it being downloaded, in addition to the installation.) The 2.1.0 that you see in a couple of places is a version number, which is the latest version, as of this writing; the win_amd64 says that I've been given the 64-bit version of Python on Windows, which is the right one for my situation. (Note, too, that all of this is automatic; I'll get the latest version that's appropriate for my installation of Python. You'll get whatever's right for your situation automatically.)
Finally, we can launch IDLE and start interacting with pygame. However, we have to launch IDLE from within our virtual environment. We'll also want all of our own modules to be loaded from our src directory (and to be saved there, too), so we'll change there and then launch IDLE from the command prompt manually.
(32Example) C:\Users\thornton\Desktop\32Example\Scripts> cd .. (32Example) C:\Users\thornton\Desktop\32Example> cd src (32Example) C:\Users\thornton\Desktop\32Example\src> python -m idlelib
The module idlelib in Python's standard library is what it sounds like; it's IDLE. What we're doing is saying "run IDLE"; it's really that simple. An IDLE window should pop up, in which case we can make sure our installation is complete by attempting to import pygame.
>>> import pygame
Assuming that this succeeded, you're in business. You've got a virtual environment with pygame installed. Next time you want to launch IDLE within your virtual environment from scratch, you can follow these steps:
Setting up a virtual environment on macOS
Your first order of business is to launch a Terminal window. Precisely how you do this is a little different from one version of macOS to another, though the simplest way is to use the operating system's search feature and look for the program called Terminal. (There are a variety of other techniques and shortcuts you can use; feel free to explore what else is available.)
What you should see in that window is a command prompt, which is a line of text usually ending in $, but preceded by some other information. For example, you might see something like this:
AlexThornton-Mac:~$
In this case, AlexThornton-Mac might be the name you've given to your computer previously. After the : character might be a ~ character, which is actually a path to a directory; on macOS, the path ~ is a shorthand for your home directory. The $ is the end of the prompt. In general, a command prompt is somewhat like the Python shell, except that instead of interacting with Python one expression or statement at a time, you're instead interacting with your operating system one command at a time.
A cursor will be blinking just after the $ character. One thing to notice right away is that your command prompt contains the path to a directory. That's telling you something important: The commands you execute are executed relative to a directory, which means that what happens when you do certain things depends very much on where you are. You're likely to have started in your home directory, which is why you may see a ~ here, but if you're curious where your home directory really is, you can use the command pwd to print the "working directory" (which means the directory where you currently are).
AlexThornton-Mac:~$ pwd /home/thornton AlexThornton-Mac:~$
You can move from one directory to another by using the command cd. You give cd either an absolute path (which is the complete path to a directory on the same drive) or a relative path (which is the name of a directory you expect to find within the directory you're already in) as a parameter. Note that parameters are separated by spaces. (Note, too, that spaces within directory and file names can be problematic, so we're going to be fastidious about avoiding them from here on out.)
AlexThornton-Mac:~$ cd Desktop AlexThornton-Mac:~/Desktop$ pwd AlexThornton-Mac:~$ cd .. AlexThornton-Mac:~$
The directory .. is special; it is the parent directory of wherever you are (i.e., the one that contains the directory you're currently in).
To create our virtual environment, we're going to need a directory to put it in. Decide where you want that directory to be, then change yourself into the directory that will contain it. For example, if I wanted it on my desktop, I might next do this.
AlexThornton-Mac:~$ cd Desktop AlexThornton-Mac:~/Desktop$
Now, we can create our new directory using the command mkdir, giving it the name of the directory we want to create as a parameter; then, we can change into it.
AlexThornton-Mac:~/Desktop$ mkdir 32Example AlexThornton-Mac:~/Desktop$ cd 32Example AlexThornton-Mac:~/Desktop/32Example$
Next, we're ready to create our virtual environment, which we can do from within the directory where we want it stored by executing the following command. Note that the space and the dot at the end are both important. The special directory . generally always means "The directory I'm in now," so what I'm really saying here is "Create me a new virtual environment in the directory where I am now."
AlexThornton-Mac:~/Desktop/32Example$ python3.10 -m venv .
It may take a little while to set this up; don't worry if it takes a minute or two. (If you see an error message saying something like 'python3.10': command not found, then this means you don't have Python set up to run from the command line properly. See the Python installation instructions for macOS from Project #0, and pay particular attention to Step 3, which explains how to set that up.)
Once it's done, let's take a look at what we ended up with. We can do that using the command ls, which means to show us a "list" of what's in the current directory.
AlexThornton-Mac:~/Desktop/32Example$ ls bin include lib pyenv.cfg AlexThornton-Mac:~/Desktop/32Example$
So what do we see here?
Let's make an additional directory for our own source code, which we'll call src. We won't use it right off, but we'll want it available to us a little later.
AlexThornton-Mac:~/Desktop/32Example$ mkdir src AlexThornton-Mac:~/Desktop/32Example$
The bin directory turns out to be an important one, because it contains a script called activate that lets us "turn on" our virtual environment. Until we've done that, the things we do will have an effect on our "global" Python installation, rather than the virtual environment that we've just created.
AlexThornton-Mac:~/Desktop/32Example$ cd bin AlexThornton-Mac:~/Desktop/32Example/bin$ source activate (32Example) AlexThornton-Mac:~/Desktop/32Example/bin$
(Note that the source activate command won't work if you've set up a different Terminal shell than the one that is default on macOS (which is called bash). If, for example, you're using tcsh as your shell, you would need to say source activate.csh instead. If you're not sure what Terminal shell you're running, the command echo $SHELL will likely tell you.)
After we ran the activate script, our prompt changed. In addition to showing us the path to a directory, it's also showing us the name of our virtual environment; this is a handy way of helping us to remember that we've activated our virtual environment.
Next, let's install a third-party library called pygame, which we can do using a tool built into Python called pip. (Once we've activated our virtual environment, anything we install using pip will be installed into our virtual environment, rather than our global Python installation.) What pip does, in short, is download a library from a community-run server on the Internet and then installs it for us. There are all kinds of details that aren't especially important here, but when what you want is simple, it's easy to get.
(32Example) AlexThornton-Mac:~/Desktop/32Example/bin$ pip install pygame Collecting pygame Using cached pygame-2.1.0-cp310-cp310-macosx_10_9_intel.whl (6.9MB) Installing collected packages: pygame Successfully installed pygame-2.1.0
Specifically what you'll see may vary a bit, but the general idea is that you'll be apprised of the installation progress. (In my case, installation was a snap because I had downloaded this once already; you may see it being downloaded, in addition to the installation.) The 2.1.0 that you see in a couple of places is a version number, which is the latest version, as of this writing; the macosx_10_9_intel says that I've been given the appropriate version for Python running (at least) macOS 10.9, which is the right one for my situation. (Note, too, that all of this is automatic; I'll get the latest version that's appropriate for my installation of Python and the operating system I'm running it on. You'll get whatever's right for your situation automatically.)
Finally, we can launch IDLE and start interacting with pygame. However, we have to launch IDLE from within our virtual environment. We'll also want all of our own modules to be loaded from our src directory (and to be saved there, too), so we'll change there and then launch IDLE from the command prompt manually.
(32Example) AlexThornton-Mac:~/Desktop/32Example/bin$ cd .. (32Example) AlexThornton-Mac:~/Desktop/32Example$ cd src (32Example) AlexThornton-Mac:~/Desktop/32Example/src$ python3.10 -m idlelib
The module idlelib in Python's standard library is what it sounds like; it's IDLE. What we're doing is saying "run IDLE"; it's really that simple. An IDLE window should pop up, in which case we can make sure our installation is complete by attempting to import pygame.
>>> import pygame
Assuming that this succeeded, you're in business. You've got a virtual environment with pygame installed. Next time you want to launch IDLE within your virtual environment from scratch, you can follow these steps: