How to Install Python Packages in Your Own Environment (Virtual Environments)

Hello, and welcome to this brief tutorial on installing your own Python packages on
CHPC resources.
This video will cover both Python 2 and Python 3.
Virtual environments are what we'll be using, and they allow you to install and modify your
own Python packages without installing a Python distribution, but they still let you use system
packages, so you can install your own packages but still use things like NumPy.
The main difference between Python 2 and Python 3 is the command, but otherwise they're very
similar.
The first thing you'll want to do is sign into a CHPC Linux resource.
Here, I'm just going to use the "ssh" command and SSH to ember.
Any Linux resource should work.
You'll want to use the Python distribution that you want to base yours off of; if you
just run the "python" command normally, it's not any, sort of, up-to-date package.
What you really want to do is use the module version of Python.
You can find which ones are available by running "module spider python."
You'll typically want either Python 2.7.11 or Python 3.5.2.
I'm going to start off with Python 3.5.2, so I'll type "module load python/3.5.2" and
if I run "which python" again, you'll see that it is the updated version, and in fact,
python and python3 are both symbolically linked to this Python module.
To create your virtual environment, you'll want to run "pyvenv --system-site-packages"
and the directory you want to create it in.
"pyvenv" is the command you'll want to use for Python 3, and the "system-site-packages"
flag here lets you use the packages that are already installed on the system version of
Python.
It inherits them from the distribution you're using.
Once this loads, you'll want to load the new distribution.
To do this, you'll have to unload the module.
So, I'll "module unload python/3.5.2" and then "source" the activation file, so "source,"
the directory you created, "/bin/activate."
When I run this command, you'll see over here on the left, there's now this text that has
the directory we created earlier.
If I run "which python," there is now a "/bin/python" inside of the directory we created up here.
You can deactivate it by typing "deactivate," and this will take you back to the default
Python that is normally accessed on the clusters.
You can get right back into it by typing "source" (the activation script).
If I run Python and try to load a package that's not installed, I should get an error
now.
I can install this package with "python -m pip install" and the name of the package if
it's on the Python Package Index.
At this point, I can run Python again, and when I run this same command, this "import
packagename," there's now no error.
Where up here there was an ImportError right here, now there's nothing, and that's a good
sign; that means it worked.
You can import something like NumPy, and it should work, because we used the "--system-site-packages"
as well when we loaded this virtual environment.
It's also possible to do more advanced installations, and if you need to do this, look on the documentation
page on the CHPC website, under "Documentation," "Software," and "Python."
On the Python page, if you scroll down, there is a link that says "Python Virtual Environment
page."
This has all of the instructions I've based this video on, but it also has some additional
information on installing packages that have dependencies.
If you click on this, you'll see the one we've already covered with the "python -m" command,
and some other useful commands, but that only really works for packages that are in the
Python Package Index, or "pypi."
There's a lot of different packages on here; there's many to choose from, but I'm going
to search for the one that I actually installed.
This 0.6.2 is the version that I actually installed in my virtual environment, and this
is where it came from.
There, you can see, it's the same version, so it's just fetching them from that website.
On occasion, you might actually have some more complicated package installs.
If there's just the source code that's not on the Python Package Index, you'll probably
need to download it, get all of the files out of the archive with the "tar" command,
and then run the "python setup.py build" and "python setup.py install" commands to install
the package.
That will, by default, install it in the virtual environment directory.
You can actually modify this with the "--prefix" command, just in case you have multiple virtual
environments, but that's typically not used.
You can see here, when the "--prefix" flag is not specified, it will automatically be
set to your virtual environment directory.
You can also use even more advanced installations if the package requires it.
This is typically done if there are dependencies that aren't in a location where they can normally
be found and you need to set something like library flags.
It follows the same premise with the "python setup.py build" and "python setup.py install"
commands.
The only thing that's really changed is the "export" commands.
The reason I'm not going to go into much depth on these is because it changes based on which
package you're trying to install.
This is for the "netcdf4" package, and there are some different variables that you have
to set for netcdf4 specifically, and this is typical.
That should cover it for Python 3 installations.
If I just "deactivate" now and then remove it entirely with the "rm -r <directory>" command
(and make sure you put the actual directory after this or you'll remove everything in
your current directory).
Python 2 is actually very similar, so you can do "module spider python" again, and in
this case, we want to use 2.7.11, and you'll typically want to use the most recent version
of Python 2.
I'll "module load python/2.7.11," and then, if I run "which python," it is, in fact, the
2.7.11 and not the default version.
This is the version I want to base my environment on.
With Python 2, the command is "virtualenv" instead of "pyvenv," but the rest of it is
identical.
You do want the "--system-site-packages" flag after the "virtualenv" command, and you also
have the directory you want to create it in.
This process can take some time to set everything up.
Once this is ready, it's almost identical to the Python 3 instructions.
You'll see that I'm still on the Python 2.7.11 version, so I can unload that and then "source"
the "/bin/activate" binary in the directory I created the virtual environment in.
Now, I have this text again at the left, for my virtual environment, and my Python executable
has changed.
The process for installing packages is pretty much the same.
If I run Python and try to import a package I don't have, I'll get an ImportError, and
I can run the "python -m pip install" and then the package name.
Once this is downloaded, I can run Python again and try to import this package.
This time, there are no errors.
You can deactivate this virtual environment with the "deactivate" command and remove all
of its contents with the "rm -r <directory to remove>."
That should cover package installations.
If you have any questions, you can email us or stop by our offices in INSCC 405.
Thanks for watching!

How to Install Python Packages in Your Own Environment (Virtual Environments)

Latest Posts