Lab 3 PyEnv

Aims

The aim of this lab is to help setup and install python and various other tools

  1. Using PyEnv for installation of python versions
  2. Using Pip to install packages
  3. Setup VSCode to edit and run python files

Why do I need PyEnv?

At present the version of python installed in our linux labs are as follows

You will notice that some of these are actually symbolic links to other versions. Most of these version of python are essential to the OS and tools that ship with RHEL and as a standard user you don’t have permission to pip install unless you use the -u flag. This can be problematic especially if we want a newer version of python or a different version such as anaconda.

DCC Python Versions

To add to the mix the different DCC tools also ship with their own version of python. Whilst the VFX Reference Platform has standardized on a version of python it is not always the same version as the OS or the other DCC tools.

PyEnv

One of the simplest methods I have discovered to manage the different versions of python is the use of PyEnv.

pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.

PyEnv allows you to change both the Global default python version as well as specify specific per folder local versions.

Note PyEnv is not fully supported under windows as it will not install the native windows version of python but will build them from source. You can either use WSL or the windows verion pyenv-win which will install native windows binaries

How it works

PyEnv basically uses a system called shims to intercept python calls in your path and select the correct python version. These are added to your path in the .bashrc (or other shells), VSCode will also pick these up.

Installing PyEnv (Linux and Mac)

PyEnv can be installed in numerous ways as documented on the main github page. For the lab install we are going to use the pyenv-installer from here

curl https://pyenv.run | bash

We now need to setup our shell to point to the installed pyenv. To do this we add the following to our .bash_profile file, this will ensure the python versions work via ssh as well.

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Once this has been added we can activate it in the terminal

source ~/.profile
This is only needed for changes to the file, next time you login it will read the current .profile

We should now be able to run the following to see what version of python we are currently using.

pyenv global
system

PyEnv Commands

The pyenv command has build in help

pyenv --help
Usage: pyenv <command> [<args>]

Some useful pyenv commands are:
   commands    List all available pyenv commands
   exec        Run an executable with the selected Python version
   global      Set or show the global Python version(s)
   help        Display help for a command
   hooks       List hook scripts for a given pyenv command
   init        Configure the shell environment for pyenv
   install     Install a Python version using python-build
   local       Set or show the local application-specific Python version(s)
   prefix      Display prefixes for Python versions
   rehash      Rehash pyenv shims (run this after installing executables)
   root        Display the root directory where versions and shims are kept
   shell       Set or show the shell-specific Python version
   shims       List existing pyenv shims
   uninstall   Uninstall Python versions
   --version   Display the version of pyenv
   version     Show the current Python version(s) and its origin
   version-file   Detect the file that sets the current pyenv version
   version-name   Show the current Python version
   version-origin   Explain how the current Python version is set
   versions    List all Python versions available to pyenv
   whence      List all Python versions that contain the given executable
   which       Display the full path to an executable

See `pyenv help <command>' for information on a specific command.
For full documentation, see: https://github.com/pyenv/pyenv#readme

We can list all the python version available using

pyenv install --list

As you can see this is a long list, best way to shorten this is to filter the output via rip grep

pyenv install --list | rg  "anaconda"

You will notice there are many version available here, typically we can use the latest version as anaconda ensures all the core libraries are compatible.

pyenv install anaconda3-2024.06-1

This will download and install the version of anaconda specified it may take some time so we may as well have a coffee break :-)

 pyenv versions
* system (set by /home/jmacey/.pyenv/version)
  anaconda3-2024.06-1

This shows at present the default python is the system one to set the new global version we can do the following

pyenv global anaconda3-2024.06-1
pyenv versions
  system
* anaconda3-2024.06-1 (set by /home/jmacey/.pyenv/version)
python -V
Python 3.12.4

Local Python

We can set a per folder python version using the pyenv local command.

mkdir PyEnvTest
cd PyEnvTest
pyenv local system

This will set the version of python to the system one, however in this case it is called python3 not python as before.

if you want to test your python install fully you can run python -m test this will take some time but tests the full installation as well as the standard modules.

Using pip

PIP is a package manager for Python packages (modules). In the case of pyenv the version of pip used will work for the current active version.

All the modules and libraries installed with pip will reside in your ~/.pyenv/versions/[current active] folder. In particular in the ~/.pyenv/versions/[current version]/lib/python[current version]/site-packages folder.

This folder can become very large if you have multiple python versions and modules installed. Currently my lab pyenv build is 2.5Gb

Some basic packages

We are going to start by installing some basic packages to make our life easier.

black

Black is a tool we can run on our code to auto format it to the correct pep8 style.

pip install black

You may see something like this in your messages

[notice] A new release of pip available: 22.2.1 -> 22.2.2
[notice] To update, run: pip install --upgrade pip

It is best to never upgrade pip as it can cause massive issues.

We will see black in action later, but first we can test it by running black as the tool is placed in the shims folder.

isort

isort your imports, so you don’t have to. This will just take all the import statements and sort them alphabetically and automatically separated into sections and by type

pip install isort
isort

MyPy

MyPy is a tool to help check types in your code for mistakes. ( Types in python you ask? More in a later lecture / lab).

pip install mypy
mypy
usage: mypy [-h] [-v] [-V] [more options; see below]
            [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: Missing target module, package, files, or command.

PyTest

PyTest is a unit testing framework for python and we will be using this in future labs and lectures.

A note of pip

All the modules we have installed so far are installed for this version of python. If you install another version of python they will not be available. So for each new install the previous modules, if required, will need to be re-installed.

This could be tedious however pip has a way of managing this for use.

pip freeze

When installing with pip it generates a manifest of the modules installed. We can list them using the following command.

pip freeze
black==23.7.0
click==8.1.7
exceptiongroup==1.1.3
iniconfig==2.0.0
isort==5.12.0
mypy==1.5.1
mypy-extensions==1.0.0
packaging==23.1
pathspec==0.11.2
platformdirs==3.10.0
pluggy==1.3.0
pytest==7.4.0
tomli==2.0.1
typing_extensions==4.7.1

We can pipe the output of this to a file called requirements.txt and then re-use it.

pip freeze > requirments.txt
# in a new version
pip install -r requirements.txt

This will install all of the modules / packages installed for the system which is not always what you would need for a project. So there are tools such as pipreqs which will only use the imports of a project. This is really beyond the scope of what we are doing now but is useful for future projects.

virtual env

Python allows us to create a new clean sandbox to test and run code and manage dependencies. We have already seen how pip can be used to install packages, however pip can also cause conflicts especially when we start to specify version of modules and libraries.

A virtual environment allows us a clean folder with it’s own install so we can mange things better. This is often seen as the preferred way to manage python projects (although docker has become popular for this now as well).

How does this differ from pyenv

We can see pyenv as being a way of installing and testing against different python versions. Virtual Env will allow us to create clean installs for the current installed python version.

creating a venv

python -m venv TestEnv
source bin/activate
python -V
Python 3.10.6
pip list
Package    Version
---------- -------
pip        22.2.1
setuptools 63.2.0

Note the packages such as black, isort and mypy are now not installed in this sandbox.

To deactivate the current environment we need to call deactivate in the terminal.

Note once activated the venv will override all of the other pythons in the system including the pyenv local and global ones.

More Reading

This article has some in depth information about how pyenv works

Previous
Next