Aims

The aim of this lab is to help setup and install python and various other tools

1. Using PyEnv for installation of python versions
2. Using Pip to install packages
3. Setup VSCode to edit and run python files

Why do I need PyEnv?

At present the version of python installed in our linux labs are

ls -al /usr/bin/python* /usr/bin/pip*
lrwxrwxrwx. 1 root root      6 Aug 19  2020 /usr/bin/pip-2 -> ./pip2
-rwxr-xr-x. 1 root root    206 Aug 19  2020 /usr/bin/pip2
lrwxrwxrwx. 1 root root      6 Aug 19  2020 /usr/bin/pip-2.7 -> ./pip2
lrwxrwxrwx. 1 root root      6 Aug 19  2020 /usr/bin/pip2.7 -> ./pip2
lrwxrwxrwx. 1 root root     23 May 26  2021 /usr/bin/pip-3 -> /etc/alternatives/pip-3
lrwxrwxrwx. 1 root root     22 May 26  2021 /usr/bin/pip3 -> /etc/alternatives/pip3
lrwxrwxrwx. 1 root root      8 Jun 18  2021 /usr/bin/pip-3.6 -> ./pip3.6
-rwxr-xr-x. 1 root root    209 Jun 18  2021 /usr/bin/pip3.6
-rwxr-xr-x. 1 root root  25440 Jun 23  2020 /usr/bin/pipewire
-rwxr-xr-x. 1 root root 437160 Jun 23  2020 /usr/bin/pipewire-media-session
lrwxrwxrwx. 1 root root      9 Aug 11  2021 /usr/bin/python2 -> python2.7
-rwxr-xr-x. 1 root root   8016 Aug 11  2021 /usr/bin/python2.7
lrwxrwxrwx. 1 root root     25 May 26  2021 /usr/bin/python3 -> /etc/alternatives/python3
lrwxrwxrwx. 1 root root     31 Aug 11  2021 /usr/bin/python3.6 -> /usr/libexec/platform-python3.6
lrwxrwxrwx. 1 root root     32 Aug 11  2021 /usr/bin/python3.6m -> /usr/libexec/platform-python3.6m

You will notice that some of these are actually symbolic links to other versions. Most of these version of python are essential to the OS and tools that ship with RHEL and as a standard user you don’t have permission to pip install unless you use the -u flag. This can be problematic especially if we want a newer version of python or a different version such as anaconda.

PyEnv

One of the simplest methods I have discovered to manage the different versions of python is the use of PyEnv.

pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.

PyEnv allows you to change both the Global default python version as well as specify specific per folder local versions.

How it works

PyEnv basically uses a system called shims to intercept python calls in your path and select the correct python version. These are added to your path in the .bashrc (or other shells), VSCode will also pick these up.

Installing PyEnv (Linux and Mac)

PyEnv can be installed in numerous ways but as we have all the development tools installed on the lab machines we can use the GitHub checkout method.

git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Next we will build the bash shell extensions which will speed things up.

cd ~/.pyenv && src/configure && make -C src

We now need to setup our shell to point to the installed pyenv. To do this we add the following to our .bashrc file.

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Once this has been added we can activate it in the terminal

source ~/.bashrc

We should now be able to run the following to see what version of python we are currently using.

pyenv global
system

PyEnv Commands

The pyenv command has build in help

pyenv --help
Usage: pyenv <command> [<args>]

Some useful pyenv commands are:
   commands    List all available pyenv commands
   exec        Run an executable with the selected Python version
   global      Set or show the global Python version(s)
   help        Display help for a command
   hooks       List hook scripts for a given pyenv command
   init        Configure the shell environment for pyenv
   install     Install a Python version using python-build
   local       Set or show the local application-specific Python version(s)
   prefix      Display prefixes for Python versions
   rehash      Rehash pyenv shims (run this after installing executables)
   root        Display the root directory where versions and shims are kept
   shell       Set or show the shell-specific Python version
   shims       List existing pyenv shims
   uninstall   Uninstall Python versions
   --version   Display the version of pyenv
   version     Show the current Python version(s) and its origin
   version-file   Detect the file that sets the current pyenv version
   version-name   Show the current Python version
   version-origin   Explain how the current Python version is set
   versions    List all Python versions available to pyenv
   whence      List all Python versions that contain the given executable
   which       Display the full path to an executable

See `pyenv help <command>' for information on a specific command.
For full documentation, see: https://github.com/pyenv/pyenv#readme

We can list all the python version available using

pyenv install --list

As you can see this is a long list, best way to shorten this is to filter the output via grep

pyenv install --list | grep " 3\.1[01]"
3.10.0
3.10-dev
3.10.1
3.10.2
3.10.3
3.10.4
3.10.5
3.10.6
3.11.0rc1
3.11-dev

You will notice that this has some very new dev versions as well as the 10 and 11 point versions. It is usually best to install the non dev versions. For now we will install 3.10.6 but if we are working with DCC tools it is best to install the VFX Reference Platform Version which is 3.10.x for 2023 but 3.9.x for last year (which is what most of our tools use).

pyenv install 3.10.6

This will download, build and install CPython 3.10.6, this may take some time as it is basically building the python and pip executables from source code.

pyenv versions
* system (set by /home/jmacey/.pyenv/version)
  3.10.6

This shows at present the default python is the system one to set the new global version we can do the following

pyenv global 3.10.6
pyenv versions
  system
* 3.10.6 (set by /home/jmacey/.pyenv/version)
python -V
Python 3.10.6

Local Python

We can set a per folder python version using the pyenv local command.

mkdir PyEnvTest
cd PyEnvTest
pyenv local system

This will set the version of python to the system one, however in this case it is called python3 not python as before. The following example is still in the directory above.

python3 -V
Python 3.6.8
python -V
pyenv: python: command not found

The `python' command exists in these Python versions:
  3.10.6

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
cd ..
python3 -V
Python 3.10.6
python -V
Python 3.10.6

Using pip

PIP is a package manager for Python packages (modules). In the case of pyenv the version of pip used will work for the current active version.

All the modules and libraries installed with pip will reside in your ~/.pyenv/versions/[current active] folder. In particular in the ~/.pyenv/versions/[current version]/lib/python[current version]/site-packages folder.

Some basic packages

We are going to start by installing some basic packages to make our life easier.

black

Black is a tool we can run on our code to auto format it to the correct pep8 style.

pip install black

[notice] A new release of pip available: 22.2.1 -> 22.2.2
[notice] To update, run: pip install --upgrade pip

We will see black in action later, but first we can test it by running black as the tool is placed in the shims folder.

isort

isort your imports, so you don’t have to. This will just take all the import statements and sort them alphabetically and automatically separated into sections and by type

pip install isort
isort

MyPy

MyPy is a tool to help check types in your code for mistakes. ( Types in python you ask? More in a later lecture / lab).

pip install mypy
mypy
usage: mypy [-h] [-v] [-V] [more options; see below]
            [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: Missing target module, package, files, or command.

A note of pip

All the modules we have installed so far are installed for this version of python. If you install another version of python they will not be available. So for each new install the previous modules, if required, will need to be re-installed.

This could be tedious however pip has a way of managing this for use.

pip freeze

When installing with pip it generates a manifest of the modules installed. We can list them using the following command.

 pip freeze
black==22.8.0
click==8.1.3
isort==5.10.1
mypy==0.971
mypy-extensions==0.4.3
pathspec==0.10.0
platformdirs==2.5.2
tomli==2.0.1
typing_extensions==4.3.0

We can pipe the output of this to a file called requirements.txt and then re-use it.

pip freeze > requirments.txt
# in a new version
pip install -r requirements.txt

This will install all of the modules / packages installed for the system which is not always what you would need for a project. So there are tools such as pipreqs which will only use the imports of a project. This is really beyond the scope of what we are doing now but is useful for future projects.

virtual env

Python allows us to create a new clean sandbox to test and run code and manage dependencies. We have already seen how pip can be used to install packages, however pip can also cause conflicts especially when we start to specify version of modules and libraries.

A virtual environment allows us a clean folder with it’s own install so we can mange things better. This is often seen as the preferred way to manage python projects (although docker has become popular for this now as well).

How does this differ from pyenv

We can see pyenv as being a way of installing and testing against different python versions. Virtual Env will allow us to create clean installs for the current installed python version.

creating a venv

python -m venv TestEnv
source bin/activate
python -V
Python 3.10.6
pip list
Package    Version
---------- -------
pip        22.2.1
setuptools 63.2.0

Note the packages such as black, isort and mypy are now not installed in this sandbox.

To deactivate the current environment we need to call deactivate in the terminal.

More Reading

This article has some in depth information about how pyenv works