Lab 1 Python Setup and Install
The aim of this lab is to help setup and install python and various other tools
- Using PyEnv for installation of python versions
- Using Pip to install packages
- Setup VSCode to edit and run python files
Why do I need PyEnv?
At present the version of python installed in our linux labs are
ls -al /usr/bin/python* /usr/bin/pip* lrwxrwxrwx. 1 root root 6 Aug 19 2020 /usr/bin/pip-2 -> ./pip2 -rwxr-xr-x. 1 root root 206 Aug 19 2020 /usr/bin/pip2 lrwxrwxrwx. 1 root root 6 Aug 19 2020 /usr/bin/pip-2.7 -> ./pip2 lrwxrwxrwx. 1 root root 6 Aug 19 2020 /usr/bin/pip2.7 -> ./pip2 lrwxrwxrwx. 1 root root 23 May 26 2021 /usr/bin/pip-3 -> /etc/alternatives/pip-3 lrwxrwxrwx. 1 root root 22 May 26 2021 /usr/bin/pip3 -> /etc/alternatives/pip3 lrwxrwxrwx. 1 root root 8 Jun 18 2021 /usr/bin/pip-3.6 -> ./pip3.6 -rwxr-xr-x. 1 root root 209 Jun 18 2021 /usr/bin/pip3.6 -rwxr-xr-x. 1 root root 25440 Jun 23 2020 /usr/bin/pipewire -rwxr-xr-x. 1 root root 437160 Jun 23 2020 /usr/bin/pipewire-media-session lrwxrwxrwx. 1 root root 9 Aug 11 2021 /usr/bin/python2 -> python2.7 -rwxr-xr-x. 1 root root 8016 Aug 11 2021 /usr/bin/python2.7 lrwxrwxrwx. 1 root root 25 May 26 2021 /usr/bin/python3 -> /etc/alternatives/python3 lrwxrwxrwx. 1 root root 31 Aug 11 2021 /usr/bin/python3.6 -> /usr/libexec/platform-python3.6 lrwxrwxrwx. 1 root root 32 Aug 11 2021 /usr/bin/python3.6m -> /usr/libexec/platform-python3.6m
You will notice that some of these are actually symbolic links to other versions. Most of these version of python are essential to the OS and tools that ship with RHEL and as a standard user you don’t have permission to pip install unless you use the -u flag. This can be problematic especially if we want a newer version of python or a different version such as anaconda.
One of the simplest methods I have discovered to manage the different versions of python is the use of PyEnv.
pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.
PyEnv allows you to change both the Global default python version as well as specify specific per folder local versions.
PyEnv basically uses a system called shims to intercept python calls in your path and select the correct python version. These are added to your path in the .bashrc (or other shells), VSCode will also pick these up.
Installing PyEnv (Linux and Mac)
PyEnv can be installed in numerous ways but as we have all the development tools installed on the lab machines we can use the GitHub checkout method.
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
Next we will build the bash shell extensions which will speed things up.
cd ~/.pyenv && src/configure && make -C src
We now need to setup our shell to point to the installed pyenv. To do this we add the following to our .bashrc file.
export PYENV_ROOT="$HOME/.pyenv" export PATH="$PYENV_ROOT/bin:$PATH" eval "$(pyenv init -)"
Once this has been added we can activate it in the terminal
We should now be able to run the following to see what version of python we are currently using.
pyenv global system
The pyenv command has build in help
pyenv --help Usage: pyenv <command> [<args>] Some useful pyenv commands are: commands List all available pyenv commands exec Run an executable with the selected Python version global Set or show the global Python version(s) help Display help for a command hooks List hook scripts for a given pyenv command init Configure the shell environment for pyenv install Install a Python version using python-build local Set or show the local application-specific Python version(s) prefix Display prefixes for Python versions rehash Rehash pyenv shims (run this after installing executables) root Display the root directory where versions and shims are kept shell Set or show the shell-specific Python version shims List existing pyenv shims uninstall Uninstall Python versions --version Display the version of pyenv version Show the current Python version(s) and its origin version-file Detect the file that sets the current pyenv version version-name Show the current Python version version-origin Explain how the current Python version is set versions List all Python versions available to pyenv whence List all Python versions that contain the given executable which Display the full path to an executable See `pyenv help <command>' for information on a specific command. For full documentation, see: https://github.com/pyenv/pyenv#readme
We can list all the python version available using
pyenv install --list
As you can see this is a long list, best way to shorten this is to filter the output via grep
pyenv install --list | grep " 3\.1" 3.10.0 3.10-dev 3.10.1 3.10.2 3.10.3 3.10.4 3.10.5 3.10.6 3.11.0rc1 3.11-dev
You will notice that this has some very new dev versions as well as the 10 and 11 point versions. It is usually best to install the non dev versions. For now we will install 3.10.6 but if we are working with DCC tools it is best to install the VFX Reference Platform Version which is 3.10.x for 2023 but 3.9.x for last year (which is what most of our tools use).
pyenv install 3.10.6
This will download, build and install CPython 3.10.6, this may take some time as it is basically building the python and pip executables from source code.
pyenv versions * system (set by /home/jmacey/.pyenv/version) 3.10.6
This shows at present the default python is the system one to set the new global version we can do the following
pyenv global 3.10.6 pyenv versions system * 3.10.6 (set by /home/jmacey/.pyenv/version) python -V Python 3.10.6
We can set a per folder python version using the
pyenv local command.
mkdir PyEnvTest cd PyEnvTest pyenv local system
This will set the version of python to the system one, however in this case it is called python3 not python as before. The following example is still in the directory above.
python3 -V Python 3.6.8 python -V pyenv: python: command not found The `python' command exists in these Python versions: 3.10.6 Note: See 'pyenv help global' for tips on allowing both python2 and python3 to be found. cd .. python3 -V Python 3.10.6 python -V Python 3.10.6
python -m testthis will take some time but tests the full installation as well as the standard modules.
PIP is a package manager for Python packages (modules). In the case of pyenv the version of pip used will work for the current active version.
All the modules and libraries installed with pip will reside in your
~/.pyenv/versions/[current active] folder. In particular in the
~/.pyenv/versions/[current version]/lib/python[current version]/site-packages folder.
Some basic packages
We are going to start by installing some basic packages to make our life easier.
Black is a tool we can run on our code to auto format it to the correct pep8 style.
pip install black
You may see something like this in your messages
[notice] A new release of pip available: 22.2.1 -> 22.2.2 [notice] To update, run: pip install --upgrade pip
It is best to never upgrade pip as it can cause massive issues.
We will see black in action later, but first we can test it by running
black as the tool is placed in the shims folder.
isort your imports, so you don’t have to. This will just take all the
import statements and sort them alphabetically and automatically separated into sections and by type
pip install isort isort
MyPy is a tool to help check types in your code for mistakes. ( Types in python you ask? More in a later lecture / lab).
pip install mypy mypy usage: mypy [-h] [-v] [-V] [more options; see below] [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...] mypy: error: Missing target module, package, files, or command.
A note of pip
All the modules we have installed so far are installed for this version of python. If you install another version of python they will not be available. So for each new install the previous modules, if required, will need to be re-installed.
This could be tedious however pip has a way of managing this for use.
When installing with pip it generates a manifest of the modules installed. We can list them using the following command.
pip freeze black==22.8.0 click==8.1.3 isort==5.10.1 mypy==0.971 mypy-extensions==0.4.3 pathspec==0.10.0 platformdirs==2.5.2 tomli==2.0.1 typing_extensions==4.3.0
We can pipe the output of this to a file called requirements.txt and then re-use it.
pip freeze > requirments.txt # in a new version pip install -r requirements.txt
This will install all of the modules / packages installed for the system which is not always what you would need for a project. So there are tools such as pipreqs which will only use the imports of a project. This is really beyond the scope of what we are doing now but is useful for future projects.
Python allows us to create a new clean sandbox to test and run code and manage dependencies. We have already seen how pip can be used to install packages, however pip can also cause conflicts especially when we start to specify version of modules and libraries.
A virtual environment allows us a clean folder with it’s own install so we can mange things better. This is often seen as the preferred way to manage python projects (although docker has become popular for this now as well).
How does this differ from pyenv
We can see pyenv as being a way of installing and testing against different python versions. Virtual Env will allow us to create clean installs for the current installed python version.
creating a venv
python -m venv TestEnv source bin/activate python -V Python 3.10.6 pip list Package Version ---------- ------- pip 22.2.1 setuptools 63.2.0
Note the packages such as black, isort and mypy are now not installed in this sandbox.
To deactivate the current environment we need to call
deactivate in the terminal.