Your User VM comes pre-configured with Python interpreters which are compatible with the Hadoop processing cluster:

  • SCL Python 2.7 (default)
  • SCL Python 3.5: /usr/bin/python3.5
  • Python 3.6: /usr/bin/python3.6

This page explains in more detail how you can manage them.

User specific package installation

This is the recommended and easiest way, for python 2.7:

pip install --user owslib

Installs the owslib library. The '--user' argument is required to avoid needing root permissions, and to ensure that the correct Python version is used. Do not use the yum package manager to install packages!

For Python 3.5 users, the command becomes:

scl enable rh-python35 "pip install --user owslib"

Python 3.6 is not an SCL Python interpreter which makes it easier to use:

pip3.6 install --user owslib

Systemwide (not recommended)

Python 2.7

Systemwide installation of (Python) packages requires root permissions. Next to that, we also need to use a special command to enable the right Python version. So as a one-liner, you can install a package like this:

sudo scl enable python27 "pip install owslib"

An alternative is to first drop into a shell with root permissions, and the right Python enabled:

[driesj@driesj ~]$  sudo scl enable python27 /bin/bash

[root@driesj driesj]# pip install owslib

[root@driesj driesj]# exit

Make sure to exit the root shell after installing packages!

Python 3.x

For Python 3.5, same as above, just replace python27 with rh-python35. For instance:

sudo scl enable rh-python35 "pip install owslib"

For Python 3.6: sudo pip3.6 install owslib

In a virtual environment

Virtual environments are a good way to avoid requiring root permissions, and to separate the dependencies of different projects.

You can create a new environment like this. If you don't want the default site-packages to be installed, omit option '--system-site-packages'.

[driesj@driesj ~]$ virtualenv --system-site-packages myEnv
New python executable in myEnv/bin/python2
Also creating executable in myEnv/bin/python
Installing setuptools, pip, wheel...done.

A virtualenv has to be activated each time you want to use it:

[driesj@driesj ~]$ cd myEnv/

[driesj@driesj myEnv]$ source bin/activate

Once activated, you can run python scripts, or install packages:

(myEnv)[driesj@driesj myEnv]$ python --version
Python 2.7.8
(myEnv)[driesj@driesj myEnv]$ pip install owslib

Python 3.x virtual environment

For a Python 3.5 virtualenv, the process is similar but we have to enable the 'scl' first:

scl enable rh-python35 bash

virtualenv --system-site-packages rh35

source ~/rh35/bin/activate

The key point here is to first enable a shell within the scl, and then to activate the virtualenv.

To simplify this, we can create an alias:

  alias py3="scl enable rh-python35 \"bash --rcfile <(echo '. ~/.bashrc; source ~/rh35/bin/activate')\""

Add this last line to you ~/.user_aliases file to make it permanent.

For Python 3.6 virtualenv you can use /usr/bin/virtualenv-3.6.