Your User VM comes pre-configured with Python interpreters which are compatible with the Hadoop processing cluster:
- SCL Python 2.7 (default)
- SCL Python 3.5
- Python 3.6
This page explains in more detail how you can manage them.
User specific package installation
This is the recommended and easiest way, for python 2.7:
pip install --user owslib
Installs the owslib library. The '--user' argument is required to avoid needing root permissions, and to ensure that the correct Python version is used. Do not use the yum package manager to install packages!
For Python 3.5 users, the command becomes:
scl enable rh-python35 "pip install --user owslib"
Python 3.6 is not an SCL Python interpreter which makes it easier to use:
pip3.6 install --user owslib
Systemwide (not recommended)
Systemwide installation of (Python) packages requires root permissions. Next to that, we also need to use a special command to enable the right Python version. So as a one-liner, you can install a package like this:
sudo scl enable python27 "pip install owslib"
An alternative is to first drop into a shell with root permissions, and the right Python enabled:
[driesj@driesj ~]$ sudo scl enable python27 /bin/bash
[root@driesj driesj]# pip install owslib
[root@driesj driesj]# exit
Make sure to exit the root shell after installing packages!
For Python 3.5, same as above, just replace python27 with rh-python35. For instance:
sudo scl enable rh-python35 "pip install owslib"
For Python 3.6: sudo pip3.6 install owslib
In a virtual environment
Virtual environments are a good way to avoid requiring root permissions, and to separate the dependencies of different projects.
You can create a new environment like this. If you don't want the default site-packages to be installed, omit option '--system-site-packages'.
[driesj@driesj ~]$ virtualenv --system-site-packages myEnv
New python executable in myEnv/bin/python2
Also creating executable in myEnv/bin/python
Installing setuptools, pip, wheel...done.
A virtualenv has to be activated each time you want to use it:
[driesj@driesj ~]$ cd myEnv/
[driesj@driesj myEnv]$ source bin/activate
Once activated, you can run python scripts, or install packages:
Python 3.x virtual environment
For a Python 3.5 virtualenv, the process is similar but we have to enable the 'scl' first:
scl enable rh-python35 bash
virtualenv --system-site-packages rh35
The key point here is to first enable a shell within the scl, and then to activate the virtualenv.
To simplify this, we can create an alias:
alias py3="scl enable rh-python35 \"bash --rcfile <(echo '. ~/.bashrc; source ~/rh35/bin/activate')\""
Add this last line to you ~/.user_aliases file to make it permanent.
For Python 3.6 virtualenv you can use /usr/bin/virtualenv-3.6.