Caltech Home > Caltech HPC Homepage > Documentation > Software and Modules
Search Search

Software and Modules

Software and Modules

Software
We have installed a number of software packages in /central/software.  These software packages are managed by the modules environment management system.

Please take a look and see what is available for your use.  If you need other software installed that will be of use to the community in general, feel free to contact us at help-hpc@caltech.edu and we can look into it, otherwise a local or group install is advisable. (see below)

Typically with many scientific software packages, the end users or groups will compile and install it themselves into their home directories or shared group directories.  This is because oftentimes the software build parameters are specific to the group or person in question and may have options compiled in that only they can use. In such cases, you may want to use a package management system like Anaconda or Spack to make building and maintaining the software environment easier. 

Modules
The module system is designed to manage your environment to make availabe softawre packages in your PATH and to also set other settings that the package may need.

To get started, it is great to look a a listing of what is available:

module avail

To load something into your environment use the load command:

module load openmpi/2.1.2

You have to specify the version as well as the package when loading it

Sometimes the modules require other modules to be loaded as well.  If this is the case, it will tell you.  You can also find out more information about the package using the show or whatis commands

module show openmpi/2.1.2
module whatis openmpi/2.1.2

If you want to remove all the currently laoded modules, you can use the purge function

module purge

If you are always using the same module, you can also add the commands to your startup files such as .bash_profile or  .cshrc depending on your shell.


Containers
If you need to run containers on the cluster, you should use singularity as the container technology. This is an application built specifically for HPC workloads, support mpi, is scheduled as any other job is scheduled, runs is userspace, and overlays the filesystem.



Spack Package Management

Some users may want to manage their own stack on software.  Spack is a tool to make it easier. It is putting software installation and modules in one package. You can find out more information at https://spack.io/ . You may also want to check out the Spack tutorial


Anaconda Package Management

If most of your software requirements revolve around scientific python packages you may want to install Conda into your home or group directory. Conda is an open source package management system and environment management system that runs on Windows, macOS and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer.

Here's a Conda Cheat Sheet.

Example 1, Conda Install into your home directory. 

Run the following in your home directory to install a Conda local to your accounts. (See next example to add shared Conda installs for larger groups)

For Python3 Conda installation

wget https://repo.anaconda.com/archive/Anaconda3-5.2.0-Linux-x86_64.sh
bash Anaconda3-5.2.0-Linux-x86_64.sh 

For Python2 Conda installation

wget https://repo.anaconda.com/archive/Anaconda2-5.2.0-Linux-x86_64.sh
bash Anaconda2-5.2.0-Linux-x86_64.sh 

(accept the license agreement, select yes to add anaconda to your path and select no when asked to install the optional vs code unless you want that.) log out of the hpc and back in and the following to verify the anaconda environment is active. 

(base) [jflilley@login1]$ which python
~/anaconda2/bin/python

Since sourcing of the Conda environment is in your .bashrc all compute nodes will have the proper paths set to use Conda packages. 

Example 2, Conda Install into your groups shared directory. Complete Example 1 installation above first. Add the secondary conda environment by creating a new .condarc file in your home directory, replacing <group name directory> with your groups directory name. 

envs_dirs: 
  - /central/groups/<group name directory>/anaconda2

Verify the environment shows up and then activate.
[jflilley@login1 ~]$ conda env list
second-shared-environment /central/groups/MICS/anaconda2/second-shared-environment base * /home/jflilley/anaconda2

conda activate second-shared-environment
which python
/central/groups/<group name directory>/second-shared-environment/bin/python

Another method to share Conda environments while allowing flexibility for other users to self manage their own, is to export an environment to yaml that can be imported into the other users personal Conda setup.