Shared storage infrastructure and software tools

This guide describes how to use the shared storage infrastructure and tools to facilitate the installation of software by the users.

Shared storage

Several network filesystems are mounted on UI and across every compute node under the /nfs path.

Currently there are three existing filesystems:

/nfs/home/$USER/

This is the location for all the users home directories.

/nfs/shared/

This filesystem is designed for storing shared data between different projects and groups.

/nfs/software/$GROUP/

This filesystem serves as the dedicated location for users to install all the libraries and applications they need on the clusters. Users can leverage Spack and modulefiles to manage software installations in this directory. This structure ensures better organization and modularity.

Software deployment and managing tools

To facilitate software deployment and environment management, we use Spack and modulefiles. These two tools work together to provide a flexible and efficient software management system in our clusters. Spack is used to install software and manage dependencies and versions, ensuring that applications and libraries are correctly compiled and configured. Once the desired software is installed, Spack generates modulefiles, which serve as an interface for users to easily load and use the installed applications.

Instead of manually setting environment variables and paths, users can simply load the appropriate module using the module command. This approach ensures a clean and modular software environment, allowing users to switch between different versions and configurations without conflicts.

Spack

Spack is a flexible package manager designed for HPC environments. It allows users to install, configure, and manage multiple versions of software packages and dependencies. It simplifies software deployment by handling dependency resolution, versioning, and compiler selection.

We have already configured and provisioned a spack environment in /nfs/software/GROUP/spack available for the users.

Using spack

As an example, we provide a step by step tutorial of how to install the application SIESTA using spack here.

The outline for installing and making new software available to users through spack and modulefiles is the following:

  1. Load spack environment
  2. Install the desired software using spack.
  3. Generate the modulefiles for the new software using spack.

Modulefiles

Modulefiles provide a convenient way to manage and switch between different software environments. They enable users to dynamically modify their shell environment to use specific software versions without affecting other applications. When Spack installs software, it can generate modulefiles to facilitate seamless environment setup.

Using modulefiles

You only need to know a few really simple commands to use any library or aplication using modulefiles:

(Optional) Load whole module environment:

module use /nfs/software/sci/spack/modulefiles/*

List available modules:

module avail

Load a specific module:

module load <module-name>

Unload a module:

module unload <module-name>

Show module details:

module show <module-name>

Summary

Whit this tools we aim to facilitate a software installation policy that follows a structured approach to ensure consistency, portability, and ease of use.

All software, when possible, should be installed under the Spack environment of each group in a self-contained manner, meaning that installations must not rely on critical system libraries or software but instead include all necessary dependencies within the Spack-managed environment. This guarantees that applications remain independent of system updates and configurations, reducing conflicts and ensuring reproducibility.

To maintain a standardized and efficient software environment and minimize redundancy, common tools and software can be installed by the SCI under /nfs/software/sci, making them accessible to all users. Individual groups should allways install their required applications and libraries in their designated /nfs/software/$GROUP directories, making use, if required, of the available libraries installed on /nfs/software/sci.