New shared storage infrastructure and software tools

This guide describes how to use the new shared storage infrastructure and tools to facilitate the installation of software by the users.

First we are describing very briefly the new storage backend and then we will talk about the new tools for managing user aplications.

New shared storage

In SCI we have deployed a brand new network filesystem (NFS) that improves the previous solution in terms of performance and availability. This new network filesystem is mounted on UI and every compute node under the /nfs path.

Currently there are three existing filesystems:

`/nfs/home/$USER/`

This will be the new location for all the users home directories. Currently we are still in the process of migrating all the user $HOME to this new storage.

`/nfs/shared/`

This filesystem is designed for storing shared data between different projects and groups.

`/nfs/software/$GROUP/`

This filesystem serves as the dedicated location for users to install all the libraries and applications they need on the clusters. Users can leverage Spack and modulefiles to manage software installations in this directory. This structure ensures better organization and modularity.

Users may access this new storage using the same tools and procedures as for the old NFS storage or any other POSIX filesystem.

New software deployment and managing tools

To facilitate software deployment and environment management, we use Spack and modulefiles. These two tools work together to provide a flexible and efficient software management system in our clusters. Spack is used to install software and manage dependencies and versions, ensuring that applications and libraries are correctly compiled and configured. Once the desired software is installed, Spack generates modulefiles, which serve as an interface for users to easily load and use the installed applications.

Instead of manually setting environment variables and paths, users can simply load the appropriate module using the module command. This approach ensures a clean and modular software environment, allowing users to switch between different versions and configurations without conflicts.

Spack

Spack is a flexible package manager designed for HPC environments. It allows users to install, configure, and manage multiple versions of software packages and dependencies. It simplifies software deployment by handling dependency resolution, versioning, and compiler selection.

We have already configured and provisioned a spack environment in /nfs/software/GROUP/spack available for the users.

Using spack

As an example, we provide a step by step tutorial of how to install the application SIESTA using spack here.

The outline for installing and making new software available to users through spack and modulefiles is the following:

Load spack environment
Install the desired software using spack.
Generate the modulefiles for the new software using spack.

Modulefiles

Modulefiles provide a convenient way to manage and switch between different software environments. They enable users to dynamically modify their shell environment to use specific software versions without affecting other applications. When Spack installs software, it can generate modulefiles to facilitate seamless environment setup.

Using modulefiles

You only need to know a few really simple commands to use any library or aplication using modulefiles:

List available modules:

module avail

Load a specific module:

module load <module-name>

Unload a module:

module unload <module-name>

Show module details:

module show <module-name>

Summary

Whit this new tools we aim to facilitate a software installation policy that follows a structured approach to ensure consistency, portability, and ease of use.

All software, when possible, should be installed under the Spack environment of each group in a self-contained manner, meaning that installations must not rely on critical system libraries or software but instead include all necessary dependencies within the Spack-managed environment. This guarantees that applications remain independent of system updates and configurations, reducing conflicts and ensuring reproducibility.

To maintain a standardized and efficient software environment and minimize redundancy, common tools and software can be installed by the SCI under /nfs/software/sci, making them accessible to all users. Individual groups should allways install their required applications and libraries in their designated /nfs/software/$GROUP directories, making use, if required, of the available libraries installed on /nfs/software/sci.