Build a uenv

CSCS User Environments (uenv) provide scientific applications, libraries and tools on Alps.

User environments are SquashFS files that store a compressed directory tree containing all of the software, tools and other information like modules and libraries required to provide a rich environment.

Each environment contains a software stack, comprised of compilers, libraries, tools and scientific applications, built using Spack.

To learn more about uenv and how to run them, please refer to the official documentation by CSCS.

This recipe will focus on building a custom user environment either locally or via CSCS CI/CD. Either way, a user environment is built from a set of recipes (YAML files), that define the compilers and software packages to be installed, along with configuration of modules and environment scripts the environment will provide to users.

In this recipe, we will install a Python package (ppafm) that provides a library and scripts to perform scanning probe microscopy simulations. The full recipe can be found here.

Locally

The main piece of software required to build a user environment is Stackinator.

Getting Stackinator

Clone the repo

git clone https://github.com/eth-cscs/stackinator.git

Checkout to a released versions since the main branch includes features for Spack v1.0, and may break older recipes.
Terminal window
```
git checkout v4.0
```
Run the bootstrap and make the binary available in your PATH
Terminal window
```
./bootstrap.sh
export PATH="<stackinator-install-path>/bin:$PATH"
```

Prepare the build environment

Move to another folder (e.g., one folder above from where you cloned Stackinator’s repo) and clone the repository with the cluster configurations
Terminal window
```
git clone https://github.com/eth-cscs/alps-cluster-config
```
These are recipes (YAML Spack files) that define the toolchains to compile and install the software described in our recipe.
If you never built a uenv, it’s a good idea to have a build cache to speed up subsequent builds. First, create an empty folder (in your $SCRATCH directory)
Terminal window
```
mkdir -p $SCRATCH/uenv-cache
```

Use a Spack installation to create generate GPG keys that are used for signing and verifying the packages

# Create a  .keys directory accessible only to you
mkdir $SCRATCH/.keys
chmod 700 $SCRATCH/.keys

# Generate the key
spack gpg create <your-name> <your-email>
spack gpg export --secret $SCRATCH/.keys/spack-push-key.gpg
chmod 600 $SCRATCH/.keys/spack-push-key.gpg

Create a file to configure the build cache
cache-config.yaml
```
root: $SCRATCH/uenv-cache
key:  $SCRATCH/.keys/spack-push-key.gpg
```
Note the path of this config file as it will be needed by Stackinator when building the uenv.

Build the uenv

In a folder of your choice, clone the empa-spack repository that contains the recipe for the ppafm package
Terminal window
```
git clone https://github.com/empa-scientific-it/empa-spack.git
```
Request an allocation on a cluster’s computing node to avoid hogging the resources on the login nodes
Terminal window
```
srun -A <account> -n1 -t180 --pty bash
```

Run stack-config (provided by Stackinator) to prepare the build directory. Adjust all the paths according to the previous steps

stack-config --recipe /path/to/empa-spack/uenv/ppafm/v0.3.2/daint \
             --build /dev/shm/$USER/ppafm \
             --system /path/to/alps-cluster-config/daint \
             --cache /path/to/cache-config.yaml

Perform the Build
Terminal window
```
cd /dev/shm/$USER/ppafm
env --ignore-environment PATH=/usr/bin:/bin:`pwd`/spack/bin make modules store.squashfs -j32
```
The call to make is prepended by env to unset all environment variables and make the build reproducible.

Installing the software

The make command generates two versions of the software stack in the build path:

A store directory, the full Spack environment with everthing that’s been built and installed
A store.squashfs file, a compressed filesystem image that can be mounted on a cluster node

The simplest method to install the software stack is to copy the store directory to the desired location (preferly out of the $SCRATCH folder).

The default installation directory is specified in the config.yaml file of our recipe. In this example, we’re using the default /user-environment directory, which should be used for deployment on Alps.

name: ppafm
spack:
    commit: releases/v0.23
    repo: https://github.com/spack/spack.git
store: /user-environment
description: Library and scripts to perform scanning probe microscopy simulations based on a CP2K calculation

Via CSCS CI/CD

Using uenv build command.