🛰️ PERLMUTTER ON NERSC

Using Blip on Perlmutter is quite simple when using the docker image. Perlmutter uses shifter to download and run images. The Blip image can be downloaded with the simple command

shifterimg -v pull docker:infophysics/blip:latest

Once downloaded, the user will want to set up the three external directories that the image uses, LOCAL_SCRATCH, LOCAL_BLIP and LOCAL_DATA. The LOCAL_SCRATCH directory can be associated to the pscratch directory on Perlmutter:

export LOCAL_SCRATCH=/pscratch/sd/${USER:0:1}/${USER}

The LOCAL_BLIP directory should point to your development version of Blip, or wherever your custom Blip files are located:

export LOCAL_BLIP=/global/cfs/cdirs/dune/users/${USER}/blip

Finally, the LOCAL_DATA directory should be used for convenience for pointing to wherever the .root or .h5 files (from LArSoft or ndlar-flow) are located:

# MiniRun4 files as an example for ND
export LOCAL_DATA=/global/cfs/cdirs/dune/www/data/2x2/simulation/productions/MiniRun4_1E19_RHC/MiniRun4_1E19_RHC.flow/FLOW

To launch into an interactive session with a GPU on Perlmutter, it is best to put the configuration parameters in a script, like the following:

#!/bin/bash
#SBATCH -A dune                 # account to use for the job, '--account', '-A'
#SBATCH -J example              # job name, '--job-name', '-J'
#SBATCH -C gpu                  # type of job (constraint can be 'cpu' or 'gpu'), '--constraint', '-C'
#SBATCH -q shared               # Jobs requiring 1 or 2 gpus should use the shared setting, all others use 'regular'
#SBATCH -t 8:00:00              # amount of time requested for the job, '--time', 't'
#SBATCH -N 1                    # number of nodes, '--nodes', '-N'
#SBATCH -n 1                    # number of tasks '--ntasks', -n'
#SBATCH -c 32                   # number of cores per task, '--cpus-per-task', '-c'
#SBATCH --gpus-per-task=1       # number of gpus to be used per task
#SBATCH --gpus-per-node=1       # number of gpus per node.
#SBATCH --gpu-bind=none         # comment this out if you don't want all gpus visible to each task

LOCAL_SCRATCH=/pscratch/sd/${USER:0:1}/${USER}
LOCAL_BLIP=/global/cfs/cdirs/dune/users/${USER}/blip
LOCAL_DATA=/global/cfs/cdirs/dune/users/${USER}/data

setfacl -m u:nobody:x /global/cfs/cdirs/dune/users/${USER}
shifter --image=docker:infophysics/blip:latest --volume="${LOCAL_SCRATCH}:/local_scratch;${LOCAL_BLIP}:/local_blip;${LOCAL_DATA}:/local_data" bash