# 🛰️ **PERLMUTTER ON NERSC**
Using Blip on Perlmutter is quite simple when using the docker image.  Perlmutter uses shifter to download and run images.  The Blip image can be downloaded with the simple command

```bash
shifterimg -v pull docker:infophysics/blip:latest
```

Once downloaded, the user will want to set up the three external directories that the image uses, LOCAL_SCRATCH, LOCAL_BLIP and LOCAL_DATA.  The LOCAL_SCRATCH directory can be associated to the pscratch directory on Perlmutter:

```bash
export LOCAL_SCRATCH=/pscratch/sd/${USER:0:1}/${USER}
```

The LOCAL_BLIP directory should point to your development version of Blip, or wherever your custom Blip files are located:

```bash
export LOCAL_BLIP=/global/cfs/cdirs/dune/users/${USER}/blip
```

Finally, the LOCAL_DATA directory should be used for convenience for pointing to wherever the .root or .h5 files (from LArSoft or ndlar-flow) are located:

```bash
# MiniRun4 files as an example for ND
export LOCAL_DATA=/global/cfs/cdirs/dune/www/data/2x2/simulation/productions/MiniRun4_1E19_RHC/MiniRun4_1E19_RHC.flow/FLOW
```

To launch into an interactive session with a GPU on Perlmutter, it is best to put the configuration parameters in a script, like the following:

```bash
#!/bin/bash
#SBATCH -A dune                 # account to use for the job, '--account', '-A'
#SBATCH -J example              # job name, '--job-name', '-J'
#SBATCH -C gpu                  # type of job (constraint can be 'cpu' or 'gpu'), '--constraint', '-C'
#SBATCH -q shared               # Jobs requiring 1 or 2 gpus should use the shared setting, all others use 'regular'
#SBATCH -t 8:00:00              # amount of time requested for the job, '--time', 't'
#SBATCH -N 1                    # number of nodes, '--nodes', '-N'
#SBATCH -n 1                    # number of tasks '--ntasks', -n'
#SBATCH -c 32                   # number of cores per task, '--cpus-per-task', '-c'
#SBATCH --gpus-per-task=1       # number of gpus to be used per task
#SBATCH --gpus-per-node=1       # number of gpus per node.
#SBATCH --gpu-bind=none         # comment this out if you don't want all gpus visible to each task

LOCAL_SCRATCH=/pscratch/sd/${USER:0:1}/${USER}
LOCAL_BLIP=/global/cfs/cdirs/dune/users/${USER}/blip
LOCAL_DATA=/global/cfs/cdirs/dune/users/${USER}/data

setfacl -m u:nobody:x /global/cfs/cdirs/dune/users/${USER}
shifter --image=docker:infophysics/blip:latest --volume="${LOCAL_SCRATCH}:/local_scratch;${LOCAL_BLIP}:/local_blip;${LOCAL_DATA}:/local_data" bash
```