Skip to content

LZ PulseFinder

Interactive session using Shifter on PDSF in SL6.4


This instruction works only for user=afan, use it as a guidance only.

ssh afan@pdsf

salloc -n 1 -p shared  -t 50:00 --image=custom:pdsf-chos-sl64:v4  --volume=/global/project:/project

shifter /bin/bash
export CHOS=sl64
source ~/.bash_profile.ext

cd forJan/PhotonDetection/
source /global/project/projectdirs/lz/releases/physics/latest/Physics/
time lzap scripts/validations/


You do not need to load ROOT, CLHEP, or Geant4. This application gets all its dependencies from cvmfs, including ROOT.


The TCling error from lzap is expected for now.

How to start LZ Slurm+Shifter+CVMFS job

ssh -A -Y afan@pdsf
$ sbatch lzOne.slr

$ cat hello2.slr
#SBATCH -t 55:00   --ntasks=1 --account lz 

# Pick one of the following lines to toggle: chos or shifter
# (toggle  '#-SBATCH' is OFF vs. '#SBATCH' is ON  )
#SBATCH -J lz-shift -p shared --image=custom:pdsf-chos-sl64:v4  # PDSF +Shifter
#-SBATCH -J lz-chos -p shared-chos  # PDSF + Chos
#-SBATCH -J lz-cori -p shared --image=custom:pdsf-chos-sl64:v2 -C haswell # Cori

#tasks to be executed

echo "start-A "`hostname`
ls -l  ${job_sh}
if [[ $SLURM_JOB_PARTITION == *"-chos" ]]
  echo  run-in-chos
  CHOS=sl64 chos  ./${job_sh}
 echo  run-in-shifter
 shifter  --volume=/global/project:/project  /bin/bash ${job_sh}
echo end-A

where 'lzOne.slr' informs Slurm what resources will you need and launches the bash script running the actuall task ''. Note, the lzOne.slr is setup to run on all 3 slurm partitions: PDSF+Chos, PDSF+Shifter, Cori+Shifter - you need only to toggle the '-' in front of SBATCH.

The bash task script '' requires sourcing of your envirement - if you use Shifter

if [[ $SLURM_JOB_PARTITION == *"-chos" ]]
   echo  task-in-chos
  echo  task-in-shifter
  #source /usr/share/Modules/init/bash
  #module use /usr/common/usg/Modules/modulefiles
  echo inShifter:`env|grep  SHIFTER_RUNTIME`
  cat /etc/*release

  export CHOS=sl64
  source ~/.bash_profile.ext

echo check if input exist
ls -l /project/projectdirs/lz/data/simulations/LUXSim_release-4.4.6_geant4.9.5.p02/full_slow_simulation/electron_recoils/FullSlowSimulation_ER_flat_10k_DER.root 

source /global/project/projectdirs/lz/releases/physics/latest/Physics/
echo run the task for 6-7 minutes
time lzap scripts/validations/
echo " lzReco completed on "`date`

The output file for this job was:

$ sbatch  lzOne.slr
   Submitted batch job 2111

$cat slurm-2111.out
Adding to following projects to this work space : PhotonDetection
# setting LC_ALL to "C"
# --> Including file '/global/u2/a/afan/forJan/PhotonDetection/scripts/validations/'
# <-- End of file '/global/u2/a/afan/forJan/PhotonDetection/scripts/validations/'
ApplicationMgr    SUCCESS
LdrfContext          INFO Opening file: /project/projectdirs/lz/data/simulations/LUXSim_release-4.4.6_geant4.9.5.p02/full_slow_simulation/electron_recoils/FullSlowSimulation_ER_flat_10k_DER.root
EventLoopMgr         INFO Histograms converted successfully according to request.
ApplicationMgr       INFO Application Manager Finalized successfully
ApplicationMgr       INFO Application Manager Terminated successfully

real    8m31.548s
user    5m25.573s
sys 0m5.361s
returned from cgroup ----