Failed to lookup Image¶
If you are trying to start many tasks at the same time with Shifter, this can create congestion on the image gateway.
If all the processes will use the same image, then you can avoid this by specifying the image in the batch submit script instead of on the command-line.
#SBATCH --image=myimage:latest shifter /path/to/app arg1 arg2
Using this format, the image will be looked up at submission time and cached as part of the job.
If your jobs needs to use multiple images during execution then the approach above will not be sufficient. A workaround is to specify the image by its ID which will avoid the lookup. Just specify the image as
id: followed by the id number which can be obtained with
shifterimg lookup. The image lookup should be done in advance to avoid the lookup occurring during the job.
# Done in advance... user:~> shifterimg lookup centos:8 76d24f3ba3317fa945743bb3746fbaf3a0b752f10b10376960de01da70685fbd # In the job... shifter --image=id:76d24f3ba3317fa945743bb3746fbaf3a0b752f10b10376960de01da70685fbd /bin/hostname
Invalid Volume Map¶
This can happen for different reasons but a common case has to do with the permissions of the directory being mounted. Let's take an example
shifter --volume /global/cfs/cdirs/myproj/a/b --image=myimage bash
In order for Shifter to allow the mount, it needs to be able to see up to the last path as user nobody. The easiest way to fix this is to use
setfacl to allow limited access to the directory. This needs to be done for the full path up to the final directory. For example:
setfacl -m u:nobody:x /global/cfs/cdirs/myproj/ setfacl -m u:nobody:x /global/cfs/cdirs/myproj/a
Note that only the owner of a directory can change the access controls, so you may need the project owner to fix some path elements.