Cori ExVivo for JGI

ExVivo is a specialized system used to run JGI applications requiring more shared memory than available on standard Cori Genepool hardware.

Access

Access to Cori ExVivo is available to all JGI users as of February 6th 2019. To use Cori ExVivo, first connect to cori.nersc.gov, load the esslurm module, and request a Slurm allocation. That request command should include an -A argument with your project name, -C skylake, and specify QOS jgi_exvivo, jgi_shared, or jgi_interactive.

Example

elvis@cori10:~> module load esslurm
elvis@cori10:~> salloc -C skylake -A fungalp -q jgi_interactive
salloc: Granted job allocation 1
salloc: Waiting for resource configuration
salloc: Nodes exvivo006 are ready for job
elvis@exvivo006:~> exit
exit
srun: Terminating job step 1.0
salloc: Relinquishing job allocation 1
elvis@cori10:~> sbatch -C skylake -A fungalp -q jgi_exvivo bioinformatics.sh
Submitted batch job 2
elvis@cori10:~>
  • jgi_exvivo is intended for production use by applications and data sets which cannot be run on Cori Genepool due to large RAM requirements. The maximum walltime for an allocation is 7 days.
  • jgi_interactive is intended for exploration and development. At most 4 nodes can be allocated to this QOS. The maximum wall time is 4 hours.
  • jgi_shared is intended for jobs which require more than 128GB RAM but less than 768GB. Use -c and --mem=###GB arguments in the Slurm invocation to request the needed resources.

Note

Multi-node allocations will not be supported on ExVivo.

Resources

Cori ExVivo contains 20 total nodes. Each node has the following configuration:

  • 2 Intel® Xeon® Gold 6140 (Skylake) processors, 36 cores total

  • 1.5 TB RAM

  • 5.1 TB available local disk, solid state drive, mounted as /tmp

The user environment on ExVivo is very similar to that of a Cori login node. Common software is available such as Cori modules, Shifter, and Anaconda.

The following filesystems are available on ExVivo:

  • User Home

  • project

  • projecta

  • projectb

  • Cori Scratch ($CSCRATCH)

  • seqfs

  • Data and Archive (read-only access)