Skip to content

Perlmutter

PerlmutterCabinetsFinal

Perlmutter is a HPE (Hewlett Packard Enterprise) Cray EX supercomputer, named in honor of Saul Perlmutter, an astrophysicist at Berkeley Lab who shared the 2011 Nobel Prize in Physics for his contributions to research showing that the expansion of the universe is accelerating. Dr. Perlmutter has been a NERSC user for many years, and part of his Nobel Prize-winning work was carried out on NERSC machines and the system name reflects and highlights NERSC's commitment to advancing scientific research.

Perlmutter, based on the HPE Cray Shasta platform, is a heterogeneous system comprising both CPU-only and GPU-accelerated nodes, with a performance of 3-4 times Cori when the installation completes. The system is scheduled to be delivered in two phases: Phase 1, with 12 GPU-accelerated cabinets housing over 1,500 nodes, and 35PB of all-flash storage, was delivered by early 2021, and Phase 2 with 12 CPU cabinets will be delivered later in 2021.

Perlmutter_in_phases

System Overview - Phase 1

System Partition # of cabinets # of nodes CPU Aggregate Theoretical Peak (FP64 in PFlops) CPU Aggregate Memory (TiB) GPU Aggregate Theoretical Peak (PFlops) GPU Aggregate Memory (TiB)
GPU-accelerated compute nodes 12 1,536 3.9 384 FP64: 59.9
TF32 Tensor: 958.1
240
Login Nodes - 15 0.07 7.5 FP64: 0.1
TF32 Tensor: 2.3
0.6

File Systems

System Specification - Phase 1

CPUs

System Partition Processor Clock Rate (MHz) Cores per Socket Threads/Core Sockets per Node Memory per Node (GiB) Local Scratch per Node (GB)
GPU-accelerated compute nodes AMD EPYC 7763 (Milan) 2450 64 2 1 256 -
Login Nodes AMD EPYC 7742 (Rome) 2250 64 2 2 512 960

GPUs

System Partition Processor Clock Rate (MHz) SMs per GPU INT32, FP32, FP64, Tensor cores per GPU GPUs per Node Memory per Node (GiB)
GPU-accelerated compute nodes NVIDIA A100 GPU 1410 108 6912, 6912, 3456, 432 4 160
Login Nodes NVIDIA A100 GPU 1410 108 6912, 6912, 3456, 432 1 40

Detailed System Information

Please check here for detailed system info.

Access

Perlmutter is not yet available for general user access. Once users have access, they can ssh to saul.nersc.gov to log on to Perlmutter.

Perlmutter will be available to users in several stages, in the following order:

  1. The NESAP (NERSC Exascale Science Applications Program) tier 1 and ECP (Exascale Computing Project) teams
  2. The NESAP tier 2 and Superfacility teams
  3. Selected general users running GPU applications
  4. Remaining general users running GPU applications
  5. Remaining users

Preparing for Perlmutter

Please check the Transitioning Applications to Perlmutter webpage for a wealth of useful information on how to transition your applications for Perlmutter.

Running Jobs

Perlmutter uses Slurm for batch job scheduling. Below you can find info on the queue policies, how to submit jobs using Slurm and monitor jobs, etc.:

During Allocation Year 2021 jobs run on Perlmutter will be free of charge.

Current Known Issues

For known system issues, check here.