Establishing Connections¶

Login Nodes¶

Opening an SSH connection to NERSC systems results in a connection to a login node. Typically systems will have multiple login nodes which sit behind a load balancer. New connections will be assigned a random node. If an account has recently connected the load balancer will attempt to connect to the same login node as the previous connection.

Connect to NERSC Computational Systems¶

Please make sure you have established a user account and have configured Multi-Factor Authentication (MFA) prior to login.

To access Perlmutter via ssh you can do the following:

ssh <user>@perlmutter.nersc.gov

or

ssh <user>@saul.nersc.gov

If you have configured sshproxy then you can run the following:

ssh -i ~/.ssh/nersc <user>@perlmutter.nersc.gov   # or 'ssh -i ~/.ssh/nersc <user>@saul.nersc.gov'

This assumes your identity file is in ~/.ssh/nersc. The sshproxy route will be convenient if you have multiple SSH connections without having to authenticate every time.

You can also connect to Perlmutter from a DTN and then connect to Perlmutter with ssh perlmutter.

NERSC IP ranges¶

Some institutions block outgoing connections except to a validated list of IPs. In order to access NERSC systems you will need to ensure the IP ranges in the table below are on the allowed list.

In this table, the "IP Range" column denotes the entire possible range of addresses that system could have. If you desire a shorter range, you can use the value in the "Current Used Range" column, but be advised that we may change the IPs for nodes to anything within the broader range without notice.

Connecting to Perlmutter with a Collaboration Account¶

Please use the direct login functionality of sshproxy to log in to Perlmutter.

X11 Forwarding¶

X11 forwarding allows one to display remote computer to your local machine, this can be done as follows:

ssh -X <user>@perlmutter.nersc.gov

Alternatively, you can make this change persistent by adding the following line in $HOME/.ssh/config:

ForwardX11 yes

SSH¶

All NERSC computers (except HPSS) are reached using either the Secure Shell (SSH) communication and encryption protocol (version 2) or by Grid tools that use trusted certificates.

The Secure Shell (SSH) Protocol is a protocol for secure remote login and other secure network services over an insecure network. The protocol is defined by IETF RFCs 4251, 4252, 4253, and 4254. Among the most widely used implementations of SSH is OpenSSH.

Connecting with SSH¶

On a system with an SSH client installed, one can log in to the Perlmutter login nodes by running the following command:

ssh <username>@perlmutter.nersc.gov   # or 'ssh <username>@saul.nersc.gov'

If the user has generated a temporary SSH key using sshproxy, then this command will connect the user to one of Perlmutter's login nodes. If the user has not generated a temporary SSH key, then SSH will challenge the user for their Iris password as well as their one-time password (OTP) before connecting them to a login node.

Users are strongly encouraged to use the most up-to-date versions of SSH clients that are available.

Password-less logins and transfers¶

Consult the documentation on using the SSH Proxy service in the MFA documentation section for ways to connect to NERSC systems without reentering your password and one-time password.

SSH certificate authority¶

NERSC generates SSH certificates for the primary login nodes using a NERSC SSH certificate authority. Most recent ssh clients support these certificates. You can add the following entry to the known_hosts file to make use of this certificate.

@cert-authority *.nersc.gov ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2yKBpvRbdD9MWiu+7wg17vBsKy46AjjuL27DmdpDYiCqRE2mN0om9b0jn4eI91RGykbcRa9wUKJ2qaD0zsD08A8HM+R14H4UsZ5hi7S+xGqscJH7uTmXy5Igo5xEOahS9Z+ecgonDCgWKJnbd/FRu4vITYXrvTlIIGHGRBYj0GzbgLHBzedoMaGNRwhVyadH2SGRaZCgbH+Swevzy0GwYfZJA9zd7EX0jiAClkSYcflIOsygmI3gHv+b35mrvXcHDeQOR/wg8knfpSiFLCkVDpfgnj27Lemzxe6k61Brhv9CUiq+t7WApVDBovhdXZn6pBg+OKeDk1G1OLvRbxJ2bw==

Key fingerprints¶

NERSC may occasionally update the host keys on the major systems. Check here to confirm the current fingerprints.

Perlmutter¶

4096 SHA256:Db9s2Fa4J3qx7An5oIMgUqUAdK7UWJGTPGoIKD44+Gs perlmutter.nersc.gov (RSA)

DTN[01-04]¶

All of the dtn nodes should have the same fingerprints:

4096 SHA256:ErZsheOUCZ4Plrk2KC7TTLyRFiBCDIHVxjiwnJDvztE dtn.nersc.gov (RSA)
256 SHA256:9zLgsb1cjyhr+YOkDhg/I8AWcJNJ1ME2j5YucHOUjyc dtn.nersc.gov (ECDSA)
256 SHA256:SOHXOjNaL4Rj98nX6rS0d9FGRRuqRZoKHrgfyjhuWEw dtn.nersc.gov (ED25519)

NoMachine/NX¶

The fingerprints for NoMachine host are encoded differently, depending on where you view them. In both cases keys are hashed using SHA256. The NoMachine client displays key fingerprints with hex encoding.

SHA256 E7 1D 95 A6 9D B4 A4 99 5B 1A 77 8E 2C FD CF FF D3 5E ED 32 BC 63 9E EB A4 46 F1 76 0F 66 49 23 nxcloud01.nersc.gov (RSA)
SHA256 E3 2B 16 1A 97 39 02 FA D0 A8 D8 78 CD F7 EE DB 15 F3 90 B3 55 B5 A9 1F A0 6A F0 F7 E7 68 57 16  nxcloud01.nersc.gov (ECDSA)
SHA256 98 C7 54 41 60 76 9B 71 33 D3 B0 43 11 0B C0 D6 9A 70 8F C9 D7 4C FA AE D9 72 48 88 90 86 24 AD nxcloud01.nersc.gov (ED25519)

If you are troubleshooting your NX connection by trying to manually SSH to nxcloud01.nersc.gov, SSH logs display key signatures using base64 encoding.

2048 SHA256:5x2Vpp20pJlbGneOLP3P/9Ne7TK8Y57rpEbxdg9mSSM nxcloud01.nersc.gov (RSA)
256 SHA256:4ysWGpc5AvrQqNh4zffu2xXzkLNVtakfoGrw9+doVxY nxcloud01.nersc.gov (ECDSA)
256 SHA256:mMdUQWB2m3Ez07BDEQvA1ppwj8nXTPqu2XJIiJCGJK0 nxcloud01.nersc.gov (ED25519)

ssh-keygen -lf <(ssh-keyscan -t rsa,ed25519,ecdsa <NERSC hostname> 2>/dev/null)

ssh-keygen -lf ~/.ssh/known_hosts | grep <NERSC hostname>

ssh-rsa 4096 3f:6b:e2:c7:95:36:ef:7c:ae:3f:be:1e:13:97:3b:08

ssh-ecdsa    256 5a:c0:06:ed:3c:28:6e:e1:30:ef:f8:8e:f3:0b:26:db
ssh-ed25519  256 65:06:f9:81:e6:21:3f:ab:67:d6:42:c0:eb:28:b4:69
ssh-rsa     4096 97:43:67:e9:13:99:6e:06:8c:26:5b:cd:7b:bc:06:bb

PuTTY Security Alert
The server's host key is not cached in the registry. You have no guarantee that the server is the computer you think it is.
The server's rsa2 key fingerprint is:
ssh-rsa 4096 f6:36:aa:ed:60:9c:d5:f1:29:af:35:81:58:f6:26:45
If you trust this host, hit Yes to add the key to PuTTY's cache and carry on connecting.
If you want to carry on connecting just once, without adding the key to the cache, hit No.
If you do not trust this host, hit Cancel to abandon the connection.

Host Keys¶

These are the entries in ~/.ssh/known_hosts.

Perlmutter¶

perlmutter.nersc.gov ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRDn86LRaU3Hl9E+OTwHtyLhLRvjNCF3e/LAuRu/Ms06FNsXTK1iIb3QMnWjzzFd4CwQwrKYZy9umHIJOyCW/SbEI4GRPrr7NJpVOvkGKVBqSo5aYY3LhqCqKgsX7lbnhwAyvYvHoy9qd/XyzjUMwvUYf1G0dmkIN4AGauEUysn2ixgfhi6+sODsLI9fNrZftnTpYecw2NPP7cn8Dwknh0Fw3450FaXZpUryghdsZp7VvtGwVjQYy5C2PFj8GKqoPh5pIrq4R63hx0OmQOQlRp8cSLtElx1vNzbTRMh43Te3/SJQ66h3sG8AgsPejJgPYEVKJhLpC8CQFj3mQFygcBcp5jOfNnEH2PWZ2Xmzi1p3qrtj01ZML6tIZX9GSdIPif0N2/Zi7+h2wsC+/d4gWdHwX0wQ+o3gYVCKl1Ipf3HJbQ+ZkJ+SecV81nWmu0sGP5nrKALYrQjSwsoO1yFxNhzFk0+XztFoSrMVsfLGG0n4ahTc/yJdB9/VL2/TcujiH+u64cDE9HhOw/JrHtUr13KC/0giaSc9qbjjdhKaXTDcdz4v9NCCqa6jb9R9+MDuufCJy/hOvvYqg99iCGAS+fKPvxJQ3byXOmzZ/dNVw0iz9jQdyTMISmn5BW23YQBtu9RbA8uvzhlV5vsTjtjadbVrWx4VEFgDLtBy8E0kACWw==

Troubleshooting¶

"Access Denied", "Permission Denied" or "Too many authentication failures"¶

Common causes of these are:

1. You are using your laptop username instead of your NERSC username (SSH will by default assume that your remote (NERSC) username is the same as your local (laptop) one. You can avoid this by:
2. Passing your NERSC username explicitly, for example if your NERSC username is elvis:
   • ssh -u elvis perlmutter.nersc.gov, or
   • ssh elvis@perlmutter.nersc.gov

3. Specifying your NERSC username in your laptop $HOME/.ssh/config file, eg

   Host perlmutter
     Hostname  perlmutter.nersc.gov
     User elvis
   

4. You mistyped or forgot your password. Check that you have the correct password by logging in to Iris. Logging into Iris also clears authentication failures, in the event that too many failed authentication attempts have prompted the system to lock your account.

5. You are using PuTTY or MobaXterm and the NERSC system you want is currently unavailable (These Windows SSH clients are known to not display the pre-authentication banners which NERSC uses to indicate down status). You can check the system status on the NERSC MOTD page.

6. You have many SSH keys in your keychain or SSH agent, and your SSH client is trying them all one at a time. After too many wrong ones are tried in a row, the SSH server will stop accepting further attempts. You can check if this might be the cause by adding -vv to your ssh command and look for failed authentication attempts in the output, eg ssh -vv elvis@perlmutter.nersc.gov.
   If this might be the cause, you can instruct your laptop SSH client to only use a specific identity, for example:

   • ssh -o IdentitiesOnly=yes -i ~/.ssh/nersc elvis@perlmutter.nersc.gov if you are using sshproxy (the identity sshproxy generates is called nersc), or
   • ssh -o IdentitiesOnly=yes -i ~/.ssh/id.rsa elvis@perlmutter.nersc.gov if you are not.

If you are still unable to login, contact NERSC Account Support at accounts@nersc.gov.

Host authenticity¶

This message may appear when a connection to a new machine is first established:

The authenticity of host 'perlmutter.nersc.gov' can't be established.
RSA key fingerprint is <omitted>
Are you sure you want to continue connecting (yes/no)?

1. Check that the fingerprints match the list above.
2. If they match accept
3. If they do not match let us know.

Host identification changed¶

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
...

Ensure that your ~/.ssh/known_hosts file contains the correct entries for the host and confirm the fingerprints using the posted fingerprints above. Add the certificate authority line to your known_hosts file if you version of ssh supports SSH certificates.

1. Open ~/.ssh/known_hosts
2. Remove any lines referring the host in question and save the file
3. Paste the host key entries from above or retry connecting to the host and accept the new host key after verify that you have the correct "fingerprint" from the above list.

SSH connection disconnects periodically¶

Some users experience their SSH or Globus connections to NERSC systems disconnect periodically and somewhat randomly. One cause of this is due to a user's SSH client using jumbo Ethernet frames, while a network switch or router between the user's computer and the NERSC network is configured not to forward jumbo frames, e.g., a home WiFi router or a router maintained by an institution's IT division.

Users who experience intermittent failures of their SSH connections to NERSC systems are encouraged to read the detailed description of the jumbo frame problem by ESnet.

Host key errors¶

An error about host keys indicates that the ssh client being used is too old and it is incompatible with NERSC systems. This error manifests as "Couldn't agree a host key algorithm ((available: rsa-sha2-512,rsa-sha2-256))" (or similar) in PuTTY, or "no matching host key type found" in MobaXterm. Update to the latest version of the software to eliminate this issue.

Other SSH connection failures¶

If users encounter unexpected failure when attempting to establish SSH connections to NERSC systems, they are encouraged to collect verbose output from the SSH client where possible, and provide that output in a ticket submitted via the NERSC Help Portal. If using a variant of OpenSSH, verbose output can be obtained by adding the -vvv flag to the ssh command. Other SSH clients may have other ways of emitting verbose output, and users are encouraged to consult the relevant documentation for obtaining such output.