VASP¶

• Use the most recent version of VASP allowed by your license.

• Check your job script, INCAR, POSCAR, and KPOINTS files for typos or other input mistakes before running.

• Check the OUTCAR, standard out, and standard error files to verify that inputs were processed correctly and for error messages.

• Nodes may be unavailable due to a scheduled maintenance or to an unexpected outage or to a large node reservation. In this case the squeue command will typically display a message similar to the following:

  elvis@perlmutter:login28:~> squeue -u elvis
  
     JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  24224393  gpu_ss11 vasp_job    elvis PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:nid[001140-008864])
  

  Your job will be held in the queue until the nodes become available. See the NERSC MOTD page for details.

• Consider whether the amount of resources (nodes, wall time) that you requested are appropriate for your job, and decrease the size of the request if possible. As a general rule, the most reliable way to accelerate your job's priority in the queue is to reduce the time limit as this gives Slurm the flexibility to fit your job in gaps between other jobs.

Example:

VASP binaries are permission-locked to file groups and are not visible to users outside of the file groups. This message appears when you attempt to run a VASP executable without being in the corresponding file group. Take the following steps:

• Once logged in to NERSC, execute the groups command to show the file groups to which you belong. You must be in the vasp5 group to access VASP 5 binaries and in the vasp6 group to access VASP 6 binaries.
• Fill out the VASP License Confirmation Request form to request access. Once your VASP license has been successfully confirmed, NERSC will add you to the file groups corresponding to your license.

Example:

ERROR: there must be 1 or 3 items on line 2 of POSCAR 

This error can occur when reading input files generated by an external program or by a text editor on a non-unix operating system. Here, the input file contains special (non-human-readable) characters that cause VASP to read the file incorrectly. To test if this is the case:

1. Back up the input file which causes the error, e.g.

   perlmutter$ mv POSCAR POSCAR_backup
   

2. Pipe the text of the input file to screen:

   perlmutter$ cat POSCAR_backup
   

3. Create a new input file (in this example, POSCAR) using a unix-based text editor (e.g. vim or nano) and paste the text from the preceding step into the new file, and save the new file.

4. Run your VASP job using the new input file.

If the preceding steps solve the issue, then possible production solutions include:

• Check to see if your external input generating program can export files to a unix-compatible format.

• Use a tool to convert the files to unix format. For example, one can convert DOS files via the dos2unix utility, e.g.

  perlmutter$ dos2unix POSCAR
  

    ERROR: noncollinear calculations require that VASP is compiled without the flag -DNGXhalf and -DNGZhalf

This error occurs when one attempts to run a noncollinear calculation using the vasp_std or vasp_gam executable. To solve, use the vasp_ncl executable instead.

This type of error can occur when one incorrectly configures how a job interacts with the file system. Here are some items to check:

• Always run jobs in your $SCRATCH directory, not in $HOME. VASP generates large files which, if run in $HOME, may cause your $HOME directory to exceed the quota. After the quota has been reached, no new files may be written, and your output will be incomplete or missing.

• When running multiple VASP calculations inside of a single job script, perform each calculation in a separate directory. VASP generates identically-named files (OUTCAR, OSZICAR, ...) for each calculation, and these files will overwrite one another if run in the same directory.

There are various reasons that a VASP job may hang, including:

• VASP jobs on GPUs may hang due to incorrect binding. Examples of this include:

  • VASP uses NCCL to transfer data between GPUs attached to different MPI ranks. If the Slurm setting --gpu-bind=single:1 is used it will cause only one GPU to be visible to each MPI rank; this inhibits NCCL and can cause GPU jobs to hang. Set --gpu-bind=none instead.

  • VASP only supports 1:1 MPI-rank-to-GPU mapping. This means that one cannot use multiple MPI ranks per GPU, nor can one use multiple GPUs per MPI rank. Make sure to select no more than four MPI ranks per node and that the number of GPUs is set equal to the number of MPI ranks in your job script.

• Running on hyperthreads, or running multiple processes per physical cores, can cause VASP jobs to hang:

  • On Perlmutter GPU, each node contains 64 physical cores but 128 "logical cores" with hyperthreading. Make sure that you do not run more than four MPI ranks per node, and, if using OpenMP, make sure that the product of MPI ranks per node and OpenMP threads-per-MPI-rank does not exceed 64. The value for the Slurm parameter -c (--cpus-per-task) should not be set less than 32.

  • On Perlmutter CPU, each node contains 128 physical cores but 256 "logical cores" with hyperthreading. Make sure that you do not run more than 128 MPI ranks per node (for a "pure" MPI calculation), or, if using OpenMP, make sure that the product of MPI ranks per node and OpenMP threads per MPI rank does not exceed 128. The value for the Slurm parameter -c (--cpus-per-task) should never be set less than 2.

  • When running VASP 6 executables with OpenMP enabled, set the OpenMP environment variables after loading the VASP module but before invoking srun to ensure that the correct OpenMP settings are selected. We recommend always setting the OpenMP variables as follows, even when not using OpenMP explicitly:

    export OMP_NUM_THREADS=1    # Set >1 if using OpenMP
    export OMP_PLACES=threads
    export OMP_PROC_BIND=spread
    

• Some plugins/third-party codes do not run on GPUs and may hang when run with the GPU version of VASP. If you have an issue running a third party code with a VASP GPU module, try running using the corresponding CPU module.

• Calculations on very small systems may hang if run with too many MPI ranks.

VASP jobs containing many atoms, many k-points, or large FFT grids may fail due to lack of memory resources. Please see the VASP Wiki Not Enough Memory page for an explanation of VASP memory usage. The general strategy to solve out-of-memory errors is to increase the number of nodes or GPUs requested as this increases the pool of available memory.

Sample out-of-memory failure when exceeding host (CPU) memory:

Sample out-of-memory failure when exceeding device (GPU) memory:

To solve out-of-memory errors, try the following steps:

• Check the memory cost table (in OUTCAR) to make sure that the cost is what you expect. Below is a sample table:

 total amount of memory used by VASP MPI-rank0 42743434. kBytes
=======================================================================

   base      :      30000. kBytes                    
   nonlr-proj:      97297. kBytes                  
   fftplans  :      63529. kBytes                    
   grid      :     228114. kBytes                    
   one-center:       9953. kBytes                  
   wavefun   :   42314541. kBytes

• Make a "test" run of your job with a short time limit, and examine how the memory cost table changes as you increase the number of nodes. Try increasing the number of nodes while leaving the number of MPI ranks-per-node constant. Alternatively, you may find it helpful to increase the number of nodes while leaving the total number of MPI ranks constant. Increase the node count until the value for "total amount of memory used by VASP MPI-rank0" multiplied by the number of ranks-per-node of your job fits inside the node's memory limit (512 GB and 256 GB for Perlmutter CPU and GPU nodes, respectively).

• For GPU jobs, one may also be limited by memory on the GPU card itself. The Slurm setting #SBATCH -C gpu may request either 40 GB or 80 GB A100 GPUs. One can explicitly request only 80 GB A100 GPU nodes by setting #SBATCH -C gpu&hbm80g instead. Please keep in mind that the 80 GB nodes are a limited commodity and that your queue time may increase significantly with this setting.

• The choice of VASP executable also influences the memory cost, with vasp_ncl > vasp_std > vasp_gam. Check if your calculation can be completed with an executable which uses less memory.

• K-point parallelization improves performance but also increases memory usage. You can reduce the memory cost by decreasing the value of KPAR in your INCAR file, with KPAR=1 giving the lowest memory usage.

Example:

 -----------------------------------------------------------------------------
|                     _     ____    _    _    _____     _                     |
|                    | |   |  _ \  | |  | |  / ____|   | |                    |
|                    | |   | |_) | | |  | | | |  __    | |                    |
|                    |_|   |  _ <  | |  | | | | |_ |   |_|                    |
|                     _    | |_) | | |__| | | |__| |    _                     |
|                    (_)   |____/   \____/   \_____|   (_)                    |
|                                                                             |
|     internal error in: mkpoints_change.F  at line: 185                      |
|                                                                             |
|     internal ERROR in RE_READ_KPOINTS_RD: the new k-point set for the       |
|     reduced symmetry case                                                   |
|      does not contain all original k-points. Try to switch off symmetry     |
|                                                                             |
|     If you are not a developer, you should not encounter this problem.      |
|     Please submit a bug report.                                             |
|                                                                             |
 -----------------------------------------------------------------------------

Take the following steps:

• Try the work-around suggestions provided by the 'BUG!' message, if any.

• Check the VASP Known Issues page and search the VASP User Forum to see if this issue has been previously reported.

• If this issue has not been reported, file a bug report with the VASP developers via the VASP Portal. Be sure to provide the VASP developers with all of the input files needed to reproduce your issue (INCAR, POSCAR, KPOINTS files, along with your job script).

• If the VASP developers provide a patch or recommend a change in the build, contact the NERSC Help Desk for further assistance, and provide NERSC with a link to your post on the VASP User Forum.

This type of issue has two common causes:

• The positions of the atoms are bad, e.g. two atoms are too close to one another (possibly as a consequence of cell volume or symmetry). This usually occurs at the start of the calculation but it can also result from a geometry update step.

• The algorithm used for the SCF is a poor match to the electronic structure of the molecule or material system.

For issues with electronic relaxation, see the VASP Developers' documentation for the ALGO parameter. For issues with geometry relaxation, see the VASP Developers' page for the IBRION parameter.

Example:

Error EDDDAV: Call to ZHEGV failed. Returncode = <value>

This message may be preceded by one or more warnings of the type:

WARNING: Sub-Space-Matrix is not hermitian in DAV

See above issue: "My VASP jobs are running but there are problems with convergence in the SCF or relaxation".

Examples of issues that should be directed to the VASP developers at the VASP Portal include:

• You encounter a !BUG! message while running.

• Your job fails due to algorithmic issues, such as SCF convergence or relaxation.

• You would like advice related to choosing pseudopotentials, INCAR parameters, or program options.

Common examples of issues that should be directed to the NERSC Help Desk include:

• Your job fails with Slurm or MPICH errors.

• You have a job which previously ran successfully on Perlmutter using a NERSC-provided module but suddenly begins failing.

• You need to create a job script for a specific purpose, such as a checkpoint/restart run or an automated workflow.

• You would like advice specific to compiling or running on Perlmutter.

When you file a VASP-related ticket at the NERSC Help Desk, your issue will be initially handled by the Consultant on Duty (CoD) who may or may not be a VASP expert. The CoD will either answer the ticket themselves or will route it to a specialist. In general, the more information that you provide in the initial ticket, the easier it is for the CoD to direct your ticket to the NERSC staff that is best equipped to answer your issue. We recommend designating your issue as one of the three categories:

• Runtime issue: this includes problems that you may have when running an individual VASP calculation, such as Slurm errors, segmentation faults, MPICH errors, and CUDA errors.

• Workflow issue: this includes most problems that do not appear in a single job but which occur when multiple VASP jobs are involved, such as:

  • Using and automated environment to run a collection of VASP jobs.

  • Running many VASP jobs in parallel using a single job script.

  • Checkpointing and restarting VASP.

  • Setting up a workflow containing VASP jobs which depend on other jobs.

  • Setting up jobs with unusual resource requirements, e.g. very long running jobs or jobs requiring a large fraction of NERSC resources.

• Build issue: this includes problems that you may face if you compile your own version of VASP, such as compiler or linker errors, or if your VASP build has runtime issues that do not appear in the NERSC-provided modules.

For runtime issues, answer the following questions when submitting your request:

• What incorrect behavior do you observe, and what error messages do you receive (if any)?

• What is the path to the location of your job on Perlmutter?

• What is your job script?

Answers to the following questions may also help NERSC diagnose your issue:

• When did the error first appear?

• Is the error reproducible when run on different nodes?

• Did you try a different VASP binary or a different module?

• Is this error specific to CPU or GPU runs, or does it appear in both?

For workflow issues, please answer the following:

• What is the "big picture" of what you would like to accomplish with the workflow?

• At which step in the workflow are you experiencing the issue?

• What is your job script?

For build issues, please answer the following:

• Do you intend to run your build on CPUs or GPUs?

• What is the makefile.include that you used to build?

• Which programming environment do you have loaded (e.g. PrgEnv-gnu)?

• Are there third party codes or other VASP plugins that you included?

• Did you modify the VASP source code prior to building?