File System Quotas¶

NERSC sets quotas on file systems shown in the table below. The table shows space quota, inode quota and Consequence for Exceeding Quota for each file system. An inode quota is a limit on the number of files and directories that can be created on a system, as each file and directory has an inode associated with it. Inode quotas are different from space (data storage) limits, which control the amount of data a user can store.

Overview¶

Policy¶

NERSC data management policy.

Quotas¶

Limits¶

You may briefly exceed your quota on Perlmutter scratch by as much as 10 TB for up to 24 hours before no new data can be written. This is known as a 'grace period'. When you are above your 20 TB quota, showquota --grace will show how much time you have left in your grace period before the file system becomes read-only.

Quota Types¶

There are two types of quotas at NERSC: user-level and directory-level.

User-level quotas operate for the Homes, Scratch, and HPSS Storage Systems. On these systems any data owned by you counts toward your quota regardless of what directory it is in. For instance, this means that if you wrote data to another user's scratch directory, it will still count against your usage for quota calculations.

Directory-level quotas (also known as fileset quotas) operate on the Community and Global Common File Systems. On these systems quotas are tied to directory paths. For instance every file that is in /global/cfs/cdirs/mxyz counts towards the quota for the mxyz project.

Quota Tools¶

NERSC offers two separate tools to retrieve your quota information, the command-line tool showquota and the web-based Data Dashboard.

Showquota¶

showquota, which works on Perlmutter login nodes and the Data Transfer Nodes, offers more control over the file systems to retrieve quota for, works with all file systems general users have access to (including the HPSS archive!) and provides many features, such as overquota grace times and JSON output.

myquota, cfsquota, prjquota, and cmnquota are relic commands from an older system. They are now symlinks to showquota, so we encourage users to switch to using showquota instead.

To see current usage for the Home and Scratch file systems you can just type:

showquota

You can also exclude the Home and Scratch file systems from the default file systems to report quotas for with -N/--no-default-fs.

For the Community File System you can specify the projects you want to check quota for with:

showquota PROJECT [PROJECT [..]]

For the Common File System you can add --cmn, and showquota will report quotas for all projects you pass as arguments:

showquota --cmn PROJECT [PROJECT [..]]

The --hpss/--hpssquota option makes showquota report HPSS Archive quotas for all projects the user is part of:

$ showquota --hpss
+----------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|                                  File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
+----------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
| username usage on HPSS charged to project123 |   23.74TiB |     1.44PiB |           1.6% |          - |           - |              - |
| username usage on HPSS charged to project456 |      0.00B |   133.00TiB |           0.0% |          - |           - |              - |
+----------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+

Or alternatively you can specify full paths, and mix and match different quota queries into a single one (unfortunately there's no way to request HPSS quotas by path, only by enabling the --hpss keyword):

$ showquota --no-default-fs --full-path $CFS/dasrepo /global/common/software/dasrepo
+---------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|                     File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
+---------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|       /global/cfs/cdirs/dasrepo |   98.54TiB |   110.00TiB |          89.6% |     32.51M |     100.00M |          32.5% |
| /global/common/software/dasrepo |   72.99GiB |   100.00GiB |          73.0% |    819.56K |       2.00M |          41.0% |
+---------------------------------+------------+-------------+----------------+------------+-------------+----------------+

Different output formats and unit of measurements are available, see showquota --help.

For scripting purposes we suggest to use the JSON formatting option -J/--json, which can be quite powerful when paired with e.g. the jq utility:

$ showquota -J dasrepo nstaff |jq '.[] | "\(.fs) \(.space_perc)"'
"home 0.2%"
"pscratch 0.7%"
"dasrepo 0.9%"
"nstaff 0.4%"

The -L/--limit/--grace option shows how much more storage and inodes you can temporarily write above your quota and how much time you have left before no new data can be written.

showquota is a Python script, and has been designed with composability in mind, so you can use it in your own Python scripts by making it available in your $PYTHONPATH variable, e.g.:

$ PYTHONPATH="$(dirname $(which showquota)):$PYTHONPATH" python3
>>> import showquota
>>> stats = showquota.report('project123', full_path=True)
>>> stats
[/global/cfs/cdirs/project123: 70.1% storage - 87.9% inode]
>>> stats[0].is_overquota()
False

Feel free to reach out by opening a ticket for more details.

The Data Dashboard¶

You can use the Data Dashboard to see usage in shared directories in the Community File System (CFS). Since CFS has a directory-level quota, projects can sometimes run into situations where they reach their quota and it's not clear which users are taking up the space. Users can use the Data Dashboard to see a breakdown of usage by user and group for each of their directories on the Community File System. When the Dashboard first loads, you will see the summary totals for each project directory to which you have access. You can click Toggle Usage Details for a project to see the breakdown by user and group.

The Data Dashboard also provides a tool to find files that are taking up large amounts of space or many inodes and are on the old side. Click My Files and Dirs for a project directory to see your largest files and subdirectories and your subdirectories with the most inodes, graphed by size and age.

Increases¶

If you or your project needs additional space for your scratch file space, CFS directory, /global/common directory, or HPSS space, you may request it via the Disk Quota Increase Form (requires login).

Quotas on the Community File System are determined by DOE Program Managers based on information PIs supply in their yearly ERCAP requests. If you need a mid-year quota increase on the Community File System, please use the Disk Quota Increase Form link above and we will pass the information along to the appropriate DOE Program Manager for approval.

Deleting your own files¶

To remain under quota, it could be useful to remove files from a directory. It can be non-trivial to find large files, especially if they are in a hidden directory.

Finding and deleting files on CFS¶

The Data Dashboard can be used to find large files in the Community File System. The dashboard shows disk space and inode usage for global directories at NERSC to which you have access as PI, PI proxy, or user. Click on the "My Files and Dirs" button for a list of your largest files in each directory.

Finding and deleting files in your home directory¶

The find command can help you discover which files are taking up the most space in your directory.

In the below example, user adele wants to find all files (-type f) in the current directory (.) and sub-directories that are at least 10 MB in size (-size +10M).

adele@perlmutter:~> find . -type f -size +10M
./.bazel/bin/bazel-real
./train/IntroToBB/examples/IOR
./test/run41/hello-coforall_real
./test/run41/hello1
./test/run41/hello2
./test/run41/hello
./test/run41/test

Note from this output that find can list files that are in hidden directories (such as the .bazel directory in the first line of output).

Adele is surprised that the IOR file is so large. An ls -lh confirms it:

adele@perlmutter:~> ls -lh ./train/IntroToBB/examples/IOR
-rwxrwx--- 1 adele adele 12M Jul 27  2017 ./train/IntroToBB/examples/IOR

To list the sizes of all the files above 10 MB in size, Adele can issue the command find . -type f -size +10M -exec ls -lh {}.

Adele realizes that all the files listed are not necessary and can be deleted. Furthermore, the entire .bazel directory can be deleted.

adele@perlmutter:~> showquota
+-------------+------------+-------------+----------------+------------+-------------+----------------+
| File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
+-------------+------------+-------------+----------------+------------+-------------+----------------+
|        home |   36.14GiB |    40.00GiB |          90.4% |     63.41K |       1.00M |           6.3% |
|    pscratch |   14.58GiB |    20.00TiB |           0.1% |     65.53K |      10.00M |           0.7% |
+-------------+------------+-------------+----------------+------------+-------------+----------------+
adele@perlmutter:~> rm -rf ./.bazel ./train/IntroToBB/examples/IOR 
./test/run41/hello-coforall_real ./test/run41/hello1 ./test/run41/hello2
./test/run41/hello ./test/run41/test
adele@perlmutter:~> showquota
+-------------+------------+-------------+----------------+------------+-------------+----------------+
| File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
+-------------+------------+-------------+----------------+------------+-------------+----------------+
|        home |   16.62GiB |    40.00GiB |          41.6% |     62.93K |       1.00M |           6.3% |
|    pscratch |   14.58GiB |    20.00TiB |           0.1% |     65.53K |      10.00M |           0.7% |
+-------------+------------+-------------+----------------+------------+-------------+----------------+