HPSS Usage Charging and Data Sharing¶

Usage Charging¶

DOE's Office of Science awards an HPSS quota to each NERSC project every year. Users charge their HPSS space usage to the HPSS projects of which they are members.

Users can check their HPSS usage and quotas with the showquota command on Perlmutter. You view usages on a user level.

nersc$ showquota --hpss
perlmutter login35> showquota --hpss
+-------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|                               File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
+-------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|                                      home |   26.13GiB |    40.00GiB |          65.3% |    390.31K |       1.00M |          39.0% |
|                                  pscratch |   26.13TiB |     6.35PiB |           0.4% |    898.54K |     209.72M |           0.4% |
|  adele usage on HPSS charged to myproject |   31.22TiB |     3.11PiB |           1.0% |          - |           - |              - |
+-------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+

Here, the "Space Used" column of the last row shows you how much data you have stored in HPSS for that project. The rest of the row (left to right) shows your assigned space quota for the project, the percentage of that you are using currently, the number of files and directories (inodes) you are using for that project, your assigned inode quota for the project, and the percentage of that you are currently using. Data stored in HPSS could potentially be charged to any project that you are a member of. (See below for details.) Here, the user has only the project "myproject".

You can also check the HPSS quota for a project by logging in to Iris, selecting that project, clicking on the "Storage" tab, and scrolling to below the CFS table.

Apportioning User Charges to Projects: Project Percentages¶

The HPSS system has no notion of project-level accounting but only of user accounting. Users must say "after the fact", in Iris, how to distribute their HPSS data usage to the HPSS projects to which they belong. If a user belongs to only one HPSS project, all usage is charged to that project. If a user belongs to multiple projects, usage is apportioned among the user's projects. By default this is split based on the size of each project's storage allocation. Users (only the user, not the project PI, Proxy, or Project Manager) can change what percentage of their HPSS usage is charged to which project in their Storage tab in Iris.

Adding or Removing Users¶

If a user is added to a new project or removed from an existing project, the project percentages for that user are adjusted based on the sizes of the quotas of the projects to which the user now belongs. However, if the user previously changed the default project percentages, the relative ratios of the previously set project percentages are respected.

As an example, user adele belongs to projects p1 and p2 and has changed the project percentages from the default of 50% for each project to 40% for p1 and 60% for p2:

If adele then becomes a new member of project p3, which has a storage allocation of 1,000 GBs, the project percentages will be adjusted as follows (to preserve the old ratio of 40:60 between p1 and p2 while adding p3, which has the same storage allocation as p1+p2):

If a project is retired, the percentage charged to that project is spread among the remaining projects while keeping their relative values the same.

HPSS Project Directories¶

A special "project directory" can be created in HPSS for groups of researchers who wish to easily share files. The files in this directory will be readable by all members of a particular Unix file group. This file group can have the same name as the project (in which case all members of the project will have access to the project directory), or a new name can be requested (in which case only those users added to the new file group by the requester will have access to the project directory).

HPSS project directories have the following properties:

• located under /home/projects (only on HPSS, not to be confused with /global/homes)
• owned by the PI, a PI Proxy, or a Project Manager of the associated project
• have suitable group attributes (including "setgid bit")

To request creation of an HPSS project directory, the PI, a PI Proxy, or a Project Manager of the requesting project should open a ticket.