The HPSS Archive System¶

Introduction¶

The High Performance Storage System (HPSS) is a modern, flexible, performance-oriented mass storage system. It has been used at NERSC for archival storage since 1998. HPSS is intended for long term storage of data that is not frequently accessed.

HPSS is Hierarchical Storage Management (HSM) software developed by a collaboration of DOE labs, of which NERSC is a participant, and IBM. The HPSS system is a tape system that uses HSM software to ingest data onto a high performance disk cache and automatically migrate it to a very large enterprise tape subsystem for long-term retention. The disk cache in HPSS is designed to retain many days worth of new data and the tape subsystem is designed to provide the most cost-effective long-term scalable data storage available.

A Beginner's Guide to HPSS¶

This section contains a few quick instructions to get you started using HPSS. For more in depth information please read the sections below.

You can log onto HPSS by using hsi

nersc$ hsi

Typing just hsi alone transfers you to an HPSS command shell which looks very similar to a regular login environment. It has a directory structure you can navagate through and most regular linux commands will work (like ls, cd, etc.). However, commands like ls will only show files and directories stored in the HPSS archive and only hsi commands will work. It's effectively like sshing to another system called hpss. To exit from the HPSS command shell, use exit.

One can execute hsi commands from any Perlmutter login node or a Data Transfer Node by either typing hsi <command> or hsi alone first, then the commands once you enter the HPSS command shell.

Common Commands¶

Here's a list of some common commands. The commands below are written assuming you are running from a login node (i.e. you haven't first invoked an HPSS command shell):

• Show the content of your HPSS home directory: hsi ls
• Create a remote directory in your home: hsi mkdir new_dir_123
• Store a single file into HPSS without renaming: hsi put my_local_file
• Store a directory tree, creating sub-dirs when needed: hsi put -R my_local_dir
• Fetch a single file from HPSS into the local directory without renaming: hsi get /path/to/my_hpss_file
• Delete a file from HPSS: hsi rm /path/to/my_hpss_file
• To recursively remove a directory and all of its contained sub-directories and files: hsi rm -R /path/to/my_hpss_dir/
• Delete an empty directory: hsi rmdir /path/to/my_hpss_dir/

The example below finds files that are more than 20 days old and redirects the output to the file temp.txt:

hsi -q "find . -ctime 20" > temp.txt 2>&1

For more details on using hsi refer to the hsi section below.

Storing groups of files in HPSS with htar¶

It's generally recommended that you group your files together into bundles whenever possible. htar is an HPSS application that will create a bundle of files and store it directly in HPSS. The next example shows how to create a bundle with the contents of the directory nova and the file simulator:

nersc$ htar -cvf nova.tar nova simulator

Listing the contents of a tar file:

nersc$ htar -tf nova.tar

To extract a specific file simulator from a htar file nova.tar

nersc$ htar -xvf nova.tar simulator

For more details on using htar refer to the htar section below.

Accessing HPSS¶

You can access NERSC's HPSS in a variety of different ways. hsi and htar are the best ways to transfer data in and out of HPSS within NERSC. By default every user has an HPSS account.

Individual files¶

hsi is used to put individual files into HPSS (usage)

Multiple files¶

It is recommended that small files be grouped together prior to transfer. htar is used to put bundles of files into HPSS, similar to how tar works (usage). Not bundling smaller files may result in issues which have been discussed in the best practices section.

Transfers between HPSS and facilities outside NERSC¶

• Globus (recommended).
• Access via pftp, and ftp inside of NERSC are offered (not recommended).

NERSC's HPSS system can be accessed at archive.nersc.gov (this is set by default for hsi, htar, and Globus). For tips on how best to use hsi and htar, see the Best Practices section section below. For a better understanding of data storage and sharing at NERSC, refer to the following resources:

• Data Storage and Sharing guide-pdf
• Data Storage and Sharing guide-video tutorial

Automatic Token Generation¶

The first time you try to connect from a NERSC system (Perlmutter, DTNs, etc.) using a NERSC provided client like hsi or htar you will be prompted for your NERSC password + one-time password which will generate a token stored in $HOME/.netrc. After completing this step you will be able to connect to HPSS without typing a password:

nersc$ hsi
Generating .netrc entry...
Password + OTP:

If you are having problems connecting see the Troubleshooting section below.

Session Limits¶

Users are limited to 15 concurrent sessions. This number can be temporarily reduced if a user is impacting system usability for others.

hsi¶

hsi is a flexible and powerful command-line utility to access the NERSC HPSS storage system. You can use it to store and retrieve files and it has a large set of commands for listing your files and directories, creating directories, changing file permissions, etc. The command set has a UNIX look and feel (e.g. mv, mkdir, rm, cp, cd, etc.) so that moving through your HPSS directory tree is close to what you would find on a UNIX file system. hsi can be used both interactively or in batch scripts. hsi doesn't offer compression options, but the HPSS tape system uses hardware compression which is as effective as software compression.

The hsi utility is available on all NERSC production computer systems and it has been configured on these systems to use high-bandwidth parallel transfers.

hsi Commands¶

All of the NERSC computational systems available to users have the hsi client already installed. To access the Archive storage system you can type hsi with no arguments: this will put you in an interactive command shell, placing you in your home directory on the Archive system. From this shell, you can run the ls command to see your files, cd into storage system subdirectories, put files into the storage system and get files from it.

Most of the standard Linux commands work in hsi (cd, ls,rm,chmod, mkdir, rmdir, etc). There are a few commands that are unique to hsi:

hsi also has a series of "local" commands, that act on the non-HPSS side of things:

hsi Syntax¶

The hsi utility uses a special syntax to specify local and HPSS file names when using the put and get commands. : (a colon character with spaces on the sides) is used to separate the local and HPSS paths; the local file name is always on the left of the colon mark, while the HPSS file name is always on the right.

You don't need to provide the separator at all if you want the destination file to use the same name of the source file; you can also combine this with a cd command, e.g. hsi "cd my_hpss_dir/; put my_local_file; get my_hpss_file"

Here are some usage examples:

• Show the content of your HPSS home directory: hsi ls
• Show the content of a specific directory: hsi -q ls /path/to/hpss/dir/ 2>&1 (-q is useful for scripting to suppress the extra output that appears at each connection to HPSS, and 2>&1 to redirect stderr to stdout)
• Create a remote directory in your home: hsi mkdir new_dir_123
• Store a single file from your local home into your HPSS home: hsi put my_local_file : my_hpss_file
• Store a single file into HPSS without renaming: hsi put my_local_file
• Store a directory tree, creating sub-dirs when needed: hsi put -R my_local_dir/
• Fetch a single file from HPSS, from a specific directory: hsi get /path/to/my_local_file : /path/to/my_hpss_file
• Fetch a single file from HPSS into the local directory without renaming: hsi get /path/to/my_hpss_file
• Delete a file from HPSS: hsi rm /path/to/my_hpss_file; use hsi rm -i if you want to confirm the deletion of each file;
• To recursively remove a directory and all of its contained sub-directories and files: hsi rm -R /path/to/my_hpss_dir/
• Delete an empty directory: hsi rmdir /path/to/my_hpss_dir/

Make sure to escape bash expansions, e.g. place quotes around * to avoid bash from replacing the symbol with the files in your local directory, e.g. hsi rm -i "*" or hsi rm -i \*.

In addition to command line, you can run hsi commands several different ways:

• Single-line execution, e.g. to create a new dir and copy a file into it: hsi "mkdir my_hpss_dir; cd my_hpss_dir; put bigdata.123"
• Read commands from a file: hsi "in command_file"
• Read commands from standard input: hsi < command_file
• Read commands from a pipe: cat command_file | hsi

hsi Storage Verification¶

HPSS provides a built-in checksum mechanism to verify data integrity while archiving to HPSS, but you can also calculate checksums for files already stored in HPSS. All checksums are stored separately from the files. The checksum algorithm used is MD5.

Some examples:

• To calculate and store the checksum of a file during a transfer to HPSS, use hsi put -c on local.file : hpss.file (specifying a destination file is optional, see the examples section above);
• To show the stored hash of a file, use hsi hashlist (-R to recurse in directories);
• You can calculate the hash of a file already stored in HPSS with hsi hashcreate hpss.file; to calculate hashes of all files in a HPSS directory recursively, use hsi hashcreate -R
• Similarly, you can verify that a file on HPSS still matches its hash using hsi hashverify (-R for directories);

The easiest way to verify the integrity of a file in HPSS is to create the checksum during the transfer, which can be then be used to verify the data on tape matches with the content on disk. Therefore the recommended approach is to use hsi put -c on to store data, and hsi hashverify before deleting the source files from the local storage.

Removing Older Files¶

You can find and remove older files in HPSS using the hsi find command. This may be useful if you're doing periodic backups of directories (not recommended for software version control, use a versioning system like git instead) and want to delete older backups. Since you can't use a linux pipe (|) in hsi, you need a multi-step process.

The example below will find files older than 10 days and delete them from HPSS:

hsi -q "find . -ctime 10" > temp.txt 2>&1
cat temp.txt | awk '{print "rm -R",$0}' > temp1.txt
hsi in temp1.txt

htar¶

htar is a command line utility that is ideal for storing groups of files in the HPSS archive. It generally works like regular tar except the tar file is created directly in HPSS. This means you don't need to use local space to temporarily store the tar file when writing or reading data. htar preserves the directory structure of stored files. htar doesn't have options for compression, but the HPSS tape system uses hardware compression which is as effective as software compression. htar creates an index file that (by default) is stored along with the archive in HPSS. This allows you to list the contents of an archive without retrieving it from tape first. The index file is only created if the htar bundle is successfully stored in the archive.

htar is installed and maintained on all NERSC production systems. If you need to access the member files of an htar archive from a system that does not have the htar utility installed, you can retrieve the tar file to a local file system and extract the member files using the local tar utility.

If you have a collection of files and store them individually with hsi, the files will likely be distributed across several tapes, requiring long delays (due to multiple tape mounts) when fetching them from HPSS. Instead, group these files in an htar archive file, which will likely be stored on a single tape, requiring only a single tape mount when it comes time to retrieve the data.

The basic syntax of htar is similar to the standard tar utility:

htar -{c|K|t|x|X} -f tarfile [directories] [files]

As with the standard unix tar utility the -c -x and -t options create, extract, and list tar archive files. The -K option verifies an existing tarfile in HPSS and the -X option can be used to re-create the index file for an existing archive.

Please note, you cannot add or append files to an existing htar file.

If your htar files are 100 GB or larger and you only want to extract one or two small member files, you may find faster retrieval rates by skipping staging the file to the HPSS disk cache by adding the -Hnostage option to your htar command.

htar Usage Examples¶

Create an archive with directory nova and file simulator

nersc$ htar -cvf nova.tar nova simulator
HTAR: a   nova/
HTAR: a   nova/sn1987a
HTAR: a   nova/sn1993j
HTAR: a   nova/sn2005e
HTAR: a   simulator
HTAR: a   /scratch/scratchdirs/elvis/HTAR_CF_CHK_61406_1285375012
HTAR Create complete for nova.tar. 28,396,544 bytes written for 4 member files, max threads: 4 Transfer time: 0.420 seconds (67.534 MB/s)
HTAR: HTAR SUCCESSFUL

Now list the contents:

nersc$ htar -tf nova.tar
HTAR: drwx------  elvis/elvis          0 2010-09-24 14:24  nova/
HTAR: -rwx------  elvis/elvis    9331200 2010-09-24 14:24  nova/sn1987a
HTAR: -rwx------  elvis/elvis    9331200 2010-09-24 14:24  nova/sn1993j
HTAR: -rwx------  elvis/elvis    9331200 2010-09-24 14:24  nova/sn2005e
HTAR: -rwx------  elvis/elvis     398552 2010-09-24 17:35  simulator
HTAR: -rw-------  elvis/elvis        256 2010-09-24 17:36  /scratch/scratchdirs/elvis/HTAR_CF_CHK_61406_1285375012
HTAR: HTAR SUCCESSFUL

As an example, using hsi remove the nova.tar.idx index file from HPSS (Note: you generally do not want to do this)

nersc$ hsi "rm nova.tar.idx"
rm: /home/e/elvis/nova.tar.idx (2010/09/24 17:36:53 3360 bytes)

Now try to list the archive contents without the index file:

nersc$ htar -tf nova.tar
ERROR: No such file: nova.tar.idx
ERROR: Fatal error opening index file: nova.tar.idx
HTAR: HTAR FAILED

Here is how we can rebuild the index file if it is accidently deleted

nersc$ htar -Xvf nova.tar
HTAR: i nova
HTAR: i nova/sn1987a
HTAR: i nova/sn1993j
HTAR: i nova/sn2005e
HTAR: i simulator
HTAR: i /scratch/scratchdirs/elvis/HTAR_CF_CHK_61406_1285375012
HTAR: Build Index complete for nova.tar, 5 files 6 total objects, size=28,396,544 bytes
HTAR: HTAR SUCCESSFUL

nersc$ htar -tf nova.tar
HTAR: drwx------  elvis/elvis          0 2010-09-24 14:24  nova/
HTAR: -rwx------  elvis/elvis    9331200 2010-09-24 14:24  nova/sn1987a
HTAR: -rwx------  elvis/elvis    9331200 2010-09-24 14:24  nova/sn1993j
HTAR: -rwx------  elvis/elvis    9331200 2010-09-24 14:24  nova/sn2005e
HTAR: -rwx------  elvis/elvis     398552 2010-09-24 17:35  simulator
HTAR: -rw-------  elvis/elvis    256 2010-09-24 17:36  /scratch/scratchdirs/elvis/HTAR_CF_CHK_61406_1285375012
HTAR: HTAR SUCCESSFUL

Here is how we extract a single file from a htar file

htar -xvf nova.tar simulator

Using ListFiles to Create an htar Archive¶

Rather than specifying the list of files and directories on the command line when creating an htar archive, you can place the list of file and directory pathnames into a ListFile and use the -L option. The contents of the ListFile must contain exactly one pathname per line. Let's assume that we want to archive only files starting with sn19 in the directory nova:

nersc$ find nova -name 'sn19*' -print > novalist
nersc$ cat novalist
nova/sn1987a
nova/sn1993j

Now create an archive containing only these files:

nersc$ htar -cvf nova19.tar -L novalist
HTAR: a   nova/sn1987a
HTAR: a   nova/sn1993j
nersc$ htar -tf nova19.tar
HTAR: -rwx------  elvis/elvis    9331200 2010-09-24 14:24  nova/sn1987a
HTAR: -rwx------  elvis/elvis    9331200 2010-09-24 14:24  nova/sn1993j

Soft Delete and Undelete¶

The -D option can be used to "soft delete" one or more member files or directories from an htar archive. The files are not really deleted, but simply marked in the index file as deleted. A file that is soft-deleted will not be retrieved from the archive during an extract operation. If you list the contents of the archive, soft deleted files will have a D character after the mode bits in the listing:

nersc$ htar -Df nova.tar nova/sn1993j
HTAR: d  nova/sn1993j
HTAR: HTAR SUCCESSFUL

Now list the files and note that sn1993j is marked as deleted:

nersc$ htar -tf nova.tar
HTAR: drwx------   elvis/elvis          0 2010-09-24 14:24  nova/
HTAR: -rwx------   elvis/elvis    9331200 2010-09-24 14:24  nova/sn1987a
HTAR: -rwx------ D elvis/elvis    9331200 2010-09-24 14:24  nova/sn1993j
HTAR: -rwx------   elvis/elvis    9331200 2010-09-24 14:24  nova/sn2005e

To undelete the file, use the -U option:

nersc$ htar -Uf nova.tar nova/sn1993j
HTAR: u  nova/sn1993j
HTAR: HTAR SUCCESSFUL

List the file and note that the 'D' is missing

nersc$ htar -tf nova.tar nova/sn1993j
HTAR: -rwx------  elvis/elvis    9331200 2010-09-24 14:24  nova/sn1993j

htar Archive Verification¶

You can request that htar compute and save checksum values for each member file during archive creation. The checksums are saved in the corresponding htar index file. You can then further request that htar compute checksums of the files as you extract them from the archive and compare the values to what it has stored in the index file.

nersc$ htar -Hcrc -cvf nova.tar nova
HTAR: a   nova/
HTAR: a   nova/sn1987a
HTAR: a   nova/sn1993j
HTAR: a   nova/sn2005e

Now, in another directory, extract the files and request verification

nersc$ htar -Hverify=crc -xvf nova.tar
HTAR: x nova/
HTAR: x nova/sn1987a, 9331200 bytes, 18226 media blocks
HTAR: x nova/sn1993j, 9331200 bytes, 18226 media blocks

htar Limitations¶

htar has several limitations to be aware of:

• Member File Path Length: File path names within an htar aggregate of the form prefix/name are limited to 154 characters for the prefix and 99 characters for the file name. Link names cannot exceed 99 characters.
• Member File Size: The maximum file size the NERSC archive will support is approximately 20 TB. However, we recommend you aim for htar aggregate sizes between 100 GB and 2 TB. Member files within an htar aggregate are limited to approximately 68GB.
• Member File Limit: htar aggregates have a default soft limit of 1,000,000 (1 million) member files. Users can increase this limit to a maximum hard limit of 5,000,000 member files.

You can work around these limitations by using tar and then hsi put to put the tar file into HPSS. If the tarballs will be very large, you can split them up by following the instructions found in the "Avoid Very Large Files" section.

Globus¶

Globus is recommended for transfers between sites (i.e. non-NERSC to NERSC).

To access the HPSS system using Globus, you first need to create a Globus account. Once you've created an account you can log in either with your Globus information or with your NERSC account information.

The NERSC HPSS endpoint is called "NERSC HPSS". You can use the web interface to transfer files. Currently, there is no explicit ordering by tape of file retrievals for Globus.

Best Practices¶

HPSS is intended for long term storage of data that is not frequently accessed.

The best guide for how files should be stored in HPSS is how you might want to retrieve them. If you are backing up against accidental directory deletion / failure, then you would want to store your files in a structure where you use htar to separately bundle up each directory. On the other hand, if you are archiving data files, you might want to bundle things up according to month the data was taken or detector run characteristics, etc. The optimal size for htar bundles is between 100 GB and 2 TB, so you may need to do several htar bundles for each set depending on the size of the data. Other best practices described in this section include:

• Grouping smaller files
• Large Retrievals
• Managing larger files
• Storage space optimization
• Accessing hpss data remotely
• Managing long-running transfers

Group Small Files Together¶

HPSS is optimized for file sizes between 100 GB and 2 TB. If you need to store many files smaller than this, please use htar to bundle them together before archiving. HPSS is a tape system and responds differently than a typical file system. If you upload large numbers of small files they will be spread across dozens or hundreds of tapes, requiring multiple loads into tape drives and positioning the tape. Storing many small files in HPSS without bundling them together will result in extremely long retrieval times for these files and will slow down the HPSS system for all users.

Order Large Retrievals¶

If you are retrieving many (> 100 files) from HPSS, you need to order your retrievals so that all files on a single tape will be retieved in a single pass in the order they are on the tape. NERSC has several scripts to help you generate an ordered list for retrievals with both hsi and htar.

When retrieving large amounts of compressed data all at once where the total size of the tape archived data is >100 Tb and each data chunk is >100 Gb, it is advised to raise a ticket and reach out to the HPSS team in order to receive detailed guidance in performing the retrieval incrementally. It helps ensure that all our users have access to HPSS simultaneously.

Generating A Tape Sorted List¶

The script, generate_sorted_list_for_hpss.py, will generate a list of tape sorted files. This list can be used for htar or hsi to extract the files. For hsi please see the description below for a more advanced script that will also re-create the directory structure you had in HPSS.

To use the script, you first need a list of fully qualified file path names. If you do not already have such a list, you can query HPSS using the following command:

hsi -q 'ls -1 -R <HPSS_files_you_want_to_retrieve>' |& egrep -v '/$|\.idx$' > temp.txt

(the stdout+stderr pipe to grep removes directories and index files from the output, keeping only files). Once you have the list of files, feed it to the sorting script:

generate_sorted_list_for_hpss.py -i temp.txt |grep -v "Generating sorted list from input file" > sorted_list.txt

The file sorted_list.txt will have a sorted list of files to retrieve. If these are htar files, you can extract them with htar into your current directory:

nersc$ awk '{print "htar -xvf",$1}' sorted_list.txt > extract.script
nersc$ chmod u+x extract.script
nersc$ ./extract.script

Ordering hsi Retrievals and Recreating Directory Structure¶

The script, hpss_get_sorted_files.py, will retrieve the files in the proper tape order and also recreate the directory structure the files had in HPSS.

To use the script, you first need a list of fully qualified file path names and/or directory path names. If you do not already have such a list, you can query HPSS using the following command:

hsi -q 'ls -1 -R <HPSS_files_or_directories_you_want_to_retrieve>' |& grep -v '/$' > temp.txt

(the stdout+stderr pipe to grep removes directories from the output, keeping only files). Once you have the list of files, feed it to the sorting script:

hpss_get_sorted_files.py -i temp.txt -o <your_target_directory, default is current directory> -s <strip string, default in NONE>

For files in HPSS under /home/e/elvis/unique_data, you might want to strip off /home/e/elvis from the target directory. You can do that by adding the -s /home/e/elvis flag.

Avoid Very Large Files¶

Files sizes greater than 2 TB can be difficult for HPSS to work with and lead to longer transfer times, increasing the possibility of transfer interruptions. Generally it's best to aim for file sizes in the 100 GB to 2 TB range. You can use tar and split to break up large aggregates or large files into 500 GB sized chunks:

tar cvf - myfiles* | split -d --bytes=500G - my_output_tarname.tar.

This will generate a number of files with names like my_output_tarname.tar.00, my_output_tarname.tar.01, which you can use hsi put to archive into HPSS. When you retrieve these files, you can recombine them with cat:

cat my_output_tarname.tar.* | tar xvf -

Storage Space Optimization¶

When working with a large amount of data, it is a good practice to delete the data from the disk cache as soon as they have been backed up on HPSS. This helps ensure optimal usage of HPSS resources. If individual files larger than 1GB are stored, that are not going to be used in the near future, users are encouraged to reach out and ask for these files to be purged from disk cache. Additionally, users may themselves clear the cache by running the hsi command migrate -f -P <filepath>. The -P will enforce that the disk copy is removed, once the tape copy has been created. Note that the migrate command will hang until migration to tape is completed. To avoid this delay, users may want to wait till the file has been automatically sent to tape by HPSS before running the migrate command. To check if data has been saved at tape level use ls -V <filepath>. Alternatively, as files should be migrated automatically by HPSS in the hours following they are stored in HPSS, issuing the migrate -P command 12 or 24 hours after the file was created could be an easier way to achieve a similar result.

Accessing HPSS Data Remotely¶

Users can access HPSS data remotely using Globus. We recommend a two-stage process to move data to or from HPSS and a remote site. Use Globus to transfer the data between NERSC and the remote site (your scratch or CFS directory would make a useful temporary staging point at NERSC) and use hsi or htar to move the data into HPSS.

Use the Xfer QOS¶

User the dedicated xfer QOS for long-running transfers to / from HPSS. You can also submit jobs to the xfer QOS after your computations are done. The xfer QOS is configured to limit the number of running jobs per user to the same number as the limit of HPSS sessions.

HPSS Usage Charging¶

DOE's Office of Science awards an HPSS quota to each NERSC project every year. Users charge their HPSS space usage to the HPSS projects of which they are members.

Users can check their HPSS usage and quotas with the showquota command on Perlmutter. You view usages on a user level:

nersc$ showquota --hpss
perlmutter login35> showquota --hpss
+-------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|                               File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
+-------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|                                      home |   26.13GiB |    40.00GiB |          65.3% |    390.31K |       1.00M |          39.0% |
|                                  pscratch |   26.13TiB |     6.35PiB |           0.4% |    898.54K |     209.72M |           0.4% |
|  adele usage on HPSS charged to myproject |   31.22TiB |     3.11PiB |           1.0% |          - |           - |              - |
+-------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+

Here, "Space Used" shows you how much data you have stored in HPSS. Data stored in HPSS could potentially be charged to any project that you are a member of (see below for details).

You can also check the HPSS quota for a project by logging in to Iris and clicking on their "Storage" menu.

Apportioning User Charges to Projects: Project Percents¶

The HPSS system has no notion of project-level accounting but only of user accounting. Users must say "after the fact" how to distribute their HPSS data usage to the HPSS projects to which they belong. If a user belongs to only one HPSS project, all usage is charged to that projects. If a user belongs to multiple projects, usage is apportioned among the user's projects. By default this is split based on the size of each project's storage allocation. Users (only the user, not the project managers) can change what percentage of their HPSS usage is charged to which project in their Storage menu in Iris.

Adding or Removing Users¶

If a user is added to a new project or removed from an existing project the project percents for that user are adjusted based on the size of the quotas of the projects to which the user currently belongs. However, if the user has previously changed the default project percents the relative ratios of the previously set project percents are respected.

As an example user u1 belongs to projects r1 and r2 and has changed the project percents from the default of 50% for each project to 40% for r1 and 60% for r2:

If u1 then becomes a new member of project r3 which has a storage allocation of 1,000 GBs the project percents will be adjusted as follows (to preserve the old ratio of 40:60 between r1 and r2 while adding r3 which has the same SRU allocation as r1+r2):

If a project is retired, the percentage charged to that project is spread among the remaining projects while keeping their relative values the same.

HPSS Project Directories¶

A special "project directory" can be created in HPSS for groups of researchers who wish to easily share files. The file in this directory will be readable by all members of a particular Unix file group. This file group can have the same name as the project (in which case all members of the project will have access to the project directory) or a new name can be requested (in which case only those users added to the new file group by the requester will have access to the project directory).

HPSS project directories have the following properties:

• located under /home/projects
• owned by the PI, a PI Proxy, or a Project Manager of the associated project
• have suitable group attribute (include "setgid bit")

To request creation of an HPSS project directory the PI, a PI Proxy or a Project Manager of the requesting project should open a ticket.

Troubleshooting¶

Some frequently encountered issues and how to solve them.

Trouble connecting¶

The first time you try to connect using a NERSC provided client like hsi or htar you will be prompted for your NERSC password + one-time password which will generate a token stored in $HOME/.netrc. This allows you to connect to HPSS without typing a password. However, sometimes this file can become out of date or otherwise corrupted. This generates errors that look like this:

nersc$ hsi
result = -11000, errno = 29
Unable to authenticate user with HPSS.
result = -11000, errno = 9
Unable to setup communication to HPSS...
*** HSI: error opening logging
Error - authentication/initialization failed

qIf this error occurs try moving $HOME/.netrc file to $HOME/.netrc_temp. Then connect to the HPSS system again and enter your NERSC password + one-time password when prompted. A new $HOME/.netrc file will be generated with a new entry/token. If the problem persists contact account support.

Manual Token Generation¶

You can manually generate a token for accessing HPSS by logging to Iris and selecting the blue "Storage" tab. Scroll down to the section labeled "HPSS Tokens" and you will see buttons to generate a token from within NERSC. This button will generate a token which you can paste into a file named .netrc in your home directory:

machine archive.nersc.gov
login <your NERSC user name>
password <token generated by Iris>

The .netrc file should only have user readable permissions. If it's group or world readable HPSS access will fail.

Cannot transfer files using htar¶

htar requires the node you're on to accept incoming connections from its movers. This is not possible from a compute node at NERSC, so htar transfers will fail. Instead we recommend you use our special xfer QOS for data transfers.

Globus transfer errors¶

Globus transfers will fail if you don't have permission to read the source directory or space to write in the target directory. One common mistake is to make the files readable, but forget to make the directory holding them readable. You can check directory permissions with ls -ld. At NERSC you can make sure you have enough space to write in a directory by using the showquota command.