NIH HPC News & Announcements
Call for beta testers for new HPC object storage system
Date: 21 July 2016 10:07:19
From: Tim Miller
Do you...
- need more storage space on biowulf?
- have at least 1 TB of data that does not change but needs to be
read repeatedly?
- want to help the HPC staff explore a new method for storing data?
If you answered "yes" to all three of these questions, then becoming a
beta tester of a new Biowulf object storage system may be for you!
About the Biowulf Object Store
------------------------------
What is object storage? It's a technology used by companies such as
Amazon, Apple, and Microsoft to store data for Web and cloud
applications. Object storage can store very large amounts of data that
doesn't change frequently (i.e. data that is written once and read many
times). However, it is organized very differently from traditional
computer storage. For example, there is no inherent concept of
directories or a filesystem hierarchy - objects are instead stored in a
flat namespace and retrieved via a unique identifier.
Biowulf object storage is provided to support computational workloads
that use a specific data access pattern in which data is written rarely but
read often. It is not intended to be used as an archive.
What You Get by Becoming a Beta Tester
--------------------------------------
1. Access to 10-100 TB of space on the object storage system. This space
will remain allocated to you after the beta period is over and may be
expanded (probably via a mechanism similar to how current data directory
quota increase requests are handled).
2. Assistance from the HPC staff in using the object storage, including
help on determining which workloads are appropriate and strategies for
making effective use of the system and integrating it into your specific
workflow.
What We Expect from Beta Testers
--------------------------------
1. Actually use the storage that you are allocated.
2. Provide feedback to the staff regarding performance, ease of use, and
suitability for your computational workflow.
How to Become a Beta Tester
---------------------------
1. Decide how you want to access the object store.
Object storage generally cannot be accessed the same way as traditional
disk-based storage systems like your /home and /data directories on
Biowulf. There are two ways to store and retrieve data on it.
Option 1: The HPC staff has purchased gateways that allow us to make the
object storage look like a disk based filesystem. However, it is still
object storage and will not perform as well as the /home or /data disks.
Also, due to technical limits of the gateways, we can only allow a few
larger groups with shared data space to use this mechanism. The
gateways will only be accessible from a few systems and not from the
Biowulf cluster nodes.
Option 2: The storage system can be accessed via a programming interface
identical to that used by commercial cloud storage (e.g. Amazon's S3). The
HPC staff has developed simple utilities for storing and retrieving data
using this interface. Users who are comfortable doing so may also write
their own programs or scripts to store and retrieve data. The programming
interface will be accessible from the head nodes and any cluster node.
Note that the two access mechanisms are mutually exclusive, i.e. data
written through the programming interface cannot be read via the gateways
and vice versa.
2. Decide how much space you would like on the object store
We're initially targeting Biowulf users and existing groups that could
use 10-100 TB on the new system. Please estimate how much space you
could reasonably use on the object storage using the same procedure as
you would use to request a /data directory quota increase.
3. Send a very brief note to staff@hpc.nih.gov describing in 3-5 sentences
how you plan to use the object storage, which method you would use to
access it, and how much space you would like. Please include how you
determined how much space you're asking for.
[Last 12 months of HPC announcements]