Rackham storage nearly full
The 6PB large storage system, called Crex, is nearly fully allocated. This is the storage system attached to Rackham and Snowy. When we have allocated all the space available on it, we cannot both grant new projects that need disk space and prolong old ones with large allocations. Please read on to learn more about the situation.
Please review the data stored in your projects and:
- Delete unnecessary data
- Compress uncompressed data (Link removed)
- Move inactive data to another storage
- Contact UPPMAX Support and ask for a reduction in your allocation
Remember that SNIC storage systems are for holding data that is actively being analysed. If you don’t expect to look at a dataset within three months, please remove it from the system to make room for more urgent needs.
Every project matter in this, but not equally. In October 2019, there were 269 PI’s with allocations of over 1 TB, 90 PI’s with allocations of 10 TB, and 11 PI’s with allocations of over 100 TB. Together, the 179 1-10 TB projects together account for 582 TB of space, the 79 10-100 TB projects for 2008 TB of space, the 11 100-1000 TB projects for 2663 TB of space. So we see that the projects with over 10 TB of quota have a great potential to conserve space.
Roughly 20% of the allocated disk space is not used by the projects. This is necessary elbow room not only for your daily activities but also for the storage system itself. Storage systems don’t perform well when they are full, they become sluggish and unresponsive. We absolutely want to avoid poor performance due to an overburdened system. This is why we will never allocate more space than what physically exists on Crex.
At the end of October, about 5.4 PB of storage was allocated in total. The total size of the Crex storage system is about 6 PB, but this includes space for reference databases and the like, so there is really only 300 TB of space left. From what we have seen historically, we believe that the community can save over 500 TB of space.
If we succeed in this, we will have enough space to be able to approve new projects and expand existing projects until the expected end of life of Rackham. If we fail to produce any free space, we must begin to deny new projects, extensions, and expansions for space reasons in early 2020.
It will be helpful to you in the long run if you act in a structured fashion. Begin by creating a plan for your data, including a description of which data you need to keep at UPPMAX, which data you can move away, and which data you can delete. In this plan, you can see what your needs are in different stages of the project and track where your data is at all times.
Please reach out to us at support@uppmax.uu.se if you need help moving data (e.g. using rsync) or any other technical problems. We are open to helping out with other questions and concerns as well.
UPPMAX also offers the possibility of paid-for storage for inactive or less active data. This is warm storage (mounted on Rackham), includes off-site backup, and is less expensive than other options (e.g. Amazon Web Services and other commercial providers). If you are interested in purchasing space on our off-load storage solution please see the following page: https://www.uppmax.uu.se/uppmax-news/?tarContentId=8146996 .