skip to content
 

Disk Space in Maths

This is a brief guide to disk space in Maths e.g. the different types of storage which are available and how to keep your usage from inconveniencing other people, with links to more detailed information.

Classes of storage

Storage class Description
Home directories Very Reliable for valuable data. Quota'd, backed up regularly (more than once a day), archived and mirrored.
Store Reliable for valuable data. No quota. Mirrored regularly to provide limited backups (2 weeks of). No archiving. /store/GROUP/CRSID/
Data Reliable Scratch Storage for longer term storage of data. Dedicated disks, no quotas, not backed up, unlikely to be lost/deleted unless there is a hardware fault. Some of these may be stored on reliable/redundant servers (e.g. /data/septal).
Scratch local access only, no quotas, not backed up, VERY likely to be lost when a machine is upgraded, re-installed or if there is a hardware fault. Mainly intended for short term use by jobs running on the machine where the scratch is local.
tmp no quotas, not backed up, files are removed after a period, or on reboot (depending on the system). Intended for smaller very short-lived files.

Windows users: your N: drive is a pointer to your home directory, and Desktop and My Documents have been redirected to subdirectories of your N: drive, so files will usually be saved somewhere in your home directory. Your C: drive, however, is equivalent to scratch space. It is not backed up and will be wiped if your PC has to be reinstalled.

What can go wrong?

If a disk becomes full, users with files and directories on that disk will be unable to continue working on their files. If this happens the COs may need to take emergency action, either by deleting any files which are growing without limit (usually output from a faulty user program_ or by deleting very large files belonging to anyone found to be using more than their fair share of the disk. One reason we have quotas is to reduce the risk of this happening to the home directory disks.

If the backup space fills up unexpectedly, then future backups may fail until the Computer Officers have fixed the problem. To help prevent this, you may wish to keep larger files in a directory called NOBACKUP. Only do this with data which is backed up elsewhere or can easily be regenerated e.g. experimental data which was downloaded from the Internet.

How much can I use?

The default home directory quota is 3GB. If you think you need an increase, please email help@maths explaining why.

There are no strict rules about the amount of Data or Scratch disk space usage on the public workstations - you will normally be able to use as much space as you need for your work. However, if you need more than 100 GBytes of long-term storage you should discuss your needs with the Computer Officers. If you need very large amounts of space, i.e. many hundreds of GBytes, you should seek funding to provide the necessary resources within your own research group.

Checking your usage

To check your overall usage, use the quota command, or quota -v to check your usage on scratch and data spaces as well as your home directory.

To obtain a sorted list of disk space used by each file and directory, type

du -sk * .??* | sort -n

More details about finding your large files and directories are on the quota page.

If you suspect that a disk is filling up, cd to the troublesome directory and type

df .

The dot means "current directory" and if you forget it, you will be told the amount of free space on all mounted disks/partitions, not just the one you were interested in.

Cutting down

Emptying the Trash is the first thing to try if you are running out of quota. If the graphical interface does not work then delete it manually with rm -rf .Trash.

The quota page has more on reducing your disk usage.

NOBACKUP directories

With the constraints of quotas disk space is no longer at a premium, and users may feel justified in keeping fairly large temporary datasets in their home directories rather than in the scratch or data areas.

For example, a user might transfer large datasets of experimental data from an archive for ease of processing (assuming they have enough quota). This may not cause problems for other users, but will cause problems with the backup system if the backup space (or tapes) fill unexpectedly.

Such data need not be backed up locally as it can always be retrieved from the archive (where it is backed up). The solution is to store the data in a directory called NOBACKUP (upper case) anywhere in your home directory.

If you prefer to retain an existing directory structure for your files you can create soft links to files in the NOBACKUP directory:

mv big.temp.file ~/NOBACKUP
ln -s ~/NOBACKUP/big.temp.file big.temp.file 

Please don't use hard links as the file may then be backed up (possibly many times).