Moving data

Basic guidelines and resources

If you are unsure about any of these instructions email help@sbgrid.org for assistance.

Internally

Overview

Generally, systems located in the EM Facility and server rooms are interconnected at 10GBe. Desktop systems are connected at 1GBe. Due to many factors, it is unlikely you will hit 10Gbe speeds, but your transfer will happen much faster on systems connected at 10Gbe.

rsync or mv?

It is safest to use rsync when moving between filesystems.

If moving between folders on the same filesystem it is safe to use the 'mv' command

Externally

transfer.sbgrid.org is a robust system that is set up for being a high speed transfer node. It can be used as an ssh and rsyncd destination. It can be reached internally by connection with just the name 'transfer. ie ssh oconnor@transfer

Overview

should already have external-ssh

rsync through ssh

rsync my source to my destination

rsyncd for paswordless deposit location

This is ideal where we may ask outside people to initiate the data transfer. Typically this
is for for directly transferring large data sets to your GPU storage from data collected from umass, janelia or somewhere else via rsycnd.

An example command would look like - rsync -av mydatafolder rsync://rsyncuser@transfer.sbgrid.org/umass-deposit-mylabname

contact help@sbgrid.org to set up a destination for directly