Transferring data with Globus

Installing your own endpoint and moving large datasets

If you are unsure about any of these instructions email help@sbgrid.org for assistance.

Globus is a managed file transfer service for moving large datasets between collections (also called endpoints). It restarts after network drops, verifies checksums, and emails you when the job finishes. For moving data inside the SBGrid network, or over ssh and rsync, see Moving data instead.

• NOTE: Globus can only see collections, not filesystem paths. If a directory such as /emdata or /standby doesn't appear in Globus, it is not a permissions problem — no collection on that machine is exposing it yet. You create that collection yourself with Globus Connect Personal (GCP).

1. Log in

Go to app.globus.org and sign in with your institutional account. Most universities and hospitals are Globus subscribers. If yours is not listed, use a Globus ID, Google, or ORCID.

2. Install GCP on the machine holding the data

GCP has to run on the machine where the data is mounted locally — your microscope workstation, a lab server, or a cluster login node. If you have previously run scp, rsync, or Aspera from a particular machine to move this data, that is the one.

No root access is needed. Downloads for all platforms are at globus.org/globus-connect-personal.

Linux

cd ~
curl -LO https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
tar xzf globusconnectpersonal-latest.tgz
cd globusconnectpersonal-*
./globusconnectpersonal -setup
./globusconnectpersonal -start &

The -setup step prints a URL. Open it in a browser, log in, name the collection something recognizable, and paste the resulting setup key back into the terminal.

macOS

1. Install the .dmg and launch Globus Connect Personal from Applications.
2. Click Log In and grant the requested consents.
3. Give the collection a recognizable name when prompted.
4. The Globus icon in the menu bar means it is running.

3. Add the paths you need

This is the step people skip. By default GCP exposes only your home directory. Everything else stays invisible until you add it.

On Linux, edit ~/.globusonline/lta/config-paths. One line per path, in the form <path>,<sharing 0|1>,<read-write 0|1>:

~/,0,1
/emdata/,0,1
/standby/,0,0

Edits are not picked up while GCP is running, so restart it:

./globusconnectpersonal -stop
./globusconnectpersonal -start &

On macOS, use the menu bar icon → Preferences… → Access tab → + to add a folder. Untick Writable for read-only paths. Changes take effect immediately.

4. Start the transfer

In File Manager, put your new collection on one side and the destination on the other, select what you want to move, and click Start.

Leave verify file integrity enabled — that checksum pass is the reason to use Globus rather than scp. If you are resuming an interrupted copy, tick sync — only transfer new or changed files.

Globus emails you when it finishes. Progress and per-file errors are under Activity.

5. Keep it running

The transfer only flows while the GCP process is alive. You do not need the GUI or a terminal open, just don't kill the process. On Linux, ./globusconnectpersonal -start & works fine headless. On macOS, leave the app running and stop the machine from sleeping.

Troubleshooting

Symptom	Fix
Collection shows offline	GCP isn't running. Check ./globusconnectpersonal -status, or the macOS menu bar.
Only your home directory is visible	Path not added, or GCP not restarted after editing config-paths. See step 3.
Path visible but not writable	The read-write flag is 0, or the underlying Unix permissions deny it.
Transfer stalls at 0 B/s	Outbound TCP 50000–51000 is blocked. Contact your network administrators.
Setup key rejected	Keys are single-use and expire. Generate a new one at app.globus.org/collections/gcp.

Directories of many small files take far longer than the same volume in a few large ones. Expect the file count, not the byte count, to set the duration.