Search Search

Transferring Files

How to transfer files using SCP, SFTP, Fuse and S3

Using SSH/SCP

Why use SSH/SCP/SFTP for file transfer?

SCP and SFTP both run over ssh and are thus encrypted. There are implementations available for all common operating systems including Linux, Windows, and Mac OS X.

Windows

GUI:
  • WinSCP
    • Host: login.hpc.caltech.edu
    • Enter your username and password.
  • FileZilla
  • Command Line:
  • Linux

  • Command Line:
    • Start Terminal (Applications->Accessories->Terminal)
      • To copy files from your computer to the central cluster
        • Type scp local_filename username@login.hpc.caltech.edu:~/username/
      • To copy files from to the central cluster to your computer
        • Type scp username@login.hpc.caltech.edu:/home/username/remote_filename .
  • Mac OS X

    Command Line:
    • Start Terminal (Applications->Utilities->Terminal)
      • To copy files from your computer to the central cluster
        • Type scp local_filename username@login.hpc.caltech.edu:~/username/
      • To copy files from the central cluster to your computer
        • Type scp username@login.hpc.caltech.edu:~/username/remote_filename

    SSHFS on Mac OS X

    If you prefer filesystem like access you may use FuseOS together with SSHFS. This works over SSH protocol and is therefore encrypted as with standard SSH/SCP/SFTP but with the added benefit of drag and drop transfers. 

    • Download and install FUSE and SSHFS here.
    • Make a local mount directory on your Mac. mkdir ~/Desktop/HPC-Mount
    • Run a command similar to the following, swapping out your username and directory name.
    •  sshfs -o allow_other,defer_permissions,auto_cache remote-username@login.hpc.caltech.edu:/home/remote-username ~/Desktop/HPC-Mount

    GUI:
    • Cyberduck.
      • Download
      • Cyberduck can be made to work with 2 factor
        • Click on "Open Connection"
        • choose "SFTP"
        • enter you username and password, then click connect
        • In the "Provide additional login credentials" box, enter 1 in the password field and hit enter if using the smartphone app.
        • You should be prompted on you cell phone to allow the connection
        • If using a yubikey, you can touch it when prompted to complete the login.

    Using Amazon S3

    If your data is in Amazon S3 you may use the awscli tools which are already installed as a module on the cluster. 

    • Log into the cluster and run module load awscli/1.15.27
    • Type aws configure and enter your Amazon Web Services API key and private key. (You generate these in the IAM credential page in the AWS console).
    • Run a command similar to the following to copy data from S3 to your cluster home directory. 
    • aws s3 cp --recursive s3://my-bucket-name/subfolder/ ~/destination-directory/
    • Run a command similar to the following to copy data from the cluster to a pre-existing S3 bucket.
    • aws s3 cp --recursive ~/source-directory/ s3://my-bucket-name/subfolder/
    • More s3 examples are available here.