SSH Access and Transfer

The ssh client

The ssh executable established secure-shell access from client to server. The student taz machine and your machine can act in both client or server role. A difference is that taz is accessible from the outside world, whereas your machine is accessible only within the firewall of which taz is a member.

We're going to set up some basic configuration features of SSH to allow easy access between your machine and taz.

From your machine to taz

Assuming your machine is one of the lab machines, you can log in to your taz account from this machine as follows:
[MACHINE]$ ssh TAZ_LOGIN@taz
TAZ_LOGIN@taz's password:
[taz]$
From anywhere else, you must use the fully qualified name:
$ ssh TAZ_LOGIN@taz.cs.wcupa.edu
The reason we can use the short name is due to the Search Domains entry of cs.wcupa.edu made in the network configuration.

Cryptographic Key

Open another shell on your machine and create your public/private key pair by doing:
[MACHINE]$ ssh-keygen
This will prompt for a pass-phrase (empty), and a place to store the public key (the default). Therefore, hit "Enter" three times. View the results by typing:
[MACHINE]$ ls ~/.ssh
id_rsa                this is the private RSA key
id_rsa.pub            this is the public RSA key

[MACHINE]$ cat ~/.ssh/id_rsa
[MACHINE]$ cat ~/.ssh/id_rsa.pub
Send the public key to taz:
[MACHINE]$ scp ~/.ssh/id_rsa.pub TAZ_LOGIN@taz:
On the other taz shell, add the key to the right config file:
[taz]$ mkdir -p ~/.ssh
[taz]$ cat id_rsa.pub >> ~/.ssh/authorized_keys
Then test it with a shell on your machine:
[MACHINE]$ ssh TAZ_LOGIN@taz
You should get in "for free" by virtue of the cryptographic key.

The SSH config file

On your machine, create and edit the SSH config file:
[MACHINE]$ nano ~/.ssh/config
add these lines:
Host taz
  Hostname taz.cs.wcupa.edu
  User TAZ_LOGIN
Now you can get into taz from your machine without giving the taz login.
[MACHINE]$ ssh taz
This will also work from an external computer running ssh, such as MAC, Linux, or Windows/Cygwin.

Running commands through SSH

Instead of logging in and running commands on taz, you can run the command directly, like this:
[MACHINE]$ ssh taz ls
If the system you are on supports the X window system , you can run your Ubuntu GUI applications remotely on your client system, typically by using the "-X" option. For example, students in Dr. Wyatt's classes could run emacs remotely:
[MACHINE]$ ssh -X taz emacs
This will usually work well within a LAN, but at Internet speeds the connection can be very slow. The Remote Desktop is a better alternative.

From taz to your lab machine

Repeat the steps in the last two subsections, except this time going the other way:
  1. on taz, create an RSA public/private key pair
  2. from your machine, retrieve the public RSA key file ~/.ssh/id_rsa.pub from taz (you should be able to do it without password)
  3. append the RSA public key onto your machine's ~/.ssh/authorized_keys file.
Then, on taz create the .ssh/config file:
Host MACHINE 
  User LOGIN

ssh-based transfer

There are a number of UNIX/Linux tools which can be used to transfer files directly using the secure socket layer. With any of these methods the network traffic is encrypted. From this point on, let's assume you've established the "easy access to taz" via cryptographic key and config file usage.

scp

The scp command acts like cp except that you include the source or target servers along with login information in the command syntax. Using taz as the target machine, the command
$ scp file taz:
$ scp -r dir taz:
copies file and dir, respectively, from your machine to your taz home directory. As can be expected, the "TAZ_LOGIN@taz" is necessary without the ssh config file defined as we have done so. The ":" is a critical piece of syntax in the above commands because scp will copy files on the same computer if it is missing. Going the other way, we can do
$ scp taz:file .
$ scp -r taz:dir .
These operations copy a file or dir, respectively, from your taz home directory to your current directory. One important difference between scp and cp is that there is no option which you can use to warn you about overwriting a preexisting file, so be careful.

recursive and exact copies

The rsync, cp and scp commands all support the -r (recursive) option to recursively copy a directory. An enhancement over the -r option is the -a option which attempts to make an exact copy, preserving file timestamps, symbolic links, ownership, etc. The -a option usage is available in the local cp command. If we compare:
$ cp -a dir1/ dir2/
and
$ cp -r dir1/ dir2/

The former will preserve symlinks whereas the latter would treat a symbolic link within dir1 as the file it points to, and copy the file. The -a option is available in the rsync command, but not for scp.

rsync

The rsync command is a complex network-based copy command like scp, but with many enhancements. You should refer to man rsync for details.

This command provides a very efficient means for copying which minimizes the amount of data transmission. These are some highlights:
  1. The ability to copy only "what is necessary" using an algorithm based on file differences, with the implication that the actual amount of data transferred for a network copy is minimal.
  2. The ability to further reduce the amount of transferred traffic by compressing/decompressing with "-z" command-line flag.
  3. The ability to copy based on file time-stamps, so that "newer" files aren't overwritten by "older" files when the "-u" command-line option is present.
  4. The ability to refine the files transferred using inclusion/exclusion rules.
A common usage is something like this with the -a, -u, -n and -v command-line options:
$ rsync -a -u -n -v DIRECTORY1/  DIRECTORY2/
When you actually key in this command, it's convenient to compress the options like this:
$ rsync -aunv DIRECTORY1/ DIRECTORY2/

The terminal "/" for rsync directory copies

A subtle issue which appears in rsync is the terminal "/" in the source directory. In particular, the following means copy the directory dir1 into the target directory dir2:
$ rsync -r dir1 dir2
In contrast, the following means copy the contents of the directory dir1 into the target directory dir2:
$ rsync -r dir1/ dir2

rsync usage examples

The following experiments are meant to illustrate rsync used with various options. Download the source archive rsync_samples.zip. For definiteness and simplicity extract it into your home directory. Open a shell and do
$ unzip ~/Downloads/rsync_samples.zip
$ cd rsync_samples
If you want to view the contents of any file, simple print it, e.g.,
$ cat setup1
Run the commands as you see them on the left and confirm the descriptions on the right.
commanddescription
./setup1
ls -RF source




ls -RF dest

source/
  a.txt  b.java  c.txt  
  D/  
    E/  
      e.txt
    f.txt
  D.conf

dest/
  D.java	 D.txt
rsync -aunv source/ dest/ (dry run) update
rsync -aunv dest/ source/ (dry run) update in reverse
rsync -aunv --delete source/ dest/ (dry run) update and delete dest files not in source
rsync -aunv --delete dest/ source/ (dry run) update and delete source files not in dest
ls dest/ see what's in dest
rsync -auv source/ dest/ update for real
ls dest/ check that the files were sent
rsync -auv source/ dest/ do it again: nothing new to do
./setup2 This script runs
./setup1
rsync -a source/ dest/
and then modifies these files:
source/
  a.txt, D.conf
dest/
  b.java, c.txt, D/f.txt
rsync -aunv source/ dest/ (dry run) send later changes in source
rsync -anv source/ dest/ no -u option
(dry run) send all changes in source

observations

  1. The folders used in the rsync command always terminate with a "/" — this is important. It means we are, in some form, copying the folder contents, not the folder itself.
  2. The "-a" option sets the target files' timestamps to that of the source, thus multiple runs with no changes in the folder contents will have no effect.
  3. The "-n" option is called a dry run and suppresses the actual completion of the operation. It should always be used in conjunction with the "-v" (verbose) option so that you get a report of what would happen without committing to it.
  4. The "--delete" deletes files in the target which are not in the source. Thus this command:
    rsync -a --delete source/ dest/
    
    will make the contents of dest exactly the same as that of source.

rsync to taz

The real importance of rsync is in its ability to do the same actions across a network connection. Like the scp command, to create a network transfer, we preface either the source or the target with this syntax:
login@host:
Here are some commands which test the transfer to taz:
commanddescription
cd
rsync -anv rsync_samples/ taz:rsync_samples/

transfer dry run
rsync -av rsync_samples/ taz:rsync_samples/ do the transfer
rsync -av rsync_samples/ taz:rsync_samples/ rsync -av rsync_samples/ taz:rsync_samples/ repeat it, confirm nothing new
Observe that the rsync command will actually create the target directory, when it doesn't exist — up to one level only.

exclusions/inclusions

The rsync command has a fairly complicated way of refining which files should participate in the copy command using these options:
--exclude=PATTERN       exclude files matching PATTERN
--exclude-from=FILE     read exclude patterns from FILE
--include=PATTERN       don't exclude files matching PATTERN
--include-from=FILE     read include patterns from FILE
--files-from=FILE       read list of source files names from FILE
We will focus on only one of these:
--exclude-from=FILE     read exclude patterns from FILE
The FILE which rsync refers to can contain lines with glob patterns common to other commands, i.e., they use the special characters *, ?, etc. For example an exclude file may be this:
a.txt
*.java
To make the exclusion/inclusion more explicit, patterns use the prefixes with + and -, respectively:
+ include-pattern
- exclude-pattern
If these prefixes are used, there is no difference between --include-from and --exclude-from specification files. One has to be careful to place the + and - characters left justified in the line with exactly one blank between the character and the file name.

The pattern notation also permits the usage of an initial "/" which means that the file name must match the pattern from the top of the directory. Thus the exclude file:
- *.txt
would exclude all the .txt files, whereas
- /*.txt
would exclude only the top level .txt files. Patterns can also employ the ** token, meaning to recursively enter all subdirectories. Thus
- D/*.txt
would exclude only the .txt files in D, but not in its subdirectories, and
- D/**.txt
would exclude all .txt files in D or any of its subdirectories.

The rsync_samples directory provides several pattern files which you can use to test.
commanddescription
cd rsync_samples
./setup1
ls -RF source






ls -RF dest

source/
  a.txt  b.java  c.txt  
  D/  
    E/  
      e.txt
    f.txt
  D.conf
  
dest/
  D.java	 D.txt
rsync -aunv source/ dest/ run it for reference
rsync -aunv --exclude-from=X source/ dest/ X:
+ *.conf
- *
rsync -aunv --exclude-from=Y source/ dest/ Y:
- D
- *.java
rsync -aunv --exclude-from=Z1 source/ dest/ Z1:
+ D
- *
rsync -aunv --exclude-from=Z2 source/ dest/ Z2:
+ D
+ D/*
- *
rsync -aunv --exclude-from=Z3 source/ dest/ Z3:
+ D
+ D/**
- *

Connect To Server

The "Connect to Server" feature offers an easy way to make remote files accessible to a local server. We'll assume you've installed your cryptographic key on taz and that you've made an entry in the .ssh/config file.

Look in the Places for Connect to Server and key in the information shown in the image.

Replace TAZ_LOGIN by TAZ_LOGIN.

The User Name and Password fields are unnecessary if you have the necessary setup.

You can close it by right-clicking and selecting Unmount.

FUSE Mount

An alternative SSH-type remote file system mounting can be done as follows. First install:
$ sudo apt-get install sshfs
Then create a mount point:
$ mkdir ~/TAZ
Mount your taz home directory onto this mount point by:
$ sshfs taz:/home/TAZ_LOGIN ~/TAZ
You can then see the contents of your taz home directory by:
$ ls ~/TAZ
To unmount this file system, do:
$ fusermount -u ~/TAZ
You cannot close it by right-clicking and selecting Unmount (on Linux).

Access through taz tunnel

Assuming you have activated the "taz tunnel" as described in the Ubuntu Desktop Installation document, you can use this to secure-copy files from your client machine, through taz, to you target machine, MACHINE.

Windows and other systems provide a number of GUI Secure-copy alternatives, possibly the best being WinSCP for Windows.

scp via a shell

For simplicity, we'll show a shell approach which would be viable for MAC/Linux/Windows+Cygwin systems. We'll assume that 2002 is the port forwarded to 22 on MACHINE.

Edit the ~/.ssh/config file, assumed to have the "tunnel" entry created and add a new entry.

~/.ssh/config
...
Host MACHINE_tunnel
  User TAZ_LOGIN
  Host taz.cs.wcupa.edu
  LocalForward 2002 MACHINE:22
  ...
 
# new entry:
Host MACHINE
  Hostname localhost
  User LOGIN
  Port 2002
With this in place, you can do
$ scp FILE_ON_CLIENT MACHINE:
# or,
$ scp -r DIRECTORY_ON_CIENT MACHINE:

FUSE Mount from MAC

I find the FUSE mount from my MAC at home to be very convenient. There's no need to install anything. If you want to mount directly to taz, you don't need the tunnel setup. Create, as indicated above, a mount target:
$ mkdir ~/TAZ
Then mount your taz home directory onto your local machine:
$ sshfs taz:/home/TAZ_LOGIN  ~/TAZ
To remove the mount use the umount command (different than on Linux):
$ umount ~/TAZ
To do the same thing to your lab machine, activate the "taz tunnel" with ~/.ssh/config entries as above. Create a dedicated directory on your MAC:
$ mkdir ~/MACHINE 
Mount your lab machine's home directory onto this by:
$ sshfs bashir:/home/LOGIN ~/MACHINE
Again, to remove the mount use the umount command:
$ umount ~/MACHINE


© Robert M. Kline