Secure Shell Transfer

Secure Shell Transfer
— print (last updated: Jun 4, 2009) — print

Personalize this online document by providing a login relevant to you. Replace the generic TAZLOGIN by the actual:

Secure Transfer

There are a number of Linux tools by which one can transfer files directly using the secure socket layer. With any of these methods the network traffic is encrypted.

scp: secure copy
rsync: robust, efficient secure copy
sftp: secure FTP client
gftp: graphical "Gnome FTP" client using the SSH2 protocol choice

The sftp behaves like a simple command-line FTP client, but with very poor features compared to the more sophisticated FTP clients ncftp and lftp, and so is generally not used in itself.

The scp command acts like cp except that you include the source or target servers along with login information in the command syntax. Using taz as the target machine, the command

$ scp file TAZLOGIN@taz:
$ scp -r dir TAZLOGIN@taz:

copies file and dir, respectively, from your machine to your taz home directory. As can be expected, the "TAZLOGIN@" is unnecessary if your login is the same as your taz login. The ":" is a critical piece of syntax in the above commands because scp will copy files on the same computer (with unexpected results) if it is missing.

Going the other way, we can do

$ scp TAZLOGIN@taz:file .
$ scp -r TAZLOGIN@taz:dir .

to copy file and dir, respectively, from your taz home directory to your current directory. One important difference between scp and cp is that there is no option which you can use to warn you about overwriting a preexisting file, so be careful. This is one reason why rsync is often a better alternative.

The rsync command is a copy command like scp with a number of enhancements.

gFTP via secure shell

The gftp executable is available on the GNOME desktop at

Applications Internet gFTP.

If it's not there, install it:

$ sudo apt-get install gftp

Use gftp from the command line or from the Applications menu. To use it from your machine to connect to, say taz, enter taz in the Host field and your TAZLOGIN in the User field, and choose SSH2 from the protocol drop-down on the far right.

Connect by clicking the icon at the top left. Once you're connected, save the connection information by bookmarking it, namely

Bookmarks Add Bookmark.

rsync

The rsync command is a very efficient means for copying which minimizes the amount of data transmission. You should refer to the on-line manual for rsync:

$ man rsync

The rsync command is a generalized copy command like scp; however, rsync offers many enhancements over scp such as these:

The ability to copy only "what is necessary" using an algorithm based on file differences, with the implication that the actual amount of data transferred for a network copy is minimal.
The ability to further reduce the amount of transferred traffic by compressing/decompressing it using the "-z" command-line flag.
The ability to copy based on file time-stamps, so that "newer" files aren't overwritten by "older" files when the "-u" command-line option is present.

The rsync, cp and scp commands all support the -r (recursive) option which is needed used to copy a directory. An enhancement over the -r option is the -a option which attempts to make an exact copy, preserving file timestamps, symbolic links, etc. This -a option usage is available in the cp command, but not in scp. One common use of

$ cp -a dir1 dir2

is to ensure that symbolic links are preserved. In contrast,

$ cp -r dir1 dir2

would treat a symbolic link within dir1 as the file it points to, and make a copy of the file.

rsync by example

The prog1 test directory discussed in Program 1 also can be used for running some tests to illustrate the behavior of rsync. If you haven't already done so, obtain the prog1 directory like this:

$ cd ~
$ wget ftp://ftp.cs.wcupa.edu/pub/rkline/gradlinux/prog1.zip
$ unzip prog1.zip
$ cd prog1

Our experiments will test rsync in order to illustrate various options. When you key in an rsync command it's convenient to combine the options like this:

$ rsync -aunv source/  dest/

But keep in mind that these options can be separated, like this:

$ rsync -a -u -nv source/  dest/

The starting point of the experiments is to create the file structure by doing this:

$ ./init.sh
Creating source and dest directories

You can see what is created by doing:

$ ls -RF source dest

revealing the following:

source/
  a.txt  b.java  c.txt  
  D/  
    E/  
      e.txt
    f.txt
  D.conf

dest/
  D.java	 D.txt

Basic rsync operations

Within prog1, run these commands

command	description
./init.sh	initialize
rsync -aunv source/ dest/	dry run update
rsync -aunv dest/ source/	in reverse
rsync -auv source/ dest/	for real
ls dest/	you sent it across
rsync -auv source/ dest/	nothing new to do
./modify.sh	modify these files: source/ a.txt, D.conf dest/ b.java, c.txt, D/f.txt
rsync -aunv source/ dest/	(dry run) send changes in `source`
rsync -anv source/ dest/	no `-u` option (dry run) changes in `source` & `dest`
rsync -aunv --delete source/ dest/	(dry run) `delete` dest files not in `source`

One observation we can make is that the command:

rsync -a --delete source/ dest/

will make the contents of dest exactly the same as that of source.

rsync to taz

command	description
scp -r prog1 TAZLOGIN@taz:	send `prog1` over via scp
rsync -av prog1/ TAZLOGIN@taz:prog1/	indicates transfer, only because of timestamps
rsync -av prog1/ TAZLOGIN@taz:prog1/	repeat: now nothing happens

Exclusions/Inclusions

The rsync command has a fairly complicated way of refining which files should participate in the copy command using these options:

--exclude=PATTERN       exclude files matching PATTERN
--exclude-from=FILE     read exclude patterns from FILE
--include=PATTERN       don't exclude files matching PATTERN
--include-from=FILE     read include patterns from FILE
--files-from=FILE       read list of source-file names from FILE

We will focus on only one of these:

--exclude-from=FILE     read exclude patterns from FILE

The FILE which rsync refers to can contain lines with glob patterns common to other commands, i.e., they use the special characters *, ?, etc. For example an exclude file may be this:

a.txt
*.java

The pattern notation also permits the usage of "/" means that the file name must match the pattern from the top of the directory. Thus the exclude file:

*.txt

would exclude all the .txt files, whereas

/*.txt

would exclude only the top level .txt files. Patterns can also employ the ** token, meaning to "cross directories" when "/" is in use.

Thus

D/*.txt

would exclude only the .txt files in D, but not in its subdirectories, and

D/**.txt

would exclude all .txt files in D or any of its subdirectories.

Patterns can also specify inclusion and exclusion more directly by prefixing with + and -, respectively:

+ include-pattern
- exclude-pattern

The prog1 directory provides several pattern files which you can use to test.

command	description
./init.sh	initialize
rsync -aunv source/ dest/	run it for reference
rsync -aunv --exclude-from=E source/ dest/	`E` is an empty file, no difference
rsync -aunv --exclude-from=X source/ dest/	`X`:	+ .conf -
rsync -aunv --exclude-from=Y source/ dest/	`Y`:	- D - *.java
rsync -aunv --exclude-from=Z1 source/ dest/	`Z1`:	+ D - *
rsync -aunv --exclude-from=Z2 source/ dest/	`Z2`:	+ D + D/* - *
rsync -aunv --exclude-from=Z3 source/ dest/	`Z3`:	+ D + D/** - *