transferring files across firewalls / piping through ssh

Operating System:  All Unix variants

Dependencies/assumptions:  Presence of gzip/gunzip

Problem:   

transferring files across firewalls /

transferring files using ssh tunneling

A problem issue in a tiered, firewalled network architecure, is how to transfer files, especially large files, between servers in different parts of the network.  There are lots of ways to do this - get your friendly network administrator to open up a temporary hole in the firewall, connect up a cable between the two servers, and transfer that way, etc.  Potentially, however, this will break your security procedures, or it might be impossible to get physical access, or your network administrator may not be available, or going through the request process would be too slow, and so on.

It's quite usual, however, for the two networks to be administered via ssh from a third network, or sometimes single ip address.  They cannot speak to each other, but this third location can speak to them both.  So the usual sidestep to the problem is to transfer the file to the third location (which is probably your desktop), then upload it to the new location.  Which is certainly fine for a small file, but when it is many gigabytes (or many hundreds of gigabytes) this may not be feasible, and certainly doubles the transfer time.

So what can be done?  The answer is to use an ssh pipe - transfer the file as a stream from the first server to the desktop, pipe this straight to an ssh connection to its new location.  The script below makes this simple (and also assumes that you'd like to gzip and gunzip the file in the stream, reducing the amount of data to be transferred (though increasing processor load).

#!/bin/ksh



# sshtransfer expects parameters:

# sshtransfer user@server1:file user@server2:file



usage() {

    print "sshtransfer expects user1@server1:file1 user2@server2:file2\n"

    exit 1

    }





testssh() {

    RESULT=failure

    RESULT=`ssh -o Batchmode=yes $1 'echo success'`

    if [[ $RESULT == "success" ]];then

        return 0

    fi

    print "Could not connect to $1 - could be connection or ssh key error.\n"

    return 1

    }



if [[ $# -ne 2 ]];then

    usage

fi



CONNECT1=`echo $1 | awk -F: '{print $1}'`

CONNECT2=`echo $2 | awk -F: '{print $1}'`

FILE1=`echo $1 | awk -F: '{print $2'}`

FILE2=`echo $2 | awk -F: '{print $2'}`



testssh $CONNECT1 && testssh $CONNECT2 && eval ssh -o Batchmode=yes $CONNECT1 \'gzip -c ${FILE1}\' 2\>/dev/null \| ssh -o Batchmode=yes $CONNECT2 \'gunzip \> $FILE2 \' 2>/dev/null



exit 0