transferring files across firewalls / piping through ssh

Operating System:  All Unix variants

Dependencies/assumptions:  Presence of gzip/gunzip

Problem:   

transferring files across firewalls /

transferring files using ssh tunneling

A problem issue in a tiered, firewalled network architecure, is how to transfer files, especially large files, between servers in different parts of the network.  There are lots of ways to do this - get your friendly network administrator to open up a temporary hole in the firewall, connect up a cable between the two servers, and transfer that way, etc.  Potentially, however, this will break your security procedures, or it might be impossible to get physical access, or your network administrator may not be available, or going through the request process would be too slow, and so on.

It's quite usual, however, for the two networks to be administered via ssh from a third network, or sometimes single ip address.  They cannot speak to each other, but this third location can speak to them both.  So the usual sidestep to the problem is to transfer the file to the third location (which is probably your desktop), then upload it to the new location.  Which is certainly fine for a small file, but when it is many gigabytes (or many hundreds of gigabytes) this may not be feasible, and certainly doubles the transfer time.

So what can be done?  The answer is to use an ssh pipe - transfer the file as a stream from the first server to the desktop, pipe this straight to an ssh connection to its new location.  The script below makes this simple (and also assumes that you'd like to gzip and gunzip the file in the stream, reducing the amount of data to be transferred (though increasing processor load).

#!/bin/ksh



# sshtransfer expects parameters:

# sshtransfer user@server1:file user@server2:file



usage() {

    print "sshtransfer expects user1@server1:file1 user2@server2:file2\n"

    exit 1

    }





testssh() {

    RESULT=failure

    RESULT=`ssh -o Batchmode=yes $1 'echo success'`

    if [[ $RESULT == "success" ]];then

        return 0

    fi

    print "Could not connect to $1 - could be connection or ssh key error.\n"

    return 1

    }



if [[ $# -ne 2 ]];then

    usage

fi



CONNECT1=`echo $1 | awk -F: '{print $1}'`

CONNECT2=`echo $2 | awk -F: '{print $1}'`

FILE1=`echo $1 | awk -F: '{print $2'}`

FILE2=`echo $2 | awk -F: '{print $2'}`



testssh $CONNECT1 && testssh $CONNECT2 && eval ssh -o Batchmode=yes $CONNECT1 \'gzip -c ${FILE1}\' 2\>/dev/null \| ssh -o Batchmode=yes $CONNECT2 \'gunzip \> $FILE2 \' 2>/dev/null



exit 0

Reconstructing user directories from a passwd file

Sometimes I'm asked to set up a box with the same users as another box. The easiest way for me to do this is to copy the /etc/passwd, /etc/group and /etc/shadow files over, and then run the following shell fragment:

for LINE in `cat /etc/passwd|sed s/' '/'SPACE'/g`;do
DIR=`echo $LINE|awk -F: '{print $6}'|sed s/'SPACE'/' '/g`
USER=`echo $LINE|awk -F: '{print $1}'|sed s/'SPACE'/' '/g`
GROUP_NUM=`echo $LINE|awk -F: '{print $4}'`
GROUP=`grep ":${GROUP_NUM}:" /etc/group|awk -F: '{print $1}'`
if [[ ! -d ${DIR} ]];then
echo $DIR doesn\'t exist
echo mkdir ${DIR}
mkdir ${DIR}
echo cp -r /export/home/appman/* ${DIR}
cp -r /export/home/appman/* ${DIR}
echo cp -r /export/home/appman/.profile ${DIR}
cp -r /export/home/appman/.profile ${DIR}
echo chown -R ${USER}:${GROUP} ${DIR}
chown -R ${USER}:${GROUP} ${DIR}
fi
done

How to change the name of a Solaris server

This script was written for Solaris 10, but it checks for the presence of all the files it wants to change and allows you to quit or continue if it doesn't find all of them. It was also written for a T2000, hence the interface name being e1000g0, but you can change that easily.


#Change oldname -> newname in the following files:
#
#/etc/hosts
#/etc/hostname.e1000g0 (or bge0)
#/etc/nodename
#/etc/dumpadm.conf
#/etc/inet/ipnodes
#/etc/inet/hosts
#/etc/mnttab
#/etc/sysidcfg

#Accomplish this with...

OLDNAME=old_name_of_server
NEWNAME=new_name_of_server
INTERFACE=e1000g0 (or bge0 and so on)

for FILE in /etc/hosts /etc/hostname.$INTERFACE /etc/nodename /etc/dumpadm.conf /etc/inet/ipnodes /etc/inet/hosts /etc/mnttab /etc/sysidcfg; do

count=0
if [[ ! -f $FILE ]]; then
echo $FILE not found
$count=$(( $count + 1 ))
fi

if [[ $count -gt 0 ]];then
GO="false"
while [[ $GO == "false" ]];do
print -n 'Continue anyway? y/n'
read ANSWER
if [[ "$ANSWER" == "y" ]] || [[ "$ANSWER" == "Y" ]]; then
echo "Okay."
GO="true"
elsif [[ "$ANSWER" == "n" ]] || [[ "$ANSWER" == "Y" ]]; then
echo "Quitting."
exit 0
else
echo "Sorry?"
fi
done
fi


sed s/$OLDNAME/$NEWNAME/g /etc/hosts > /etc/hosts_new
sed s/$OLDNAME/$NEWNAME/g /etc/hostname.$INTERFACE > /etc/hostname.$INTERFACE_new
sed s/$OLDNAME/$NEWNAME/g /etc/nodename > /etc/nodename_new
sed s/$OLDNAME/$NEWNAME/g /etc/dumpadm.conf > /etc/dumpadm.conf_new
sed s/$OLDNAME/$NEWNAME/g /etc/inet/ipnodes > /etc/inet/ipnodes_new
sed s/$OLDNAME/$NEWNAME/g /etc/inet/hosts > /etc/inet/hosts_new
sed s/$OLDNAME/$NEWNAME/g /etc/mnttab > /etc/mnttab_new
sed s/$OLDNAME/$NEWNAME/g /etc/sysidcfg > /etc/sysidcfg_new


#Make sure you back up the current settings...

cp hosts hosts_orig
cp hostname.$INTERFACE hostname.$INTERFACE_orig
cp nodename nodename_orig
cp dumpadm.conf dumpadm.conf_orig
cp inet/ipnodes inet/ipnodes_orig
cp inet/hosts inet/hosts_orig
cp mnttab mnttab_orig
cp sysidcfg sysidcfg_orig



#Feeling brave?

mv hosts_new hosts
mv hostname.$INTERFACE_new hostname.$INTERFACE
mv nodename_new nodename
mv dumpadm.conf_new dumpadm.conf
mv inet/ipnodes_new /inet/ipnodes
mv inet/hosts_new inet/hosts
mv mnttab_new mnttab
mv sysidcfg_new sysidcfg


#Now set the new name:

uname -S $NEWNAME


#Finally, restart:

shutdown -i 6 -g 0 -y

add ssh key pair for ssh logins without password authentication

If you're trying to write a script that needs to log in to a number of remote servers, and are trying to avoid typing in passwords (or hard-coding passwords in an expect script, or similar) - this could be a lifesaver.

Create a key:

$ ssh-keygen -t dsa -f ~/.ssh/id_dsa -C "you@example.com"
Generating DSA keys: Key generation complete.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~/.ssh/id_dsa
Your public key is:
1024 35 [really long string] you@example.com
Your public key has been saved in ~/.ssh/id_dsa.pub

For some versions of ssh-keygen, you'll need to specify ssh-keygen -d

The -C is an optional comment.

Create a list of servers you want to be able to log into freely and save it to a file called serverlist.  Then you'll be able to run the fragment:

for server in `grep -v "^ *#" serverlist`;do
  echo $server
  cat ~/.ssh/id_dsa.pub | ssh username@$server 'cat - >> ~/.ssh/authorized_keys'
done

You may need to create the directory .ssh on the target server.


Now you should be able to ssh to the servers listed without being prompted for a password.