Automating File Transfers With SFTP

In my last blog I talked about automating file transfers using FTP. There are three issues with using FTP. First, your password is sent across the network in clear text making it available to anyone with a protocol analyzer. Second, your data is also sent in clear text. Third, you have to record your password somewhere on the module, either in the macro or in the .netrc file. Because of these issues many sites have mandated or are thinking of mandating that SFTP, the secure FTP subsystem of SSH, be used instead of FTP. However, you just cannot replace ftp with sftp in your command macro (figure 1) and expect it to work (figure 2).

&attach_input

sftp  172.16.1.116

nd

MYPASSWORD

put foo

&if (command_status) ^= 226 &then &goto ERROR1

get bar

&if (command_status) ^= 226 &then &goto ERROR2

quit

&

&label ERROR1

..display_line Could not put file

quit

&return

&

&label ERROR2

..display_line Could not get file

quit

&return

Figure 1 – sftp1.cm – replacing the ftp command with sftp in a command macro

sftp1

Connecting to 172.16.1.116...

dup2: Bad  file number.

Error on line 3 of sftp1.

command_processor: Object not found. nd. In command  macro

%phx_vos#m15_mas>SysAdmin>Noah_Davids>sftp1.cm

ready  12:39:59

Figure 2 – running sftp1.cm

The problem is an incompatibility between the way that native VOS commands and POSIX open source treat input. However, SFTP, like FTP allows you to create a file with a list of requests (figure 3) which it will automagically run (figure 4).

put foo

get bar

Figure 3 – sftp2.input – file containing requests for sftp to execute

sftp -b sftp2_input [email protected]

[email protected]'s  password:

sftp> put foo

Uploading foo to /SysAdmin/Noah_Davids/foo

sftp> get bar

Couldn't stat remote file: No such file or directory

File "/SysAdmin/Noah_Davids/bar" not found.

Ready  13:06:45

Figure 4 – execution with an input file of requests and a password prompt

You’ll notice in figure 4 that sftp is prompting for a password. There is no way to put the password in the input file. The ONLY way to bypass the password prompt is to use public key authentication (figure 5).

sftp -b sftp2_input [email protected]

sftp> put foo

Uploading foo to /SysAdmin/Noah_Davids/foo

sftp> get bar

Couldn't stat remote file: No such file or directory

File "/SysAdmin/Noah_Davids/bar" not found.

Ready  13:12:28

Figure 5 – execution with an input file of requests, using public key authentication to eliminate the password prompt

You can find an article on how to set up public key authentication here – http://members.cox.net/ndav1/self_published/publickey_authentication_setup.html

Unfortunately, sftp will just execute all the requests in the input file one after another, there is no mechanism for testing if the transfer worked or not.

Since command macros cannot be used are there any alternatives? The gnu_tools product ships with a program called expect. It can be used to send commands, wait for any number of different responds and do something based on what it sees in the output stream. I am no expect expert but the script in figure 6 will help get you started. If you search the web for “expect scripts” you will find many references that will help you customize my simple example.

# If we get an end-of-file (eof) it means that  the sftp process

# terminated. Report the last command that was sent.

#

proc goteof {cmd} {

puts "sftp connection unexpectedly closed n"

puts "last text received wasn"

puts "<<"

puts $cmd

puts ">>nnn"

exit

}

# This procedure does regular expression matches looking for key strings
# in the output collected from the command that was execured. In this case
# I am just reporting the type of error but other things can be done
# as well. I also only check for 2 errors. There are others. You will have
# to add them as you find them.
#
proc checkforerrors {buf cmd} {
if [regexp {.*not found} $buf] {
puts “$cmd FAILED : not foundnnn”
return 1
}
if [regexp {.*Permission denied} $buf] {
puts “$cmd FAILED : access problemsnnn”
return 1
}
return 0
}# set the timeout to -1 so there is no timeout. The default is 10 seconds
# and most file transfers take longer than that. I decided to set no
# timeout, you can change that.
set timeout -1

# start sftp
spawn sftp [email protected]

# wait for the sftp prompt but if we get an authentication prompt, ending in
# yes/no report a security issue and exit.
expect {
“yes/no?” { puts
“nnnWe have connected to the wrong server or the server has been reloaded.”
puts
“The server key must be validated before this script can be run again.nnn”
exit }
sftp>
}

# change to a convient directory for testing
send “cd sftp_testr”
expect {
eof { goteof “cd sftp_test” }
sftp>
}

# call check for errors passing it all collected characters up the
# sftp> prompt. Also the command that was sent to sftp. If the
# checkforerrors procedure returns 1 exit the script. Again, you can
# do other things.
if {[checkforerrors $expect_out(buffer) “cd sftp_test”] == 1} { exit }

send “put foor”
expect {
eof { goteof “put foo” }
sftp>
}
if {[checkforerrors $expect_out(buffer) “put foo”] == 1} { exit }

send “get barr”
expect {
eof { goteof “get bar” }
sftp>
}
if {[checkforerrors $expect_out(buffer) “get bar”] == 1} { exit }

send “quitr”
expect eof
puts “script donenn”

Figure 6 – sftp3.exp – expect script to automate SFTP

expect sftp3.exp

spawn sftp [email protected]

Connecting to 172.16.1.116...

The authenticity of host 172.16.1.116(172.16.1.116)' can't  be established

+.

RSA key fingerprint is  37:f4:1a:56:64:af:ab:8a:7c:0b:36:47:c5:6c:1d:1a.

Are you sure you want to continue connecting (yes/no)?

We have connected to the wrong server or the server has been reloaded.
The server key must be validated before this script can be run again.ready 10:41:10

Figure 7 – execution of sftp3.exp expect script with security warning

expect sftp3.exp

spawn sftp [email protected]

Connecting to 172.16.1.116...

sftp> cd sftp_test

sftp> put foo

Uploading foo to /SysAdmin/Noah_Davids/sftp_test/foo

sftp> get bar
Couldn’t stat remote file: No such file or directory
File “/SysAdmin/Noah_Davids/sftp_test/bar” not found.
sftp> get bar FAILED : not foundready 10:42:11

Figure 8 – execution of sftp3.exp expect script without warning but wiuth a failed transfer

Finally, in my FTP blog I mentioned that FTP was able to read files that were still open and this on occasion has resulted in incomplete files being transferred, I suggested that if the FTP macro waits for a file to appear at a location and then transfers it, the macro should check to be sure that it is no longer locked. The same problem can occur with SFTP and the solution is the same. You can place the file lock check in a command macro (figure 9) and then when the file is no longer locked call expect with the appropriate script.

&label AGAIN

&if (exists new_file) = 1 & (locked new_file) = 0  &then &goto EXPECT

display_line new_file not ready for transfer as of (time)

sleep -seconds 15

&goto AGAIN

&

&

&label EXPECT

expect sftp3.exp

Figure 9 – command macro testing for file lock before calling expect

Automating File Transfers With SFTP

PARTNERS

TOPICS

QUICK LINKS