Lots of people have created automated processes to transfer files using FTP. There are several different ways to do this, some better than others. This blog will discuss the different approaches and identify the advantages and disadvantages of each. It will also cover an issue which can result in incomplete files being transferred.
The simplest approach is to put your user ID, password and all your FTP requests into a command macro (figure 1) and just execute it (figure 2).
&attach_input
ftp 172.16.1.116
Noah_Davids
MYPASSWORD
put foo
&if (command_status) ^= 226 &then &goto ERROR1
get bar
&if (command_status) ^= 226 &then &goto ERROR2
quit
&
&label ERROR1
..display_line Could not put file
quit
&return
&
&label ERROR2
..display_line Could not get file
quit
&return
|
Figure 1 – ftp1.cm – simple command macro
ftp1
Connected to 172.16.1.116.
220 phx_vos-m16 FTP server (FTP 1.0 for Stratus STCP) ready. (Compatible with OS
+ TCP/IP)
User (172.16.1.116:Noah_Davids): 331 Password required for nd.
Password:
230 User Noah_Davids.CAC logged in.
200 PORT command successful.
150 Opening data connection for foo (172.16.1.34,49253)0.
226 Transfer complete.
80 bytes sent in 0.01 seconds (6.85 Kbytes/sec)
200 PORT command successful.
550 bar: No such file or directory.
recvrequest: Bad file number.
Get transfer did not complete
Could not get file
221 Goodbye. |
Figure 2 – ftp1.cm – execution of simple command macro
The draw back of this approach is that your user ID and more importantly your password are in clear text as part of the macro. Anyone who has access to the macro can see it. The second approach is to use the .netrc file to hold your user ID and password. (figure 3). The command macro (figure 4) then just includes the FTP requests to execute (figure 5).
machine 172.16.1.116
login Noah_Davids
password MYPASSWORD |
Figure 3 – .netrc file
&attach_input
ftp 172.16.1.116
put foo
&if (command_status) ^= 226 &then &goto ERROR1
get bar
&if (command_status) ^= 226 &then &goto ERROR2
quit
&
&label ERROR1
..display_line Could not put file
quit
&return
&
&label ERROR2
..display_line Could not get file
quit
&return |
Figure 4 – ftp2.cm – command macro without user ID and password
ftp2Connected to 172.16.1.116.
220 phx_vos-m16 FTP server (FTP 1.0 for Stratus STCP) ready. (Compatible with OS
+ TCP/IP)
331 Password required for nd.
230 User Noah_Davids.CAC logged in.
200 PORT command successful.
150 Opening data connection for foo (172.16.1.34,49256)0.
226 Transfer complete.
80 bytes sent in 0.04 seconds (2.09 Kbytes/sec)
200 PORT command successful.
550 bar: No such file or directory.
recvrequest: Bad file number.
Get transfer did not complete
Could not get file
221 Goodbye. |
Figure 5 – ftp2.cm – executing command macro without user ID and password
The .netrc file works only if the owner is the only one with access to the file (figure 6). If anyone else has read access (figure 7) it will not work and a password prompt is displayed (figure 8).
display_access .netrc -all%phx_vos#m15_mas>SysAdmin>Noah_Davids>.netrc w Noah_Davids.*
n *.* |
Figure 6 – correct ACLs on .netrc file
display_access .netrc -all%phx_vos#m15_mas>SysAdmin>Noah_Davids>.netrc w Noah_Davids.*
r *.CAC
n *.* |
Figure 7 – incorrect ACLs on .netrc file
ftp2
Connected to 172.16.1.116.
220 phx_vos-m16 FTP server (FTP 1.0 for Stratus STCP) ready. (Compatible with OS
+ TCP/IP)
User (172.16.1.116:Noah_Davids): 331 Password required for put foo.
Password:
530 Login incorrect.
Login failed.
221 Goodbye. |
Figure 8 – password prompt when .netrc has the wrong ACLs
You can place all of the commands in the .netrc file by creating an FTP macro. If the macro is named “init” (figure 9) it is automagically executed after login, This makes the VOS command macro just 1 line long (figure 10), the ftp command. The major problem with this approach is there is no error recovery; you cannot check the status of any of the FTP requests with the (command_status) function (figure 11).
machine 172.16.1.116
login Noah_Davids
password MYPASSWORD
macdef init
put foo
get bar
quit |
Figure 9 – .netrc file with init macro
ftp 172.16.1.116 |
Figure 10 – ftp3.cm – macro when used with .netrc file containing init FTP macro
ftp3
Connected to 172.16.1.116.
220 phx_vos-m16 FTP server (FTP 1.0 for Stratus STCP) ready. (Compatible with OS
+ TCP/IP)
331 Password required for nd.
230 User Noah_Davids.CAC logged in.
put foo
200 PORT command successful.
150 Opening data connection for foo (172.16.1.34,49276)0.
226 Transfer complete.
80 bytes sent in 0.01 seconds (9.52 Kbytes/sec)
get bar
200 PORT command successful.
550 bar: No such file or directory.
recvrequest: Bad file number.
Get transfer did not complete
quit
221 Goodbye. |
Figure 11 – ftp3.cm – output when using init macro – no error recovery
Many macros that I have seen just loop waiting for a file to appear in a directory, when it does the macro uses FTP to transfer it (figure 12).
&attach_input&label AGAIN
&if (exists new_file) = 1 &then &goto FTP
display_line No new_file as of (time)
sleep -seconds 15
&goto AGAIN
&
&
&label FTP
ftp 172.16.1.116
put new_file
&if (command_status) = 226 &then ..display_line OK
&else &do
..display_line
..display_line ERROR
..display_line
&end
quit |
Figure 12 – ftp4.cm – macro waiting for a file to appear and then transfer
The problem with this approach is that FTP can read a file that is still open and being written. For a large file or if your timing is just bad the result will be that only part of the file will be transferred (figure 13).
1234567890
abcdefghij
1234567890
abcdefghij
1234567890
abcdefghij
1234567890
abcdefghij
1234567890
abcdefghij |
Figure 13 – only part of the file is transfered
1234567890
abcdefghij
1234567890
abcdefghij
1234567890
abcdefghij
1234567890
abcdefghij
1234567890
abcdefghij
this is the last record |
Figure 14 – complete file
It is not enough to check to see if the file exists, you must also check to see if the file is locked (figure 15).
&attach_input
&label AGAIN
&if (exists new_file) = 1 & (locked new_file) = 0 &then &goto FTP
display_line new_file not ready for transfer as of (time)
sleep -seconds 15
&goto AGAIN
&
&
&label FTP
ftp 172.16.1.116
put new_file
&if (command_status) = 226 &then ..display_line OK
&else &do
..display_line
..display_line ERROR
..display_line
&end
quit |
Figure 15 – ftp5.cm – checking that a file is unlocked before transferring it
So the best approach is to use the .netrc file to hold your user ID and password and a VOS command macro with the ability to check the command_status after each request to control the actual requests. Also before transferring a file check to make sure that it is not locked.
Some of you are wondering about SFTP; automating file transfers with SFTP is completely different and will be discussed in my next blog.