Lots of people have created automated processes to transfer files using FTP. There are several different ways to do this, some better than others. This blog will discuss the different approaches and identify the advantages and disadvantages of each. It will also cover an issue which can result in incomplete files being transferred.
The simplest approach is to put your user ID, password and all your FTP requests into a command macro (figure 1) and just execute it (figure 2).
&attach_input |
Figure 1 – ftp1.cm – simple command macro
ftp1 |
Figure 2 – ftp1.cm – execution of simple command macro
The draw back of this approach is that your user ID and more importantly your password are in clear text as part of the macro. Anyone who has access to the macro can see it. The second approach is to use the .netrc file to hold your user ID and password. (figure 3). The command macro (figure 4) then just includes the FTP requests to execute (figure 5).
machine 172.16.1.116 |
Figure 3 – .netrc file
&attach_input |
Figure 4 – ftp2.cm – command macro without user ID and password
ftp2Connected to 172.16.1.116. |
Figure 5 – ftp2.cm – executing command macro without user ID and password
The .netrc file works only if the owner is the only one with access to the file (figure 6). If anyone else has read access (figure 7) it will not work and a password prompt is displayed (figure 8).
display_access .netrc -all%phx_vos#m15_mas>SysAdmin>Noah_Davids>.netrc w Noah_Davids.* |
Figure 6 – correct ACLs on .netrc file
display_access .netrc -all%phx_vos#m15_mas>SysAdmin>Noah_Davids>.netrc w Noah_Davids.* |
Figure 7 – incorrect ACLs on .netrc file
ftp2 |
Figure 8 – password prompt when .netrc has the wrong ACLs
You can place all of the commands in the .netrc file by creating an FTP macro. If the macro is named “init” (figure 9) it is automagically executed after login, This makes the VOS command macro just 1 line long (figure 10), the ftp command. The major problem with this approach is there is no error recovery; you cannot check the status of any of the FTP requests with the (command_status) function (figure 11).
machine 172.16.1.116 |
Figure 9 – .netrc file with init macro
ftp 172.16.1.116 |
Figure 10 – ftp3.cm – macro when used with .netrc file containing init FTP macro
ftp3 |
Figure 11 – ftp3.cm – output when using init macro – no error recovery
Many macros that I have seen just loop waiting for a file to appear in a directory, when it does the macro uses FTP to transfer it (figure 12).
&attach_input&label AGAIN |
Figure 12 – ftp4.cm – macro waiting for a file to appear and then transfer
The problem with this approach is that FTP can read a file that is still open and being written. For a large file or if your timing is just bad the result will be that only part of the file will be transferred (figure 13).
1234567890 |
Figure 13 – only part of the file is transfered
1234567890 |
Figure 14 – complete file
It is not enough to check to see if the file exists, you must also check to see if the file is locked (figure 15).
&attach_input |
Figure 15 – ftp5.cm – checking that a file is unlocked before transferring it
So the best approach is to use the .netrc file to hold your user ID and password and a VOS command macro with the ability to check the command_status after each request to control the actual requests. Also before transferring a file check to make sure that it is not locked.
Some of you are wondering about SFTP; automating file transfers with SFTP is completely different and will be discussed in my next blog.