SFTP is the FTP subsystem that is part of SSH, it allows you to do encrypted file transfers. In the past week I have had 2 issues about SFTP not properly transferring files. In one case OpenVOS was the source of the file and in the other case OpenVOS was receiving the file. In both cases the files contained ASCII data and the problems had to do with the differences between how Microsoft Windows and OpenVOS terminate lines in a text file.
Let’s start with OpenVOS as the source. If you create a text file using your favorite editor you create a sequential file. For example
d test %phx_vos#m16_mas>SysAdmin>Noah_Davids>test 10-09-05 08:13:49 mst 1234567890
|
Figure 1 – display of a text file
The display_file_status command shows that it is a sequential file and that while the data byte count is 10 there are actually 20 bytes in the file.
display_file_status testname: %phx_vos#m16_mas>SysAdmin>Noah_Davids>testfile organization: sequential file. . . next byte: 20blocks used: 1. . . record count: 2data byte count: 10
|
Figure 2 – display_file status
The dump_file command shows the actual structure of the file. Each “line” is preceded and terminated with 2 bytes that indicate its length. These length bytes have to start on an even byte boundary so there is also a filler byte. The significant thing to notice is that there are no typical line termination characters, either a line feed (0x0A) or a carriage return line feed (0x0D0A) sequence.
dump_file test%phx_vos#m16_mas>SysAdmin>Noah_Davids>test 10-09-05 08:14:07 mstBlock number 1000 00053132 333435FF 00050005 36373839 |..12345…..6789|010 30FF0005 FFFFFFFF FFFFFFFF FFFFFFFF |0……………|020 FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |…………….| = FF0 FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |…………….| ready 08:14:07 |
Figure 3 – dump_file showing the actual structure of the file
When you transfer the file using SFTP, OpenVOS strips the length bytes from the line and appends a line feed character. The result is that while the original file contained 10 data types the transferred file contains 12.
C:Documents and SettingsnoahMy Documentstemp>"C:Program FilesPuTTYpsftp" [email protected]Using username "nd".[email protected]'s password:Remote working directory is /SysAdmin/Noah_Davidspsftp> get test test.txtremote:/SysAdmin/Noah_Davids/test => local:test.txtpsftp> quit C:Documents and SettingsnoahMy Documentstemp>dir Volume in drive C has no label. Volume Serial Number is 38B1-9C13 Directory of C:Documents and SettingsnoahMy Documentstempblog - sftp 08/27/2010 01:50 PM <DIR> .08/27/2010 01:50 PM <DIR> ..08/27/2010 01:50 PM 12 test.txt 1 File(s) 12 bytes 2 Dir(s) 39,471,644,672 bytes free |
Figure 4 – Transferred file length doesn’t match source file length
In addition, if you display the file in something like Notepad, it displays as one line with funny symbols where the lines should terminate.
Figure 5 – display of file in NotepadIf you use a utility to get a hex dump of the file you will see that at the end of each line there is a line feed character (0x0A). The standard Microsoft Windows line termination sequence is carriage return line feed (0x0D0A), Notepad doesn’t know how to interpret just the line feed character, although there are other editors that will display the file correctly.
Figure 6 – Hex dump of the transferred file
The result is that the transferred file appears to be corrupt, the length is different and you cannot display it – at least in Notepad. Converting the file to a stream file before transferring it will correct the length issue; both files will now show the same length but the line termination issue will remain.
Now what if OpenVOS is receiving a file from a Microsoft Windows System? As I already said, Microsoft Windows uses a carriage return line feed sequence to terminate a line.
Figure 7 – Hex dump of a file created on a Microsoft Windows systemThe OpenVOS SFTP subsystem will create a stream file. The line feed characters will be taken as line termination characters and the carriage return characters will be treated as part of the line’s data.
|
%phx_vos#m16_mas>SysAdmin>Noah_Davids>pc1.txt 10-09-05 08:52:21 mst 000 61626364 650D0A66 6768696A 0D0AFFFF |abcde..fghij….| 010 FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |…………….| = FF0 FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |…………….| ready 08:52:21 |
Figure 8 – dump file of the transferred file
The display command will display the file correctly.
|
%phx_vos#m16_mas>SysAdmin>Noah_Davids>pc1.txt 10-09-05 08:54:25 mst abcde fghij ready 08:54:25 |
Figure 9 – display of the transferred file
But when you go to actually read the file the carriage return character will show up at the end of every line.
test_system_calls tsc: s$attach_port p pc1.txt tsc: s$opentsc: s$seq_read pBuffer length = 600000000 61626364 650D |abcde. |tsc: s$seq_read pBuffer length = 600000000 66676869 6A0D |fghij. |
|
Figure 10 – carriage return characters show up when you read the file
Again, the result is that the file appears to be corrupt.
The following Perl script can be used to add or remove the carriage return characters. You want to add them before transferring the file to a Microsoft Windows system and remove them after transferring the file from the Microsoft Windows system.
# cr.pl begins here## cr# version 1.0 10-08-27#use strict;use warnings;use Getopt::Long; my ($inFile, $outFile, @files, $add, $remove);my ($INFILE);my ($result, $count, $verbose, $addremove); $result = GetOptions ('in=s' => $inFile, 'out=s' => $outFile, 'add' => $add, 'remove' => $remove, 'verbose=s' => $verbose);if (($result != 1) || !defined ($inFile) || !defined ($outFile)) { print "nnUsage:n"; print "perl cr.pl -in PATH -out PATH [[-add] | [-remove]] [-verbose]}n"; exit; } if (defined ($add) && defined ($remove)) { print "You can only specify -add or -remove not bothnn"; print "nnUsage:n"; print "perl cr.pl -in PATH -out PATH [[-add] | [-remove]] [-verbose]}n"; exit; } @files = glob ($inFile);if (@files < 1) {print "nNo files found for " . $inFile . "nn";}if (@files > 1) {print "nMore than 1 file found for " . $inFile . "nn";} open (OUT, ">".$outFile) || die "Can't open output file " . $outFile . "nn";open ($INFILE, $files[0]) || die "Can't open input file " . $files[0] . "nn"; if (!defined ($verbose)) { $verbose = -1; } $count = 0;while ($_ = <$INFILE>) { if (defined ($remove)) { s/r//; print OUT $_ ; } else { s/n//; print OUT $_ . "rn"; } $count++; if (($verbose > 0) && ($count % $verbose) == 0) { print "Line " . $count . " of " . $files[0] . " processedn"; } } close $INFILE;#
|
Figure 11 – Perl script to remove or add carriage return characters
Why does FTP seem to work? FTP has two modes, ASCII and binary. In ASCII mode transfers from a Microsoft Windows system to OpenVOS result in a sequential file with the carriage return and line feed characters both removed from the records. Binary mode transfers from the Microsoft Windows system to Open VOS result in a stream file with the carriage return and line feed characters present – exactly like the binary mode transfer of SFTP. ASCII mode transfers from OpenVOS to a Microsoft Windows system result in a file with carriage return and line feed characters at the end of each line while binary mode transfers result in a file with just the line feed character at the end of each line. Again just like binary mode transfer of SFTP. The FTP RFC goes into detail about how to terminate lines but the bottom line is that in ASCII mode both client and server are free to terminate the lines as necessary for the local operating system while in binary mode client and server have no such discretion.