Many times the only way, or at least the fastest way, to resolve network communications problems is by collecting a trace of the communication and sending it to an expert for analysis. When the expert is not part of your organization this may present security concerns since the trace will contain application data. However, typically what is important are the Ethernet, IP, and TCP, UDP or ICMP protocol headers and not the application data. When that is the case you can either not collect the data in the first place or strip the data after it is has been collected.
Using packet_monitor you can collect all the headers but none of the data with the following command
packet_monitor -interface #INTERFACE -numeric -time_stamp -verbose -pkt_hdr
|
Where INTERFACE is the device name of the IP interface as displayed by the “ifconfig -all” command. The trace will look like
dir icmp type
+ tcp
hh:mm:ss.ttt dir len proto source destination src port ds
+t port type
11:40:21.234 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 78, ID 2212, Flg/Frg 0, TTL 3c, Prtl 6
Cksum 788b, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1332611210, ack 3416994988, window 8192, 80 data bytes, flags Push Ack
+.
X/Off 05, Flags 18, Cksum c3aa, Urg-> 0000
11:40:21.235 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len a8, ID 2213, Flg/Frg 0, TTL 3c, Prtl 6
Cksum 785a, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1332611290, ack 3416994988, window 8192, 128 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum ce9f, Urg-> 0000
11:40:21.236 Rcvd Ether Dst 00:00:a8:43:52:22 Src 00:23:54:52:18:6e Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 28, ID 3032, Flg/Frg 4000, TTL 80, Prtl 6
Cksum e6ba, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq 3416994988, ack 1332611418, window 16176, 0 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum 183c, Urg-> 0000
11:40:21.952 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len a8, ID 2214, Flg/Frg 0, TTL 3c, Prtl 6
Cksum 7859, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1332611418, ack 3416994988, window 8192, 128 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum 59a9, Urg-> 0000
11:40:21.953 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 98, ID 2215, Flg/Frg 0, TTL 3c, Prtl 6
Cksum 7868, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1332611546, ack 3416994988, window 8192, 112 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum 2b4e, Urg-> 0000
|
The problem with this approach is that sometimes the problem turns out to be application related and you need the application data. If you haven’t collected a full trace to start with you have to either reproduce the problem or wait for it to happen again. It is much easier to collect a full trace by adding the “-hex_dump” and “-length 1500” arguments and then stripping the application data from a copy when initially sending the trace out for analysis. The following figure shows the command and trace but in the interest of space I edited the trace to remove most of the application data.
packet_monitor -interface #sdlmux.m16.11-3 -numeric -time_stamp -verbose -pkt_hd
+r -hex_dump -length 1500
dir icmp type
+ tcp
hh:mm:ss.ttt dir len proto source destination src port ds
+t port type
13:52:04.672 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 5dc, ID 9eaa, Flg/Frg 0, TTL 3c, Prtl 6
Cksum f68e, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1335114362, ack 3417069804, window 65535, 1460 data bytes, flags Push A
+ck.
X/Off 05, Flags 18, Cksum 8d33, Urg-> 0000
offset 0 . . . 4 . . . 8 . . . C . . . 0...4... 8...C...
0 22 cd 33 93 25 7b 85 39 7 c4 3b 7e c9 a9 d5 d9 "M3>%{>9 <D;~I)UY
10 63 25 a7 80 6 d7 4f c9 e7 7a 91 1e 4b e7 b7 a5 c%'><WOI gz><Kg7%
20 4f 4c bf 1d 2a 3d 72 53 99 41 b8 c4 26 24 31 4d OL?<*=rS >A8D&$1M
. . .
590 98 bc af 74 d1 71 88 3f 3d 90 22 d3 91 86 92 4e ></tQq>? =>"S>>>N
5a0 da cc d8 7 18 e7 9e 55 c8 f1 af d3 30 0 35 4 ZLX<<g>U Hq/S0 5<
5b0 ac e8 f0 82 ,hp>
13:52:04.692 Rcvd Ether Dst 00:00:a8:43:52:22 Src 00:23:54:52:18:6e Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 28, ID 9324, Flg/Frg 4000, TTL 80, Prtl 6
Cksum 83c8, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq 3417069804, ack 1335120266, window 16384, 0 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum aad4, Urg-> 0000
No tcp data.
13:52:04.692 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 5dc, ID 9eab, Flg/Frg 0, TTL 3c, Prtl 6
Cksum f68d, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1335120266, ack 3417069804, window 65535, 1460 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum 2626, Urg-> 0000
offset 0 . . . 4 . . . 8 . . . C . . . 0...4... 8...C...
0 a0 95 bb 79 99 93 c9 52 ac 11 69 fd d5 a9 39 b8 >;y>>IR ,<i}U)98
10 fd 4 2e ec 3e eb 87 9d 3f 96 a9 91 2e b2 c8 91 }<.l>k>> ?>)>.2H>
20 6a b3 7e 9f cc 79 6f e2 9f 5 c6 a0 e4 95 57 9c j3~>Lyob ><F d>W>
. . .
590 88 d8 2e b6 54 c1 25 95 c4 38 d9 0 55 36 32 58 >X.6TA%> D8Y U62X
5a0 2d ba 81 2c e5 51 8a 3b ef cd 98 29 a1 c2 82 24 -:>,eQ>; oM>)!B>$
5b0 90 33 6e e8 >3nh
13:52:04.694 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 3a4, ID 9eac, Flg/Frg 0, TTL 3c, Prtl 6
Cksum f8c4, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1335121726, ack 3417069804, window 65535, 892 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum d4c2, Urg-> 0000
offset 0 . . . 4 . . . 8 . . . C . . . 0...4... 8...C...
0 23 2a 5a 1a e3 34 e7 b4 62 d7 ee 55 7c 38 f7 a #*Z<c4g4 bWnU|8w<
10 cb b6 95 4 6 d8 b8 e 7d 88 68 a7 24 7a ed bd K6><<X8< }>h'$zm=
20 57 ce 14 43 6c 17 56 5a 25 7d 9b f5 88 d9 97 29 WN<Cl<VZ %}>u>Y>)
. . .
350 8 a3 86 6 24 bc cc b9 d6 3f af ab f bd 38 ca <#><$<L9 V?/+<=8J
360 da b5 6 8a bf 2b 49 90 a2 d4 27 f5 79 a1 9 1a Z5<>?+I> "T'uy!<<
370 5c 87 6b ae f0 d2 e8 45 14 b3 12 b5 >k.pRhE <3<5
13:52:04.695 Rcvd Ether Dst 00:00:a8:43:52:22 Src 00:23:54:52:18:6e Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 58, ID 9325, Flg/Frg 4000, TTL 80, Prtl 6
Cksum 8397, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq 3417069804, ack 1335120266, window 16384, 48 data bytes, flags Push Ack
+.
X/Off 05, Flags 18, Cksum dde1, Urg-> 0000
offset 0 . . . 4 . . . 8 . . . C . . . 0...4... 8...C...
0 84 2c 64 69 88 5d c5 b2 bc 6c ca 4e af 15 be 2e >,di>]E2 <lJN/<>.
10 3f 87 93 79 a0 b7 5d d5 3c 35 7f 2d db 7e be 44 ?>>y 7]U <5<-[~>D
20 a8 24 6d 96 6f f0 79 c1 d6 9c c3 be 64 4b 7d 4c ($m>opyA V>C>dK}L
|
There are several ways to strip the application data from the trace. You can manually edit the trace; for short traces this might be possible but for longer traces it is infeasible. I have previously published two Perl scripts that can be used. The first pm21line.pl is designed to put all the headers on 1 line and strip out the application data. The script uses IO indication so it must be run under the bash environment.
bash
bash-2.05$ perl pm21line.pl < full_trace.out > 1line_trace.out
bash-2.05$ exit
exit
ready 17:21:00
d 1line_trace.out%phx_vos#m16_mas>SysAdmin>Noah_Davids>1line_trace.out 10-10-1
+0 17:21:08 mst
13:52:04.672 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP) IP Ver/HL 45, ToS 0, Len 5dc, ID 9eaa, Flg/Frg 0, TTL 3c, Prtl 6
+TCP from 164.152.77.128.22 to 164.152.77.50.6991 seq 1335114362, ack 34170
+69804, window 65535, 1460 data bytes, flags Push Ack.
13:52:04.692 Rcvd Ether Dst 00:00:a8:43:52:22 Src 00:23:54:52:18:6e Type 0800
+(IP) IP Ver/HL 45, ToS 0, Len 28, ID 9324, Flg/Frg 4000, TTL 80, Prtl 6
+TCP from 164.152.77.50.6991 to 164.152.77.128.22 seq 3417069804, ack 13351
+20266, window 16384, 0 data bytes, flags Ack.
13:52:04.692 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP) IP Ver/HL 45, ToS 0, Len 5dc, ID 9eab, Flg/Frg 0, TTL 3c, Prtl 6
+TCP from 164.152.77.128.22 to 164.152.77.50.6991 seq 1335120266, ack 34170
+69804, window 65535, 1460 data bytes, flags Ack.
13:52:04.694 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP) IP Ver/HL 45, ToS 0, Len 3a4, ID 9eac, Flg/Frg 0, TTL 3c, Prtl 6
+TCP from 164.152.77.128.22 to 164.152.77.50.6991 seq 1335121726, ack 34170
+69804, window 65535, 892 data bytes, flags Push Ack.
13:52:04.695 Rcvd Ether Dst 00:00:a8:43:52:22 Src 00:23:54:52:18:6e Type 0800
+(IP) IP Ver/HL 45, ToS 0, Len 58, ID 9325, Flg/Frg 4000, TTL 80, Prtl 6
+TCP from 164.152.77.50.6991 to 164.152.77.128.22 seq 3417069804, ack 13351
+20266, window 16384, 48 data bytes, flags Push Ack.
|
The one line format is useful when the trace contains packets from multiple connections and you are interested in only a subset of those connections. After creating the one line format you can easily filter on the unique features of the connections you are interested in.
The second script match.pl displays a file and allows you to match on multiple strings. For example
perl match.pl -file full_trace.out -match Ether -match IP -match Cksum -match TC
+P -match seq -dots
****************************** full_trace.out ******************************
. . .
13:52:04.672 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 5dc, ID 9eaa, Flg/Frg 0, TTL 3c, Prtl 6
Cksum f68e, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1335114362, ack 3417069804, window 65535, 1460 data bytes, flags Push A
+ck.
X/Off 05, Flags 18, Cksum 8d33, Urg-> 0000
. . .
13:52:04.692 Rcvd Ether Dst 00:00:a8:43:52:22 Src 00:23:54:52:18:6e Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 28, ID 9324, Flg/Frg 4000, TTL 80, Prtl 6
Cksum 83c8, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq 3417069804, ack 1335120266, window 16384, 0 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum aad4, Urg-> 0000
. . .
13:52:04.692 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 5dc, ID 9eab, Flg/Frg 0, TTL 3c, Prtl 6
Cksum f68d, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1335120266, ack 3417069804, window 65535, 1460 data bytes, flags Ack.
X/Off 05, Flags 10, Cksum 2626, Urg-> 0000
. . .
13:52:04.694 Xmit Ether Dst 00:23:54:52:18:6e Src 00:00:a8:43:52:22 Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 3a4, ID 9eac, Flg/Frg 0, TTL 3c, Prtl 6
Cksum f8c4, Src a4984d80, Dst a4984d32
TCP from 164.152.77.128.22 to 164.152.77.50.6991
seq 1335121726, ack 3417069804, window 65535, 892 data bytes, flags Push Ac
+k.
X/Off 05, Flags 18, Cksum d4c2, Urg-> 0000
. . .
13:52:04.695 Rcvd Ether Dst 00:00:a8:43:52:22 Src 00:23:54:52:18:6e Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 58, ID 9325, Flg/Frg 4000, TTL 80, Prtl 6
Cksum 8397, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq 3417069804, ack 1335120266, window 16384, 48 data bytes, flags Push Ack
+.
X/Off 05, Flags 18, Cksum dde1, Urg-> 0000
. . .
|
Sometimes the data is not the only thing that should be kept secret. The other identifying parts of the trace are the IP addresses and port numbers. If you wish to hide your internal IP addresses you will need to make the changes manually using the global search and replace function in your favorite editor. However, the IP addresses appear in four places in each packet. The first two are in hex format and the second two are in dotted decimal notation, or possibly as a name if you did not use the -numeric argument (I recommend that you always use -numeric, it faster since name resolution doesn’t happen and it makes the network relationships obvious). The port number or name only appears in 2 places.
13:52:04.695 Rcvd Ether Dst 00:00:a8:43:52:22 Src 00:23:54:52:18:6e Type 0800
+(IP)
IP Ver/HL 45, ToS 0, Len 58, ID 9325, Flg/Frg 4000, TTL 80, Prtl 6
Cksum 8397, Src a4984d32, Dst a4984d80
TCP from 164.152.77.50.6991 to 164.152.77.128.22
seq 3417069804, ack 1335120266, window 16384, 48 data bytes, flags Push Ack
+.
X/Off 05, Flags 18, Cksum dde1, Urg-> 0000
|
Keep in mind that changing the IP addresses is not quite as simple as picking an arbitrary number, you need to maintain the relationships between addresses. Any two addresses that are on the same subnet or network before the change must be on the same subnet or network after the change, similarly any two addresses that are on different subnets or networks before the change must be on different subnets or networks after the change. Changing the relationship can have profound effects on the interpretation of the trace.