Someone sent me the following trace and asked if it represented retransmissions because the “packet numbers” were duplicated
T 0536 TCP host.subdomain.domain.com host2.sub2.dom2.com 49957 4000 A T 0536 TCP host.subdomain.domain.com host2.sub2.dom2.com 49957 4000 A T 0536 TCP host.subdomain.domain.com host2.sub2.dom2.com 49957 4000 A T 0536 TCP host.subdomain.domain.com host2.sub2.dom2.com 49957 4000 A T 0536 TCP host.subdomain.domain.com host2.sub2.dom2.com 49957 4000 A T 0486 TCP host.subdomain.domain.com host2.sub2.dom2.com 49957 4000 PA R 0000 TCP host2.sub2.dom2.com host.subdomain.domain.co 4000 49957 A R 0000 TCP host2.sub2.dom2.com host.subdomain.domain.co 4000 49957 A R 0000 TCP host2.sub2.dom2.com host.subdomain.domain.co 4000 49957 A R 0000 TCP host2.sub2.dom2.com host.subdomain.domain.co 4000 49957 A R 0044 TCP host2.sub2.dom2.com host.subdomain.domain.co 4000 49957 PA T 0536 TCP host.subdomain.domain.com host2.sub2.dom2.com 49957 4000 A T 0536 TCP host.subdomain.domain.com host2.sub2.dom2.com 49957 4000 A T 0536 TCP host.subdomain.domain.com host2.sub2.dom2.com 49957 4000 A
I was a little confused by her use of “packet numbers”; she was referring to the last 2 numeric columns. I explained that these are the source and destination port numbers, not packet numbers. Unfortunately, I could not answer her question because she did not capture enough information from the TCP packet header to be able to tell if these are retransmissions or not.
To determine if any TCP packet is a retransmission you must capture the sequence number of the packet. At the beginning of every TCP connection the two hosts select random sequence numbers, and then they increment the sequence number for every byte of TCP data that they send. The receiving host echoes back in the acknowledgment field the sequence number it expects to see next.
The following example shows what I recommend as the minimum set of control arguments for packet_monitor; assuming you are interested in more than just the information that a packet was sent or received. The interface argument identifies the IP interface to monitor. If you do not provide an interface name packet_monitor will monitor all interfaces. Packet_monitor can use a large amount of streams memory, using the interface argument will help reduce this amount. The -verbose argument is what will actually print the sequence and acknowledgment numbers, again without this there is no way to identify retransmissions. It will also print other information useful in analyzing traces, for example the time to live (TTL) and window size (window) values. The -filter -host and -port arguments help identify a specific connection and reduce the number of packets you have to look at.
packet_monitor -interface #sdlmux.m16.11-2 -verbose -filter -host 172.16.1.34 -p +ort 23 dir icmp type tcp dir len proto source destination src port dst port type Xmit IP Ver/HL 45, ToS 0, Len 29, ID 8569, Flg/Frg 0, TTL 3c, Prtl 6 Cksum 9eaf, Src ac100174, Dst ac100122 TCP from phx_vos-m16.51094 to 172.16.1.34.telnet seq 448953077, ack 1766884569, window 65535, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum 9c85, Urg-> 0000
Rcvd IP Ver/HL 45, ToS 0, Len 29, ID 45db, Flg/Frg 0, TTL 3c, Prtl 6 Cksum de3d, Src ac100122, Dst ac100174 TCP from 172.16.1.34.telnet to phx_vos-m16.51094 seq 1766884569, ack 448953078, window 8192, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum c984, Urg-> 0000
Xmit IP Ver/HL 45, ToS 0, Len 29, ID 8576, Flg/Frg 0, TTL 3c, Prtl 6 Cksum 9ea2, Src ac100174, Dst ac100122 TCP from phx_vos-m16.51094 to 172.16.1.34.telnet seq 448953078, ack 1766884570, window 65535, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum d883, Urg-> 0000
Rcvd IP Ver/HL 45, ToS 0, Len 29, ID 45dc, Flg/Frg 0, TTL 3c, Prtl 6 Cksum de3c, Src ac100122, Dst ac100174 TCP from 172.16.1.34.telnet to phx_vos-m16.51094 seq 1766884570, ack 448953079, window 8192, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum c982, Urg-> 0000
Xmit IP Ver/HL 45, ToS 0, Len 28, ID 8580, Flg/Frg 0, TTL 3c, Prtl 6 Cksum 9e99, Src ac100174, Dst ac100122 TCP from phx_vos-m16.51094 to 172.16.1.34.telnet seq 448953079, ack 1766884571, window 65535, 0 data bytes, flags Ack. X/Off 05, Flags 10, Cksum 098b, Urg-> 0000
Rcvd IP Ver/HL 45, ToS 0, Len 28, ID 45dd, Flg/Frg 0, TTL 3c, Prtl 6 Cksum de3c, Src ac100122, Dst ac100174 TCP from 172.16.1.34.telnet to phx_vos-m16.51094 seq 1766884571, ack 448953079, window 8192, 0 data bytes, flags Ack. X/Off 05, Flags 10, Cksum e98a, Urg-> 0000
The above sequence numbers are being increased by 1 because each packet contains only 1 byte of data, if the packets included 10 bytes, 23 bytes or 1023 bytes the sequence numbers would go up by 10, 23 or 1023.
So what does a retransmission look like? With a retransmission the sequence number is either smaller than or equal to the previous sequence number and has at least 1 byte of data, packets with no data do not count. In the following example the sequence number 448953089 is repeated multiple times. Every time but the first is a retransmission.
packet_monitor -interface #sdlmux.m16.11-2 -verbose -filter -host 172.16.1.34 -p +ort 23 dir icmp type tcp dir len proto source destination src port dst port type
Rcvd IP Ver/HL 45, ToS 0, Len 28, ID 4d43, Flg/Frg 0, TTL 3c, Prtl 6 Cksum d6d6, Src ac100122, Dst ac100174 TCP from 172.16.1.34.telnet to phx_vos-m16.51094 seq 1766885133, ack 448953089, window 8192, 0 data bytes, flags Ack. X/Off 05, Flags 10, Cksum e74e, Urg-> 0000
Xmit IP Ver/HL 45, ToS 0, Len 29, ID 9191, Flg/Frg 0, TTL 3c, Prtl 6 Cksum 9287, Src ac100174, Dst ac100122 TCP from phx_vos-m16.51094 to 172.16.1.34.telnet seq 448953089, ack 1766885133, window 65535, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum 9745, Urg-> 0000
Xmit IP Ver/HL 45, ToS 0, Len 29, ID 919a, Flg/Frg 0, TTL 3c, Prtl 6 Cksum 927e, Src ac100174, Dst ac100122 TCP from phx_vos-m16.51094 to 172.16.1.34.telnet seq 448953089, ack 1766885133, window 65535, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum 9745, Urg-> 0000
Xmit IP Ver/HL 45, ToS 0, Len 29, ID 91a0, Flg/Frg 0, TTL 3c, Prtl 6 Cksum 9278, Src ac100174, Dst ac100122 TCP from phx_vos-m16.51094 to 172.16.1.34.telnet seq 448953089, ack 1766885133, window 65535, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum 9745, Urg-> 0000
. . .
Xmit IP Ver/HL 45, ToS 0, Len 29, ID 91f9, Flg/Frg 0, TTL 3c, Prtl 6 Cksum 921f, Src ac100174, Dst ac100122 TCP from phx_vos-m16.51094 to 172.16.1.34.telnet seq 448953089, ack 1766885133, window 65535, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum 9745, Urg-> 0000
Rcvd IP Ver/HL 45, ToS 0, Len 29, ID 4f04, Flg/Frg 0, TTL 3c, Prtl 6 Cksum d514, Src ac100122, Dst ac100174 TCP from 172.16.1.34.telnet to phx_vos-m16.51094 seq 1766885133, ack 448953090, window 8192, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum 7744, Urg-> 0000
Besides the interface and verbose arguments I would include the time_stamp and numeric arguments. The time_stamp argument puts a time stamp on every packet displayed, this makes it easy to correlate messages in log files with the trace and allows you to estimate how long events, like a retransmission sequence actually last. The numeric argument reduces overhead by not looking up host or port names.
packet_monitor -interface #sdlmux.m16.11-2 -verbose -time_stamp -numeric -filter + -host 172.16.1.34 -port 23 dir icmp type + tcp hh:mm:ss.ttt dir len proto source destination src port ds +t port type 14:30:17.604 Xmit IP Ver/HL 45, ToS 0, Len 29, ID 9b7b, Flg/Frg 0, TTL 3 +c, Prtl 6 Cksum 889d, Src ac100174, Dst ac100122 TCP from 172.16.1.116.51094 to 172.16.1.34.telnet seq 448953090, ack 1766885134, window 65535, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum 9b43, Urg-> 0000
14:30:17.606 Rcvd IP Ver/HL 45, ToS 0, Len 29, ID 5442, Flg/Frg 0, TTL 3 +c, Prtl 6 Cksum cfd6, Src ac100122, Dst ac100174 TCP from 172.16.1.34.telnet to 172.16.1.116.51094 seq 1766885134, ack 448953091, window 32768, 1 data bytes, flags Push Ack. X/Off 05, Flags 18, Cksum 1b42, Urg-> 0000
If you know the problem is at the TCP or IP layer this is typically all the information you need. However, if the problem could be at the application layer you also need the actual data. You can display the data with the -hex_dump argument. By default only the first 128 bytes of data are displayed. To be on the safe size I always set length to 1500 which will display all of the data in the packet.
packet_monitor -interface #sdlmux.m16.11-2 -verbose -time_stamp -numeric -hex_du +mp -length 1500 -filter -host 172.16.1.34 -port 23 dir icmp type + tcp hh:mm:ss.ttt dir len proto source destination src port ds +t port type 10:25:19.387 Rcvd IP Ver/HL 45, ToS 0, Len 3b, ID f8f, Flg/Frg 0, TTL 3 +c, Prtl 6 Cksum 1478, Src ac100122, Dst ac100174 TCP from 172.16.1.34.49562 to 172.16.1.116.telnet seq 110526313, ack 1206659552, window 32768, 19 data bytes, flags Push Ack +. X/Off 05, Flags 18, Cksum d3ee, Urg-> 0000 offset 0 . . . 4 . . . 8 . . . C . . . 0...4... 8...C... 0 54 68 69 73 20 69 73 20 6f 6e 6c 79 20 61 20 74 * This is only a t 10 65 73 74 * est
10:25:19.389 Xmit IP Ver/HL 45, ToS 0, Len 3b, ID f969, Flg/Frg 0, TTL 3 +c, Prtl 6 Cksum 2a9d, Src ac100174, Dst ac100122 TCP from 172.16.1.116.telnet to 172.16.1.34.49562 seq 1206659552, ack 110526332, window 8192, 19 data bytes, flags Push Ack +. X/Off 05, Flags 18, Cksum 33dc, Urg-> 0000 offset 0 . . . 4 . . . 8 . . . C . . . 0...4... 8...C... 0 54 68 69 73 20 69 73 20 6f 6e 6c 79 20 61 20 74 * This is only a t 10 65 73 74 * est
10:25:22.263 Xmit IP Ver/HL 45, ToS 0, Len 28, ID f994, Flg/Frg 0, TTL 3 +c, Prtl 6 Cksum 2a85, Src ac100174, Dst ac100122 TCP from 172.16.1.116.telnet to 172.16.1.34.49562 seq 1206659571, ack 110526332, window 8192, 0 data bytes, flags Ack. X/Off 05, Flags 10, Cksum 7b7a, Urg-> 0000 No tcp data.
Before sending a packet_monitor trace with data to anyone remember to sanitize the data – both text and hex values to remove proprietary information. Under some circumstances you may also wish to sanitize the IP addresses. The IP addresses occur in two places, in the protocol header decode line they appear as standard dotted decimal notation. In the line immediately above the decode line they appear as hex values.
10:25:22.263 Xmit IP Ver/HL 45, ToS 0, Len 28, ID f994, Flg/Frg 0, TTL 3 +c, Prtl 6 Cksum 2a85, Src ac100174, Dst ac100122 TCP from 172.16.1.116.telnet to 172.16.1.34.49562 seq 1206659571, ack 110526332, window 8192, 0 data bytes, flags Ack. X/Off 05, Flags 10, Cksum 7b7a, Urg-> 0000 No tcp data.