TCP Backlog capture is an effect that prevents a server application on the Stratus module listening on port X from successfully creating a connection with any client trying to connect to port X. Connections to other ports even from the same clients work fine. The typical solution is to shut down and restart the server application. This post will explain why backlog capture happens and other less drastic solutions.
The TCP backlog is the queue of pending connections, that is connection requests that have been received but not yet been accepted. In applications compiled on pre OpenVOS 17.1 releases and in applications compiled on OpenVOS 17.1 but without the POSIX libraries, connection requests are not sent an acknowledgment until they are accepted by the application. The client will resend the connection request packet between 3 and 10 times at increasing intervals. Exact details will depend on the client operating system and how it is configured. At some point the server application accepts the connection request and the STCP stack sends an acknowledgment. As long as that acknowledgment is received before the client has given up, the connection will be completed and communication can start.
The backlog capture scenario requires several conditions
- There is a temporary network failure between server and client (but not between client and server). This allows the client’s connection requests to be received by the server but prevents the server’s connection acknowledgments from being received by the client.
or
The server application delays calling the accept function for some reason.
- The client continues to send connection requests; that is when the current request times out the client sends a new one. The client’s timeout value must be shorter than the server’s timeout value.
- A stateful firewall or some other device somewhere between the client and server times out the client’s connection requests and discards the server’s connection acknowledgments without sending a response back to the server. This timeout value must be shorter than the server’s timeout value.
Unfortunately this combination of conditions is not as unlikely as might first appear.
It typically starts with a network outage that prevents packets from the server from reaching the client but allows packets from the client to reach the server. When the client’s connection request packet reaches the server it is placed in the backlog queue. When the server application makes a call to the accept code the code pulls the connection request from the backlog queue and sends a connection acknowledgment. Since the server does not get an acknowledgment for its packet, because the network is broken, it retransmits the connection acknowledgment. The client’s TCP stack will also retransmits the connection request because it hasn’t gotten an acknowledgment. After a while the client’s TCP stack gives up and the client application sends a new request. The server continues to respond to the first request. After around a minute and half (by default) the server times out and pulls the next request from the backlog queue. The client meanwhile has resent the second request and might be on the third. These requests are also in the backlog queue. It is possible that some other client could get a connect request placed in the backlog queue but for it to complete successfully the client would need a timeout that is greater than the server timeout multiplied by the connection request’s position in the backlog queue. Once the network outage is corrected the stateful firewall perpetuates the outage by silently dropping connection acknowledgments that it no longer has a state for because it only holds state for a time that is shorter than the server timeout.
How do you recognize this?
First the backlog is full or at least filling. You can display the backlog queue for a connection with this command.
analyze_system -request_line (string match backlog -or syncnt (byte 3Bx) dump_st +
cbq -full -lport 7654 -faddr 0x) -quit
OpenVOS Release 17.0.2au, analyze_system Release 17.0.2au
Current process is 154, ptep 91CC7100, Noah_Davids.CAC
syncnt 6
backlog 5
ready 11:41:19
|
The lport value is the local port number, 7654 in this case. The foreign address (faddr) value of 0 limits the output to just the LISTENING socket. If the backlog value plus 1 equals the syncnt value the backlog is full. If the syncnt is greater than 0 and going up the backlog queue is filling. The size of the backlog queue is set by the application when it calls the listen function.
Second you need to look at a network protocol trace. The trace will show that the last connection request does not get acknowledged while there are connection acknowledgments for a previous connection request and those acknowledgments are not getting any kind of response.
The trace in figure 1 shows this. Each connection is labeled with a stream index and also color coded. The first connection request from the client is in frame 1 at time 0. The acknowledgment is sent in frame 2 and then retransmitted in frame 3. Frame 4 is the client retransmitting the connection request. Since STCP is already sending connection acknowledgments frame 4 triggers just a TCP acknowledgment in frame 5. Frames 6 and 7 are retransmitted connection acknowledgments. Frame 8 is a retransmitted connection request from the client. Frame 9 is the TCP acknowledgment triggered by the retransmitted connection request and frame 10 is another retransmitted connection acknowledgment. Frame 11 is another connection request from the client but for a new second connection. Note that there is no response and frame 12 shows a retransmitted connection request. Frame 13 shows a connection acknowledgment but it is still for the first connection. Frame 14 is a retransmission of the second connection request. Frame 14 is another retransmission of the second connection request. Frame 15 shows a new, third connection request from the client and frames 16 and 17 are retransmissions. Frame 18 is another retransmission of the connection acknowledgment to the first connection. Frame 19 is yet another connection request with frames 20 and 21 the retransmissions and frame 22 is the fourth new connection request. Finally in frame 23 STCP gives up on the first connection request and sends a reset followed in frame 24 with a connection acknowledgment to the second connection request with a retransmission in frame 25. At this point I think you get the idea.
STCP took from frame 0 to frame 23 to time out, a total of 106.9 seconds. While the client does not send a reset when it times out, the time between the start of the second and third connection requests (frame 11 to frame 15) is 29.152 seconds and between the third and fourth (frames 15 and 19) is 25.8 seconds; much faster than STCP.
What can you do to prevent this?
The best solution is to upgrade to OpenVOS 17.1 and recompile the application with the POSIX libraries. Doing this will prevent the problem from ever happening since STCP will send a connection acknowledgment immediately, without waiting for the application to call accept. It may however change other behaviors of your application and careful testing of your application should be done. If upgrading and recompiling are not possible you can
- Reconfigure the stateful network device that is dropping the connection acknowledgments to respond with a reset. When STCP receives the reset it will move on to the next connection request in the backlog queue. The result is that the connection requests timed out by the stateful device will be quickly drained from the queue. Any further connection acknowledgments will be handled by the client which will either send its own reset or acknowledge them.
- Stop the client from making connection requests until the server backlog queue is emptied.
- Close the server application’s listening socket and reopen it. Typically, the application is not designed to do this so you have to stop and restart the server application.
- Configure syn_rcvd_abort value to something that is shorter than the client’s timeout value. The value is changed with the analyze_system request set_stcp_param
analyze_system -request_line 'set_stcp_param syn_rcvd_abort 15' -quit OpenVOS Release 17.0.2au, analyze_system Release 17.0.2au Current process is 151, ptep 91530AC0, Noah_Davids.CAC Changing tcp SYN_RCVD timeout (syn_rcvd_abort) from off to 15 ready 11:07:44In the above example the timeout is set to 15 seconds. The valid values are between 1 and 180 with a value of 0 meaning to use the default (about 100 seconds).
This will prevent a single client from creating a large buildup in the backlog queue. How short the value should be will depend on many clients could be accessing the server at the same time. The problem with this approach is that affects all clients and all server ports, you cannot have one value for one set of ports and another for another set of ports. In addition, too low a value might cause some connections going over high latency links to fail when they could succeed if the timer was longer. Choosing an optimal value is more art than science