In this post I wish to discuss a common coding error that can result in a module running out of or at least seriously depleting the available number of stcp device clones for use in creating TCP sockets.
Every call to the socket function results in STCP creating a new clone of the stcp device. The maximum number of clones that can be created is controlled by the clone_limit field in the devices.tin entry.
/ =name stcp.m17
=module_name m17
=device_type streams
=access_list_name stcp_access
=streams_driver stcp
=clone_limit 5120
=comment Provides TCP API
|
You can see how many clones are currently in use by dumping the device structure in analyze_system and looking at the clone_count value. If clone_count equals clone_limit then calls to the socket function will return an e$clone_limit_exceeded: “The clone limit for the device has been exceeded” error.
as: match clone; dump_dvt -name stcp.m17
clone_limit: 5120
clone_count: 42
cloned_from: -27271
remote_clone_limit: 0
|
In general the socket call that creates the clone device is followed by either a connect call or bind and listen calls. At that point you can see a corresponding entry when you execute the “netstat command”.
netstat -numeric -all_sockets Active connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp 0 0 172.16.124.217:23 192.168.109.22:50038 ESTABLISHED tcp 0 0 172.16.124.217:22 172.16.124.24:54987 ESTABLISHED tcp 0 0 10.20.1.1:37 10.20.1.26:1528 TIME_WAIT tcp 0 0 10.20.1.1:37 10.20.1.27:1579 TIME_WAIT tcp 0 0 172.16.124.217:61780 192.168.109.22:23 ESTABLISHED tcp 0 0 172.16.124.217:22 172.16.124.50:17421 ESTABLISHED tcp 0 0 172.16.124.217:22 172.16.124.50:17658 ESTABLISHED tcp 0 0 *:23 *:* LISTEN tcp 0 0 *:6666 *:* LISTEN tcp 0 0 *:21 *:* LISTEN tcp 0 0 *:3000 *:* LISTEN tcp 0 0 *:7 *:* LISTEN tcp 0 0 *:9 *:* LISTEN tcp 0 0 *:13 *:* LISTEN tcp 0 0 *:19 *:* LISTEN tcp 0 0 *:37 *:* LISTEN tcp 0 0 *:901 *:* LISTEN tcp 0 0 *:1414 *:* LISTEN tcp 0 0 *:81 *:* LISTEN tcp 0 0 10.20.1.1:37 10.20.1.9:3633 TIME_WAIT tcp 0 50 10.10.1.1:52653 10.10.1.200:3001 ESTABLISHED tcp 0 0 10.10.1.1:52624 10.10.1.200:3001 FIN_WAIT_1 tcp 0 0 10.20.1.1:61704 10.20.1.3:48879 ESTABLISHED tcp 0 0 *:3001 *:* LISTEN tcp 0 0 *:3002 *:* LISTEN tcp 0 0 *:3003 *:* LISTEN tcp 0 0 *:4000 *:* LISTEN tcp 0 0 172.16.124.217:4000 172.16.124.78:1024 ESTABLISHED tcp 0 0 172.16.124.217:4000 172.16.124.227:1025 ESTABLISHED tcp 0 0 *:4001 *:* LISTEN tcp 0 0 *:4002 *:* LISTEN tcp 0 0 *:4003 *:* LISTEN tcp 0 0 *:4004 *:* LISTEN tcp 0 0 *:22 *:* LISTEN tcp 0 0 *:4005 *:* LISTEN tcp 0 0 *:4006 *:* LISTEN tcp 0 0 172.16.124.217:4006 172.16.124.203:49231 ESTABLISHED tcp 0 0 *:4007 *:* LISTEN tcp 0 0 *:4008 *:* LISTEN tcp 0 0 *:4009 *:* LISTEN tcp 0 0 172.16.124.217:4008 172.16.124.203:49262 ESTABLISHED tcp 0 0 *:4010 *:* LISTEN tcp 0 0 *:4011 *:* LISTEN tcp 0 0 *:4012 *:* LISTEN tcp 0 0 *:4013 *:* LISTEN tcp 0 0 *:4014 *:* LISTEN tcp 0 0 *:4015 *:* LISTEN tcp 0 0 *:80 *:* LISTEN tcp 0 0 *:9182 *:* LISTEN tcp 0 0 *:445 *:* LISTEN tcp 0 0 *:139 *:* LISTEN tcp 0 0 10.20.1.1:53495 10.20.1.9:48879 ESTABLISHED tcp 0 0 10.20.1.1:61703 10.20.1.3:48879 ESTABLISHED tcp 0 0 10.20.1.1:61707 10.20.1.3:48879 ESTABLISHED tcp 0 0 10.20.1.1:61705 10.20.1.9:48879 ESTABLISHED tcp 0 0 10.20.1.1:61709 10.20.1.9:48879 ESTABLISHED tcp 0 0 10.20.1.1:61710 10.20.1.9:48879 ESTABLISHED tcp 0 0 172.16.124.217:61789 172.16.124.203:4000 ESTABLISHED tcp 0 400 172.16.124.217:22 172.16.124.50:17674 ESTABLISHED |
If you count up the number of lines you will see more than 42, that is because not every entry shown by netstat uses an stcp device clone. For example, OSL connections and the X25_cpc connections used with the NIO. Take a look at socket_count.cm for more details.
If a socket call is made without a connect or bind or the bind fails you can create the opposite issue, the value of clone_count is larger than the number of entries shown by netstat.
as: match clone; dump_dvt -name stcp.m17
clone_limit: 5120
clone_count: 4131
cloned_from: -23179
remote_clone_limit: 0
as:
|
I am not including the netstat output again, but trust me it hasn’t changed from the previous example.
This situation, an extra (4131 – 42) and apparently unaccounted for, STCP device clones was created by the following code fragment. The code calls the socket function followed by the bind function. If the bind fails it loops. Many applications would add a timer to wait 1, 60, or 300 seconds and try again, that just delays the inevitable, assuming of course that the condition causing the error does not go away.
tryAgain = 1;
while (tryAgain)
{
if ((socks0 = socket (AF_INET, SOCK_STREAM, 0)) < 0)
{
if (debugFlag)
perror ("badService: can't create listening socket");
}
else {
/* build a sockaddr structure holding the address we will bind to.
The IP address is INADDR_ANY meaning we will listen on all active
IP addresses */
bzero ( (char *) &serv_addr, sizeof (serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = htonl (INADDR_ANY);
serv_addr.sin_port = htons (portNumber);
/* now bind to the address and port */
if (bind (socks0, (struct sockaddr *) &serv_addr,
sizeof (serv_addr)) < 0)
{
if (debugFlag)
perror ("badService: can't bind address, trying again");
}
else
tryAgain = 0;
}
}
|
The most common error is that another process already has bound to the requested port. Regardless of the reason for the error the solution is to close the socket after reporting the bind error.
tryAgain = 1;
while (tryAgain)
{
if ((socks0 = socket (AF_INET, SOCK_STREAM, 0)) < 0)
{
if (debugFlag)
perror ("goodService: can't create listening socket");
}
else {
/* build a sockaddr structure holding the address we will bind to.
The IP address is INADDR_ANY meaning we will listen on all active
IP addresses */
bzero ( (char *) &serv_addr, sizeof (serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = htonl (INADDR_ANY);
serv_addr.sin_port = htons (portNumber);
/* now bind to the address and port */
if (bind (socks0, (struct sockaddr *) &serv_addr,
sizeof (serv_addr)) < 0)
{
if (debugFlag)
perror ("goodService: can't bind address, trying again");
if (close (socks0) < 0)
if (debugFlag)
perror ("goodService: can't close old socket");
}
else
tryAgain = 0;
}
}
|
This post has been about reaching the clone_limit because of a coding error but what if there is no error, what if the application environment is really using all those clone devices. Well then, assuming you have not reached the system limit of 16,000 you can raise the limit. You need to update the clone_limit field of the stcp device in the devices.tin file and recreate the devices.table. If you are on a 17.1 or later release you can use the update_device_info command to raise the limit for the current boot and rely on the updated devices.table to take care of the next boot. On releases before 17.1 your only real option is to reboot. You should set the limit to a value that corresponds to your current needs plus expected growth; you should not just raise the limit to 16,000. Even if you do not have an application bug that is consuming clone devices now there is no guarantee that you will not have one in the future. An application that is consuming all the available clone devices will also consume a great deal of streams memory and exhausting streams memory will negatively affect existing TCP connections.