I wrote the following remarks about conducting effective pre-production testing for my “VOS Corner” column in the Stratus User Group newsletter for December, 1996. They are still relevant today, almost 13 years later.
If I could make one suggestion to our customers (just one request!), it would be to be sure that you conduct a stress test of new releases of your application before placing them into production. I’ve been involved in a number of situations recently where the customer deployed an application that passed all of the functional tests but failed soon after being placed into production because it could not handle the production load.
As I’m sure you understand, software can be unpredictable and tricky to test. Knowing that a program works at 10 transactions per second is not the same as knowing it works at 100 transactions per second. Knowing it works with 10 virtual circuits is not the same as knowing it works with 1000 virtual circuits. In each case I’ve seen, the program worked fine in the lab or test environment, but failed miserably in production due to some underlying software defect or capacity limit. Generally, if I get involved, the limit was within VOS, not the application, but no matter where it is, the effect is the same on your customers. If you want to know whether your application will work in production, you must test it as if it is in production.
I’ve listened to many excuses over the years why you can’t test your software to the same limits as in production. I don’t believe any of them. You can capture the production data stream and files to use in test. You can rig up synthetic test generators to simulate good and bad input. You can bypass device-sensitive interface code in order to feed high-speed data streams to servers. There are lots of tricks you can pull to get 90% of your application working flat-out, even if you can’t get 100% of it working in test mode. Stop thinking of the problem as simulating production and start thinking of it as simply getting as much of your code as possible to execute, as fast as possible. Be sure to drive your system to 100% CPU utilization; that’s where the “interesting” problems tend to show up.
By exercising your application software to its limits, you are also exercising VOS as well. You will find capacity problems in your lab, not in production. You will know what and where your capacity limits are. You will exit your test phase with a much higher degree of confidence that your application deployment will be successful. You will keep your boss and customers happy. Aren’t those good reasons?