Based on my experience in customer projects I have found out the following three key points to be true:
- – Customer testing in production can be a happy customer
- – No matter what development and testing practices you use, the production brings out the truth
- – I get to know the customer so when we do another project, later on, we all know each other and things get easier
Let’s just take a few steps back to look at this in detail.
Label Applicator Line in production
There is a saying in Finland which directly translates as “The customer will test in production”, used jokingly by companies that do not practice testing systematically (or at all). This is of course the topic of many, many heated internet discussions where testing puritans… Oops, sorry. I got a bit carried away…
Twenty years ago, there was a great movement towards Test Driven Development. TDD was seen as the golden standard for software development. There were some who even said that if you do not do TDD, you do not appreciate your customers. So, it became a dogma and as with all dogmas, people start to practice it blindly, locking themselves into this certain mindset and never wonder why I’m doing this.
Despite all these testing best practices and guidelines, the customer still tests in production to some extent. And now I will explain why this happens.
At this point I have to add a disclaimer: This is my personal view of the matter. My view does not necessarily be that of my employer, my colleagues, or even other programmers in the world. I base my views on the experience I have from working in the software industry for over 20 years.
Recently I was working on our new product, the Label Applicator Line. It is an automated machine that captures images of mail items running on a conveyor belt, reads their address and possible barcode labels, adds a new barcode label if needed, and registers them for sorting at later stages. This machine has a high throughput of 6000 mail items per hour. It is 10 meters long, 5 meters wide and 2,5 meters tall. The OCR recognition rate is high.
How does one test it at the office?
I desktop tested all the software I wrote for it, of course. I unit tested every component except the camera imaging software, which needs an actual camera to operate. The camera’s imaging software was tested at the same time when I did the integration tests on the machine. I sat for days in our lab with the machine. I even let the machine run through the weekend and checked the logs on Monday to get a good understanding of the events.
But my tests were aimed at ensuring that the software on the machine will not crash the moment the client takes it in use. I could test how it performs under a heavy load. However, the problem was my limited test material. The client, a big postal operator, has tens of thousands of unique shipments every day. I, on the other hand, had just a dozen unique shipments.
Now, one could argue that the client needs a separate test/lab environment that runs parallel to the production line. And while this is true, one has to keep in mind the costs, the volume of test material (actual mail items), and the floor space that the extra machines will occupy. In a perfect world, with an unlimited budget, space, and time, the client could have built two postal centers; one for production, one for testing, and hire twice the amount of people needed. But in this world, that is not an option.
So, the client tests new equipment in the production environment. Of course, the client has to run production at the same time and have a backup system to be used when the equipment under testing fails. It will fail at the first run(s), you can count on that. There is no way to avoid it since I, the developer, does not have a postal center and a gazillion amount of test material. This is the unfortunate truth, no matter which way one puts it.
And a lot of times even the development phase must be done in production. Luckily, it can be done in iterations to ease the pain. Real problems start in integration. Those will be solved too, of course.
But. When a product has been tested with gazillion packages, with quite an amount of workforce and a couple of weeks of time, it will work. It will be tweaked like a customer wants. Many observations are being made during testing and those are used in optimizing the software, for example, camera settings can be changed to match the production environment (ambient light changes from place to place). And what is best, the workforce gets to know the machine from the start. Human mind has great capability to work around little quirks which existence developers do not even know about!
So, all-in-all, a) customer testing in production can be a happy customer, b) no matter what development and testing practices you use, the production brings out the truth and c) at the same time I get to know the customer so when we do another project later on, we all know each other and things get easier.
