Zhengziying's Blog

Recently I saw an email thread in which people in a team were asked to sign-off on a hotfix. The BVT run of that hotfix only passed 97% (11 failures out of 380 test cases in total). I told them that something in the team’s engineering practice is wrong when there is such a thread asking people to sign-off on a BVT result of a hotfix.

That is because:

• The reason there is a request to signoff is because the BVT is not 100% passing. If it were 100% passing or just 1 failure, they wouldn’t need such a signoff thread.
• BVT should be 100% passing for all the time or just 1-2 intermittent failure at worst. It’s a big problem that a BVT has 11 failures (97% passing).
• It’s an even bigger problem that BVT is only 97% passing on a hotfix. The code (including test automation) in a hotfix branch should be in very good quality and very reliable.
• If this 97% passing BVT result were due to test environment issue, they should have very quickly fixed the environment issue and get another BVT run (which should have got result in at most a couple hours). It would have been another problem if they weren’t able to quickly fix test environment issues that caused large amount of failures in BVT.

Actually, not just in this specific team, not just on BVT result , no modern established engineering team should be doing any kind of signoffs. Signoff means making decisions heavily based on human judgments, which shouldn’t be in the equation when it comes to determine whether a piece of code is good enough to ship or not. Signoff should be prohibited in day-to-day work in software engineering. The use of the word “signoff” usually indicates issues in the team’s engineering practice and culture.

Instead of “signoff”, the right word to use is “override”. It’s like in the airports. In many airports, passengers go through the screening machines and only if the screening machines beeps, there is an officer to do pat-down. After pat-down, the officer can let us go (overriding the outcome of a machine), or send us back if they do find something suspicious (confirming the outcome of a machine). Things should work the same way in shipping software: the code just goes out of the door when the automated validation says “Good”. We only stops the train when the test automation beeps, then it comes human inspection and judgment: we can still let the code ship (overriding), or pull the code back if we do find something suspicious.

Of course, test automation sometimes may be more complicated than a screening machine. For a sizable product, there is a whole lot of work to do to make sure test automation not to beep too often nor too rarely.