originally posted on one of my several now defunct blogs, called On Engineering, on 14th November 2012
Have you not noticed a book recently? Forgotten that you were reading whilst you were reading? That’s the author’s ideal: their books should melt away whilst you are reading them, so that the content transcends the medium and becomes the event.
Can this happen with a technical book? Honestly, I don’t believe it can; technical books are so full of references, tables and figures, footnotes and diagrams that you can not escape their structure, their architecture for long. I could briefly get lost in an alloy phase diagram in “Engineering Materials”, but I couldn’t read the book page for page, for hours on end like I could a Julian Barnes or an Iain M. Banks.
An engineer’s job does (or at least should) include reading up on things, whether that be a new book or browsing the web for information. This being an engineering blog, I thought the occasional review of interesting resources that I have encountered might end up being something that I could write about. This is the first in this unforeseeably long or short series or reviews.
The book that kickstarted this whole thought process was one I came across as background reading for my post on whether Software Engineering is Engineering: it was the ebook How Google Tests Software
How Google Tests Software (HGTS) was written (developed and compiled, perhaps?) by three gurus in the art of software testing: James Whittaker, Jason Arbon and Jeff Carollo. In style, it is what could be expected of Google from an outsider’s viewpoint - quite chatty, breezy, somewhat at odds with the incredibly technical and mathematical work that they do. It is also replete with excellent word selection, suggesting that whilst coding is at the heart of their work, this trio is also at home communicating with people. Indeed, being bright and capable of communication is a key aspect of their respective rises to the upper echelons of Google (and, in Whittaker’s case, Microsoft) management. James Whittaker certainly has literary form, having written “How to break software” and “Exploratory Software Testing” prior to HGTS.
In truth, and from my perspective thankfully, HGTS is only semi-technical. There is not much in the way of code snippets or significant jargon; it’s more a case of using dialect (“dog-fooding” for internal pre-Alpha software testing, for example). The book reminded me a little of the classic aerospace book “Flight without formulae” {Link} in that there is a minimum of code and a maximum of description. This suited me down to the ground. Someone in the software development world may be disappointed at not having chunks of test code to try to understand or to try out, but this book describes in a lively way the key principles of how to manage testing, how to manage testers and how testing has to become integrated both into the product and into the company itself. This makes the book worthwhile reading for software developers, I’m sure - but also for us.
The essential message of the book is entirely relevant even at my mechanical end of the engineering spectrum: it is that {software} testing and quality must go hand in hand with development.
In the book, we learn how Google went so far as to kill off the group called “Testing Services” and to resurrect it as “Engineering Productivity.” More than merely a rebranding, the switch ensured that the software developers were testing their new code all the way through the development process: the Productivity Team gave them the tools to do so.
Software testing consists of several levels, from quality checks on portions of code, through to logic and functionality tests on components and upwards to full interdependent systems and finally user testing.
Equally, there are several levels of test engineer involved: there is the SWE (Software Engineer), who principally develops code, but also tests the same code for “low-level” bugs. There is the SET, the Software Engineer in Test, who aids the SWEs in writing test code and the frameworks for such testing, and finally there is the TE, the Test Engineer, who is involved in the user-side testing of an app or a site.
The test team is kept small by design, making it a limited resource that thereby keeps a large enough balance of responsibility on the side of the SWEs to keep things as bug-free and as smooth as possible. The idea is that if Testing were to become a huge department, like in the bad old days, software quality would become worse, not better, since SWEs would once more feel released from the constraint of having to consider testing and quality as being an integral part of what they create. Google (as would any other company) would slow down to become a bureaucratic monster, no longer nimble, no longer smart.
The sheer complexity of what the testers do is incredible and totally beyond my ken. Tests that range from small to enormous, automated bots that trawl websites for bugs, whole tracking systems for bugs: these are all impossible creatures for me.
Intellectually, though, they become analogies and hints for improving our own ways of working. Let’s take the structure: We have technical, production and quality departments: why not eliminate the quality department as we know it and create a Productivity Improvement Team? Indeed, why is quality treated separately? If we focus on productivity, we automatically have to eliminate quality issues. Google tracks bugs with Buganizer? Well, we could move on from quality catalogues (aka rogues’ galleries) to active tracking and destroying of our own quality stumbles - for everybody. Google trawls websites for usability issues? We could do much more collecting of warranty and benchmark data for our parts and those of our competitors. Google raises bug alerts on competitors’ sites? Hmm, well, perhaps that’s an analogy too far, but the notion of making our industry a better place is a noble one.
Google uses what they term the “ACC” Analysis methodology, where teams think through Attributes, Components and Capabilities to determine an initial test plan for that product for each instance where a component is broken or a capability not met. That is, they think through what would happen and how a user would be affected if a particular component were suboptimal or broken, and assess how frequent that type of failure would be. It all sounds very similar to the FMEA methodology in our world.
Tellingly, though, Google doesn’t seem to let itself get bogged down in documentation or specifications. “…I suppose there is some fairytale world where every line of code is preceded by a test, which is preceded by a specification. Maybe that world exists. I don’t know. But in the innovative and fast-paced world that I live in, you get what you get. Spec? Great! Thank you very much, I will put it to good use. But… demanding a spec won’t get you one… Nothing a spec writer … can do will help us find a problem that a real user will encounter.”
I would be interested to know if Google needs to pass audits in the same way we do.
Google can be very clear on how it should manage clever people: “…I am a big believer in keeping things focused. Whenever I see a team trying to do too much, say working on five things and doing only 80% of all five, I have them step back and prioritise. Drop the number of things that you are doing to two or three and nail those 100%.”
So - the way Google has set up its development teams with quality at their heart, then set up productivity teams that provide the tools for quality to succeed sounds like a benchmark for us to meet.
These and many more are the lessons to be drawn from How Google Tests Software. I would certainly recommend you delving into the book for even more on how to recruit clever people, how to work with barefoot managers, and how to ensure that jobs and roles are not entities in themselves, but part of a community in their own right.
Could Google learn something from us? Well, if Google really wanted to know how to bog itself down in administration, they could always learn from us and introduce PPAPs to the software world. That would help, I’m sure.
And: did you not notice your browser there for a few minutes? If so, then it’s a sign that this post was in some way or other interesting; if not, then you probably disappeared down a few tangents via those links - the very nature of blogs and the internet (or your browser crashed…)
