Testing - Child's Play?

Whether it be watching children’s tv or movies with my 2 boys, driving in the car with them or going for walks with them; its always the same. Question after question after question after question…..

My eldest boy is at that wonderful age where he is learning about his world. To do that means he is continually bombarding my wife and I with a flurry of questions about everything he sees.

A recent example as we were watching ‘Toy Story’ and Woody finds himself no longer the favourite toy as Buzz Lightyear comes along:-

Child: ‘Daddy, why does Andy no longer like Woody?’

Dad: ‘Its not that he doesn’t like him, its just he has a new toy and he is playing with it instead, much like you do with your new toys’

Child: ‘But Woody is sad’

Dad: ‘Yes, he is sad but he will be alright. Andy will play with him again soon’

Child: ‘But why is he not playing with him daddy?’

Dad: <repeats answer from above>

Child: ‘But why daddy?’

And so it continues. I have heard it said that ‘kids make great Testers’…..

Now 1 of 2 things are at play here. Either my answer is not making any sense to the child or he is not satisfied with the information he has and wants more information. Now the method he is using to get the info is to repeat the exact same question, but that’s what a child will do as he doesn’t realize to frame the question in another way. However, the intention is similar to any good Tester.

A good Tester will ask many, many questions in an effort to paint a picture in their mind as to the problem. In other words, they want to make sense of the issue or at least clarify it. To do so, good Testers will not ask the same question over and over, instead, they will ask a question, use the answer to ask another but phrased slightly differently. An example is below:-

Tester: ‘The app appears to be responding with an incorrect message format, according to the spec’

Stakeholder: ‘It would appear that way, although that spec is only a guideline and the 3^rd party that the app integrates with is not that fussy regards format’

Tester: ‘Ok, it may not be strict regards format but the spec is our frame of reference, correct?’

Stakeholder: ‘well it is supposed to be, but as said the app’s integration with the 3^rd party behavior is the key frame of reference’

Tester: ‘Ok, if that’s the case then why do we have a spec, why is it not always reflecting the app’s behavior?’

Stakeholder: ‘S’pose we should update that but we just ‘know’ to use whatever comes back from the 3^rd party as the correct behavior’

Tester: ‘if that’s the case then how would we ever know if there is an issue with the app integrating with the 3^rd party?’

Stakeholder: ‘We would usually just know, but fair point, we need to ensure our specs are correctly representative of behavior’

Tester: ‘well otherwise, we are saying that our testing will only ever tell us what is being returned at that time. That’s not testing, that is merely confirming’

Above, the Tester didn’t just ask the same question which could have been ‘but why’ each time when the stakeholder explained the spec wasn’t the key reference but instead, the Tester used rational reasoning and the previous answer to frame the following question. Eventually painting the picture not only for himself but also for the stakeholder, that all is not as it should be.

And that’s the crux, Testers ask questions. NO!!!!. Testers ask GOOD questions, questions that require a non-closed answer as much as possible.

Now earlier I mentioned that it is a popular opinion that ‘kids would make good testers’. That’s not strictly true, their questions are immature which is perfectly understandable, they are kids after all. However, they display one of the key traits that makes a good tester, they ask question after question after question….

Being a good Tester? It’s not really child’s play J

Need To Know

Bernard Woolley: But, you only need to know things on a need-to-know basis.  
Sir Humphrey Appleby: (quite animated) I need to know “everything”. How else can I judge whether or not I need to know it?  
Bernard Woolley: So that means you need to know things even when you don't need to know them. You need to know them not because you need to know them but because you need to know whether or not you need to know. If you don't need to know, you still need to know so that you know that there is no need to know.  
Sir Humphrey Appleby: Yes!
 http://www.youtube.com/watch?v=vh1eCDotdSc (about 1.35 onwards)

The above quote comes from the popular 1980s sitcom ‘Yes Minister’ which was an amusing look at the high handed and snooty aspect of British politics at the time. The show provided a different way of looking at politics, often in a comical way. The actual scene from above has Sir Humphrey, who was the Permanent Secretary to the Cabinet Minister and is quite manipulative in his dealings with the Minister, in the aim of ensuring his own agenda is fulfilled. Alongside him is Bernard, who is the private Secretary to the Minister (a body man of sorts), loyal to the Minister and quite naïve.

When I saw this scene, it got me thinking of how this “need to know” relates to the Testing field. I have on many occasions, been asked ‘what makes a good tester’. My language I find varies dependent on audience, but a few central themes never change. Creativity, inquisitive, understands the role (as in it’s not about being a Yes person and following the ‘green bar’) and crucially, ability to focus and de-focus. It is this final point that I want to elaborate on.
The main difference between an average tester and a good tester is that ability to appreciate how and what they are testing and how that relates to the bigger picture. Too many testers can test their feature/application/story at that time and see it almost as an individual entity. That’s fine, there is value in that aspect (they can focus).
However, the really good testers will do that AND be able to place it into the bigger puzzle (defocus). In other words, ‘I am testing feature x, ok that works, great (focus). Now, what about feature y and application z, they have dependencies on feature x and there is a knock on from application abc, as that feeds into feature x. Ok, I need to pull together a map/compass as to how I am going to test those dependencies. Oh, also, a customer might do d, e and f that rely and interact with feature x. I need to test that too’ (defocus)

Remember what service a tester provides. The tester provides information on a feature/application/software. That information shouldn’t just relate to the feature/application/software in isolation, it should also be information on as many aspects of that feature/application/software as possible, in the time allowed.
Now, an average tester might say ‘oh, I didn’t know I had to do that and I am testing the story/feature as that’s what the requirement is, I don’t need to know what feature y and application z do’.

 My counter to that is ‘you should want to know everything, how else will you judge what you need to know’. Sir Humphrey was right in this instance.

Raising Babies, Testing Software – Much the same thing!

Shortly after becoming a Father for the second time, I thought I was about to become a slave to ‘routine’. ‘Sure the first one had his bottle at this time, so we’ll do this with the newborn, ‘sure when the first one cried during the night, we did this, let’s do this with the newborn’, ‘sure the first one slept during these hours, so will the newborn’ and so on.
Now as anyone who has had more than one child will testify, it’s not as simple as this. Things that worked the first time around can be pretty much obsolete this time, they were outdated, irrelevant and just didn’t work. Essentially, routine went not only out the window, so much as straight through the glass.
So what do you do? Do you persevere with the tried and tested even though it may be outdated and doesn’t fit anymore or do you invent different ways?

After all necessity is the Mother of innovation

The answer is of course the latter. You adapt your approach to the situation at hand, try things, they may work, they may not, toss them out (the idea, not the baby), tinker the approach, reinvent it. You end up finding a rhythm that suits and an approach that works. You end up with the same result (all being well) a healthy growing baby, but raised a different way than his older sibling.
Now what if we were testing software and not raising children?
Consider the following. Test Manager Seamie is given the responsibility for testing the initial release which is a new product going to market, after weeks of coming up with an approach, reviewing, refactoring and essentially agonizing, the test approach that best suits the product (and the company’s testing ethos) is in place. Given the company’s commitment to agile development, Seamie has put in place a mostly automated approach based on end user scenarios. He has agreed that the Test team will work approximately ¼ of a sprint behind development and this gives the automation time to catch up and test the stories and functionality in an automated fashion.
Now, there is nothing whatsoever wrong in Seamie’s approach. It is what I like to term ‘agile lite’ in that it takes whatever facets of agile that best suit the company’s make-up and utilizes them. Not fully blown agile (not any company claiming to be agile that I have ever come in contact with actually is) but that’s ok. It’s ok to adopt an agile approach to being agile
As it turns out, Seamie’s approach is very successful. The product is delivered on or around time, the testing effort found a number of issues (mainly early), the test pack is 95% automated and quick to run, customer beta testing was a success and straightforward and the testing effort suited the Test team and the Development team who were both equally comfortable with the style. – GOLDEN CHILD

Due to its success, another Product Owner in the same company but responsible for a different area has come to Seamie asking him to lead the testing for a similar type product. However, this product offers different functionality and isn’t so open to automation. In addition, the development team here are ‘pure agilists’, they want their Test team to work with them in getting a story ‘done done’ once development is complete. Seamie tries to adopt the same approach to project 2 as he did in project 1. This leads to confusion over where the overall effort is and what is required, friction between the Test and Development teams and to the eventual breakdown of communication resulting in long delays to project 2 and damaged reputations – PROBLEM CHILD

A short story, straight to video and a little general maybe. But this is a common problem today in the Testing world. I have seen this on a number of occasions throughout the years. In people terms it is what we would deem as ‘set in our ways’, it’s the way it’s’ always been so it is the way it is. What could and should Jamie have done for project 2?

Well, for starters the best way to approach any project, irrelevant of how ‘similar’ it may seem to a previous one is to treat it as standalone initially. Ask the question, ‘if I had a blank sheet in front of me with ANYTHING or ANY OPTION open to me, how would I test this?’ Re-invent testing each time! You may come out at the same destination each time (hopefully not or you should maybe re-consider your career as you are most likely getting stale in Testing) but each time new ideas should come out, mistakes from last time can be called out and made priorities this time, tools, ‘what ifs’, etc. should all come out.

After that, ask this question ‘what makes this project different (no matter how small) from previous projects?’ each project, even subsequent releases, have something new or different than before. Make sure you know, inside out, what those are and their usage. How will they be used, how could they be used, and what will a customer think of doing here? All these will help you build a picture as to what ‘extra’ you have to bring to your testing effort.

Finally and only finally, what facets of a previous project can I re-use? This has to be the question asked last. Why? Because, it keeps your mind fresh, sharp and doesn’t allow you to become lazy. If you ask this question up top, you probably won’t get to questions 1 and 2.
Don’t be a slave to routine because ‘it’s the way I did it before’. Things that worked previously may work this time but may not, new things may work but may not. But ask the above questions first – don’t assume and worse still, don’t shoe horn them in as Testing, as is raising babies, is not a ‘one size fits all’ business.
What you will find though is that after a few projects, you will start to notice patterns and this will eventually equate to different ways of doing similar things. Different approaches, tools, mindsets that all lead us to releasable software and a tested (as best we can) product. As mentioned earlier, we end up with the common result but their journeys may take different paths.

A healthy growing baby but raised a different way than his older sibling!

Negative Testing or Negative About Testing

Just a few quick thoughts regards something I have been noticing and thinking about recently.

It seems to me, more and more, we read or hear nothing but negativity about testing. That is to say, when testing comes up in many people's blogs, it is only to criticize the lack of ability in the testing craft, highlight the deficiencies of checking, complain about what Testers think, what Testers are, etc. As Testers, I think we do a very bad PR job on what testing is and brings.

I am a big admirer of some of James Bach's ideas and principles. However, I am not a fan of his abrupt approach to Testers, who don’t sign up to his ideas and principles. To be a Tester, you need to possess an open mind, you need to see all sides of everything and then make your own judgment call. Sometimes, your call won't tally with others, that’s fine, but it doesn’t give anyone the right to belittle you, because your way of seeing testing is different. There is still a way to be a person, sometimes; I think James fails the test here.

Testing is a specialized industry, you need to be a different type of thinker than a regular programmer to be a Tester, be more of a questioner. As opposed to damning Testers constantly and surrounding testing with an air of negativity, we should be trumpeting committed people involved, embracing the committed who are in the Testing domain (even if their thoughts on Testing differ from ours).
Even at interviews, negativity reigns: -

I want to be a Tester because I like breaking things

is an answer that is so stereotypical, it gives clichés a bad name. However, it is so limited. If I wanted to hire someone to break things, I would hire my 2 year old son! But that's the aura that has been created about testing, to break things, to prove things wrong, to dispel assumptions - Negativity!

The purpose of testing is surely to provide information on the product. That's it, pure and simple. Whether that be via the 'frowned upon' checking approach, manual testing, 'super cool' exploratory testing, automated regression suites or whatever individual approach; It is all testing and it is all to provide information - good and bad. However, the message that seems to filter to the top is testing is to provide bad news.

Think of it this way:-
The message about Testing needs to change. We need to promote testing as a valuable industry, a valuable asset to projects, a place where you can advance and hone your skills, create new trends, create new ways of testing, become the difference, gain respect, gain enjoyment, make a career.

Now, read the above paragraph again, but change the spin and think of it positively!

The Principled Tester

What type of Tester are you?
Are you the ‘process driven’ tester? Are you the ‘checks pass so we are ok’ driven tester? Or are you the ‘fly by the seat of my pants, full on exploratory son of a gun’ tester?

All have their roles in the Test industry, there will always be companies who need each of the types (and more!) mentioned above.

The type of Tester you are really comes from within. It depends on a number of things; such as attitude, intelligence and experience. Overall though, every Tester worth their salt will have their principles. That is to say, they hold dear various things that are important to them when it comes to Testing. Some don’t, some are very much 9-5 type Testers with no real genuine interest or passion in what they do or the consequences of their actions. In my eyes, those people shouldn’t even be classified as Testers, they are hired hands doing (more than likely) a below average job. Eventually those people are, or will be, weeded out.

But back to the true Testers, yes, they have principles. I would hazard to guess if we pulled into a room, some of the world’s best Testers, their principles would differ…but probably only by a hairs breadth!
Their language explaining it may vary, their rating of their principles may differ, and some may even see them as not being principles but just part of their make-up. However, hair splitting aside, they will be similar. Core principles are common among good Testers.

Now, whilst I wouldn’t attempt to place myself in such exulted company as among the best Testers in the world (whomever they may be), I believe I have ‘some game’ and see myself very much so, as a Principled Tester.

I place my Testing principles at the very forefront of what I do for a living and indeed, influence me in my job every day. I wish not to impose these on anyone, but for the record they are below. I call it my T.E.S.T.S. approach.

Test the right things
Preparation
Knowledge of AUT with a Test Journey in Mind

End the day never satisfied
Must do better mantra
How can I improve that?

Strength of Purpose
Willing to admit you may be wrong – Humility
Stand by your theories and thoughts – don’t be swayed because others have
Stay true to your testing compass
Take ownership for testing the product/application

Thoroughness
Methodical mindset
Attention to detail
Going the extra mile, run the extra Test, answer the nagging ‘what if’

Speak the truth to power
Your job is to provide information not ‘the sugar coat message’
No Testing is better than poor Testing
Keep others honest


They are in no particular order, but to get the T.E.S.T.S. handle in there, I had to play with the order ever so slightly

Whether we like it or not, Testing is a results driven business. We have project delivery time constraints and managerial pressure and expectation for the right message.
One would think this type of environment wouldn’t lend itself to allowing for principles? It can!

You can still deliver on time, give the right message and be principled.
How?
Project delivery on time can be achieved as long as you provide the evidence which details what you have found as a result of your Testing, on time. Testers are not in the project delivery business, that is not their job. If someone else wants to deliver based on your evidence, with or without your recommendation, that decision is theirs.

Managerial pressure for the right message will always be there, but if the message you provide is that you have found issues/defects that are detrimental to what the project states it should do, then you have delivered the right message because it has provided information related to the health of the project.

Again, what Managers do with that message is not the Testers job.
So you have delivered on time, gave the right message AND stuck to your principles.

First, Fast, Now Testing

Testing in today's world has to be done yesterday. Such are customer demands and in turn, project demands. Good Project Managers and Companies at least attempt to ensure that testing is not squeezed as a result and that the due time and attention is given to ensure at least adequate testing, is carried out. But a lot of the time, their hands are tied.

So how do we ensure adequate testing in a world that is deadline driven?
Any UK and Ireland Sports fans among us will be familiar with 'Sky Sports News'. Essentially this is an around the clock news channel dedicated to sport, almost sycophantically at times reveling in the 'exclusives'. Their philosophy is 'First, Fast, Now'.

I liken agile testing to this, or any testing in today's world. Many companies carry out 'sky sports testing'. The need to do everything immediately, quickly quickly, 'best we can'.

In some instances, that may be fine, as long as you can be confident that you have covered all you think you should and provided the stakeholders with as much information they need to make their decisions on whether to go live. You have done as much as you can. This is a skill that the better testers have. Usually not through design but neccessity.

However, sometimes as Testers, we need to push back, sometimes we need to incorporate a 'steady as she goes' mentality. Take the time to do the testing, investigate, explore, provide more information that merely the confirmatory type.

Hand on heart, how many times do we ever do this?

Admittedly, we are testers and under the direct management of others who are under the direct management of those higher, who want the project done without delay. 'Sure, we can get the customers to test it'. So we might not be able to take our time and perform the due diligence.

Again, that's fine. As Testers, our primary role is to provide information to the health and status of a project. The time we are given to do this is usually someone else's call. As long as that someone is aware that there is the chance that 'Sky Sports Testing' might not lead to a 'Sky Sports Product'.

Wouldn't it be great though, just a little more often, to be asked by PM, 'how long would you like to test' as opposed to 'here is what is left to test, what can you do?'

I’m a tester, I can’t waste time doing testing activities?

I was involved in a discussion one day about the regression testing activities and more implicitly, the post execution activities. Someone was making a point that validating the results and outcomes of the Tests was ‘taking too much my time’.
This point intrigued me and my initial thoughts were ‘ok, so what did these validations prevent you from doing that was more important, than a key aspect of your actual job?’ I simply don’t get this.
As a tester, you have many aspects to your job but the general thesis is this, you create and execute Test Cases (be it exploratory or scripted checks) with the intention of finding out information to provide feedback on the health of a project. As part of this execution, there is an inevitable ‘wash up’ activity that involves verifying the results. That is, to say, did the outcomes match what was expected or is there a potential issue? There is no way around this, the results need to be verified (do you ever hear TV talent shows say before a result is announced, ‘the results have been counted but we don’t need to VERIFY them’ – NO).
If you don’t verify the results, how can you:-

• Provide meaningful information?
• Find out if there is a problem?
• Carry out further testing?
• Improve your testing?

How you do this verification is a different discussion. If it is manual, it will take longer than if you have automation that can spit out results and potentially examine them as well.
It is my belief that automation has been the best and the worst thing that has happened to testing in the past number of years. It has helped improve monotonous checking, increase the speed of provision of information through quicker execution and when used properly, extremely powerful to various areas of testing such as reporting and data entry. However, I believe automation has made testers and testing LAZY. Some Testers (as well as Managers) now see automation = testing and anything that cannot be automated is not worth testing. Further to that, some testers are seeing executing automated Tests as their entire testing activities…period. In other words, the automation runs when it is finished, I am done and nothing else is required. This is equivalent to saying it NEEDS to provide Passes and the Green bar so as to not mean I spend time verifying the results.
This mindset is damaging to the core aspects of testing and actually quite insulting to the good intentions of the keepers of the testing industry.
All Testers want to be providing value, being involved in the creation of the next big Testing thing, creating solutions. However, all Testers need to be true to the core aspects of their job; the creation of Test Cases, Test Case execution and Test Case execution FOLLOW UP.
There are no workarounds here, no quick fixes, its simply part of ‘doing your job’. If you are a tester who doesn’t believe in these facets of your job, my advice is quite simple ‘Testing is a waste of your time, time for a career move’.