Test Retreat

Welcome to the homepage of Kris Corbus

Archive for the category “testing”

Test Software Guide

In the last decade test software development has moved from being a cult technique to an increasing part of the mainstream. I was lucky enough to be at the beginning of this story, with early experiences on the ‘birth project’ of and a co-author of the Manifesto for Test Software Development. Trendig started using test techniques in 2000 and we’ve since successfully used them on our projects world-wide. We’ve learned a huge amount about using test methods in enterprise settings and are committed to sharing this learning to help foster their intelligent adoption.

The Essence of Test Software Development

It’s been over a decade since the developers of test methods first started to talk about their approaches. In this time test thinking has changed from a niche activity to an approach that is widely used. However, like any popular technique, test software development has suffered from semantic diffusion, so much of what we see under the name of test doesn’t bear much resemblance to what the early pioneers were doing. So I think it’s important to revisit the essential elements of test thinking.

I’ve always seen the essence of test thinking resting on two contrasts with traditional code-driven software engineering:

Test Development

      • is adaptive rather than predictive

      • is quality-oriented rather than process-oriented

Code-driven engineering expects us to come up with a predictive plan that precedes development. The plan lays out the people, resources and timelines for the overall project. Software design is also done up-front, with implementation expected to conform with this design. Success is measured according to how well development follows this plan.

Test plans are a baseline that we use to help us control changes. Test teams plan just as carefully as traditional teams, but the plans are constantly revising to reflect the things we learn during a project. Success is based on value delivered by the software.

Code-driven engineering seeks a process which provides enough structure to reduce individual variations to insignificance. Such an industrial process is more predictable, copes better when people transfer, and is easier to define skills and career paths.

Test engineering sees software development as a primarily human activity, where the people involved and how they bond as a team are the primary driver behind success. Processes (and tools) can enhance a team’s effectiveness, but are always second-order influences.


April April

As you may be guessed by now – there is no such thing as Test Software Development and I am not the author of this text. Original title is “Agile Software Development” and it was written and published by Martin Fowler. My creativity was only to replace some of words, like “agile” with “test.” 

Stay healthy and stay funny!



Please Apply Critical Thinking

Last week I gave a software testing training and when we arrived to chapter about experienced based testing, one of my students asked: “do you mean monkey testing?” I tried to stay as calm as I can and asked back: “what is monkey testing?” She started to describe it with several examples. And to each of her example as summary I asked: “so you mean negative testing? …could it be that in this case you mean reliability testing?” and so on. Another students jumped in and said that he is doing monkey testing as well and gave few examples and I again asked questions: “so you mean stress testing?

After few minutes they stopped, all class looked at me and asked what is then monkey testing. I asked back: “how do you know that it exist?” Almost immediately both replied: “our developers call it like this. I was new to testing and thought it is a thing.

I like to believe that developers, when they named something monkey testing, did not want to harm testers. I also think it is absurd that monkey testing has a wikipedia page, but Lisa Crispin does not. 20 years ago when I studied computer science, I was not taught what software testing is. I keep hearing that this is still the case in majority of universities and technical schools where program developers learn their craft. How you call something you don’t understand and don’t want to understand? Witchcraft? Magic? But if you don’t think it is important or valuable?

Testing is about seeing behind visible, hearing unspoken, sensing strange vibes. Listen to what people are saying, but don’t assume that it is 100% correct. There is no one truth! Each of us has our own truth, coloured by our experiences, knowledge, relationships. Don’t believe something just because it has wikipedia page, people who wrote and edit it are biased as well, just like you and me.

What to do? Apply critical thinking!

A statement by Michael Scriven & Richard Paul, presented at the 8th Annual International Conference on Critical Thinking and Education Reform, 1987:

Critical thinking is the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action. In its exemplary form, it is based on universal intellectual values that transcend subject matter divisions: clarity, accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness.

Pretty complex statement, right? Here is simpler version from Wikipedia:

Critical thinking is the analysis of facts to form a judgement.

How do you form a judgement? It is much easier to believe somebody who shares information with confidence. Especially if that somebody is a person with status or power. Teen years are partly so hard for parents, because they are not ready to loose their status and power as main information bearers for their children. It is easy to manipulate those who depend on you, impossible if they don’t. To learn and to apply critical thinking looks like lifelong task. You cannot question everything. What is important to question?

My story on critical thinking

Very early I learned that grownups, who say lying is bad, lie themselves. I remember that situation very vividly because it changed my life. Since that moment I never believed one-sided story. I always question motives and seek opposite version to come up with my decision. This is how I run.

What do you think – monkey testing does exist?

How Much Time Takes Your Smoke Test?

The first sign of smoke is always an early indicator that fire is not far away. That is the idea behind the smoke test – quick test to get confirmation that after a change system works as usual.

Every tester tend to agree to this, but the fun starts when I ask: “how quick is “quick”?” or more specific: “how much time takes your smoke test?” In trainings usually I get  answers between 10 minutes and several hours. In case of hours I ask what is the difference between smoke test and regression test. We wanted to check the smoke, not to retest all the features, right?

My smoke test consist of one main pathway, nothing fancy, just straight through. Tipically it takes 30-40 seconds for automated test and around 5 minutes for manual test. If I observe “smoke” than first step is to inform a team (during development phase) or a customer and operations team (if software is already on production). In case of smoke team needs to investigate why it is not working and also why unit and integration tests did not find it, what do we need to improve to get feedback earlier. If smoke in on production in most of cases first step is roll back and only when system is stable again, we can analyze what happened and why. If there is no smoke, than continue with regression tests or few selected checks of new features.


How To Explain #ExploratoryTesting in 15 Minutes

Every other week I explain basics in software testing, one of them is exploratory testing. It depends from group to group, but sometimes I have only 5-10 minutes on the topic. I love challenges! But I am also aware that I am still learning myself. This is why I asked my peers during Exploratory Testing Peer Conference: how to explain exploratory testing in 5 minutes?

How Did I Explain It So Far

Because I give trainings frequently, I experiment a lot with explanations and observe reactions of students. This is what I have used so far:

+ I used to start the topic with question if they are doing exploratory testing, if yes, then how? Sometimes I needed to interrupt and explain that monkey testing a.k.a. klicki bunti (German expression for mindless clicking) is not exploratory testing. Some students felt bad to find out the difference between their approach and real exploratory testing, so I stopped to ask for student’s experience.

+ my favourite example is sightseeing in unknown city and following unexpected events on the way to planned location. Students seem to benefit from it as well. But one thing I am missing in this case – note taking. Now I mostly use sightseeing as light introduction to the subject. Works well if I traveled to unknown city and day before had time for sightseeing, or if students traveled to unknown city and are planing to have time for sightseeing, or simply several people in the group are open-minded travellers.

+ for in-house trainings once I tried to apply exploratory testing on their system, it didn’t work well. I had not enough information what the system should do and I needed time to gather information. Mission impossible if you have only 10 minutes for the whole thing.

+ once I had group of 14 and almost everyone was somehow involved with football. I used it to encourage the group to look for similarities with exploratory testing. It worked very well, but again almost none of students mentioned or thought about note taking. I retired this analogy for explorative testing, but started to use it to explain test levels.

+ job interview is example which is used by my colleague. It works very well for him, because we can use several techniques, note taking inclusive. This piece almost always cracks up the group, because we use exaggerated situations like:
A: can you please tell us, why did you moved to Australia?
B: I always wanted to live aboard. Besides I hade trouble with mafia in Europe, so I thought why not to develop software somewhere far away!

+ in case I used example without note taking, I explained it separately. I some trainings I showed notes, I took for my group during Anne-Marie‘s exploratory testing workshop with BigTrak robots. Because it takes additional time, I prefer to choose examples with note taking.

I always suggest my students to read “Explore it!“, but I definitely needed more ideas what I can use during training!

Peer suggestions

This is what I got suggested during my session. Most of the ideas are still raw. To visualise authors idea from my comment, I used italic and named the author, who suggested it. The one I started to use in my trainings I listed up as the last one.

Exploratory testing as performance

Jokin: I could explain how to play guitar vs I can show how to play guitar.

I love this approach! I use it with something else, not guitar playing, but in our Berlin office we have a guitar and some of my colleagues really play it.

Lightning talks

Rick had idea to present exploratory testing in lightning talk. I cannot imagine how I could use it in my setting, but may be this could be useful for others.

Flow charts

Alex made her own flow chart, it’s always going back to exploratory testing, showing that you have to learn it anyway.

You can read about it more here.

Selective Attention

Maaret reminded about gorilla videos where you are supposed to count the passes so you don’t see the gorilla, and if you see the gorilla you cannot count the passes. In my topic introduction I forgot to mention that I have used this few times and stopped because too many knew the videos already.

unexpected truth

James on the spot sales pitch: “Systems are weird. Are you looking for trouble? ET can help you to find unexpected truth, about what you really got.”


Ash used go-karts a lot with you people; many were excited about them; planned a lot which car to take; interesting things happen when you’re actually in the car. Lisa used Black Stories.

Improv game

Mel: there are so many improv games to use, can get even physical; “What’s next?” I know people who are practising improv, they say it is their life mindset. I understand the idea, but I do not use it, because I am not really into it.

Get the chocolate

Alex: two groups, goal is to get the chocolate on the other side of the room; 1) write down all steps to get the chocolate, 2) allow them to go directly; then put the chocolate to another place; mean but it will stick. I used similar approach for test automation and we have another game which we play to explain agile approach and team dynamics.

Autonomy / mastery / purpose

Anne-Marie: this concept is often appealing to people.

Escape rooms

Mor: describe exploratory testing using escape rooms; looking for riddles, paths.

Don’t teach it

Martin: you have a room full of tired people, don’t teach exploratory testing; just keep referencing to it! When I heard this I started to laugh, because this is exactly how I am building my next training.

testing mathematics

Maaret suggested puzzle with two ways to solve it – scripted and exploratory. Approach is based on children game where you think of one thing, write it down and hide(fold the paper); people can ask question, you can only answer yes/no; either have them write down all questions in advance, or have them ask, hear the answer and formulate the next question based on the answer.

What I am using now

I never have two identical trainings because there are no two identical groups. Testing mathematics has become my favourite way how I explain exploratory testing. I set it in timebox and limit number of questions. After first round I encourage people to analyse how they did and and how they could do it better, I also give feedback which my observations. Then we have the second round where they can apply the learnings. At the end I explain how they can transfer this approach for software testing.

Besides sightseeing and job interviews I started to use another examples. Both were inspired by Mor’s idea of escape rooms – investigative journalism and crime investigation.

In November I will be presenting “How to explain exploratory testing in 10 minutes” at Agile Testing Days. Come and see me explaining it!

My Story of Test Automation

I use to think I am not technical and very far away from the thing called Test Automation (TA). Then I started to work at this one company, who were having huge quality problems. I asked how automation looks like and found out that they do not have any (yes, even no unit tests) and that they currently work on defining CI pipeline. This is how my operation and test automation journey started. It turned out that even I thought I am not technical enough and knew a bit about TA, others knew less or were not motivated to introduce it to the company.  Next few years I worked closely together with operation team, supported creation of CI pipeline, defined processes and introduced automation.

One of the first thing that I did – I asked developers from each project to explain how the build process works in his/her project. I found out that many developers did not know how it works. I found out that some developers did not know how to build release and were deploying snapshots. Why did I ask developers about something that usually should be done by operations? Because problems do not appear in operations or in development, they appear where everyone should work together. Another thing – we wanted to create general approach and process to enable employee switchover in case of necessity (vacation or illness).

Parallel to building CI pipeline I started to automate. Just as every other beginner I made many, many mistakes on the way. At the beginning, I was too focused on the tool and too less on the mindset about automation. During TestBash Brighton 2015 several other attendees stayed in my hotel, and I remember that one morning  we all came together at the breakfast table. It was very energised and inspiring morning. That breakfast was an eye-opening moment for me. Richard Bradshaw was sitting at the table as well and told me about his Lego automation workshop, I could see immediately how to implement some of his ideas into project I was currently working on. I was sitting there trembling and panicking – is it too late to change our approach? Some of the projects were already automating everything – a test for each acceptance criteria in a user story. Developers were not involved in creation and maintenance of test automation, but hoped that GUI automation will be their safety net and supported customers wish to automate ALL tests. We started to observe problems with testing and some people got very disappointed. I saw it but did not understand where it comes from and how to fix it. TestBash and ongoing conversations with others helped me to see the cause. I started to learn more about mindset and realised that by automating everything, I have build technical debt. We needed test automation strategy!

If you think that this story has happy ending, I have to disappoint you. One thing I have learned in 20 years of IT – it is not about tools, methods and processes. It is about people and people are stubborn. I saw my mistakes, I learned a lesson and of course it was hard for myself to accept that some of my mistakes were so huge and I misled others. But denial or to hide evidence has never been something what I do. I went back to the teams I was working with, explained what I have discovered and what I want to change and… they refused my suggestions! They thought it is OK how it is running. I tried to point out to problems we were having, but they found another reason and explanation. They just wanted to keep to automate everything…

Soon after that all happened, I felt urge to speak about it and make people aware of pitfalls. I started to give workshops for other in-house teams, our project managers and for customers. I had so much fun by doing that that I got the idea to do workshops and trainings full time. Fast forward – here I am, working as full time trainer and guess what I talk the most in my trainings? Yes, mindset! And very little about tools and concrete examples, because I still need to learn a lot and you could have better ideas than I do. This is my happy ending.

If you want to learn more about automation tests, I suggest you to visit evil testers website.


Approach To Test Automation

Test automation – frequent topic in testing community, but for businesses and developers it is still something new and unknown. This blog post is very simple, high level my lessons learned in the field.


Start by choosing your test automation strategy.  Business sees Test Automation as way to save money. The problem is, if you don’t have strategy and simply write 1000 automated tests, then sooner or later your team will become slower and slower. To pick up speed and save money, test automation has to be flexible and lean. Forget about testing-all-the-things, it is not possible and is wasteful. If your system has 1000 automated tests, your team has to maintain 1000 automated tests. Depending how you created your tests and how oft you will change your software – it can be a LOT.

Martin Fowler created great thinking tool: Test Automation Pyramid

Focus on unit tests, because this is the fastest way how developer can find out that her/his new code broke already existing code.

Testing in Agile world

If you are living in agile world, I would suggest you to read “Agile testing” by Janet Gregory & Lisa Crispin. They have great chapter on test automation with thought provoking mind map. I learned from Janet, test automation approach to write e2e tests for epics, integration tests for stories and unit tests for tasks.

Remember: in agile we believe in zero-bug policy. That means as well that all automated test need to be active and results has to be green. all the time.

e2e test Criteria

Test Automation criteria is part of your strategy because it answers the question: What to automate? Typical approach for e2e is to automate:

  1. only standard features
  2. only features with high risk
  3. only most used configurations

Many teams use that criteria to reduce test automation which has grown too big. My suggestion in trainings is – from very beginning automate only really necessary test cases. If you doubt, then don’t automate on e2e level, talk with your team how you could catch it on integration or unit level.


The biggest challenge in Test Automation is not to write a script, but to decide what to automate and what not to automate. You will easily find people who can automate 1000 tests, but not so easy to find people who can create test automation strategy and efficient test automation design. Just like there are many software developers, but not so many software architects.

If you have strategy in place then you will understand how wrong are statements similar to this: “I often write e2e automated tests for bugs raised by customer because if we made that mistake once, we will likely make it again”. We have to learn from bugs raised by customers, we need to understand what went wrong, but we cannot test everything and it is very unprofessional to automate everything. Testers and developers should learn to think profitably.

If you want to learn more, I suggest you to check what Angie Jones wrote about test automation, or better, her free course on test automation strategy.


Few days after I published this post, Lisa, in cooperation with Joe, published amazing collection of Test Automation Models. My suggestion: explore Rob Meaney’s Risk Mapping and Katrina Clokie’s DevOps Bug Filter. Maybe it is something for your team.

Words Has Meanings: Learning

This is conversation between two friends

Kris: I have to tell you a secret… I have big problem with “learning” if we talk about ET (exploratory testing).

Lisa: Hmm… It’s always been taught to me as “learning about our product”. But I am not an ET expert by any means.

Kris: For me “learning” means “to change behaviour”

Lisa: oh, interesting! for me it’s just gathering knowledge

Kris: Gathering information doesn’t include using it. You use information by changing your behaviour (huge excursion in my experience as mother and trainer). But only thing what we want to change is the software. So no real learning on human side.

Lisa: We might learn about features we are missing?

Kris: That is functional or contractual acceptance testing. No learning.

Lisa: So if I do exploratory testing on some feature, and then I realize, before customers can use this feature, they need another capability that we haven’t even thought of before – I didn’t learn something? Or if I find something is really hard to use – I didn’t learn something?

Kris: How do you know that they need something else? Hard to use – do you mean usability? Testing usability?

Lisa: How do I know? Because I’m exploring as a particular persona or role or job, I have a scenario of what I want to accomplish with the app, and I run into a roadblock. The persona is blocked, Lisa has learned that we didn’t provide a necessary capability.

We might be splitting hairs on words and semantics, but exploratory testing is called a process of learning, and it has seems correct to me.

Kris: For me it is very important to clarify what we mean by words. I always liked language (in Latvian we have more then 30 words how we call a mother), but as older I get as more aware I am about layers of language, coded messages and communication in general. People, who name things, make mistakes (just like everybody else). I am not looking for fight… It just feels wrong to call it “learning” if we only gather information.

Lisa: OK

Kris: The way how I understand whole team approach is that if we have situation as you described it, the tester in the team should be able to identify and categorise the problem. Is it functionality, usability or performance? then to decide what following tests should the team run to gather missing information.

Am I aiming too high?

Lisa: IME it works best if the tester collaborates with the team to do all that. We don’t want to be the safety net so that everyone relies on us to point out issues. We want to help teammates prevent the issues from happening in the first place by building shared understanding of features, using good tech practices for code correctness, fast feedback loops with automation, exploratory testing before committing changes…

More a consultant role than doing all the testing for the team.

You know, you have some interesting ideas here, it would be nice to discuss this on the AgileTestingFellow slack.

Kris: Maybe… but I don’t feel there yet. And I didn’t mean that tester is a safety net. What I meant is that tester has this knowledge about testing techniques and approaches and she/he guides the team. This is what I understand with tester as testing coach for the team.

Lisa: Sure but we have to be continually helping everyone else ramp up those skills. If we make all decisions ourselves, we take autonomy about testing away from the rest of the team. Guides, yes, that’s the ticket

Kris: Yes, helping others. But tester should be able to recognise it her-/himself first. Only then she/he can guide others.

Lisa: I like what you say there about recognizing it in ourselves.


Post Scriptum

I really, really LIKE how James Lyndsay phrased it:
“Systems are weird. Are you looking for trouble? Exploratory testing can help you to find unexpected truth, about what you really got.”

Exploratory testing is technique to find out, gather information. As a team we will decide later what we want to do with information with gathered. Maybe we will ignore it, because risk is too low, maybe we will use it and make decision based on it. And maybe we will learn out of it, adapt our behaviour and leave the issue in the past. If it will not reappear, then we really can tell: “we learned something!”

Mini Test On Production

Yesterday I changed my Twitter handle, today I want to test, what happens if I do not update the change on WordPress and publish a blog post.

I expect to happen one of these:

  1. cannot publish blog post
  2. cannot publish blog post, error message that there is no such handle
  3. can publish blog post, but no tweet on Twitter
  4. can publish blog post, there is tweet on Twitter for ghost account with old handle
  5. can publish blog post, tweet on Twitter with new handle
  6. something else

UPDATE after the test

Test was successful. Blog was published with right Twitter handle.

What I like a lot, that WP updated automatically to the new handle. If I choose to write another blog post than it is already filled with the new handle. No human interaction necessary! 


Update: I found one thing that broke – Twitter widget in the sidebar.

#30DaysOfTesting – ECommerce Task II & IV

Previous in this series: Task I & III

Task 2: Read and share an interesting blog about ecommerce testing

Task 4: Find and share a useful video on youtube about ecommerce testing

One testing website I like and read since I started to test software is Software Testing Help. Vijay has done amazing job by collecting all kind of testing ideas and helping so many rookie testers. 8 Important Segments Of Testing eCommerce Websites is very good place where to start if you are starting to test retail software.

For advanced testing or as Daniel says at the end – to put a smile on your face – do some penetration testing.

#30DaysOfTesting – eCommerce Task I & III

Task 1: Look up some definitions for ‘ecommerce’, from these create and share your own definition

Task 3: Join the #ecommerce channel on https://testers.chat and introduce yourself!


For me e-commerce is a system where you can exchange all kind of goods and services. I used word “system” because there is more what the eye meets. In those digital transformation projects where I worked and e-commerce was a part of the project, companies were unprepared how big their e-commerce system can be.

#ecommerce channel

For those who use Slack : you will find Testers.chat under testersio.

I like that this time MinistryOfTesting involve other testing communities into challenge and created channel on testers.chat.

Each of us have some good tools in their toolbox. If we put those together we could help the domain to get better and lighten up entry for rookies.

Post Navigation