How Much Time Takes Your Smoke Test?

The first sign of smoke is always an early indicator that fire is not far away. That is the idea behind the smoke test – quick test to get confirmation that after a change system works as usual.

Every tester tend to agree to this, but the fun starts when I ask: “how quick is “quick”?” or more specific: “how much time takes your smoke test?” In trainings usually I get  answers between 10 minutes and several hours. In case of hours I ask what is the difference between smoke test and regression test. We wanted to check the smoke, not to retest all the features, right?

My smoke test consist of one main pathway, nothing fancy, just straight through. Tipically it takes 30-40 seconds for automated test and around 5 minutes for manual test. If I observe “smoke” than first step is to inform a team (during development phase) or a customer and operations team (if software is already on production). In case of smoke team needs to investigate why it is not working and also why unit and integration tests did not find it, what do we need to improve to get feedback earlier. If smoke in on production in most of cases first step is roll back and only when system is stable again, we can analyze what happened and why. If there is no smoke, than continue with regression tests or few selected checks of new features.

 

How To Explain #ExploratoryTesting in 15 Minutes

Every other week I explain basics in software testing, one of them is exploratory testing. It depends from group to group, but sometimes I have only 5-10 minutes on the topic. I love challenges! But I am also aware that I am still learning myself. This is why I asked my peers during Exploratory Testing Peer Conference: how to explain exploratory testing in 5 minutes?

How Did I Explain It So Far

Because I give trainings frequently, I experiment a lot with explanations and observe reactions of students. This is what I have used so far:

+ I used to start the topic with question if they are doing exploratory testing, if yes, then how? Sometimes I needed to interrupt and explain that monkey testing a.k.a. klicki bunti (German expression for mindless clicking) is not exploratory testing. Some students felt bad to find out the difference between their approach and real exploratory testing, so I stopped to ask for student’s experience.

+ my favourite example is sightseeing in unknown city and following unexpected events on the way to planned location. Students seem to benefit from it as well. But one thing I am missing in this case – note taking. Now I mostly use sightseeing as light introduction to the subject. Works well if I traveled to unknown city and day before had time for sightseeing, or if students traveled to unknown city and are planing to have time for sightseeing, or simply several people in the group are open-minded travellers.

+ for in-house trainings once I tried to apply exploratory testing on their system, it didn’t work well. I had not enough information what the system should do and I needed time to gather information. Mission impossible if you have only 10 minutes for the whole thing.

+ once I had group of 14 and almost everyone was somehow involved with football. I used it to encourage the group to look for similarities with exploratory testing. It worked very well, but again almost none of students mentioned or thought about note taking. I retired this analogy for explorative testing, but started to use it to explain test levels.

+ job interview is example which is used by my colleague. It works very well for him, because we can use several techniques, note taking inclusive. This piece almost always cracks up the group, because we use exaggerated situations like:
A: can you please tell us, why did you moved to Australia?
B: I always wanted to live aboard. Besides I hade trouble with mafia in Europe, so I thought why not to develop software somewhere far away!

+ in case I used example without note taking, I explained it separately. I some trainings I showed notes, I took for my group during Anne-Marie‘s exploratory testing workshop with BigTrak robots. Because it takes additional time, I prefer to choose examples with note taking.

I always suggest my students to read “Explore it!“, but I definitely needed more ideas what I can use during training!

Peer suggestions

This is what I got suggested during my session. Most of the ideas are still raw. To visualise authors idea from my comment, I used italic and named the author, who suggested it. The one I started to use in my trainings I listed up as the last one.

Exploratory testing as performance

Jokin: I could explain how to play guitar vs I can show how to play guitar.

I love this approach! I use it with something else, not guitar playing, but in our Berlin office we have a guitar and some of my colleagues really play it.

Lightning talks

Rick had idea to present exploratory testing in lightning talk. I cannot imagine how I could use it in my setting, but may be this could be useful for others.

Flow charts

Alex made her own flow chart, it’s always going back to exploratory testing, showing that you have to learn it anyway.

You can read about it more here.

Selective Attention

Maaret reminded about gorilla videos where you are supposed to count the passes so you don’t see the gorilla, and if you see the gorilla you cannot count the passes. In my topic introduction I forgot to mention that I have used this few times and stopped because too many knew the videos already.

unexpected truth

James on the spot sales pitch: “Systems are weird. Are you looking for trouble? ET can help you to find unexpected truth, about what you really got.”

Games

Ash used go-karts a lot with you people; many were excited about them; planned a lot which car to take; interesting things happen when you’re actually in the car. Lisa used Black Stories.

Improv game

Mel: there are so many improv games to use, can get even physical; “What’s next?” I know people who are practising improv, they say it is their life mindset. I understand the idea, but I do not use it, because I am not really into it.

Get the chocolate

Alex: two groups, goal is to get the chocolate on the other side of the room; 1) write down all steps to get the chocolate, 2) allow them to go directly; then put the chocolate to another place; mean but it will stick. I used similar approach for test automation and we have another game which we play to explain agile approach and team dynamics.

Autonomy / mastery / purpose

Anne-Marie: this concept is often appealing to people.

Escape rooms

Mor: describe exploratory testing using escape rooms; looking for riddles, paths.

Don’t teach it

Martin: you have a room full of tired people, don’t teach exploratory testing; just keep referencing to it! When I heard this I started to laugh, because this is exactly how I am building my next training.

testing mathematics

Maaret suggested puzzle with two ways to solve it – scripted and exploratory. Approach is based on children game where you think of one thing, write it down and hide(fold the paper); people can ask question, you can only answer yes/no; either have them write down all questions in advance, or have them ask, hear the answer and formulate the next question based on the answer.

What I am using now

I never have two identical trainings because there are no two identical groups. Testing mathematics has become my favourite way how I explain exploratory testing. I set it in timebox and limit number of questions. After first round I encourage people to analyse how they did and and how they could do it better, I also give feedback which my observations. Then we have the second round where they can apply the learnings. At the end I explain how they can transfer this approach for software testing.

Besides sightseeing and job interviews I started to use another examples. Both were inspired by Mor’s idea of escape rooms – investigative journalism and crime investigation.

In November I will be presenting “How to explain exploratory testing in 10 minutes” at Agile Testing Days. Come and see me explaining it!

My Story of Test Automation

I use to think I am not technical and very far away from the thing called Test Automation (TA). Then I started to work at this one company, who were having huge quality problems. I asked how automation looks like and found out that they do not have any (yes, even no unit tests) and that they currently work on defining CI pipeline. This is how my operation and test automation journey started. It turned out that even I thought I am not technical enough and knew a bit about TA, others knew less or were not motivated to introduce it to the company.  Next few years I worked closely together with operation team, supported creation of CI pipeline, defined processes and introduced automation.

One of the first thing that I did – I asked developers from each project to explain how the build process works in his/her project. I found out that many developers did not know how it works. I found out that some developers did not know how to build release and were deploying snapshots. Why did I ask developers about something that usually should be done by operations? Because problems do not appear in operations or in development, they appear where everyone should work together. Another thing – we wanted to create general approach and process to enable employee switchover in case of necessity (vacation or illness).

Parallel to building CI pipeline I started to automate. Just as every other beginner I made many, many mistakes on the way. At the beginning, I was too focused on the tool and too less on the mindset about automation. During TestBash Brighton 2015 several other attendees stayed in my hotel, and I remember that one morning  we all came together at the breakfast table. It was very energised and inspiring morning. That breakfast was an eye-opening moment for me. Richard Bradshaw was sitting at the table as well and told me about his Lego automation workshop, I could see immediately how to implement some of his ideas into project I was currently working on. I was sitting there trembling and panicking – is it too late to change our approach? Some of the projects were already automating everything – a test for each acceptance criteria in a user story. Developers were not involved in creation and maintenance of test automation, but hoped that GUI automation will be their safety net and supported customers wish to automate ALL tests. We started to observe problems with testing and some people got very disappointed. I saw it but did not understand where it comes from and how to fix it. TestBash and ongoing conversations with others helped me to see the cause. I started to learn more about mindset and realised that by automating everything, I have build technical debt. We needed test automation strategy!

If you think that this story has happy ending, I have to disappoint you. One thing I have learned in 20 years of IT – it is not about tools, methods and processes. It is about people and people are stubborn. I saw my mistakes, I learned a lesson and of course it was hard for myself to accept that some of my mistakes were so huge and I misled others. But denial or to hide evidence has never been something what I do. I went back to the teams I was working with, explained what I have discovered and what I want to change and… they refused my suggestions! They thought it is OK how it is running. I tried to point out to problems we were having, but they found another reason and explanation. They just wanted to keep to automate everything…

Soon after that all happened, I felt urge to speak about it and make people aware of pitfalls. I started to give workshops for other in-house teams, our project managers and for customers. I had so much fun by doing that that I got the idea to do workshops and trainings full time. Fast forward – here I am, working as full time trainer and guess what I talk the most in my trainings? Yes, mindset! And very little about tools and concrete examples, because I still need to learn a lot and you could have better ideas than I do. This is my happy ending.

If you want to learn more about automation tests, I suggest you to visit evil testers website.

 

Approach To Test Automation

Test automation – frequent topic in testing community, but for businesses and developers it is still something new and unknown. This blog post is very simple, high level my lessons learned in the field.

Strategy

Start by choosing your test automation strategy.  Business sees Test Automation as way to save money. The problem is, if you don’t have strategy and simply write 1000 automated tests, then sooner or later your team will become slower and slower. To pick up speed and save money, test automation has to be flexible and lean. Forget about testing-all-the-things, it is not possible and is wasteful. If your system has 1000 automated tests, your team has to maintain 1000 automated tests. Depending how you created your tests and how oft you will change your software – it can be a LOT.

Martin Fowler created great thinking tool: Test Automation Pyramid

Focus on unit tests, because this is the fastest way how developer can find out that her/his new code broke already existing code.

Testing in Agile world

If you are living in agile world, I would suggest you to read “Agile testing” by Janet Gregory & Lisa Crispin. They have great chapter on test automation with thought provoking mind map. I learned from Janet, test automation approach to write e2e tests for epics, integration tests for stories and unit tests for tasks.

Remember: in agile we believe in zero-bug policy. That means as well that all automated test need to be active and results has to be green. all the time.

e2e test Criteria

Test Automation criteria is part of your strategy because it answers the question: What to automate? Typical approach for e2e is to automate:

  1. only standard features
  2. only features with high risk
  3. only most used configurations

Many teams use that criteria to reduce test automation which has grown too big. My suggestion in trainings is – from very beginning automate only really necessary test cases. If you doubt, then don’t automate on e2e level, talk with your team how you could catch it on integration or unit level.

summary

The biggest challenge in Test Automation is not to write a script, but to decide what to automate and what not to automate. You will easily find people who can automate 1000 tests, but not so easy to find people who can create test automation strategy and efficient test automation design. Just like there are many software developers, but not so many software architects.

If you have strategy in place then you will understand how wrong are statements similar to this: “I often write e2e automated tests for bugs raised by customer because if we made that mistake once, we will likely make it again”. We have to learn from bugs raised by customers, we need to understand what went wrong, but we cannot test everything and it is very unprofessional to automate everything. Testers and developers should learn to think profitably.

If you want to learn more, I suggest you to check what Angie Jones wrote about test automation, or better, her free course on test automation strategy.

Update

Few days after I published this post, Lisa, in cooperation with Joe, published amazing collection of Test Automation Models. My suggestion: explore Rob Meaney’s Risk Mapping and Katrina Clokie’s DevOps Bug Filter. Maybe it is something for your team.

Words Has Meanings: Learning

This is conversation between two friends

Kris: I have to tell you a secret… I have big problem with “learning” if we talk about ET (exploratory testing).

Lisa: Hmm… It’s always been taught to me as “learning about our product”. But I am not an ET expert by any means.

Kris: For me “learning” means “to change behaviour”

Lisa: oh, interesting! for me it’s just gathering knowledge

Kris: Gathering information doesn’t include using it. You use information by changing your behaviour (huge excursion in my experience as mother and trainer). But only thing what we want to change is the software. So no real learning on human side.

Lisa: We might learn about features we are missing?

Kris: That is functional or contractual acceptance testing. No learning.

Lisa: So if I do exploratory testing on some feature, and then I realize, before customers can use this feature, they need another capability that we haven’t even thought of before – I didn’t learn something? Or if I find something is really hard to use – I didn’t learn something?

Kris: How do you know that they need something else? Hard to use – do you mean usability? Testing usability?

Lisa: How do I know? Because I’m exploring as a particular persona or role or job, I have a scenario of what I want to accomplish with the app, and I run into a roadblock. The persona is blocked, Lisa has learned that we didn’t provide a necessary capability.

We might be splitting hairs on words and semantics, but exploratory testing is called a process of learning, and it has seems correct to me.

Kris: For me it is very important to clarify what we mean by words. I always liked language (in Latvian we have more then 30 words how we call a mother), but as older I get as more aware I am about layers of language, coded messages and communication in general. People, who name things, make mistakes (just like everybody else). I am not looking for fight… It just feels wrong to call it “learning” if we only gather information.

Lisa: OK

Kris: The way how I understand whole team approach is that if we have situation as you described it, the tester in the team should be able to identify and categorise the problem. Is it functionality, usability or performance? then to decide what following tests should the team run to gather missing information.

Am I aiming too high?

Lisa: IME it works best if the tester collaborates with the team to do all that. We don’t want to be the safety net so that everyone relies on us to point out issues. We want to help teammates prevent the issues from happening in the first place by building shared understanding of features, using good tech practices for code correctness, fast feedback loops with automation, exploratory testing before committing changes…

More a consultant role than doing all the testing for the team.

You know, you have some interesting ideas here, it would be nice to discuss this on the AgileTestingFellow slack.

Kris: Maybe… but I don’t feel there yet. And I didn’t mean that tester is a safety net. What I meant is that tester has this knowledge about testing techniques and approaches and she/he guides the team. This is what I understand with tester as testing coach for the team.

Lisa: Sure but we have to be continually helping everyone else ramp up those skills. If we make all decisions ourselves, we take autonomy about testing away from the rest of the team. Guides, yes, that’s the ticket

Kris: Yes, helping others. But tester should be able to recognise it her-/himself first. Only then she/he can guide others.

Lisa: I like what you say there about recognizing it in ourselves.

 

Post Scriptum

I really, really LIKE how James Lyndsay phrased it:
“Systems are weird. Are you looking for trouble? Exploratory testing can help you to find unexpected truth, about what you really got.”

Exploratory testing is technique to find out, gather information. As a team we will decide later what we want to do with information with gathered. Maybe we will ignore it, because risk is too low, maybe we will use it and make decision based on it. And maybe we will learn out of it, adapt our behaviour and leave the issue in the past. If it will not reappear, then we really can tell: “we learned something!”

Mini Test On Production

Yesterday I changed my Twitter handle, today I want to test, what happens if I do not update the change on WordPress and publish a blog post.

I expect to happen one of these:

  1. cannot publish blog post
  2. cannot publish blog post, error message that there is no such handle
  3. can publish blog post, but no tweet on Twitter
  4. can publish blog post, there is tweet on Twitter for ghost account with old handle
  5. can publish blog post, tweet on Twitter with new handle
  6. something else

UPDATE after the test

Test was successful. Blog was published with right Twitter handle.

What I like a lot, that WP updated automatically to the new handle. If I choose to write another blog post than it is already filled with the new handle. No human interaction necessary! 

 

Update: I found one thing that broke – Twitter widget in the sidebar.

#30DaysOfTesting – ECommerce Task II & IV

Previous in this series: Task I & III

Task 2: Read and share an interesting blog about ecommerce testing

Task 4: Find and share a useful video on youtube about ecommerce testing

One testing website I like and read since I started to test software is Software Testing Help. Vijay has done amazing job by collecting all kind of testing ideas and helping so many rookie testers. 8 Important Segments Of Testing eCommerce Websites is very good place where to start if you are starting to test retail software.

For advanced testing or as Daniel says at the end – to put a smile on your face – do some penetration testing.

#30DaysOfTesting – eCommerce Task I & III

Task 1: Look up some definitions for ‘ecommerce’, from these create and share your own definition

Task 3: Join the #ecommerce channel on https://testers.chat and introduce yourself!

e-commerce

For me e-commerce is a system where you can exchange all kind of goods and services. I used word “system” because there is more what the eye meets. In those digital transformation projects where I worked and e-commerce was a part of the project, companies were unprepared how big their e-commerce system can be.

#ecommerce channel

For those who use Slack : you will find Testers.chat under testersio.

I like that this time MinistryOfTesting involve other testing communities into challenge and created channel on testers.chat.

Each of us have some good tools in their toolbox. If we put those together we could help the domain to get better and lighten up entry for rookies.

Explaining Software Testing

Michael Bolton is good in summarising fundaments of software testing in a tweet. In my eyes this piece of information deserves a little blog post.

The last sentence is something what I teach my ISTQB Foundation Level students – you cannot assure that there are no bugs, but you can collect information about bugs you have found. You can analyse them e.g. with aim to create set of actions how to secure your software developing process from reappearance of that type of bugs.

Most of developers want to know if a feature does work or not, but we as testers can only say that Version A did work on Machine B under Circumstances C.  Project managers and customers want to know if it will work on production. We as testers cannot assure it, because we did not (and we should not) test on production environment with production data. Based on tests on similar environments and under similar circumstances we can suppose that it can work.

Sounds very logical, but it can escalate very quickly, because DEVs, PMs and customers think that testers are there to save the word and to give certificates that the software they are working on, works perfectly as described.

What to do if you still get asked:

– “What do you do whole day long if you cannot tell me will it work or not?!” 

In heated situation like this it is too late to explain semantics, fundaments of testings or why software development process is complex. You should done it as soon as possible, when you started to work on a new feature, on new project, in new team or new company.  Remember – one of most important skill of a tester is communication skills. We need those not to be able to talk about the weather, but for explaining what is testing and how testing can help to DEVs, PMs, customers and all others. Daily.

Michael suggests 4 step plan how to learn fundaments of software testing:

  1. Learn how to test: How can a trainee improve his/her skill sets in testing?; To The New Tester
  2. Declare your commitments: A Tester’s Commitments
  3. Recognise that all testing is heuristic: Heuristics for Understanding Heuristics
  4. Learn to tell the testing story: How is the testing going?

Looks simple, but as you start to read the linked resources you will understand that studies can take years. Take deep breath. To be able to explain testing to others, you have to learn it first for yourself. Do not panic if it does not work on a first try. Make experiments, try new approaches, improve your skills. One day you will master it!

#30daysofsecuritytesting – February

February is over, but my 30 days of security testing challenge is not done yet. I have done only 11 of 30 so far: I, II, IV, V, VII, IX, X, XII, XVIII, XX, XXX, and I am not thinking to give up. This has been amazing learning journey! I always wanted to learn more about web security but never had real reason or time to do it. Challenge helped me to realise it is not so difficult as I thought it would be. Those 30 tasks are like a map with turning points and there is so much information if you know what you are looking for.

One of my information sources is YouTube – you will be very surprised to find out how many conferences upload the talks on YouTube. Like this talk from Troy Hunt.

In his talk Troy shows several examples with insecure passwords. It is something what I could definitely use for testing.

Thank you: Melissa Eaden, Claire Reckless and Dan Billing for putting this challenge together. I am very intrigued how it will go on.