My Story of Test Automation

I use to think I am not technical and very far away from the thing called Test Automation (TA). Then I started to work at this one company, who were having huge quality problems. I asked how automation looks like and found out that they do not have any (yes, even no unit tests) and that they currently work on defining CI pipeline. This is how my operation and test automation journey started. It turned out that even I thought I am not technical enough and knew a bit about TA, others knew less or were not motivated to introduce it to the company.  Next few years I worked closely together with operation team, supported creation of CI pipeline, defined processes and introduced automation.

One of the first thing that I did – I asked developers from each project to explain how the build process works in his/her project. I found out that many developers did not know how it works. I found out that some developers did not know how to build release and were deploying snapshots. Why did I ask developers about something that usually should be done by operations? Because problems do not appear in operations or in development, they appear where everyone should work together. Another thing – we wanted to create general approach and process to enable employee switchover in case of necessity (vacation or illness).

Parallel to building CI pipeline I started to automate. Just as every other beginner I made many, many mistakes on the way. At the beginning, I was too focused on the tool and too less on the mindset about automation. During TestBash Brighton 2015 several other attendees stayed in my hotel, and I remember that one morning  we all came together at the breakfast table. It was very energised and inspiring morning. That breakfast was an eye-opening moment for me. Richard Bradshaw was sitting at the table as well and told me about his Lego automation workshop, I could see immediately how to implement some of his ideas into project I was currently working on. I was sitting there trembling and panicking – is it too late to change our approach? Some of the projects were already automating everything – a test for each acceptance criteria in a user story. Developers were not involved in creation and maintenance of test automation, but hoped that GUI automation will be their safety net and supported customers wish to automate ALL tests. We started to observe problems with testing and some people got very disappointed. I saw it but did not understand where it comes from and how to fix it. TestBash and ongoing conversations with others helped me to see the cause. I started to learn more about mindset and realised that by automating everything, I have build technical debt. We needed test automation strategy!

If you think that this story has happy ending, I have to disappoint you. One thing I have learned in 20 years of IT – it is not about tools, methods and processes. It is about people and people are stubborn. I saw my mistakes, I learned a lesson and of course it was hard for myself to accept that some of my mistakes were so huge and I misled others. But denial or to hide evidence has never been something what I do. I went back to the teams I was working with, explained what I have discovered and what I want to change and… they refused my suggestions! They thought it is OK how it is running. I tried to point out to problems we were having, but they found another reason and explanation. They just wanted to keep to automate everything…

Soon after that all happened, I felt urge to speak about it and make people aware of pitfalls. I started to give workshops for other in-house teams, our project managers and for customers. I had so much fun by doing that that I got the idea to do workshops and trainings full time. Fast forward – here I am, working as full time trainer and guess what I talk the most in my trainings? Yes, mindset! And very little about tools and concrete examples, because I still need to learn a lot and you could have better ideas than I do. This is my happy ending.

If you want to learn more about automation tests, I suggest you to visit evil testers website.

 

Approach To Test Automation

Test automation – frequent topic in testing community, but for businesses and developers it is still something new and unknown. This blog post is very simple, high level my lessons learned in the field.

Strategy

Start by choosing your test automation strategy.  Business sees Test Automation as way to save money. The problem is, if you don’t have strategy and simply write 1000 automated tests, then sooner or later your team will become slower and slower. To pick up speed and save money, test automation has to be flexible and lean. Forget about testing-all-the-things, it is not possible and is wasteful. If your system has 1000 automated tests, your team has to maintain 1000 automated tests. Depending how you created your tests and how oft you will change your software – it can be a LOT.

Martin Fowler created great thinking tool: Test Automation Pyramid

Focus on unit tests, because this is the fastest way how developer can find out that her/his new code broke already existing code.

Testing in Agile world

If you are living in agile world, I would suggest you to read “Agile testing” by Janet Gregory & Lisa Crispin. They have great chapter on test automation with thought provoking mind map. I learned from Janet, test automation approach to write e2e tests for epics, integration tests for stories and unit tests for tasks.

Remember: in agile we believe in zero-bug policy. That means as well that all automated test need to be active and results has to be green. all the time.

e2e test Criteria

Test Automation criteria is part of your strategy because it answers the question: What to automate? Typical approach for e2e is to automate:

  1. only standard features
  2. only features with high risk
  3. only most used configurations

Many teams use that criteria to reduce test automation which has grown too big. My suggestion in trainings is – from very beginning automate only really necessary test cases. If you doubt, then don’t automate on e2e level, talk with your team how you could catch it on integration or unit level.

summary

The biggest challenge in Test Automation is not to write a script, but to decide what to automate and what not to automate. You will easily find people who can automate 1000 tests, but not so easy to find people who can create test automation strategy and efficient test automation design. Just like there are many software developers, but not so many software architects.

If you have strategy in place then you will understand how wrong are statements similar to this: “I often write e2e automated tests for bugs raised by customer because if we made that mistake once, we will likely make it again”. We have to learn from bugs raised by customers, we need to understand what went wrong, but we cannot test everything and it is very unprofessional to automate everything. Testers and developers should learn to think profitably.

If you want to learn more, I suggest you to check what Angie Jones wrote about test automation, or better, her free course on test automation strategy.

Update

Few days after I published this post, Lisa, in cooperation with Joe, published amazing collection of Test Automation Models. My suggestion: explore Rob Meaney’s Risk Mapping and Katrina Clokie’s DevOps Bug Filter. Maybe it is something for your team.

Words Has Meanings: Learning

This is conversation between two friends

Kris: I have to tell you a secret… I have big problem with “learning” if we talk about ET (exploratory testing).

Lisa: Hmm… It’s always been taught to me as “learning about our product”. But I am not an ET expert by any means.

Kris: For me “learning” means “to change behaviour”

Lisa: oh, interesting! for me it’s just gathering knowledge

Kris: Gathering information doesn’t include using it. You use information by changing your behaviour (huge excursion in my experience as mother and trainer). But only thing what we want to change is the software. So no real learning on human side.

Lisa: We might learn about features we are missing?

Kris: That is functional or contractual acceptance testing. No learning.

Lisa: So if I do exploratory testing on some feature, and then I realize, before customers can use this feature, they need another capability that we haven’t even thought of before – I didn’t learn something? Or if I find something is really hard to use – I didn’t learn something?

Kris: How do you know that they need something else? Hard to use – do you mean usability? Testing usability?

Lisa: How do I know? Because I’m exploring as a particular persona or role or job, I have a scenario of what I want to accomplish with the app, and I run into a roadblock. The persona is blocked, Lisa has learned that we didn’t provide a necessary capability.

We might be splitting hairs on words and semantics, but exploratory testing is called a process of learning, and it has seems correct to me.

Kris: For me it is very important to clarify what we mean by words. I always liked language (in Latvian we have more then 30 words how we call a mother), but as older I get as more aware I am about layers of language, coded messages and communication in general. People, who name things, make mistakes (just like everybody else). I am not looking for fight… It just feels wrong to call it “learning” if we only gather information.

Lisa: OK

Kris: The way how I understand whole team approach is that if we have situation as you described it, the tester in the team should be able to identify and categorise the problem. Is it functionality, usability or performance? then to decide what following tests should the team run to gather missing information.

Am I aiming too high?

Lisa: IME it works best if the tester collaborates with the team to do all that. We don’t want to be the safety net so that everyone relies on us to point out issues. We want to help teammates prevent the issues from happening in the first place by building shared understanding of features, using good tech practices for code correctness, fast feedback loops with automation, exploratory testing before committing changes…

More a consultant role than doing all the testing for the team.

You know, you have some interesting ideas here, it would be nice to discuss this on the AgileTestingFellow slack.

Kris: Maybe… but I don’t feel there yet. And I didn’t mean that tester is a safety net. What I meant is that tester has this knowledge about testing techniques and approaches and she/he guides the team. This is what I understand with tester as testing coach for the team.

Lisa: Sure but we have to be continually helping everyone else ramp up those skills. If we make all decisions ourselves, we take autonomy about testing away from the rest of the team. Guides, yes, that’s the ticket

Kris: Yes, helping others. But tester should be able to recognise it her-/himself first. Only then she/he can guide others.

Lisa: I like what you say there about recognizing it in ourselves.

 

Post Scriptum

I really, really LIKE how James Lyndsay phrased it:
“Systems are weird. Are you looking for trouble? Exploratory testing can help you to find unexpected truth, about what you really got.”

Exploratory testing is technique to find out, gather information. As a team we will decide later what we want to do with information with gathered. Maybe we will ignore it, because risk is too low, maybe we will use it and make decision based on it. And maybe we will learn out of it, adapt our behaviour and leave the issue in the past. If it will not reappear, then we really can tell: “we learned something!”

Mini Test On Production

Yesterday I changed my Twitter handle, today I want to test, what happens if I do not update the change on WordPress and publish a blog post.

I expect to happen one of these:

  1. cannot publish blog post
  2. cannot publish blog post, error message that there is no such handle
  3. can publish blog post, but no tweet on Twitter
  4. can publish blog post, there is tweet on Twitter for ghost account with old handle
  5. can publish blog post, tweet on Twitter with new handle
  6. something else

UPDATE after the test

Test was successful. Blog was published with right Twitter handle.

What I like a lot, that WP updated automatically to the new handle. If I choose to write another blog post than it is already filled with the new handle. No human interaction necessary! 

 

Update: I found one thing that broke – Twitter widget in the sidebar.

#30DaysOfTesting – ECommerce Task II & IV

Previous in this series: Task I & III

Task 2: Read and share an interesting blog about ecommerce testing

Task 4: Find and share a useful video on youtube about ecommerce testing

One testing website I like and read since I started to test software is Software Testing Help. Vijay has done amazing job by collecting all kind of testing ideas and helping so many rookie testers. 8 Important Segments Of Testing eCommerce Websites is very good place where to start if you are starting to test retail software.

For advanced testing or as Daniel says at the end – to put a smile on your face – do some penetration testing.

#30DaysOfTesting – eCommerce Task I & III

Task 1: Look up some definitions for ‘ecommerce’, from these create and share your own definition

Task 3: Join the #ecommerce channel on https://testers.chat and introduce yourself!

e-commerce

For me e-commerce is a system where you can exchange all kind of goods and services. I used word “system” because there is more what the eye meets. In those digital transformation projects where I worked and e-commerce was a part of the project, companies were unprepared how big their e-commerce system can be.

#ecommerce channel

For those who use Slack : you will find Testers.chat under testersio.

I like that this time MinistryOfTesting involve other testing communities into challenge and created channel on testers.chat.

Each of us have some good tools in their toolbox. If we put those together we could help the domain to get better and lighten up entry for rookies.

Explaining Software Testing

Michael Bolton is good in summarising fundaments of software testing in a tweet. In my eyes this piece of information deserves a little blog post.

The last sentence is something what I teach my ISTQB Foundation Level students – you cannot assure that there are no bugs, but you can collect information about bugs you have found. You can analyse them e.g. with aim to create set of actions how to secure your software developing process from reappearance of that type of bugs.

Most of developers want to know if a feature does work or not, but we as testers can only say that Version A did work on Machine B under Circumstances C.  Project managers and customers want to know if it will work on production. We as testers cannot assure it, because we did not (and we should not) test on production environment with production data. Based on tests on similar environments and under similar circumstances we can suppose that it can work.

Sounds very logical, but it can escalate very quickly, because DEVs, PMs and customers think that testers are there to save the word and to give certificates that the software they are working on, works perfectly as described.

What to do if you still get asked:

– “What do you do whole day long if you cannot tell me will it work or not?!” 

In heated situation like this it is too late to explain semantics, fundaments of testings or why software development process is complex. You should done it as soon as possible, when you started to work on a new feature, on new project, in new team or new company.  Remember – one of most important skill of a tester is communication skills. We need those not to be able to talk about the weather, but for explaining what is testing and how testing can help to DEVs, PMs, customers and all others. Daily.

Michael suggests 4 step plan how to learn fundaments of software testing:

  1. Learn how to test: How can a trainee improve his/her skill sets in testing?; To The New Tester
  2. Declare your commitments: A Tester’s Commitments
  3. Recognise that all testing is heuristic: Heuristics for Understanding Heuristics
  4. Learn to tell the testing story: How is the testing going?

Looks simple, but as you start to read the linked resources you will understand that studies can take years. Take deep breath. To be able to explain testing to others, you have to learn it first for yourself. Do not panic if it does not work on a first try. Make experiments, try new approaches, improve your skills. One day you will master it!

#30daysofsecuritytesting – February

February is over, but my 30 days of security testing challenge is not done yet. I have done only 11 of 30 so far: I, II, IV, V, VII, IX, X, XII, XVIII, XX, XXX, and I am not thinking to give up. This has been amazing learning journey! I always wanted to learn more about web security but never had real reason or time to do it. Challenge helped me to realise it is not so difficult as I thought it would be. Those 30 tasks are like a map with turning points and there is so much information if you know what you are looking for.

One of my information sources is YouTube – you will be very surprised to find out how many conferences upload the talks on YouTube. Like this talk from Troy Hunt.

In his talk Troy shows several examples with insecure passwords. It is something what I could definitely use for testing.

Thank you: Melissa Eaden, Claire Reckless and Dan Billing for putting this challenge together. I am very intrigued how it will go on.

#30daysofsecuritytesting – Task XX

All previous posts in this series: I, II, IV, V, VII, IX, X, XII, XVIII, XXX

Task 20: Read about DOS/DDOS attacks. Share examples/stories via social media.

screen-shot-2017-02-28-at-17-19-30

I started the task by looking the definition of DOS/DDOS attacks, to be sure that we are on the same page, one of the first results in DuckDuckGo was this interesting website. Emergency readiness team, cool! DOS/DDOS attack they classify as security tip.
Security header check: D

If there is US CERT team, there should be EU CERT team, right?
screen-shot-2017-02-28-at-17-57-10
Yes! only Europeans call it Computer Emergency Response Team. The website is a collection of articles and I definitely weekly will check top stories, hall of fame or latest info from security vendors.
Security header check: F

Continue my research in German websites. Again very interesting first DuckDuckGo result. Cyber-Sicherheitsrat Deutschland in English “Cyber-Security Council Germany”. Cool name, but there is something strange. Founded by a “group of reputable individuals” and “the cost of an annual subscription is 2,500 Euros. There is also a one-off admission fee of 1,000 Euros”. What kind of security group is it? And is it only me or that home photo I have seen somewhere else?
Security header check: F.

The real institution in Germany is Bundesministerium des Innern (BMI)

screen-shot-2017-02-28-at-21-11-22
Security header check: D

And Bundesamt für Sicherheit in der Informationstechnik, which has listed “common sense” as one of the suggestions for internet security.

screen-shot-2017-02-28-at-21-53-06
Security header check: C

OK. It is very interesting but what was the subject? DOS/DDOS attacks.

You will find information about latest DOS attacks on EU page above, but I liked the story about possiblly first DOS attack. Especially interesting are the comments.

Security and WordPress

keytrust2

Week ago I learned about security headers  and found weaknesses of this blog. In my previous post I did not write, that beside mine I checked other websites, which I know are run by WordPress. All of them have the same problems. No matter where the WordPress page is hosted or stored. I contacted Happiness Engineers and got feedback that my question will be forwarded to Network Engineers. Still waiting for an answer from them.

One thing what made me especially concerned was the blog post “stopmullware-on-the-security-of-27-of-the-websites-on-the-internet” written by Scott Arciszewski, which got deleted shortly after I read it – the post was about priorities of Automattic, hint – security is not one of them. Here are some tweets which should explain why it was deleted:

Use Google and read yourself who is Scott and what he does. And make your own picture of the situation.

But Scott is not the only one who is alarmed. Here is another article about WordPress vulnerability. In fact I have a feeling that suddenly everyone writes about how insecure is WordPress. German media people seem to be little slow – they are informing about update bug just now.

Long story short – I spent whole week reading and understanding how it works all together – touched certificates and domains as well. My aim was to find out, what can I do to improve the security of my website. My current answer – as long as I use WordPress, I cannot fix security header issues. But optimist inside me is really looking forward to WordPress Network Engineer answer. May be there is a way.