I have been talking about Whole Team Quality via Whole Team Testing for a couple of years now. During my workshops, I am often asked if testing can only be extended for programmers in a team. Pretty interesting question it is and my answer is obviously "no". Though I usually explain in my workshops on how to extend testing to roles beyond programmers in a team i.e. for UX or PO roles, I realized that I have not given deep thought to it and to how exactly testing could be extended to other disciplines in a meaningful way.
I read books, discussed with my colleagues, did my research and the outcome has been what I would like to name as QX i.e. Quality Experience. If QA (read that as Quality Advocates) and UX professionals collaborate in a meaningful way, I firmly believe they can co-create a Quality Experience for everyone associated with the product.
So, what is QX after all?
QX stands for Quality Experience. For sake of understanding, you can call it a marriage between QA (read it as Quality Advocacy please) and UX. After some need-based discussions and interactions with my UX colleagues, I realized that we can achieve a lot more if we work closely together regularly. The key idea of QX is about facilitating the collaboration between QA and UX so that they can contribute to what I would call a "Quality Experience" of the product. This is both for the end-user and for those who build the product itself.
I believe that with some process optimizations, mindset enablers for testers as well as UX designers, and following some heuristics I have created, it is possible to kickstart the QX journey if that idea interests you.
But what's the need? Is QA alone not enough to cater to product quality (or UX alone not enough for better user experience)?
Well, I am afraid, it is not. Not at-least when you believe that Quality is value to someone (who matters) and when multiple stakeholders matter at the same time.
Let me explain. Imagine that a new design change required in a product is a revenue booster for the company but it is also likely to impact the user experience. Testers often end up with oracle problem in such situations and can not decide what their quality criteria should look like. Of course, the Product Owner can be consulted here for a final decision but that's not the point. We testers are in the information business, after all (yes, even if you follow the Modern Testing). I find it important for testers to be able to make a comprehensive information gathering and present that information to decision-makers so that they can make an informed decision based on that.
Now, if testers lack the tools and mindset to figure out how to go about solving such problems and gathering information that would matter, their job would be poorly done. And if you are still not convinced, I highly recommend you to read Weinberg's latest book around System Design Heuristics. This book could not have come at any other better time for me. I started reading it while working on the QX concept and it has given me some interesting insights to sharpen my thinking around this topic.
On the other hand, let's assume that a UX/Interaction designer has been given a problem to solve or has been asked to create a new solution for some product. How do they ensure they have gathered enough information to do that job right? How about historical incidents or say hidden technical challenges or simply put edge case scenarios, cross-functional dependencies, and so on? I believe that having this information at hand can greatly benefit how UX designers can approach the problem and solve that in a better way.
Therefore I think that an engagement between UX and QA can help both of them to perform their job even better. And hence QX seems to be a good way to go about it.
Well, how does it work?
Here are some ideas that I experienced to be working quite well. See if they work for you too?
1. Cross-discipline training for QA and UX
For successful collaboration, it is important that QA and UX understand each other well and that they speak and understand each other's language. More important is to understand the mindset with which they both operate.
I have experienced difficulties in understanding UX's point of view sometimes and my UX colleagues made the same experience. But a conscious effort made towards understanding each others' language helped us solve those issues and facilitating the collaboration thereby.
That said, we recommend that testers and UX designers start from understanding each other's roles and mindset. Attending cross-discipline training should help but if that is not possible, try doing "pairing sessions" at least.
2. Process changes or optimizations
A great deal of it depends on what kind of team set up you have. Some teams may have dedicated UX designers, some organizations have UX teams as a "lateral" service provider for different teams and some teams simply don't have UX people. Their designs are usually outsourced or made by engineers themselves. Some recommendations I would like to make here are:
1. Involve testers as early as possible and make them also part of the design process/discussions. UX will thank them for lots of useful info which might result in the faulty design if missed.
2. Early and frequent communication between UX and QA would help. Try "brainstorming" sessions for early design stages. Ask tester for hidden scenarios or technical hacks to go around things. Ask them for user complaints or known production issues surrounding the design under discussion.
3. Testers may perform focused UX testing and consult UX from time to time for their design-related findings. A "pair testing" session with UX expert can greatly benefit testers for more test ideas surrounding usability and human-computer interaction under different contexts.
4. Instead of creating a misinformed bug report surrounding usability and UX, testers can always consult UX colleagues first for feedback around their findings and use them as their oracles. Most of the time, UX people have information and insights (from their interaction with real users, test sessions they perform, qualitative and quantitive data analysis they do, etc.) that explain why that design is made a certain way.
5. Not every design change goes via the standard UX lead design process. Engineers sometimes have to make decisions that may result in a change in product design or impact certain product behaviors. Such changes can always be sent to UX designers for their feedback. The tester can play a big role in making this activity happen on behalf of the engineering team.
3. UX testing heuristics for testers
Mere exchange with UX colleagues without having proper knowledge of how to test for better user experience can be futile. Which is why I recommend testers to get good at UX testing too. By that, I don't mean typical usability testing or accessibility testing alone. After giving a deep thought to all possibilities involved, I have come up with the following heuristic for testers.
Keep in mind that it can also be used as a means to facilitate collaboration and have a better discussion with UX colleagues. Not everything might be applicable for all the contexts. Choose those that fit best in your context. Here you go:
Problem -To come up with relevant test ideas, Testers and UX must be on the same page in terms of understanding the problem they want to solve with the proposed design. UX often get first-hand information from the PO about the problems, which testers sometimes are lacking. Trying to understand what problem UX wants to solve with their solution, can open up lots of possibilities for testers and confine them to think in the right direction. It would also spare them from coming up with the right questions and perform better impact analysis of the change.
User Needs -
This is highly subjective and might vary from project to project. The key idea is to understand what User Needs are getting addressed through the design and if you as a tester can foresee any challenges/impact of that. This can also be very well made as a part of 'understanding the problem' part. It does not matter how you do it, but what matters most is that you know what aspects of user needs you are dealing with.
User vs Business Needs
Finding Balance - Solving the Oracle Problem
When you as a tester are unable to decide if the proposed solution is good for the product but bad for users and vice versa, use some of the methods below to make your decision-making more concrete and practical. Based on the configurations of teams, there may be no dedicated UX expert to cater to these needs. It is therefore advisable for testers that they wear UX hats and find the balance by asking the right questions before it is too late.
Plenty of efforts could be saved if testers too are involved in design/UX decisions early where dedicated UX expert is absent or designs have been made by third party services etc.
What Must Not Change?
Whatever we ultimately do, what are the things you don’t want to be changed? (from System Design Heuristics by Jerry Weinberg)
Introducing design changes in an existing product that are targeted at a specific goal by UX designers, can harm things that are not meant to be affected by that. I highly doubt if there are deliberate efforts made to analyze this regression impact on the design level itself. This is why, if a tester asks this question right in the early phase of a design change, it is likely to save lots of rework for the later.
“If we don’t start that way, it’s all too easy to lose track of the unchangeable.” - Jerry Weinberg.
What are the visible and invisible parts of the product that are impacted by this change? And what is the impact?
Running out of test ideas to decide if the solution is perfect? Pour some creativity in.
Exactness, Intuitive and Counter-intuitive Design
Most of the time, testers are more focused on the technical and functional aspects of the product so much that they unknowingly tend to ignore looking for obvious problems. A deliberate attempt needs to be made to look at the product like an unexperienced user to see if they will understand what we expect them to. To do this:
Well, I wish I could work on this and develop this idea even further. But for the benefit of time, I would stop here. Maybe I would get back to this again when I have more ideas. But in the meantime, feel free to comment below and share more ideas if you like. And do not forget to tell me how you find the QX idea so far.
And, I recommend you to read System Design Heuristics by Jerry Weinberg. The book gave me lots of ideas to ponder upon.
Lately, I happened to have an interesting discussion with my colleague Dirk Meißner on whether programmers should have reasonable understanding of testing or not. A lot has been talked and written about how testers need to be great with their technical skills so that they can contribute effectively and remain valuable. Sure, that's helpful and I too insist that it's high time that testers get over with their traditional way of working (and thinking). However, what surprises me is that there is not enough awareness or enough discussions happening around programmers learning to understand testing to amplify their effectiveness.
Does it matter? Why?
It absolutely does. At least now, if it did not before. "Whole Team Testing" is new cool (again) especially in DevOps contexts. And it has it's own reasons to be that way.
Let me explain. Agile teams typically have one tester dedicatedly looking after testing and related activities in team. This tester is usually busy testing (and automating often) stories for each sprint with primary focus on acceptance criteria. If the tester is "cool kid" then they go over the board and test things beyond acceptance criteria too. Cool! Let's park this thought here for a moment. Okay?
James Bach, in his interesting article "Seven Kinds of Testers“ beautifully explains the key patterns surrounding testing styles and how testers typically fit into one pattern or the other or combination of more some times (or checkout this thought-provoking tweet series by Michael Bolton). In over eight years of hands-on experience of testing, I have found myself to be of one kind (or maximum two) at a time and by the time I wish to change my hat (or style) it is usually almost the time to deploy the feature in production. Pity!
The point is, there is limit on how much versatile a tester can be in limited period of time for each story they test. Sure, it's not impossible but I would say it's not very easy either, given the time constraints. Now, imagine that we add "programmers" from team as other kinds of tester (based on their skills, expertise and experiences) working on same story. Do you not think it will most likely add more coverage to the quality of that feature, without having to spend really additional time on it? Do you not think that testing wisdom of a programmer would help tester and the team to ship quality product? I'm sure, now you do!
When I say programmers should contribute to testing, it does not have to be only in a way that they will need to test the software like testers do. Even if they develop the required mindset, it's already a good start. Indeed it will be great if they can test it but I feel that if they could at-least understand modern testing, it will greatly benefit the project teams.
How exactly programmers can learn to test and start off with it, or how testers can help them to onboard with testing is another topic. It requires a dedicated post (more on that later).
This post is about identifying programmers with testing mindset or skills that can help them test better, while you interview them. I recommend watching out for these skills/traits in interview:
1. Quest for Context
The scholar John von Neumann once said, "There's no sense being exact about something if you don't even know what you're talking about." In a world that is growing increasingly dependent on highly complex, computer-based systems, the importance of defining what you want to make before making it -- that is, knowing what you're talking about -- cannot be stressed enough. (Exploring Requirements: Quality before Design by Weinberg & Gause).
Developing software is not just about writing a program that will do the stuff but it is more about building a product that your customer would like to use. In past, I have come across programmers who were excellent coders but failed to care enough about purpose of the program they used to write. I rarely see programmers questioning the user story beyond acceptance criteria and technical implementations if any (unfortunately, it's not very different for majority of testers either).
If I were to hire a programmer, I would expect him to ask Context Revealing questions. By that, I don't mean just questioning the business value of the user-story. There are things beyond that which matter. What if we find out that other team has worked on similar solution before? What if we could re-use some components developed by other teams? What happens when particular feature development requires specific technology expertise and team does not have it? What happens when stakeholders' understanding of technical details defer from engineering team's? What if implementation of some solution requires tools not available with team or required access levels for that matter?
Sure, one can eventually find these things out when they start working on the ticket but what's the point in findings things with accident and when it's already late?
Programming interviews typically include coding challenge that typically gets assessed for candidate's technical skills, familiarity with known technological issues, understanding of best programming practices, problem solving skills etc. which are indeed important. However, I am yet to see a programmer being assessed for the kind of questions they asked before jumping on to the coding challenge itself. See if they are questioning the very purpose of challenge, see if they question the business value, check if they ask about other elements of Project Environment and Product Elements for that matter. And, please check if they ask questions about testing and Quality Criteria if nothing else. Most of the programmers I got to interview usually assumed that that there will be a tester in team who will QA their code. They just have to write the code and throw it in tester's bucket. You better watch for such kind if your team does not have a tester.
Just like testing, good software development should be treated like an intellectual activity. The better one understands it the more ways one can contribute to product quality. And it all begins with asking questions.
2. Interactional Expertise
If you are unfamiliar with the idea of Interactional Expertise, I suggest you start from understanding it first. Even better if you could read Tacit and Explicit Knowledge by Harry Collins. I personally found it to be very useful learning when I was introduced to it by my friend Iain McCowatt.
The purpose of mentioning Interactional Expertise as a skill here is that, I find it to be very important skill when it comes to have technical discussions with non-technical people. Or, even when it is about having meaningful technical discussions in short period of time.
Bringing up technical topic for discussion in planning meetings or grooming/estimation meetings is usually like opening the Pandora's box. Over the years, I have been a part of deep discussions in meetings with only conclusion of carrying them to next meeting or scheduling separate meeting for that. Then again, special meetings for explaining those technical things to non-technical people were required. Does that not sound familiar to you?
I feel that spending so much time on deep discussions very often is unnecessary and it can be significantly controlled if all of us (not just programmers or testers) learn the skill of explaining things in short (and to the point) when needed without losing the substance of it or compromising with the impact an elaborated version would make. Same goes with explaining technical things to non-technical people. As techies, we can't expect the whole world to understand the language we speak (it would be nice if that happens though) but we can make things simple by learning the art of explaining those to other in language and context they would understand better.
Added advantage would be when you onboard new members in team. Regardless of what role they are hired for, person's IE skills would help them onboard much better and the skill will definitely help for better collaboration and communication. In fact, when testers and programmers both have great interactional expertise then sessions like pair testing or programming will be super productive. Imagine what value it can add for Mob Programming and Mob Testing sessions. I have worked with some programmers who were master of explaining technical things to non-technical members of team as if they were putting a child to sleep by telling a story. Short, sweet and yet satisfying. That's what I mean by Interactional Expertise.
Next time when you interview a programmer, look for these. It will help you. When I interview testers for it, I usually ask them to explain some technical concept in 50 words for example and again same concept in 100 words. It helps me analyse how good (or bad) they are with their Interactional Expertise. Asking programmers to write a technical bug report or user story can also be helpful trick to evaluate them for their IE.
3. Understanding of Testability
May be I am wrong about it, but I honestly feel our industry still lacks required seriousness (and awareness) for building testable products. This is not just about programmers being unaware of it but even testers.
The only times I hear of the word "test" in programmer interviews are when they talk about their unit tests or TDD or automated tests at the max. And it's a pity!
Building testable products is an important part of software development and it is important that programmers understand how to bake testability right from the beginning. Sure, skilled testers can certainly be advocates for testability but it won't hurt if programmers too understand what it means to them and how they can contribute to it.
While interviewing programmer, I suggest you pay special attention to their solution if that demonstrates at-least few aspects of Intrinsic Testability as explained in Heuristics of Software Testability by James Bach. If not, at-least make an attempt to discuss other aspects of software testability (listed in the heuristics) with candidate in general and gauge their fitness for your requirement.
For sure, the skills mentioned above are equally important for testers too but since "Whole Team Testing" thing is picking up, I wanted to make it explicit for traditional non-testers. Next time when you interview a programmer, please try and see if it helps.
If we need technical testers, we also need programmers who understand testing. And that is reasonable to ask for, isn't it?
Oh and by the way, I will be touching on some of the related topics in my talk for Online Testing Conference. Feel free to join if the topic interests you.
During my ET/SBTM workshops, I have been often asked if it is possible to perform Session Based Testing in typical Agile/DevOps environments.
I think if tester knows how to perform SBT (especially with different session types), there should be no trouble in doing it regardless of the development methodology their team follows. (If you are unaware of typical session types in SBTM, then I recommend you to checkout this informative series by my friend Simon ‘Peter’ Schrijver.) However, I understand that probably, the fast paced way of working through sprints, shipping small chunks of software regularly, keeps everyone mainly focused around achieving the sprint goal. And for testers, this often means focusing only on ‘acceptance criteria’ and moving on to the next ticket, which is waiting to be tested and deployed. And that might force some testers to just forget every awesome technique they know and run behind getting “the stuff done”.
If you are a tester (sorry, Agile Tester) stuck in such situation and feel bad about it, then this post is for you. For a while, I too was stuck in similar situation. Not that I was not getting to perform ET in SBTM way, but I constantly felt something was missing and that there was still more I could do. A couple of important things were getting skipped off my typical sessions, and all I could do was procrastinate them. And that was bad, very bad! (yeh)
When I was done with feeling guilty about it, here is what I did (and I strongly recommend you too try it out). I created some more session types for myself some of which I perform twice a week and some I perform daily. And they have been helping me greatly so far. Well, what are they?
1. Critical Thinking Session
I know, a good tester should think critically at every possible occasion. But there is no harm if you dedicate a special slot for it. How often do agile testers spend time on reading through backlog and preparing their notes in advance so that they know what to ask, know what extra information they would need for certain ticket or prepare a risk list associated with some ticket, to make everyone aware of it before it’s too late?
A dedicated ‘Critical Thinking’ session aims to solve this problem. Schedule at-least two of such sessions, one before beginning of new sprint and one before your feedback/refining sessions so that you spend enough time for ‘critically thinking’ about stuff that would be soon on your plate. The sooner you prepare yourself with it, the better you will be able to shape your further sessions around actual testing of those items.
Medium sessions of roughly 60 minutes works great for me. Sometimes a short session of 45 mins is enough. The more you practice with Critical Thinking sessions and the more time you spend with applications you test, I guess you will eventually need lesser time for CT sessions.
In case you are wondering what I really mean by “Critically Thinking” about backlog items then please checkout “Mary Had A Little Lamb” heuristic by Jerry Weinberg or “Huh?Really?So?” by James Bach.
2. Monitoring Session
Monitoring production logs especially after deployments has benefits of its own. And if tester does it regularly then there is a lot they can discover through those logs.
If as a tester, you are not yet doing production deployments then I suggest you start doing it and make sure to check production logs (or logs on other environments for that matter). If for some reason, you do not own deployments then try to spend at-least one short session for monitoring your application’s production logs, every day. That’s what I mean by “Monitoring session”.
Because of such dedicated monitoring sessions, I have come across some elusive bugs that were hard to catch on testing environment. Production logs are also very interesting means to learn about different ways in which you can test your application against disfavored use or extreme use (refer HTSM by James Bach – SFDIPOT – Operations). Not only that, it can also help you identify some illusive integration level bugs which might not be caught easily otherwise.
Monitoring sessions can also help you identify technical bugs, errors or warnings, which might not be directly affecting the end user but still warrant attention and fixing. It’s hard to identify such issues in regular functional testing which usually has big scope of its own.
3. Bug-visit Session
This session is about visiting the bugs in backlog and going through them carefully. Sometimes, over the period of time, some bugs become irrelevant or can also become important to be fixed on priority. Revisiting those bugs helps take appropriate action on time.
If there are bugs logged by other teams or customer care team for example, then they can also serve as a great ‘test ideas’ that can be extended to other features of application. Information gathered through such bug-visit sessions can help you create your own risk-list or project specific heuristic (I have explained that in detail with HEEENA).
4. Test for Testability Session
I can't stress enough on importance of this session, especially in this changing era of Software Development. One of the key role a skilled tester can play in modern software development is of "advocate of testability". Checkout this heuristic for Software Testability by James Bach. Once you understand the dynamics of testability you will realise that the sooner you care for it (and advice where needed) the better testable product your team will create. If I am to give you an example, please check "stats for nerds" on youtube videos you watch. Would such information not help you if your product starts storing such information, that you as a tester can access easily and shape your tests based on that?
I suggest you create the checklist for Intrinsic Testability in particular and test your design right from the beginning against it. Please dedicate a compulsory session (short should be good to begin with) for testability even before you get the build for testing against acceptance criteria. In fact, you could also pair with programmer or PO when they work on user stories that will latter come to you for independent testing. This in tern will help you as a tester to better test the product for session/charter you care for (and you won't get lost in figuring things out to help you test better).
So, that’s about four additional session types I have created for myself to do even more effective SBTM in agile environment. I am finding them very useful and hope you too will benefit from them.
Feel free to get in touch if you have any questions or would like to discuss it further.
Header image credit - blog.xebia.fr
A passionate & thinking tester. Trainer & student of the craft of testing. Father, Foodie and dog lover. Chief Editor and Co-founder of Tea-time with Testers magazine.