Summative vs Formative: Which Test is Right for You?

Summative Vs. Formative Usability Testing

When conducting user research, UX researchers generally preach iterative usability testing throughout the development lifecycle. But did you know that there are actually two different types of usability testing? Formative testing identifies usability problems to be fixed in later iterations whereas summative testing describes the current state of the usability of an interface. While both types of testing are conducted to improve the usability of a design, they are performed at different stages of development. So how are these website usability testing services conducted differently and when do you need them?

What is formative testing?

Typically conducted in the early stages of the design and development process, formative usability testing is about problem discovery and provides solutions to solve usability issues. This type of user testing is performed in small sample sizes, with 5-8 participants, and is used to quickly assess what’s working and what isn’t. It is an incredibly powerful support tool in shaping the direction of your design or to test your website. Testing at a small scale allows you to maximize your research spend, as 5 testers can identify as high as 85% of discoverable problems. Taking a discovery-based approach early on in development pushes the iteration process along as testers continually find new usability issues. While typically associated with qualitative evaluations, formative testing can also be conducted with quantitative metrics like problem frequency.

When do you conduct formative testing?

Formative testing is especially powerful early on in the design process as a tool to inform the direction of the product. We recommend to conduct at least two formative tests:

– Wireframe Prototype – First, validate workflows by testing before adding working functionality. Use card sorting to see how users group information and use those insights to shape your navigation and UI.
– Minimum Viable Product (MVP) – Save development money by conducting another test before development to see what usability issues still exist and fix them before your development team invests time and resources into building your product.

Formative testing in the early design stages allows you to determine whether or not your solutions are viable for your users before you start development. Further formative tests after development can help identify key revisions to be made and are a crucial piece of the iterative redesign process. 

What is summative usability testing?

Summative evaluations are meant to assess how well a design performs in a user test. Usually conducted when a product is fully developed, summative tests are a great way to evaluate a design against competitors or usability goals. By measuring usability through quantitative metrics like time on task and success rate, we are able to compare our design’s usability to previous iterations or market competitors. In order to make these kinds of statistical observations, you have to make sure that your sample data is as accurate as possible when conducting a summative test. That is why summative usability tests are conducted at a much larger scale than usability tests, in order to narrow the confidence intervals of our data. Larger sample sizes reduces sampling error, which allows us to confidently make business decisions with insights gained during summative tests.

When do you conduct summative testing?

Summative tests are primarily conducted for two purposes: to see how you stack up against competitors or how close you are to reaching internal goals. Performing a summative test before launching a product can help you determine whether or not your product is ready to launch. Using confidence intervals, you can get a picture of what your completion rates will look like before you launch. If your tests show that you won’t reach your 90% completion rate goal, then you know that there are still issues to be fixed before going live. Likewise, summative tests conducted after launch with other competitors allows you to figure out where you fit in the market. If you aren’t matching up with your competitors, your test insights can help guide the next redesign direction.

Takeaways

Think of formative testing as a medical diagnosis and summative testing as a health check up. Formative testing helps you get to the root of what usability issues are out there and how you can remedy them, summative testing can tell you where you stand compared to competitors or internal usability goals. Utilising both forms of testing, at different stages of the design and development process, can help inform the design decisions you make. Knowing the difference between formative and summative can help you choose which metrics to measure and with what sample size. 

Need more information regarding formative and summative usability testing? Interested in investing in some user testing of your own? TestMate offers user testing Australia wide! Contact us for more information today.