My job during this internship is to help the QA team expand its test coverage of various software features (to date, the ContentTranslation and Visual Editor tools). These two tools each have dedicated QA people who perform smoke tests and regression tests. A smoke test is a quick inspection of the major functionality of the software to ensure that it’s working as expected. For example, with the ContentTranslation tool, you’d check to see if you could enable it, navigate to the main page, select an article to translate, specify what language to use, navigate to a secondary page where the actual translation takes place, successfully translate a few paragraphs of text, and finally publish the translated article. There are more steps than that, but you get the idea. In regression testing, you go through the functionality in a more detailed way, making sure that nothing has been broken by any recent changes that have been made to the code. For example, instead of just translating a few paragraphs of text, you test other page elements such as tables, templates, captioned images, different types of references, and specially-formatted items like math functions – and you check them using three different translation engines!
Many (maybe most?) companies use test cases – actual scripted instructions – to guide the testing. At Wikimedia, it’s a little different! The type of testing we’re doing (at least in the two groups I’ve worked with) is called “exploratory testing.” I told you a little bit about that in my second blog post, but to recap, exploratory testing is not as linear as traditional testing, which goes from test design to test execution and then to evaluation. In exploratory testing, rather than following a script, the tester is free to investigate and experiment with the product, and follow her own instincts about what to look into at any given time. This is a very practical approach, especially when you consider that most testers have extensive experience with their product and know exactly how to evaluate it without having to refer to instructions. Not only that, but they are perfectly aware of the priorities and risks attached to each individual release – so they know where to focus their energies to make the best use of limited time. That’s something that a set of test cases can’t do!
But don’t get me wrong, it’s not that test cases are tossed aside.* The exploratory tester is free to make use of any tool. It’s just that the tester’s skills and experience – and not some spreadsheet – drive the process.
* In fact, my very first project was to create a set of test cases for ContentTranslation, and it was that exercise that taught me all the ins and outs and bells and whistles of the feature. Invaluable! But now that I can find my way around, I use the test cases mainly as a reference and as a launching pad for further discovery.
BarbV 