Guerilla software testing (because sometimes you have to)

28 July 2006 by Rod McLaren

Our friend Kirsten pinged us from New Zealand, asking “do you have a test plan template I could use????” She’s a fairly technology-literate project manager working on a new version of a software tool for a HR programme that her company runs every year. She’s managing the project from Auckland, and the vendor is in the UK.

It’s usually better to test hand-in-hand with the development team, and ideal to build tests into the development process (for instance, test-driven development advocates this explicitly). This also allows you to put the user experience at the centre of your design, development and testing. But sometimes you’re at arms-length from your developers, and have to find a way to quickly test what they’re handing you so that you can accept it, or not. Similarly, sometimes you can’t test with the intended user audience – nasty. And lastly, you might not have much time. Kirst certinly didn’t.

It’s not an ideal situation at all, but sometimes it happens. So this is what we sent back in haste.

General stuff

1. Evaluate what worked well and what didn’t last year. You’ve probably done this already. The user rage from last time probably describes exactly the scenarios (see below) that aren’t being served well!

2. Write your plan and be ready run it:

• Be really clear what you’re testing and why. (This avoids all that “but I thought you tested X” stuff that other people will throw at you.
• running the tests is also about having the right people in the right place with the right kit etc, which takes some management.
• It can really help during testing to have some kind of permanent log being generated so that if you find weirdness, a tech person can go through it and see that “and then they did this and oh! It dies at that point. Interesting.”
• Have a consistent way of capturing bug reports. Where something doesn’t perform as expected, the tester will write down what they did, what they got back from the product, and what they expected instead. The tech folk can later compare this with the permanent log mentioned above.
• Keep reports in one place, like a bug reporting tool. We use one called Mantis, though RT and Bugzilla and FogBugz are good too. If there’s only one person managing the list of bugs, you can use Excel. Each bug would want: a number, a title (descriptive), a longer description (where the meat of the bug report goes, and something describing its severity, what happened, what the user expected instead, and so on. Later you can add columns showing who owns the fixing of the bug etc.
• Have a consistent way to evaluate and prioritise bugs (scenarios, see below, are useful for this). Some stuff is not going to be worth fixing.
• be aware that testing often finds things that aren’t merely “bugs” (broken things), but things that could be done a lot better. Think of those latter improvements as “bugs in the experience” rather than “in the functionality” when the developers complain.
• Testing without a formal test plan and Scenario testing (pdf file) look like helpful overviews.

First, functional testing

I’d be testing for whether it functions correctly, probably with a more technical group:

3. ideally, BEFORE you do anything like test a front-end, you’re testing that the functions work separately of whether they display ok.

• You probably do this by having someone drive the logic bit and call every function (save this bit of data, combine those 2 bits, etc). If that’s not possibly jump on to 2 and have the vendor put debugging data on the interface as well as the buttons – then you can press buttons, and the programmer can sit there and go “ok it’s trying to add X to Y. Was that what we wanted?”
• scenarios: this bit can be really helped by writing some scenarios in advance and running through them.

Manager X adds these new staff, but one works for two bosses. Manager Y needs to delete some because they’ve moved to X. Kirst needs to remind Manager Z to do the HR software thing.

Describe the important scenarios, and in terms of what the users needs to be able to do, rather than “then they select X from the dropdown, pausing to check the Y box, then…”

• afterwards get the tech people to sift through the database to make sure that you’re saving the data you want in the format you want.

4. Functional testing on the front-end.

• make them perform every function possible to make sure it works. Run through every feature BUT start with the important stuff.
• then break testing: make them perform every function, but try every thing they can think of to break it. Submit it with empty data, crappy data, weird characters, and so on.
• You could do all of that with one clever tester and yourself. Then move onto weirder edge cases like: two users try open the same record at the same time. If they can both open it, have them make different changes and then try save it at the same time. Does it save both changes? Or only one? Or try having the max number of concurrent users on it. And so on.
• again, afterwards get the tech people to sift through the database to make sure that you’re saving the data you want in the format you want.

Arguably your vendor should be doing all of that before it gets to you, unless you have data sources that you don’t allow them access to.

Then, user experience testing

5. By now, you know that it works, but not whether a real user can make it work. So I’d also be testing for whether it functions well, with a the actual user group.

• This section might be pretty brief if you and a couple of others are the only users.
• If you can get actual users, give them tasks described like “add a user’s employment report thing to the system” rather than ”... then press this button, then… ”
• Or, if you don’t have access to real users (not a fun situation), then test with people who are (a) non-technical, (b) who haven’t worked on the project so far but© might understand the domain, rather than your Granny.
• If you can’t get those real users, get your fake users to run through the scenarios.

Related stuff

Delicious for ‘test plan’ plan has some good material. From that we found:

Testing Without a Formal Test Plan

Scenario testing (pdf file) (both of these first two look pretty useful)

Might be some stuff in here too.

name		Remember
email
http://
Message
	Textile Help

Why Windows Mobile will rule the world People over process and IT in mobile workforces