Crowdsourced Testing: A Quick Overview

• I first heard of it (and the broader crowdsourcing topic) at the Jul 2008 TMF in a talk by James Whittaker (who said it was part of Microsoft’s vision of the future).

• Piqued my interest as I was at the BBC wrestling with extensive browser/OS coverage requirements triggered by the BBC Charter.

• uTest.com the only player in the game back then.

• uTest had raised about $12m at that stage (now $37m), all of it after the credit crunch.

Crowdsourced Testing: A Quick Overview

• Unlike crowdcasting (e.g. logo design) where the winner takes all, in testing everyone can contribute a piece of the overall solution.

• That makes testing ideally suited to engaging a motivated crowd.

• Go to the brilliant Zappers event run by TCL (uTest’s UK affiliate) and you’ll meet excellent crowd testers who are motivated by non-monetary factors (more on that shortly).

• The existing maturity of test management tools has provided an excellent basis for the (essential) crowd management.

Introducing Utest.com

• • • • • • • • • US-based company, pilot system launched in Feb ’08.

Circa 49,000 in the crowd (circa 3000 in the UK). But how many active?

Sales team in US (at least some of them), handing over to TCL (uTest’s UK affiliate) to project manage the cycles.

A functional management interface (but not commensurate with the funding received or the time since launch).

iPhone app available (but bugged as of last summer).

Fixed price model for the client, testers paid per bug or artefact on the other side of the equation.

TCL PM drives crowd selection, but with a client steer.

Daily reporting (if desired) with an optional test lead to sanity check submissions at minimal cost (approx £8 per day).

Offers functional and non-functional (although I’m not sure about the repeatability of the latter). Covers mobile, web and desktop.

Alternatives To uTest

I’ve briefly looked at, but not used: – Bugfinders (UK company, client pays per bug).

– Centre4Testing (Leverages C4T’s candidate pool, UK crowd, formally engaged to deliver coverage. In quoting for my case study they wanted a lot more info and lead time than uTest. About 10% dearer than uTest on the 1 st cycle but about 50% cheaper for subsequent cycles. Also slightly cheaper on 1 st cycle if you exclude uTest’s new client discount).

I’m aware of, but have not looked at: – Bugpub – 99tests (Seemingly not associated with 99designs.com

) I’m sure there are (and will be) others.

Crowdsourced Testers: Some Motivations

• Practising their skills, breaking things, trying the latest software, having fun, increasing their market exposure through rankings, networking*, etc… • For many (most?), money is not a material factor and they have a day job for subsistence.

* I’ve since met the uTest team lead for my case study and would happily engage him in a full-time role.

Crowdsourced Testing: Some Pros

• • • • • • • Significantly more eyeballs on the case £ for £; compares extremely favourably with a single contractor.

Externalise accommodation costs (desks, PCs etc).

Ultra rapid ramp-up time (vis-à-vis recruitment) and feedback loop (potentially get bugs within minutes).

Flexible engagement model for reactively rolling testers on and off the project (e.g. for post-release testing).

Mitigates costs of test environment provision in the face platform proliferation.

Evening and weekend productivity at zero marginal cost.

Cheap form of usability testing if you know how to frame the survey.

Crowdsourced Testing: Some Cons

• • • • • • • • Lack of direct accountability (some ‘soft’ sanctions like star ratings and feedback to platform operator).

System knowledge vested in un-contracted resources.

Could be unsettling to in-house testers.

Could be seen as de-valuing testing within the organisation.

If it’s a desktop application and it gets leaked – a watermark system may be scant consolation.

Testers may (probably?) care less than their internal counterparts.

Need to provide strangers with access to your test instance.

Could confer unwarranted confidence; relies on stakeholders understanding these downsides.

Crowdsourced Testing: Applicability

• • • • • • • • Start-ups that can’t sustain in house test resource.

Organisations that aren’t exclusively reliant on the crowd (not withstanding my point above).

Agile teams that want out-of-hours productivity (especially for smoke testing purposes) and/or to mitigate the effect of having a single embedded tester working in mini-water falls.

Public facing systems that need to undergo testing out in the wild (corporate systems involving sensitive/financial data are much less appropriate).

Organisations where the testing workload is highly bi-modal.

Places that lack the time/environments to test in-scope OS/browser combinations.

Environments where exploratory testing has been culturally accepted.

Environments that may want to target users in specific geographies to enable, for example, localisation testing out in the wild.

Case Study: The Context

• • • • • • • • One of Europe’s largest price comparison sites with profits measured in the millions.

Wholly reliant on TDD (within Kanban) save for 1 over-worked manual tester located on the continent.

Requirement was for post-release testing of a UK-only web proposition in public beta.

No time or money to recruit contract testers.

3 week window of opportunity for testing but with scope for it to slip.

No meaningful collateral from which to derive tests (just a load of binned post-it notes).

Engaged by the Programme Director who wanted to de-risk the testing and was open to suggestions. He knew about uTest but had no time/patience with working through the relevant questions.

I wanted crowdsourcing experience so I offered take this off his plate.

Case Study: The Solution

• • • • • • • I became the uTest interface and clarified the commercials, legals, process etc. Need to read the terms – uTest can quote you as a case study unless you opt out.

uTest sales team willing to progress paperwork over the weekend.

Commissioned a UK-only crowd for 3 calendar weeks of exploratory testing at a cost of $3000 which factored in a $600 new client discount. We paid $250 to have a team lead (in addition to the TCL PM) to sanity check submissions. $3,250=£2,075.

uTest provided a fixed IP to enable test traffic to be removed from merchant billing.

Coverage: Windows: Chrome, FF, IE 7/8/9 (+ Mac/Safari).

Testers were given a minimal briefing with goals comparable in detail to a charter within session-based test management.

TCL PM provided lightweight daily reports tailored to our requirements (weekdays only).

Case Study: The Outcome

• • • • • • • TCL invited 35 testers of which 17 accepted with 10 submitting defects. Around 80 defects (from memory) with a rejection rate circa 10%. Some useful site review feedback was also provided (at no extra cost).

The reporting and defect sanity checking worked well and made the internal triage process more effective.

Bug detection continued during weekends and evenings through to the early hours of the morning.

Rightly or wrongly, the client was delighted with the results.

Programme Director – who is a contractor - has vowed to “try and use uTest in as many future roles as possible as it worked brilliantly”.

Whilst recognising that the bar had been set low for adding value (post-release, no internal testers, non-complex web app etc) I also felt positive about the experience.

I felt it was too cheap; I wonder what time horizon the uTest backers have in mind for turning a profit.

Conclusions

• • • • • • • A rich and burgeoning topic area in which venture capitalists are quite active.

Still a young area with massively un-tapped potential, but established enough to not be dismissed out of hand.

uTest looks dominant but there are other options.

In the right hands and circumstances it can be a very powerful addition to your context-sensitive toolbox.

My uTest experience was positive but the bar was set low.

Its disadvantages will be unacceptable for some clients.

In my view, crowdsourced testing does have legs which will see its adoption rise over the coming years.