About fifteen years ago now, I was reorganizing how search results are displayed in a popular travel site (one you’ve probably used). The goal of the project was to to increase readability and scannability. To do this, I needed to move a few bits of information around. In my rush to prepare the prototype for the initial usability tests, I neglected to move all the bits of information to their new home.
The majority of the evaluators caught the mistakes – unprompted.
Commenting on mistakes – below the fold – unprompted?
I’ll happily take that as proof the scannability & readability improvements were successful. For, if the problems weren’t obvious from even this smallest-level engagement – we still had work to do.
**
Fast forward a decade, I’m building a proof-of-concept for a startup client. We agreed upon the smallest, most unique functionality necessary to communicate the value of the product. I went off to build it and they went off to find a cohort of interested, beta users.
A couple weeks later, the prototype was ready to interact with and I mass-created a few dozen accounts for the initial users, including a one for my client
I waited. I waited for bug reports, questions, for server performance issues, for emails, for phone calls. For I knew there would be some. Some bringing up issues I hadn’t dealt with yet, some bringing up edge cases that are only exposed during actual use. Some wanting to do something we hadn’t even considered yet.
A week went by – and nothing. No bug reports, no emailed questions of ‘How do I?’. Complete radio silence. On my way out the door, to meet my client in-person, I quick checked the access log – not a single person had logged in since they received their credentials.
Not even my client.
“I haven’t received any bug reports from any of the initial users,” I started after the Arnie Palmers arrived.
“Good?”
“Not good. No one’s logged in since the site’s been up. I don’t think the idea is compelling for the initial users – or for you.
“What do you think we should do?”
“Shut it down and find something more compelling to you and your interested users.”
The next day, we did.
**
This morning I received an email from a different client. One of their news products was throwing a series of admin-only messages into a publicly-available view. The issue was easy enough to resolve, though in resolving it, it was clear the issue had existed for least two years. Which means, this part of the site has had zero human engagement for that entire time. For if there was any engagement, I would have received emails from concerned users notifying us of the bug. Doch.
**
If you’re looking for a cheap way to measure actual human engagement and attention – deliberately insert an obvious and miniscule mistake and wait.
If a flood of corrections don’t come in, you’ve got a much bigger problem – nobody actually cares.