I arrived in San Francisco yesterday for the Lean Startup conference, where I gave my 5-minute Ignite format talk on the parallels between Lean hospital design and Lean startups – Lean is Lean. There will be video available of the talk online and I'll share it when I can. Follow the #LeanStartup hashtag or @LeanStartup on twitter for the latest updates from the event.
When I got to my hotel, I saw the front page headline on the San Francisco Chronicle: “Arrest totals down sharply in Oakland.” The article says arrests are down 44%. That sounds great… oh, but wait… is crime actually down 44%??
Nope… from the article:
The arrest figures, obtained under the state's Public Records Act, raise questions about the effectiveness and assertiveness of the Police Department, which is struggling under the weight of job cuts, low morale and the demands of federal court oversight.
…
Criminologists and law enforcement experts said that while they would expect arrests to fall somewhat along with manpower, the trend was troubling. Nationally, arrests were down 11 percent in the same period, the FBI said.
Ah, there's just fewer police officers (and morale is worse) so there's just been fewer arrests. Maybe the Chronicle headline should say “Criminals Getting Away with Crime in Oakland?”
In the Lean Startup approach, Eric Ries and company talk about “vanity metrics” — these are often things that are easy to measure (such as the number of hits to the KaiNexus.com website, recently redesigned) but don't really impact the business and its core (like how many customers and how much revenue does KaiNexus have?). More on vanity metrics here and here.
It seems that “number of arrests” might be a vanity metric for the Oakland police department. How do you measure the safety and quality of life in Oakland?
The talk about vanity metrics reminded me of something I saw in the news recently from Indiana: “Nearly a third of Indiana hospitals had at least one reportable error in 2011.”
Wait, so two thirds of Indiana did NOT have a reportable error? Not reporting the errors doesn't mean the errors didn't occur and we know the underreporting of errors in healthcare is a chronic and serious problem.
The 41 patients with bed sores was the highest number since 2005, when the reports began. But 2011's 17 patients with foreign objects left behind, following surgery, were the fewest ever.
“Reported errors” are down, but does that really mean healthcare in Indiana is safer? 41 patients, statewide, seems like an unbelievably low number. The federal government (the AHRQ) says “pressure ulcers” (bed sores) affect 2.5 million patients per year. Something doesn't add up.
More on the “improvement” (is it?) in Indiana: “Medical errors drop at hospitals.”
When I talked about the high number of hospitals with “zero” errors in Indiana, the Lean Startup folks I was chatting with were appropriately skeptical… can we trust self-reported data? How do we get more transparency about healthcare errors? Or, if Indiana healthcare is REALLY that safe, how do we spread those results around the world?
What do you think? Please scroll down (or click) to post a comment. Or please share the post with your thoughts on LinkedIn – and follow me or connect with me there.
Did you like this post? Make sure you don't miss a post or podcast — Subscribe to get notified about posts via email daily or weekly.
Check out my latest book, The Mistakes That Make Us: Cultivating a Culture of Learning and Innovation:
Here are my live notes from the conference via Google Docs:
LINK
Eric Ries kicked off the event saying “no success theatre and no vanity metrics” from the presenters. I love it.
Hi Mark. “Vanity metrics” evolve because of the wrong incentive systems, in my opinion. (Maybe root cause analysis needs to happen.) Thanks for drawing attention to this practice and giving it a name. How do we stop its use?
Giving credit where it’s due (other than me), the term “vanity metrics” isn’t something I coined. It’s in wide use in the Lean Startup community. I think Eric Ries coined the term.
How do we stop its use? I dunno… in an organization, leaders (and boards) need to make sure people aren’t just reporting the things that make them look good. We need critical thinking to drill into the metrics. If you’re a hospital CEO and people are saying “we didn’t make any medical errors last month” then GO AND SEE. Is reality the same as the data?
[…] about people gaming a system and what drives that behavior, whether that’s in healthcare (such as not reporting errors) or in sports. We can criticize the individuals involved all we want, but what they are doing is […]