Posted by: Jeremy Reimer on Wed Jan 9 23:27:41 2013.
Eric Ries is not very fond of what he calls "vanity metrics"-- numbers that show things going up or down (like hits on a website or number of new customers per month) because they don’t actually measure what was done to cause the change.
This leads to people making the following conclusions:
In my experience, when the numbers go up, people think the improvement was caused by their actions, by whatever they were working on at the time. That’s why it’s common to have a meeting in which marketing thinks the numbers went up because of a new PR or marketing effort and engineering thinks the better numbers are the result of the new features it added.
Unfortunately, when the numbers go down, it results in a very different reaction: now it’s somebody else’s fault. Thus, most team members or departments live in a world where their department is constantly making things better, only to have their hard work sabotaged by other departments that just don’t get it.
He suggests two solutions to this problem, both of which need to be implemented. First, people need to work in cross-functional teams, not traditional departments like marketing or engineering. Second, metrics need to actually give real information about what caused the change. Primarily, he suggests using A/B testing on the product (giving different versions with and without a new change to different groups of customers).
For the entrepreneur working with a small team of a few (or even one!) the first solution is irrelevant, but the second could prove invaluable.
In other news, I made a new comic. Go read it!