Thursday, February 18, 2010

The not-so-trivial measurement problem

Local VC legend Fred Wilson has a great blog post today titled "How Unique Is A Unique Visitor?" in which he points out that in a typical day he uses seven different browsers. My own count is similar: (1) iPhone Safari, (2) iPhone Twitter client's internal browser (3) my laptop (Firefox/Linux), (4) my laptop (Firefox/Windows, running in VMWare/Linux), and (5-7) several different university computers.

Ponder that, imagine your own usage, and then think about Wilson's point: a "unique visitor" (if measured by a browser cookie) is overcounting me by 6x! If you're monetizing your site with ads, this is a huge error margin. It suggests an inverse of the problem magazines and newspapers face. They know circulation (# of copies distributed), and then they multiply this by a (made up) factor that represents "readers per copy," and they get "audience size."

With cookie-based web audience sizing, Fred is showing that you're not measuring "unique people" but rather "unique browsers", and (inverse of the magazine problem), you need to divide by "browsers per person" to get back to "unique people".

Fred's take is: (1) web log measurement (if it relies on cookies) probably over-states by the above factor, (2) other single-scheme measurements (comscore, compete, alexa, etc) are subject to other problems (esp., extrapolating from non-uniform and small-sized populations), (3) he's bullish on comScore's hybrid approach.

No comments: