Why are they broken? Because I like them, because I like charts, graphs and numbers, and because the tools are awesome. Ok, that doesn’t really explain why I think analytics are broken, but it might help explain why I feel this way. So if the tools are so awesome and easy to use, why are analytics not use properly? Because it is the actual process that is broken (well, I’m talking for the rich web experience world because that is what I know).
How it should be
I guess people think implementing analytics is easy so they don’t give it enough time. The part that is not working currently is not the implementation but the analysis. And the analysis part is the most important, if you don’t do it, there is nearly no point to implementing analytics. In my head you could divide the analysis part in two, first would be building an hypothesis document about what the final data is going to be (ex: we are going to try this interface and it is going to have this impact on data), and secondly an analysis document that review the previous hypothesis, tries to explain why things went well or not and what to do the next time to improve. The thing is, none or only part of that is happening right now. Having made these steps will also make implementation more coherent (and no, just tracking everything and not caring is not a solution).
The problem: finding the right person
I think that the whole problem comes from the fact that it is not the right person in charge of the analytics. Most of the time one of two things happens. Either the person in charge of analytics is a so-called analytics expert paid by the client (so separated entity than the production company), that is never talked to, that just provides an excel sheet with stuff to track, he usually works way before the project is completed (so project could change a lot meanwhile), and once this is done, he is never heard of again. Most of the time the goal is to integrate the data of this campaign into the data of the main site. The problem is that campaigns (rich experiences) needs to be tracked differently so the data probably integrates badly anyway. The second case I have seen is where the developer is the one in charge of analytics. I think this approach is already a bit better because you can’t get someone closer to the project than the developer. The problem is that most of the time, he doesn’t have access to the data once the project is completed, so no post analysis can be done or at least not by the person who would know what to analyze.
Who is that right person?
For me the right person would either be an information architect or a usability expert. These people are close enough to the project plus they are the one who need to check if they what they did was right using the real numbers. These guys could write proper hypothesis before the project, write thoughtful analysis and they can point out what to improve next time. Actually, an IA or a UX person that doesn’t care about analytics seems pretty phony to me, because they can’t prove what they preach. Analytics are easy, these people should be able to pick it up.
So what now? Well we can hope this gets better. UX and analytics are still pretty new fields (IA is a bit older) so I think they might evolve in the right direction, but we have to keep pushing so that things get done right. And don’t worry, if it doesn’t, I’ll complain again!