|
|
 |
| |
By Gary Angel
Expert Author Article Date: 2007-10-03 As someone who has worked on various measures of Engagement quite a bit, I read Avinash's "Engagement" Is Not A Metric, It's An Excuse" with considerable interest. As always, Avinash makes some excellent points and the post is well worth reading, but I don't ultimately find in it much reason to change the direction of the work we're doing (which often involves building Measures of Engagement). There are five separate points in Avinash's argument concerning Engagement - some powerful, some not. I'd like to consider each and see what they add up to. Point 1: Each business is unique so it's hard to generalize measures of Engagement. I think this is true. I know of no satisfactorily generalizable Engagement Metric and I see only very modest hope of every attaining one. That being said, the desire for comparability is an illusion across much of web analytics. People always ask me if X% conversion or Y% lead generation is good. And I'm forced to answer that the question as stated is meaningless and always will be. Claiming that X% Conversion is good is akin to claiming that there is some "natural fair" price for a commodity. It's an easy mistake to make - but it's completely wrong. Since I'm assuming we can all agree that this lack of comparability doesn't imply that measuring Conversion Rate on your own site is meaningless, I don't think the non-comparability of Engagement is fatal. Does it present difficulties in understanding? It may. I've found that more complex data definitions almost always create problems of explanation. When we use neural networks to generate complex visitor segments based on the information in hundreds of behavioral data points it's nearly impossible to explain (without a great deal of gloss) how a segment is built. That's a significant drawback to the technique - but one that is often outweighed by the ability to capture many complex behaviors in a single segment. The same may well be true for Engagement - when you capture multiple behaviors in a single measure you have both simplified your reporting and complicated your vocabulary. The benefits of the trade-off need to be judged by individual cases. Point 2: It is nearly impossible to define engagement in a standard way that can be applied across the board. This strikes me as Point 1 re-stated with slightly different words. Point 3: Engagement tries to measure something deeply qualitative. I think this point is misguided on several levels. I'm not sure it's true and to whatever extent it is true, I'm not sure it's meaningful. Let's start with a simpler metric - impressions. Impressions has a specific meaning - an ad was displayed where a person could see it. It has an implied meaning, that of making an impression. But "impressions" doesn't truly measure making an impression. It measures a behavioral fact about how often a visitor was in a particular situation where a psychological impression might have occurred. Most measures of Engagement are similar. They measure a specific set of Behaviors that in theory may represent a psychological state of engagement. If you think about it, you'll see that even seemingly uncomplicated metrics like Page Views are this way. Did a visitor actually view (read or see) a page? That psychological question isn't actually answered by the metric - but we tend to assume that the behavioral proxy is a reasonable measure. The same is true for measures of engagement. When you are measuring Engagement you're typically measuring a set of behaviors that you believe are a legitimate proxy for establishing differing levels of psychological engagement. It's an empirically verifiable fact that on many web sites these proxies are predictive of final outcomes and are, therefore, measuring something real. It is, of course, a much deeper question whether your measurements of engagement are right. But that isn't what Avinash is getting at. So I'd tend to say that almost every web metric we use is a proxy for a psychological state and that this is untroubling. We don't actually care about the psychological state - if the behavioral measure is good, it's good. I think there's a second issue lurking here that's deeper. Some types of questions are either extremely difficult or impossible to answer with behavioral analysis. We often talk about "why" questions as being like this. We see drop-off in a step in a conversion process. We can see the drop off, but the data will often tell us nothing about why visitors drop-off at a particular point or at all - their state-of-mind isn't revealed by the behavior. I think Avinash is trying to suggest that Engagement is this sort of question; however, I think it's clearly not true. Measures of Engagement are typically scores based on accomplishing some specific set of site actions - they are a pure representation of "What Happened." Perhaps there is too much psychological superstructure in the word engagement for some people's taste, but I don't think it's reasonable to suggest that a typical Engagement metric isn't measuring a quantitatively different and potentially significant variance in site usage. It is. Point #4: One of my personal golden rules is that a metric should be instantly useful. This is just wrong and I'm surprised anyone still believes this. Old myths die hard. I thought I had conclusively shown that no single metric is ever instantly useful. For the complete and I think absolutely irrefutable argument, you can read this extended post on why a 100% Conversion Rate is a bad thing! ( and the follow-up). If you don't want to dive that deep consider this simple case. I tell you your Conversion Rate went from 30% to 32%. Is that good? If you think so, what if I add that your cost of traffic doubled? What if I told you your Conversion Rate dropped from 10% to 7%. Bad? What if you sourced ten times the traffic for the same cost? Still think you can make instant decisions based on the change in a single metric? Almost anything we are ever going to care about is a system - and it is virtually impossible to represent a system in a single directional metric. I'll make a second point here and one that I think gets to the heart of a fundamental mis-understanding about reporting. The idea that you can with a single, simple uncomplicated metric tell a meaningful story is WRONG. Badly, completely and totally wrong. There is not a single idea that makes reports both worth less and misused more than this widely propagated idea. It is not so. And analysts that strive for this ideal are sorely misguided. Conversion-Rate, as we have seen, is only significant in the context of a particular level and cost of traffic. If you ever build a report that leaves a decision-maker with the impression that they can act based on movement in a single number (like Conversion Rate) without providing the key contextual numbers (traffic and cost of traffic for example) then you've failed in your job. Giving a decision-maker too little information is much worse - and much more dangerous - than the sin of providing too much. Both are sins, but one is venal and the other mortal! So typically, you have a choice between building complex metrics like Engagement if you want to represent real-world trends concisely (these aren't instantly actionable either) or you can present a decision-maker with a matrix of 15-20 simpler metrics. Which is better? Not always a trivial call, but lots of times the former is better. It's true, you'll have to do some spadework within your organization so that everyone understands and buys into the Engagement metric. But once you've accomplished that, you're reporting on a few numbers not 15 or even 50. That's a big win when it comes to actually absorbing trends on an ongoing basis. I can tell you from personal experience that when you give a decision-maker a report and say: "Here's our campaign data broken out across twenty different success metrics on our site," the first question I'll get back is "but which campaign is best overall?!" Point #5: Most of all engagement is a proxy for measuring an outcome from a website. Conversion is not enough, as mentioned above, so we try something else. The problem that we'll define engagement as a measure of some kind of outcome but we won't give it the sexy name of engagement. I'll go along with Avinash part-way here. If your measure of Engagement is just view / visitors then you might as well it call it that. But if your measure is based on views, interactions, events and weighted content areas, what exactly are you supposed to call it? And what, really, is wrong with calling it engagement? Some words carry too much baggage to be deployed effectively by the analyst. I just don't see anything in Avinash's post that makes me think Engagement is one of those words. In my experience, Engagement isn't a proxy for measuring an outcome on a site at all. It's more often a means of aggregating a set of outcomes into a single visitor score or segment. Since a set of outcomes can't reasonably be described by just appending all of the events into one long name, the analyst is always going to have to pick a name that reasonably represents the overall concept. In many situations, and for many sites, "Engagement" does a pretty good job of carrying that baggage. So I guess I score the argument this way: points 1-2 are compelling if you had unrealistic expectations of using an Engagement Metric to compare your site to others; point 3 is interesting but almost certainly misguided; point 4 is flat-out wrong; and point 5 misses the true object of most real attempts to use Engagement. Add it all up, and I think the biggest idea I'd take away is that if you are going to call something Engagement it probably shouldn't be reducible to a single simple metric (good point - I've made that mistake) and you should probably make sure you are comfortable with the trade-off between the long-term economy of collapsing multiple metrics with the price you pay both in explanation and loss of clarity (this isn't in the original post at all but I think it's what ultimately matters). As is so often the case, the real-world complexity of being an analyst is rarely reducible to simple rules of "do this" or "do that." Reporting is much more complicated than most people ever realize, and it involves an unending series of trade-offs to which there is no perfect answer. Can an Engagement metric be misused as an excuse? Of course. Can it be immensely valuable? Of course. Comments About the Author: Gary Angel is the author of the "SEMAngel blog - Web Analytics and Search Engine Marketing practices and perspectives from a 10-year experienced guru. |
 |
| Newsletter Archive | Submit Article | Advertising Information | About Us | Contact |
