Posted by: Matt Shanahan

Customer engagement is a relative measure.  More is better, and less is worse. To qualify the level of engagement of a customer, you compare that customer’s engagement metrics to all your other customers.  And in the B2C world, this is a straightforward process.  Let’s face it an individual only has 24 hours/day of attention that can be devoted to your product.  So it’s fair to compare traditional engagement metrics directly between individuals.

And while traditional engagement metrics can be used to compare individuals, how do you determine the comparable level of engagement for an organization? In the B2B world where we’re dealing with significant variances in sizes, attention capacity between two organizations can be enormously different which makes comparison of engagement trickier. How do you compare the level of engagement for a large global organization with 5,000 active users to a medium-sized locally focused organization with 1,000 employees? No longer do we have the built-in apples-to-apples comparison found in the B2C world.

In the B2B environment, engagement metrics need to be normalized to account for the differences in customers’ sizes and contracts.  Incorporating firmographic information (e.g., # of employees, revenues) to compare against actual engagement metrics (e.g., # of active devices, # of active users), lets us take into account organizational capacity, allowing engagement metrics to be scored and normalized and therefore comparable.  This approach lets you determine how much corporate attention you have—essentially, engagement at the corporate level. 

Getting back to our example, if that large, global company with 5,000 active users out of 15,000 employees, it will be significantly less engaged than the medium company with 1,000 active users out of 1,500 employees.  

By overlaying the actuals against the firmographics for the company, we normalize traditional engagement metrics, making them useful in the B2B arena.  This approach let us extend our understanding from “are the individuals within the organization engaged” to “is the organization itself engaged?”

Engagement Metrics, Firmographics Engagement Metrics, Firmographics

Posted by: Matt Shanahan

When it comes to measuring engagement, it’s hard to argue with the approach of measuring everything.  The incremental cost of collecting three or three-dozen quantitative measures is negligible. Why not have duration and frequency of visit, click-through rate, bookmark data, RSS subscriptions, downloads and blog comments metrics available at your fingertips?

My view gets a bit more controversial when we start talking about how to derive meaning from all the engagement data points that are collected. The general model today is to sift through the data points and, based on instinct and/or experience, determine what knob to dial up or down for improvement.  An organization might create a feature that helps improves content resyndication, roll out a program that increases downloads, or create a promotion that increases repeat visits.  All are fine ideas, but they are mostly guesses, and the approach is often shotgun.

I find it fascinating that all of these objective analytics are translated into activity through a mainly subjective lens.  Call me a stickler, but it seems we need an approach these analytics in a way that’s, well, a bit more analytical.

What’s missing is a quantitative platform for comparison—a normalized metric for figuring out which engagement metrics matter. For Scout Analytics in the paid-content arena, this metric revolves around understanding value.  Once you have a quantitative way of understanding the value of your service to your customers, the engagement metrics come alive. Demand Rating™ is that missing metric. It enables organizations to rank customers based on the value that your service provides to them, and group customers into comparables sets to investigate differences. 

For example, you can group customers receiving high value from your service and then compare their engagement metrics to those receiving low value. The delta between these groups will offer insight into engagement metrics that drive consumption. Or, you might compare value across segments —maybe geographical, organizational, or firmographic—to understand which engagement metrics are important to that particular segment.

In the paid-content space, Demand Rating™ quells a lot of controversy because it’s the key for quantifying contribution of each engagement metric.

Demand Rating, Engagement Metrics Demand Rating, Engagement Metrics

Posted by: Pete Horadan

So I’ve laid the foundation for the idea of unique signatures, now let’s look at them in more detail.

On the access side, we start by creating a Device Signature for each machine that accesses an application. We track over 80 unique attributes of the machine, and utilize a proprietary statistical method for understanding the patterns and the evolution of change to the patter for a given machine. Using this approach, we can tell with great accuracy, which machines accessing a system are unique and create associations between devices and users.

Another of our algorithms—keystroke dynamics—can actually get to the individual level if required.  Keystroke dynamics works in conjunction with our Device Signature, analyzing user-specific typing rhythms. While typing a string of characters, every person exhibits a series of keystroke timing events, or a “biometric signature.” The measurement and comparison of user-specific typing rhythms is called keystroke dynamics. Keystroke dynamics derives a unique Biometric Signature of the user—essentially enabling systems to infer individual people using the system, independent of location or system. 

Accurate contextual information has been the focus of our latest technology break-throughs here at Scout Analytics.  Using our experience in creating complex algorithms that produce a highly accurate device and biometric tracking, we’ve developed new algorithms that take the incoming IP address and translate that information into insights about the business or legal entity associated with the user.  By linking IP address information with firmographic and geolocation data from online databases, a Network Signature is derived that can be used for segmentation of behavioral profiles.

The predictions produced by behavioral analytics are only as good as the quality of the data behind it.  We’re focused on doing it right.

Firmographics, Signature Accuracy, Firmographics, Signature

Posted by: Pete Horadan

I talked earlier about the insufficiencies of current tracking approaches and their impact on behavioral analytics.  Let’s now talk about how it should be done—how we do it here at Scout Analytics.  Basically, our technology is centered around a unique signature concept.  We analyze dozens of unique attributes and utilize patent-pending algorithms to derive a unique signature for the device, network, even the individual user (biometric). 

You can think of signatures sort of like a genome—while there are overlapping characteristics—each carries a unique and identifiable pattern and sequence.

Of course, this isn’t easy or everyone would be doing it.  As you can imagine, machines, networks and even people change in different environments and over time—you install a new version of Flash, or change to Silverlight.  The cast on your broken right hand changes how you use the computer.  You change network carriers.  The trick is to have algorithms that take into account enough factors and have enough intelligence to sense the change as a progression of a previous signature, rather than a different signature.

We have algorithms for recording and tracking a Devices Signature, a Network Signature and a Biometric Signature.  Each of these unique signatures works to improve the accuracy of tracking and context during analysis.  So next time I’ll give a little more insight into each.  The development of these techniques is something that truly sets us apart, and it’s what makes Demand Rating™ work.

Behavioral Analytics, Engagement Metrics Accuracy, Behavioral Analytics, Engagement Metrics

Posted by: Pete Horadan

Last week, we looked at the serious acurracy issues associated with using cookies to track user behavior and engagement.  This week we’ll explore the accuracy of another often-used factor, IP address. In short, IP address provides even less information about users than a cookie. 

The accuracy issue starts with the fact that one user may have many IP addresses.   For an example, let’s look at me. I access different sites (e.g., TechCrunch) from home through my personal broadband, while at my coffeehouse through their broadband, or through the network at my corporate office. When I’m on the road, I broaden my access points to include hotels and airports. When using IP address to count visitors, the information collected for me might look something like this:

IP Address                    Domain                         Location
24.16.13.198             comcast.net                Woodinville, WA  
208.54.4.23               tmobile.net                   Issaquah, WA
70.103.221.189    scoutanalytics.com              Issaquah, WA
12.204.178.67          marriott.com                  New York, NY

Of course this is incorrect. What appears to be four unique visitors, is really only me. 

The flip side is true as well, namely one IP address may have many users.  The common example is found when users behind an office firewall may all appear to come from the same address. In this case, paid content providers have no visibility into individual users or their different behaviors. So, when trying to analyze user counts from IP addresses, the problem is almost intractable.  Does a given IP address represent one user, many users, or just a fraction of a user?

On their own, IP addresses have serious limitations in terms of establishing unique visitors or tracking behavior, seriously over- or under-estimating the true picture. Improperly used, they can lead to faulty analysis.  IP addresses are best used to provide hints which I’ll discuss more in future posts.

Behavioral Analytics, Engagement Metrics Accuracy, Behavioral Analytics, Engagement Metrics

Posted by: Pete Horadan

So we’ve been making the point all along that Demand Rating™ provides a new lens on customer behavior. Realistically though, the usefulness of behavioral analytics and engagement metrics are gated by accuracy.  So how do you get accurate engagement metrics?  What’s really knowable?

Let’s start with usage tracking.  The traditional approach is through the use of tracking cookies.  A cookie is a small piece of data that is stored on a device by the application. Each new browser that accesses an application is given a cookie, and the application relies on that cookie to track user behavior. The issue, of course, is that the cookie method of tracking is wildly inaccurate, and it’s simple to see why.

At the highest level, tracking based on cookies don’t track individuals at all—they track cookies.   Users access applications through a variety of browsers and machines—from the office, at home, from their mobile devices.  Cookies can and are often reset by the user or even automatically by the browser whenever it shuts down.  It is practically never true that an individual user will utilize a single cookied browser for access.

In fact a recent study at Penn State shows that “cookies are about just as inaccurate in estimating unique visitors as unique network addresses,” overestimating audience size by 7 -30 times! So, do you really want to know that 90 cookies accessed your application?  Or that three people did?  The whole flawed cookie process results in a huge overstatement of usage and a total inability to track actual people. 

In a future article, I’ll talk about a better approach that avoids these problems.

Behavioral Analytics, Engagement Metrics Accuracy, Cookies, Engagement Metrics

Posted by: Matt Shanahan

John Lovett has an interesting post Building a Culture of Measurement.  The title and content caught my eye because it was embodied a big force behind the development of Demand Rating™.  As John stated in his post, “Culture consists of values, beliefs, legends, taboos and rituals that all companies develop over time.”  Below are some of our values and beliefs behind Demand Rating. 

Scout Analytics set out to measure one of the most critical variables in sales, marketing, and product management: customer demand.  Our culture is marked by pursuit of an astonishingly simple measurement that has dramatic impact on results.   The Demand Rating measurement was originally sparked from a book called How to Measure Anything and the following excerpts have directly shaped our culture:

Measurement is a set of observations that reduce uncertainty where the result is expressed as a quantity.

If it matters at all, it is detectable/observable
If it is detectable, it can be detected as an amount (or range of possible amounts)
If it can be detected as range of possible amounts, it can be measured

In business cases, only a few key variables merit deliberate measurement efforts. 
The rest of the variables have an “information value” at or near zero. 
In other words, most measurements do not reduce uncertainty.

Guessing which customers represent the best sales opportunities is still widely based on intuition and experience.  It doesn’t have to be that way.  Learn more about Demand Rating in this blog.

Customer Demand, Demand Rating Behavioral Analytics, Demand Rating