Tag: online marketing analytics
Connect With Us
Revealing Patterns of Change
This is Fun, But Not When You Are Under the Gun!!!
Gartner has launched a new focus area called “Pattern Based Strategy”, based on the need of businesses to capitalize on large amounts of data and the new rules for business process adaptation.
Here is a great verbatim quote from the Gartner web page.
The depth of the recent recession blindsided most businesses. As the economy starts to recover, many business leaders are thinking, “If I had seen this coming sooner, I could have acted faster, decreased my risk and enhanced my opportunities for growth.” There is a way to see things coming. It’s a framework for proactively seeking and acting on the early and often-termed “weak” signals forming patterns in the marketplace. It’s also about the ability to model the impact of patterns on your organization and identify the disciplines and technologies that help you consistently adapt. It’s called Pattern-Based Strategy.
The key to Pattern Based Strategy is automatically revealing intelligence that is hidden in the data/information. Companies today are running more lean than ever before. Employees across all organizations are inundated with work and overloaded with data. . There is a great need for technology that will make our jobs easier and make us more productive. At Gartner, the idea that emerged, led by Yvonne Genovese, is called Pattern-based Strategy (PBS).
We are victims of too much information, missed opportunities and ‘@#$% I wish I could have seen that!‘ moments. Connecting this to a rather timely/charged topic – Think about a recent attempted terrorist attack by the Nigerian traveler who bought a one-way ticket, paid in cash, checked no bags, boarded an international plane. There were a very large number of ‘red flags’ in the sequence of events, and there was a large volume of data hiding all this intelligence. A Hope Strategy is to hire tons of people and make them search the data for red flags, more importantly sequences of red flags. This may work sometimes. But it is a poor and expensive strategy, and rarely does it produce the desired results on time! (making it quite useless, actually!)
As companies start to incorporate intelligence from data into their operations, one of the primary issues is the ability to have the intelligence automatically come to you. ‘Digging for insight’ is a poor, time consuming, expensive strategy. We need the technology to work for us. Second, it is also important to start focusing the insight with a particular business function/strategy in mind. Sales, Marketing, Operations, etc.
Connecting this back to what we do, Emcien provides analytics that automatically reveal customer buying patterns in sales data. The analytics reveals the popular choice combinations, key differences by region, key trends and new emerging segments. This is an example of technology working for you, bringing insights back so that you can act on it.
Quoting a Regional Practice Manager and the Senior Architect for Siebel -
Emcien offers rigorous and repeatable detection of buying patterns, enabling your customers to act on them, while supporting your product objectives (margin, inventory, velocity, …)
Quoting a former Oracle Practice Manager and Senior Siebel Architect -
Emcien offers rigorous and repeatable detection of buying patterns, enabling your customers to act on them, while supporting your product objectives (margin, inventory, velocity, …). Emcien’s offerings readily integrate with Siebel, enabling immediate improvements to revenues. Few projects offer such potential for improving the customer experience and increasing revenues, with so relatively little development or integration efforts.
Automatically revealing patterns is required today as we all drown in data, and do not have time to hope that someone may find the intelligence that the organization needs to act on. Thanks to Gartner for launching this focus area!
Comments Off
How Good Is Your Demand Intelligence?
I was meeting with Martin, the CEO of a fortune 1000 Company. He talked about sales productivity as a significant opportunity and an area of strategic focus for 2010. “We have been selling this product for years. Yet, every day is ground hog’s day for our sales reps. What I mean by that is every sales call and quote request is treated as if we have never sold this product before!”.
Maximize Sales Productivity On Every Channel With Demand Intelligence
In the B2B markets there are distinctive patterns in the product choices that customers make. It isn’t really customer intelligence as much as demand intelligence. The B2B markets are different from B2C. The average purchase value is typically larger, the frequency of the same end-customer is lower, but there are distinctive patterns in the product purchases. The purchase patterns exist by product choice combinations, customer type, vertical, usage, geographic region, price point and so on.
“I know that there are patterns in what our customers buy,” added Martin. He was previously the VP of Sales and has knowledge of what products customers buy. “We need our sales reps to have access to that intelligence so that they can be better advisors to customers and close the deal faster. I have sat in sales meetings with a company exactly like the one we sold to 3 months ago, and watched my sales rep grill the prospect on product requirements. That hurts our sales more than anything else.”
In a recent blog, Michael Gerard from IDC wrote a very interesting article on the same topic. He mentions a story where a CIO from a $10B+ company had to continuously teach a vendor sales reps what he had purchased from them in the past. The article goes on to state that this can lead to poor credibility on the sales front lines.
There is a solution for this problem. Every company is sitting on tons of sales data. It is a wealth of data that can reveal what their customers are buying and where they are willing to spend their money. Emcien offers analytics that auto-detects the choice combinations in sales data. What are they buying? What are popular product choice combinations? What are popular choice combinations by vertical? What combinations are profitable? Which ones are not? This is the demand intelligence that makes each day NOT be ground hog’s day!!
As a first step, it is invaluable to arm your sales reps with this intelligence so that they can be smart on every deal. In fact, every sales channel can benefit from this intelligence. Here are a few examples -
- Sales reps can use demand intelligence to be better advisors to customers, convert requests to quotes faster and recommend good choices to close the deal. Even a simple report on what is the fastest selling product choices will empower your sales reps, and drive to the bottom line.
- Your ecommerce site can use demand intelligence to quickly recommend the best products to help the customers to self-serve on and make better decisions.
- Inside sales team can use demand intelligence to quickly complete quotes and close the deal. I was talking to an inside sales rep and he told me that his biggest challenge is quickly responding to a quote with a price. ”We have done this so many times before,” he said. “Why does it take us to long to get a price? We lose deals because we cannot respond fast. The first to respond with a quote locks in the deal 90% of the times, even if the price is higher. We lose deals by being slow.”
- Call centers can use demand intelligence to cross sell/up sell based on buying patterns. The turn over in call centers is high. Automating the cross/up sell with demand intelligence will dramatically improve productivity and profit.
Quoting Michael Gerard “This is only the tip of the iceberg of course.” Demand intelligence can dramatically improve your sales performance, customer satisfaction and profit margins.
Follow the Money!
Buying patterns and the economy are constantly changing. Some products and categories that were popular are not anymore. You cannot control your customers’ tastes or the economy. But if you follow how the money is being spent, you can make a lot more! Unlike clicks and page views, buying patterns are very reliable as they are based on actual sales. Money changed hands. An economic transaction occurred!
Track sales transactions to understand your customer’s buying patterns, establish a more relevant product mix, satisfy more people and sell more.
Your customers speak to you when they buy. If you can listen to what your customer wants you can manage the buying process and you can influence and even control it. “Why would I want to do that?” you may ask. By better understanding your customer buying patterns you can establish a more relevant product mix that will satisfy more people. You can also guide them to more profitable choices at point of sale based on product availability or close substitution. You will satisfy more people and sell more. You will also make it easy for them to buy your products and services.
The Analytics of Buying Patterns
First, take the guessing out of the equation. You need to know what your customers are purchasing and what they want to buy from you in the future. This intelligence is available in your sales transaction data. Customers buy your products and services in distinct patterns.
Products and services have become more complex and companies offer a dizzying array of choices. However, with analytics the sales data will reveal popular combinations of choices. These popular combinations are guides on how you can make your products and services easier to buy. How you can make is easier for customers to do business with you.
There is also the issue of product profitability. Some of the choice combinations are more profitable than other. Again the analytics will reveal which combinations are moneymakers, and which ones not! Once again – if you have access to this intelligence, you can stock the right product mix and guide customer to better choices. If you stock inventory in your store you can leverage this intelligence to plan an optimal inventory mix. That means making the most money from the least amount of inventory investment while satisfying your customers’ needs.
Whether you are running an online store or a brick ‘n mortar store – this is a key principle to selling more and maximizing your capital utilization.
Demand Sensing And Demand Shaping
Forecasting and planning is a challenge in the best of times. The times we are in make this a herculean task. Market demand shifts continually as economic conditions change, products change, prices fluctuate, competitors act, new products are introduced, marketing creates promotions,……. The list is quite endless. Current planning and forecasting methods are reactive and struggle to keep up with these shifts.
The solution is “Demand sensing and Demand Shaping” – active methods to predict what demand will arise and keep ahead of the market. Demand sensing is the ability to detect what choices customers are buying patterns and the trends associated with these choices. Demand sensing can help you to quickly see market shifts to plan your product mix and offering.
Customer Buying Patterns "Customers who bought this SKU also bought this other SKU"
Demand shaping is the ability to guide customers to the best choices at point-of-sale. This is the key to increase revenue and supply chain efficiency. However, demand shaping needs product intelligence at point-of-sale to guide customers to the best choices. Some of the ways to demand shape are –
- If you offer many products or SKUs, there are typically strong buying patterns in the demand. For example – This printer is often bought with this unbleached paper, this ink cartridge and cable. Then, when a customer selects the printer at point of sale, you want to automatically show him the other items that have strong buying patterns. The customer will thank you for this recommendation because usually they need this additional stuff, and you just saved him a ton of effort thinking about it, and a ton of time searching for it. And you made more money in this sale!
- If you offer a product with many attributes, every sale will begin with the customer calling out a few attributes. The opportunity to demand shape is to recommend a good choice based on the partial list of attributes the customer has called out. Demand Shaping requires the ability to complete the order with the right attributes. The best way to complete the order is to have sales intelligence these attributes are bought with these other attributes. It is the Amazon-esque way to look at products with many attributes.
- The biggest opportunity of Demand Shaping is guiding customers to close-enough SKUs. Most customers describe the products they want to buy with a ‘kinda-sorta’ attribute description. As the number of product features grow, there are a large number of SKUs that are similar or close-enough that they can satisfy the customer. So there is a significant opportunity to guide a customer to a similar or close-enough SKU at the point of sale. The recommended SKU may differ in attributes that the customer did not “call out” or specify. If you can offer up this SKU it is a win-win. You have served the customer. You have won the sale. You have moved your inventory. And your competitor did not get this customer.
As product choices and the number of SKUs grow, these techniques are mandatory for an efficient supply chain and for a good customer experience in this customer-centric world.
I just read an article by Mark Pearson, Six secrets of Supply Chain Planning Masters.
Quoting Mark Pearson’s article – Think of demand sensing as predicting what demand will arise, as opposed to simply reacting to incoming orders. Shaping demand, on the other hand, is all about steering customers toward available products and services. Compared to laggards, more than four times as many masters said they can predict demand with greater than 80 percent accuracy levels. And nearly twice as many masters said their ability to shape demand was “good” or “excellent.
Comments Off
Do Customer Buying Patterns Exist?
Customers have to make choices in order to buy configurable products. Do they make these choices at random, or are there patterns? When we look at the sales history for a configurable product, like a car or a computer, can we tell if customers have just been flipping coins and rolling dice? Or do their choices hang together and make sense? To answer this question, we would have to look at how they buy combinations of options. In the previous post, I took a pizza as a simple configurable product, and looked at how customers ordered pairs of toppings. Just by looking at the sales numbers we could detect that the selection of pineapple and Canadian bacon are not independent. Even if we had never heard of a Hawaiian Pizza, we could discover it in the data.
Even more information is hidden in combinations of three toppings at a time, or four toppings at a time. Any combination of toppings will have appeared on some of the pizzas that have been sold (or maybe none). The relative popularities of all the different combinations has a clear message: customers are not flipping coins. Some toppings naturally go together, and others do not. Pepperoni, broccoli, and anchovies is just unlikely. If a particular pizza restaurant has a few “house specials”, like the Meat Lovers and the Veggie Delight, we can see them in the data, even if we don’t know their names.
What is true of the pizza is also true of other configurable products: computers, trucks, tractors, lighting fixtures, industrial fans, and so on. All products that have variety. Customers make choices, but not by rolling dice. There are combinations that go together and combinations that do not. A pizza maker can juggle the preferences of his customers in his head. But when a product has 30 or more features, intuition is overwhelmed. The number of combinations explodes so fast that the unaided human mind can’t see the patterns. At this point, mathematical models and intense number crunching can reveal the patterns and let the product manager for a line of trucks be as confident as a pizza maker.
Do Customer Buying Patterns Exist?
Buying patterns are real, and they manifest themselves in how customers buy combinations of options. With the computing power we have available today we can detect and capture them. These patterns can then be used to design “house specials”, forecast future sales, and guide customers to what we want to sell them.
So, who else is talking about customer buying patterns?
Intel Talks about Changing technology buying patterns
Customer Buying Patterns have Changed. What’s Your Plan?
An entire report that summarizes the results of a consumer usage and purchasing pattern survey conducted in March of 2007. The survey was conducted with In-Stat’s Technology Adoption Panel (TAP) — a dynamic, online panel of more than 19,000 technology users and decision makers. Over 1,400 technology users responded to this focused survey.
Findings in this report include consumers’ time spent on PCs, when they last purchased a personal-use PC, the PC’s features/form factor/usage, the desired features of future PC purchases, changes in usage patterns, and consumers’ thoughts about new technologies.
The changing patterns include -
- When consumers are likely to make their next PC purchase.
- The features consumers state they want
- The features consumers state they really want, based on changes in their usage/buying patterns.
- How consumers view new technologies
However – buying patterns are constantly changing. As social networking grows, we are watching new markets emerge every day. There is gold for companies who can continually detect these patterns and offer the right products and feature mix.
Customer Buying Patterns – What you can learn From Pizza Sales
There are 1,140 ways of ordering a pizza with 3-toppings, if the pizza offers 20 choices
First order take rates tell us about the relative popularity of different options. For example, consider a small set of possible pizza toppings.
|
Topping |
Take Rate |
|
Pepperoni |
40% |
|
Mushrooms |
20% |
|
Pineapple |
3% |
|
Canadian Bacon |
3% |
|
Green Peppers |
10% |
Customer buying patterns really start with second order take rates, which tell us about pairs of options, or toppings. Second order take rates tell us about relative popularity, but they also reveal something deeper: dependence. If you know that a pizza has pineapple on it, there is a very good chance that it also has Canadian bacon. This is dependence. In this case the reason is that there is a widely known “Hawaiian Pizza” that has both of these toppings. In general, customers don’t flip coins or roll dice. They select options that “hang together” in some way. The patterns can be seen in the combined take rates. Let me illustrate with three examples that are contrived to illustrate some important points. First consider pepperoni and mushrooms together.
|
|
|
Mushrooms |
|
|
|
|
No |
Yes |
|
Pepperoni |
No |
18% |
12% |
|
Yes |
32% |
8% |
|
In this table you can see that pepperoni has a 40% take rate, since 32% of pizzas have pepperoni without mushrooms, and 8% have pepperoni with mushrooms. In the same way, 20% have mushrooms, because 12% have mushrooms without pepperoni and 8% have mushrooms with pepperoni. This illustrates the first law of second order take rates: the first order take rates must be preserved. But this table contains no new information. Customers are apparently ordering pepperoni and mushrooms independently. This is revealed by the fact that 8% is exactly 20% of 40%. Knowing that a pizza has pepperoni does not give us any clue about whether or not it has mushrooms. Similarly, knowing that it has mushrooms is useless in guessing if it has pepperoni.
As a second example, consider the two toppings that are on every Hawaiian pizza: pineapple and Canadian bacon.
|
|
|
Canadian Bacon |
|
|
|
|
No |
Yes |
|
Pineapple |
No |
97% |
0% |
|
|
Yes |
0% |
3% |
For simplicity, I have made this an example of complete dependence: a pizza has pineapple if and only if it has Canadian bacon. Notice that the first order take rates (3% for each) are preserved.
The third example is the really important one: partial dependence. This is illustrated here by pineapple and green peppers.
|
|
|
Green Peppers |
|
|
|
|
No |
Yes |
|
Pineapple |
No |
89% |
8% |
|
Yes |
1% |
2% |
|
In this case, the first order take rates are also preserved: 3% for pineapple and 10% for green peppers. But these two choices are not independent. The take rate for pineapple and green peppers together is 2%, which is much greater than 10% of 3%, which would be only 0.3%.
Exercise for the reader: show that if we know that the pizza has green peppers, then there is a 20% chance that it has pineapple. (Much greater than its 3% first order take rate.) And if we know that it has pineapple, then there is a whopping 67% chance that it has green peppers!
So second order take rates capture information about how customers are combining toppings, and we can use that information to make predictions.
Comments Off
Value of SKU Intelligence (What Are Customers Buying?)
One of the most frequent questions we are asked about Emcien’s methodology is “why hasn’t anyone done this before?” The answer seems obvious to us, but we should probably write it down once and for all so we can just point people to a definitive document.
Exploding SKU’s to attributes generates a LOT of data. It is only useful if this translates to actionable intelligence. As we all know, we don’t need more data. We need actionable items and recommendations to improve business. The simplest answer is that collecting all the attribute data for SKUs has not been done before, because the algorithms have not existed to farm intelligence from them. In this blog I am going to address –
- Why having the data with the SKU attributes is invaluable!
- What analytics capabilities are needed, if you have that data
- Value of having SKU Intelligence (To drive SKU velocity)
Consider a product that has attributes, and offers lots of variety, also called configurations. Almost all products come under that class today! Think car, computer, or cell phone. The product has features, and each feature has alternative options. For example, a car has an engine (V-6 or V-8), a body type (sedan or convertible), and a color (red, green, blue, black, or white). Any specific car has many choices for each of the features. Customers make choices on the features. Do you want the cloth seats or the leather seats? Do you want the DVD player? How about the iPod connector? Make a choice for about 30 features and you are done. This applies for shampoo, toothpaste, computers, light fixtures, consumer electronics, … all products that have attributes choices.
We consider a configurable product that has a large number of features, each of which many alternative choices/options. Notice that we have finessed the hierarchy problem by allowing only two levels: feature and option. A customer will typically call out only a few attributes during a purchase. They expect you to know how you can complete that spec to fill the order. That is the single biggest opportunity at every point of sale!
If we have data with attributes for every SKU, we can begin to talk about buying patterns! A buying pattern is groups of options that are bought together, like the red color with the convertible body style. Or the DVD player with the leather seats. Or the pattern might involve 3 options, 4 options, 5 options…. many options.
Auto detect most popular attributes in fastest sellers
This brings me to a very important point. If you have sales data with attributes, you need an analytics engine that will automatically detect and tell you what attributes are bought together and are highly popular. A reporting tool that makes you query every choice combination will NOT work! You will be very old by the time you get an answer to most popular choices, as there are millions of attribute combinations. Emcien offers an analytics engine with cluster analysis that will tell you what attribute groups are popular. This answers the questions – what are customers buying and what attributes are popular. This knowledge can also be used for planning the SKU definition and knowing what products need to be on the top landing pages of your web site/ store aisles. (SKU Definition is the list of attributes in the SKU.)
Once the SKU definition is in place, 75% of the cost structure and efficiency of your supply chain has been fixed ! Your supply chain operates under the assumption that the SKU definition is correct. What does this mean? This means that if you offer 2 SKUs with slightly different attributes, that could have been consolidated into one, the supply chain will suffer that cost and inefficiency. The SKU definition has to be optimized before you send it into the supply chain. This is a key driver to SKU inventory.
Customers buy products based on choices at the attribute level. If you cannot gather demand intelligence at the attribute level, you are out of touch with your customer. Customers DO NOT buy SKUs. They do not know the SKU numbers, and they do not care.
SKU Challenges Based on Supply Model
Your business falls into one of these categories based on your supply model (see table). Knowing what attributes customers are buying can dramatically improve your demand response. You will be able to improve -
- SKU definition -This means knowing how many SKUs you need and what needs to be in the SKU
- Demand Forecasting – You will be able to forecast demand at the attribute level, which is the level that customers are buying
- SKU Inventory - plan what to stock to have highest turns.
- Sales Productivity and Efficiency – If you know what attributes are selling together, you can implement an automated recommendation engine (EmcienMatch) and your sales rep can recommend a good choice to the customer during the ordering process. This is probably the single biggest value of knowing what attributes customers are buying. Your sales reps are a trusted advisor to your customers. If the sales person know that all customers of a certain type bough a particular configuration, he could recommend that choice to the customer. Win! Customer is happy because he done! Sales person is happy because he looks smart! You are happy because you increased your sales repeatability!
For our Nerdy Math readers, here is technical nugget that you will love:
SO – here is another reason why what Emcien does has not been done before. Statistics deals with numerically valued variables. A numerically valued random variable X has a domain that can be classified as a ratio scale, an interval scale, an ordinal scale, or a nominal scale. In a nominal scale, the numbers {0,1,2,3,…} are just labels and have no numerical significance. In an ordinal scale the numbers provide an ordering, so that 2>1 in some appropriate sense. An interval scale allows real numbers (like 3.27) and differences are significant, but ratios are not. The classic example is temperature. A difference of 10 degrees is bigger than a difference of 5 degrees, but 100 degrees is not twice as hot as 50 degrees. Finally, a ratio scale allows all of the usual numerical operations on real numbers. Most of statistics assumes a ratio scale, and almost all of it assumes at least an ordinal scale. This leads directly to the mean, variance, covariance, and correlation. It also leads to metric spaces with distance functions.
But we want to consider a random variable X that represents a choice between cloth seats and leather seats. You might argue that leather > cloth, because leather is more expensive. But what about color? Is there any sense in which green > red? We are really interested in nominal scales. We may assign 0, 1, and 2 to “none”, V-6, and V-8 respectively, but these are just labels. That has a profound effect on the statistics we can use. And beyond statistics, we will have to develop notions of proximity or closeness that do not depend on distance functions. So it hasn’t been done before because the mathematics you need is not the popular stuff that’s in the textbooks.
Comments Off
Part II: Analytics to Action….. The Holy Grail
Can-you-have-too-much-information?
… in my last blog we talked about reporting, BI and data mining, and ? the information overload. So how can help business users with solutions for better decision making, as opposed to drowning them in more data and pretty charts? That is the Holy Grail and the purpose of all this data!
Lets start by defining analytics. So, what is analytics? Neil Raden of Hired Brains, a market research and management-consulting firm, has said that, “the proper term for interacting with information at the speed of business, analyzing and discovering and following through with the appropriate action, is ‘analytics’. I agree. In the information age, this must be done by specialized applications built on analytics based on the requirements of the actions/recommendations required by a business function. Dumping data on a users lap with the message – “Figure it out!” is NOT analytics, and it not very useful either. (I am reproducing a picture I really like as it conveys the message so very well! The Picture is from mathewingram.com/work)
So – how can we transform the user experience for analytics? As mentioned earlier, this can only be accomplished by focusing the analytics on a business problem with the mission to deliver actionable tasks. The challenge is selecting a business problem that the analytics truly delivers unique capabilities and intelligence that is relevant to that problem. This level of focus can be perceived as very limiting, and hence many choose not to go this route. Why limit the scope of the analytics to one specialization, when we can claim that we can do everything! To that I say – you are better off doing one thing very well, as opposed to many with mediocrity at best.
I am going to bring this back to Emcien, as this is a company that has focused analytics on a very specific business problem. The problem is one of product variety, product variants, and lots of attribution. In this age of product variety, that is a problem that is causing tremendous challenges to various business functions.
The analytics automatically detects what features customers are buying, where you are making money. This SKU or configuration intelligence is leveraged for:
SKU Intelligence Analytics Used to Drive Application Specific Recommendation
- Better forecasting at the mix level - The application uses the analytics intelligence to determine the exact product mix with very high accuracy based on true demand sensing.
- Improving the customer experience at the point of sale - The application uses the analytics intelligence and guides the buyer to a good configuration based on the few features they have called out. And by the way – customers love it when you can recommend a configuration based on the few features they ask for. They want you to stop asking more questions and recommend a good choice.
While the analytics may throw out volumes of data, the user can relax, as he does not have to crawl through volumes of date wondering what it is telling him. Converting analytics to actions and recommendations minimizes human interpretation and error on a day-to-day basis. For analytics to be functional in business applications, this is a mandatory requirement in today’s business environment.
So – when you are evaluating BI tools, Analytics, Data mining….. what ever they are calling it! Ask yourself, how am I adding value to the company? What am I giving my business users? Am I adding more work to their busy schedule by piling on data on their computers???? If the answer is YES, please don’t do it. They will thank you for it.
If the data has not been converted to recommendations the business can act on, you will not get value from your investment!
Recommendation engine for configurable products
The Amazon recommendation engine has received a lot of attention and imitation. It has been successful at increasing sales by pointing out that people who bought book x also bought book y. This simulates a helpful book store employee who has an extensive mental map of how books relate to each other. Recommendations have been most successful for books, movies, and music. Companies that sell complex configurable products could also benefit from a system of automated recommendations. Products like trucks, tractors, computers, lighting fixtures, valves, and industrial fans. These are products that the buyer can customize to his own preferences or needs. The buying process is complex, and sales agents make recommendations, just like the book store expert. But many of these products are far more complex than books.
How is a book different from a truck? Could a recommendation engine for books be used to recommend trucks? The purpose of this note is to consider the ways in which the two cases are similar and different, and to explore what a recommendation engine for configurable products might look like.
Books and trucks can both be described in terms of attributes (also called features or characteristics). Books can be described by their genre, author, language, and publication date for instance. Trucks can be described by their engine, transmission, wheelbase, gross vehicle weight, and many other attributes. Books, movies, and music can be classified in sufficient detail with ten or fewer attributes, while configurable products usually have a lot more. Heavy-duty trucks can have anywhere from 30 to 300 attributes. For books, the attributes are used to classify existing books. For configurable products, each attribute represents a choice for a customer who is ordering the product. Each attribute has several alternative options, so an order is really a list of option choices.
Amazon uses attributes to let customers search for a book, but the recommendation engine does not use them. The recommendation engine remembers each customer’s purchase history. For example, Joe has already bought “War and Peace” and “Crime and Punishment”. When he buys “The Sound and the Fury”, a connection is made between “The Sound and the Fury” and “War and Peace”, and another connection is made between “The Sound and the Fury” and “Crime and Punishment”. More accurately, the weights of these connections are increased. The connection between “The Sound and the Fury” and “War and Peace” depends on the number of customers that have bought both books. Commonality depends on common purchase by individual customers.
One big difference between books and trucks is that an individual will buy many books, but is unlikely to buy many trucks. So the basis for the connections between trucks can’t be the common purchases of individual buyers.
Another difference is that only books that have already been written are of interest. No customer is looking for a book that hasn’t been written yet, and you certainly can’t recommend one. But because of the very large number of choices, a buyer who is customizing a truck may arrive at a configuration that has never been built before. So our recommendation system must be able to accommodate trucks that don’t exist yet.
The helpful book store employee will jump from one book to another book that his customer might like. When we look at the helpful sales agent, he is doing something quite different. Based on the first, and usually most important, choices that the buyer makes, he will suggest ways of completing the order. The buyer may really only care about 10 of the 30 choices, and want guidance on what else to choose. The book store employee jumps from one complete item to another complete item. The sales agent is finding a path from a partial order to a complete order.
So we have an initial set of requirements for a recommendation engine for configurable products. It must help a buyer who has made some choices to make additional choices, so as to arrive at a complete order. From the seller’s point of view, we would want these recommendations to guide the buyer toward items that are advantageous. This might mean popular, in stock, and/or profitable. Its concept of the product (e.g. truck) must be broad enough to encompass configurations that have never been built before, and its concept of commonality must be based on something other than common purchases.
To show one way this might be done, consider a different way of making connections. When a customer buys a red truck with a V-8 engine, he is making a connection between red, a value of the Color attribute, and V-8, a value of the Engine attribute. So the customer’s vote is not recorded as a connection between two complete trucks, but as a connection between two attributes. More precisely, it is recorded as a connection between two values, or options, of the two attributes.
When a customer buys a truck with n attributes, we will record his “vote” n times, once for each of the options he has chosen. Then we will also record it nC2 more times for each pair of options. The symbol nC2 stands for the number of different pairs of attributes there are, and can be computed as n*(n-1)/2. For example 10C2=45. The relative popularity of the different pairs of options (like red and V-8) tells us something about how customers are buying the product, and gives us a different basis for making recommendations. (If you want pineapple on your pizza, you probably want Canadian bacon.)
If pairs of options are interesting, then so are triples of options. The same customer vote can be recorded for nC3 different sets of three options (10C3=120). Every option of every attribute has its own popularity (i.e. number of votes). Expressed as a fraction of total units sold it is known as a first-order take rate. Every pair of options also has a popularity, and as a fraction is defined as a second-order take rate. Similarly, each triple has a third-order take rate. Of course we can also define fourth, fifth, and higher-order take rates.
If customer purchases (votes) are recorded in this way, then we will have captured the buying patterns in the form of first, second, third, and higher-order take rates. A new customer who begins to select the options that are important to him will trigger these patterns, which will then serve as the basis for recommending a complete truck. So we see that two complete trucks may be related because they both contain instances of the same fifth-order take rate. This can be true even if neither truck has ever been built before.
We conclude that a recommendation engine for configurable products could be built by mapping customer purchases into a structure of take rates that record the relative popularity of options, pairs of options, triples of options, and so forth.
Comments Off
Part I: Reporting, Business Intelligence, Data Mining, Analytics: Actionable Tasks!
Software vendors use so many big words and confuse customers. Our customers have often asked us to clarify – so here I go. The definitions in this article are based on research of these terms, and the collective opinion of many of our customers and prospects. Over numerous conversations with our customers and the discussions of the terminology, the clarifications always go back to the origin of the terms and then move on to change in usage. Hence this article folows that flow. I would love your feedback as it is important to help buyers understand this.
Business Reporting
Business Reporting, as the term suggests presents the data from the database in an easy to read format. This originated when business users were frustrated that all the data was locked up in databases. There was a lot of data, but no one could get access to it without calling on IT folks. Hence Business Reporting was born.
Business Intelligence
This is a fancy name for business reporting. Business intelligence (BI) is a broad category of technologies that allows for gathering, storing, accessing and analyzing data to help business users make better decisions. In a 1958 article, IBM researcher Hans Peter Luhn used the term business intelligence. He defined intelligence as: “the ability to apprehend the interrelationships of presented facts in such a way as to guide action towards a desired goal.”
In 1989 Howard Dresner (later a Gartner Group analyst) proposed Business Intelligence as an umbrella term to describe “concepts and methods to improve business decision-making by using fact-based support systems.” Then in the late 1990s the usage became widespread (Remember the Bubble!). Then. everything with any data reporting was called Business Intelligence. So today, Business Intelligence is a glorified term for “Business Reporting”.
Data mining
Simply put, Data mining is hitting the data with all mathematical methods available to a mathematician! The data source can be almost anything – news papers articles, financial reports, sales data, medical data, … . This means that the data can have structure or can be un-structured. And the mathematical methods that can be applied can include neural networks, genetic algorithms, statistics on steroid and anything else they can think of.
One may ask – why are they doing this? What are they mining? Well, the simple answer is that they are mining the data looking for patterns; any patterns that can reveal relationships. So the methods used are varied and the kinds of data that are mined can come from a myriad of sources.
The results of data mining are lots of data! In fact – the result of Business Reporting and BI has been data overload. Now that’s the bad news. In a world of information overload, the last thing that we need is more data. We have less time today than we have ever had before. Business users do not need more data. They need quick conclusions on what the data is saying, converted into actionable tasks. Simply put – “Please tell me what to do”.
… More on the discussion of analytics to action in the next blog.
Comments Off