While the name “recommendation engine” might seems to imply it all, there is so much potential in this technology approach that it’s worth digging deeper. As the importance of engaging customers and other stakeholders grows- as the reality of remote engagement itself continues to grow- recommendation engines and their many different uses are rapidly becoming a fundamental capability for organizations.
Possibly the most common recommendation engine experience we all have is when streaming video services suggest new content for us to watch. Not surprisingly those companies use recommendation engine technology to find and suggest this new content to us, based on all our previously watched content. While streaming content recommendations, and perhaps even product recommendations from our favorite e-commerce websites are easy to imagine utilizing this technology, we should ask how else recommendation engines can benefit organizations.
There are any number of additional use cases, but many question arise in considering them: Is the ROI sufficient to justify the time and money to implement them? And how would one get started, given that the best approaches and technologies are relatively new, and true expertise hard to find? Many organizations believe that recommendation engines must be so complex, timely, and costly, and this can often cause them to shy away from exploring the significant potential value.
In this article we demystify recommendation engines, covering what they are, what organizations can gain from them, and even provide an example approach to start building one. The article itself is geared towards organizations that are newer to graph databases and to graph data science, or that may not have the internal technical resources to explore this on their own.
What is a Recommendation Engine?
Generally, recommendation engines are a class of algorithms and models that are used to suggest ‘things’ to users. These algorithms use patterns in user behavior to find and serve the most likely item(s) of interest to the user. The earliest and most widely used form of recommendation engine is the “people also bought” algorithm, built using a market basket analysis approach.
Nowadays, the complexity of recommendation engines can vary widely, as organizations take different approaches to constructing and deploying them. For example, some recommendation engines are built to learn more about a user over time, so as to continuously improve recommendations. Others might use “collaborative filtering”, which uses the abstracted behavior patterns and preferences of the many to find the most relevant products to serve to particular users. Each organization must choose the approach that works best for them, given constraints around the availability of data, industry, or user type as well as desired functionality.
What is the real value of Recommendation Engines?
Ultimately, the value of any recommendation engine is in being able to tap into and traverse the breadth of valuable data across an organization- whether it be structured and/or unstructured data- in order to take people, content, products (as a few examples, but really any other entities in the data) and to connect them in ways that are only possible programmatically, and only possible at scale with graph databases (e.g. Neo4j) and graph data science technology. By connecting potentially valuable “entities” in your data to other potentially impacted/interested entities, there is an opportunity to create net-new value through the those new connections.
Obvious examples as mentioned above include e-commerce recommendations- where recommending the right product to the right person at the right time in order to create enough value in the buyer’s mind to purchase and/or add on to a purchase can be the difference in them staying or changing to a new vendor; or with streaming services, mining the content in ways that surface new recommended content to keeps viewers from jumping to the many other streaming options out there. It starts to become very clear that the effectiveness of the recommendation technology to produce only the most relevant recommendations is core to producing value.
Beyond the most common use cases mentioned above, recommendation technology can help in so many other ways- really only constrained by the needs and imagination of each organization. Some examples of other valuable use cases include:
Looking across medical research, patient profiles, clinical trials and more to help doctors recommend optimal treatment plans
Using customer purchase history data to recommend even quantity of products to purchase,
Leveraging customer user profiles and internal store data to recommend new more convenient store locations,
Using social media data to recommend staffing and capacity needs for upcoming events,
Using event and employee data to recommend relevant events/programs to employees,
Using internal structured and unstructured data to surface (recommend) an organization’s top X employees to focus on for retention efforts,
Using internal and external (e.g. legislation, news) structured and unstructured data to recommend the top X unknowns or possible risk areas for a larger company to consider,
Finding and recommending de-duplication opportunities across compliance requirements.
The list really could be endless based on the unique internal and external needs and/or opportunity areas of an organization. Many are calling data “the new gold” for businesses. Those that can imagine beyond the traditional recommendation use cases will set themselves up to compete much more effectively by mining their data for value in this unique way.
How to build a Recommendation Engine
To begin building recommendation engines, you need interconnected data in a graph database such as Neo4j. Though not required, storing your data in this way enables you to traverse your data much more efficiently particularly at scale, enabling lightning fast insights by avoiding the expensive joins that would be required in a relational database. For more information on the benefits of a graph database, read this post on what is a graph database or this post on graph databases and knowledge graphs.
When building a recommendation engine for the first time, one approach is to follow a simple three step question approach that breaks down the process, using the collaborative filtering approach mentioned above. Once you have your interconnected data loaded into a graph database, such as customer purchase history data, it is important to start this approach out with with a single user.
Starting with a single user enables you to filter the data so you can easily see the results without extra noise. Formulate a question that can be answered with one relationship, or one hop, from your starting user. The first question could be “what products has this user purchased in the last 6 months?”. With that question and a single user, query the database and pull all answers to that question, as shown in the image below.
Next, build on the query by asking a question that can be answered in two relationships, or two hops, from your starting user. The question might be, “what users have also purchased the products that our starting user purchased?”. The results give you a number of users who are connected to your starting user through a similar product that they purchased.
Lastly, query the database to ask a question that is 3 relationships, or hops, away from your starting user. In our example, the question is “what other products have our related users bought, that our starting user has not bought?”.
With that final result, in 3 simple questions, you just created a list of potential products as shown below, that you might recommend to your starting user, in essence creating a simple recommendation engine. Clearly, there is much more to ensuring that those products will fit the starting user, but this is a way to show the backbone of the process where the relevance is based on the fact that people who buy one thing often buy other similar products.
This 3 step question process can be applied to any interconnected data, driving recommendations and insights from your data in minutes. After going through this kind of process, you can continue to build on it, adding more context through questions and through more data, increasing both your question complexity and the possible insights you can gain.
What barriers are there to developing a good Recommendation Engine?
Outlined above is a simple example of leveraging customer behavior data in order to understand purchasing patterns for the purpose of recommending products to customers. It is evident though that the most popular products will always be the most recommended, if we use only this collaborative filtering approach. While it can be effective, some of the downsides of this approach are:
- It is not well suited to account for new product launches (e.g. no one or very few people have purchased them),
- If you are trying to give your customers a wider variety of recommended products that may be a fit, but may not connected through similar customers, but instead perhaps through other means.
Graphs can also solve this problem by matching products to customers along any other dimension (e.g. weather data for a customer’s particular location could drive very tailored product recommendations), even using unstructured data (with Natural Language Processing or NLP) such as reviews, user guides, descriptions and more to find that relevance. Using movies as another example, recommendations could be made based on the cast, genre, production company or even filming location and a whole host of other dimensions, leveraging a user’s rating data to find the connectedness across those dimensions.
Graphable delivers insightful graph database (e.g. Neo4j) / machine learning (ml) / natural language processing (nlp) projects as well as graph and Domo analytics with measurable impact. We are known for operating ethically, communicating well, and delivering on-time. With hundreds of successful projects across most industries, we thrive in the most challenging data integration and data science contexts, driving analytics success.