Backlog prioritization sometimes leads to a choice between two strategies. Each one has its metrics or composite KPIs. Attraction or conversion rate optimization, attraction or retention are among them. In very moment limited budget and deadlines are constraints. Some problems require complex approach – correlations between revenue/transaction frequency and micro KPIs like web conversion are unstable and have low statistical power. It is especially evident when we speak about delayed conversion when the sales funnel and customer map [CJM] are not linear. For example a funnel gap between online and offline channels. Field of non-linear management. In this case machine learning [ML] techniques may give a value. One may use a Meta Model to predict 'monetary' metrics with use of web micro conversion, fraction of new visitors and other metrics. Predictive power may be used to make a choice in favor of a macro strategy. Secondly ML helps to reduce a dimension and simplify KPIs.
In the given example ensemble voting, binning of target and dimension reduction are involved. Two groups of metrics are considered. First (I) corresponds to web conversion [bounce rate, micro conversion, depth]. Second (II) corresponds to attraction of new visitors from organic channel [visits, viewers, views]. The target function is a number of commercial offers per day converted into the sale. Features and target are normalized to the [0,1] interval. Relatively small data with daily sampling is used: four years of recording. Web metrics are parsed from analytical platform through Python API. Sales data is received from local CRM/SQL database. Correlation matrix below shows that there is no significant correlation between converted Offers and Web metrics.
However we may notice that Depth/CR and Visits/Viewers/Views are mutually correlated groups. We may apply dimension reduction techniques. The target is binned into the 5 equal intervals/classes. It is a price for a small amount of data. Ensemble of nonlinear voting estimators [Python sklearn library] is used: KNeighborsClassifier, Decision Tree, Ada Boost, Gradient Boosting, Support Vector Classifier, Naive Bayes and Multi-layer Perceptron with three hidden layers. Hyper tuning is applied to KNeighborsClassifier only. Let's look at the prediction power of groups and metrics:
'Conversion features' I group:
Average accuracy of cross validation: 81%
Standard deviation of accuracy: 0.4%
'Organic traffic features'– II group:
Average accuracy of cross validation: 81%
Standard deviation of accuracy: 0.6%
The result is acceptable in both cases. Algorithm is able to learn prediction of sales using both groups. We may improve it only by tuning of models or by rough binning – less intervals. Now we may use mutual correlations of Depth/CR and Visits/Viewers/Views. We may transform initial traffic set to Views metric only (1D) – great simplification.
'Views' only– II group:
Accuracy of out-of-sample 50/50 test=82.0%
Average accuracy of cross validation: 81%
Standard deviation of accuracy: 0.4%
Less features – less risk of over-fitting, simpler interpretation. Fundamentally we may experiment with leaving only micro web conversion from the second group.
'Micro conversion' only– I group:
Accuracy of out-of-sample 50/50 test=82.0%
Average accuracy of cross validation: 82%
Standard deviation of accuracy: 0.5%
Comparable results. However Visits/Viewers/Views give higher input in scattering of composite KPI according to Principle Component Analysis – see the figure below. It is clear that traffic has higher volatility (seasonality influence).
So Product Owner may take into account Prediction Power, Risks and Budget for any of two strategies. In this very case Conversion Rate Optimization (CRO) is cheaper than SEO or Ads and has less influence of seasonality. In this case CRO is preferred until results are updated. The proposed method may be generalized in decision making when non-linearity and the implicit connections between behavior of user and monetization are inevitable. However more simple instruments are preferable if the funnel is linear, short and simple. Combination of both approaches with intuition may give a confidence when making “expensive” decisions.
Sergey Kamenshchikov
Code
In the given example ensemble voting, binning of target and dimension reduction are involved. Two groups of metrics are considered. First (I) corresponds to web conversion [bounce rate, micro conversion, depth]. Second (II) corresponds to attraction of new visitors from organic channel [visits, viewers, views]. The target function is a number of commercial offers per day converted into the sale. Features and target are normalized to the [0,1] interval. Relatively small data with daily sampling is used: four years of recording. Web metrics are parsed from analytical platform through Python API. Sales data is received from local CRM/SQL database. Correlation matrix below shows that there is no significant correlation between converted Offers and Web metrics.
However we may notice that Depth/CR and Visits/Viewers/Views are mutually correlated groups. We may apply dimension reduction techniques. The target is binned into the 5 equal intervals/classes. It is a price for a small amount of data. Ensemble of nonlinear voting estimators [Python sklearn library] is used: KNeighborsClassifier, Decision Tree, Ada Boost, Gradient Boosting, Support Vector Classifier, Naive Bayes and Multi-layer Perceptron with three hidden layers. Hyper tuning is applied to KNeighborsClassifier only. Let's look at the prediction power of groups and metrics:
'Conversion features' I group:
Average accuracy of cross validation: 81%
Standard deviation of accuracy: 0.4%
'Organic traffic features'– II group:
Average accuracy of cross validation: 81%
Standard deviation of accuracy: 0.6%
The result is acceptable in both cases. Algorithm is able to learn prediction of sales using both groups. We may improve it only by tuning of models or by rough binning – less intervals. Now we may use mutual correlations of Depth/CR and Visits/Viewers/Views. We may transform initial traffic set to Views metric only (1D) – great simplification.
'Views' only– II group:
Accuracy of out-of-sample 50/50 test=82.0%
Average accuracy of cross validation: 81%
Standard deviation of accuracy: 0.4%
Less features – less risk of over-fitting, simpler interpretation. Fundamentally we may experiment with leaving only micro web conversion from the second group.
'Micro conversion' only– I group:
Accuracy of out-of-sample 50/50 test=82.0%
Average accuracy of cross validation: 82%
Standard deviation of accuracy: 0.5%
Comparable results. However Visits/Viewers/Views give higher input in scattering of composite KPI according to Principle Component Analysis – see the figure below. It is clear that traffic has higher volatility (seasonality influence).
So Product Owner may take into account Prediction Power, Risks and Budget for any of two strategies. In this very case Conversion Rate Optimization (CRO) is cheaper than SEO or Ads and has less influence of seasonality. In this case CRO is preferred until results are updated. The proposed method may be generalized in decision making when non-linearity and the implicit connections between behavior of user and monetization are inevitable. However more simple instruments are preferable if the funnel is linear, short and simple. Combination of both approaches with intuition may give a confidence when making “expensive” decisions.
Sergey Kamenshchikov
Code