Day 1 Workshops

01:Experiment Design

The Scientific Way to Compare Your CRM Tactic

Matan Block-Temin & Jonathan Inbar

The Motivation for Comparing Tactics

  • //Optimizing tactics for each customer persona
  • //Different campaigns have different objectives
  • //Objectives may change over time
  • //Testing and validating

What is a Journey?

A journey is a sequence of campaigns put together to implement a CRM tactic

Measuring Campaign Uplift

Total campaign uplift = (Test results + Control results) – (Estimated organic results)

  • //Test results = test responses X test AVG KPI
  • //Control results = control responses X control AVG KPI
  • //Estimated organic results = (test group + control group) X control response rate X control AVG KPI

Comparing Tactics, Scientifically – Two analysis methods:

  1. Measuring a tactic – Super Control Group (SCG)
  • SCG – What is it?

Excluding percent of total population from all journey communication

  • SCG – Pros and cons
  • SCG – How to measure?

Apply same methodology as the campaign analysis:

Total journey uplift = (Journey results + Control results) – (Estimated organic results)

  1. Streams – comparing journeys
  • Stream – What is it?

Dividing total population into two different communication journeys in order to define the optimal tactic

  • Streams – Pros and cons
  • Streams – How to measure?

Compare between both stream results to define the wining tactic.

02:Content Repurposing

Always Have Something for Everyon

Hadas Avrech & Scott Shapiro

  • 1. Use assets from your website in your email marketing
  • 2.Cross-hit your active users! Distribute the same content across all relevant channels
  • 3.Refresh and republish your best performing historical pieces
  • 4.Use technology to reuse content
  • 5.Use your customers as content generators

03:Email Marketing

Personalize Email Like You Personalize Your Website

Ella Aharon & Shefa Weinstein

Examples of dynamic content

Real Time Content:
  • //Real time pricing / Inventory
  • //Jackpots
  • //Loyalty points
Advanced HTML:
  • //Forms / Surveys
  • //Carousel
  • //Pop ups

Personalizing emails with Dynamic Content

Welcome Emails:
  • //Scrape the homepage banner
  • //Show latest bestsellers
  • //Get them into your store
2nd Transaction Emails:
  • //Recommended products
  • //Complementary products
  • //Similar deals
Monthly loyalty / Account summary emails:
  • //Current status level
  • //Offers for their specific level and how to redeem them
  • //Leaderboards
  • //Summary of winnings
Email Headers and Footers:
  • //Loyalty points
  • //News feeds
  • //Social feeds
  • //Closest store

04:DIY Hack for Advanced Campaign Analysis

Measuring Campaign Effectiveness and Long-term Impact

Nitay Alon & Yohai Sabag

Business question: Do we make a change in the behavior of our customers?
Relevant distribution region: All
Scientific tool: Subsample
How to accomplish this in Excel:

  • 1. Add a random column using RAND() function next to your data column
    • a. Create random values for all the observations
    • b. Copy and paste the random values (as values) – this fixes the value of the random number
  • 2. Sort the data by the random variable column
  • 3. Select the (control size) top observations
    • a. Compute the average of the sample
    • b. Compare it to the average of the control group
    • c. Mark 1 if the sample average exceeds the control and mark 0 if else
  • 4. Repeat steps 1 – 3 as needed
  • 5. Compute the proportion of times that the test subsample average is bigger than the control average

Data based decision: Look at the winning odds of each group. If it’s close to 50-50, keep exploring if you’re in an A/B setting. Otherwise, consider stopping if you’re in a Test/Control setting.

Business question: Do we make a change for most of the target population?
Relevant distribution region: The central mass (the Filet Mignon)
Scientific tool: Trimmed mean
How to accomplish this in Excel:

  • 1. Use excel TRIMMEAN() function
    • a. First select the range of the data
    • b. Set the percent of trimming, say 0.2 to trim 20% of the data

Data based decision: If a trimmed mean t-test is significant, then there’s a statistically significant effect on the majority.

Business question: How much money is expected to be seen in the next iteration?
Relevant distribution region: The average and standard deviation of the logged data
Scientific tool: Log normal distribution
How to accomplish this in excel:

  • 1. Apply the LN() function over the data you wish to simulate from
  • 2. Compute the average and standard deviation of the post-transformation data
  • 3. Using RAND() function create 10,000 new values
    • a. Copy and paste as values
  • 4. Use NORM.INV() function on the values from step (2) and (3) to sample data from normal distribution
  • 5. Using EXP() function transform the data from step (4) into Log-Normal data

Data based decision: Take a sample of known size from the simulated data and compute the sum. Compare sums between alternatives (Automate this process and repeat 10K times to get a better feeling).

05:VIP Segmentation

Methodologies to Define Your Most Valuable Customers

Roni Cohen & Nadav Yekutiel

VIP Segmentation: Methodologies to better define your most valuable customers

Every business should know who their VIP customers are to ensure the satisfaction and loyalty of those customers.

There are several questions we should answer to accurately define the VIP segment to promise optimal results.

What attributes should you look at?
  • //Different metrics and timeframes to be considered
  • //Correlation matrix to find attributes with highest correlation to customer’s future value
What is the ideal segment size?
  • //Pareto analysis shows us which % of customers generates a mass of the income
Fixed or dynamic threshold?
  • //Fixed thresholds are easier to implement, track and measure
  • //Dynamic thresholds capture customer behavioral changes
How often do we recalculate?
  • //Daily update promises most updates VIP list
  • //Periodic update is more stable and easy to work with
  • //Optimal solution: Grace Period – Recalculate list on a daily basis but hold close customers in segment for a certain period of time
Should we split our VIP into sub segments?
  • //Allows more personal messages for each sub segment
  • //Understand customer potential by monitoring highest achieved tier

06:Stack Architecture

Build a Marketing Tech Stack to Meet Your Business Objectives

Leigh Noy & Andrew Samuels

A marketing technology stack is the layering and interconnection of different technologies that make marketing happen.

What comprises a stack?

    • //Data Layer
    • //Planning Layer
    • //Execution Layer

Building your stack:

Two main approaches:
      • //Single Vendor Suite
      • //Best of Breed
Things to bear in mind
      • //Be thoughtful
      • //Visualize your stack
      • //Focus on the big picture
      • //Think long-term
Determining factors
      • //Company’s stage of maturity
      • //Team and talents
      • //Your existing tools
      • //The scope of the funnel and customer experience you’re trying to create
      • //Your business model
More to consider
      • //Align with IT
      • //Create a use case wishlist
      • //Test vendors
      • //Understand R&R


“Architecture is about aging well, about precision and authenticity.” – Annabell Selldorf