In [47]:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from scipy.stats import chi2_contingency, norm, zscore
from scipy.stats import skew, kurtosis
from scipy.stats import ttest_ind, mannwhitneyu, shapiro
import statsmodels.api as sm
1. Load and Preprocess the Data¶
In [21]:
control_df = pd.read_csv('control_group.csv')
test_df = pd.read_csv('test_group.csv')
In [22]:
control_df.head()
Out[22]:
| Campaign Name;Date;Spend [USD];# of Impressions;Reach;# of Website Clicks;# of Searches;# of View Content;# of Add to Cart;# of Purchase | |
|---|---|
| 0 | Control Campaign;1.08.2019;2280;82702;56930;70... |
| 1 | Control Campaign;2.08.2019;1757;121040;102513;... |
| 2 | Control Campaign;3.08.2019;2343;131711;110862;... |
| 3 | Control Campaign;4.08.2019;1940;72878;61235;30... |
| 4 | Control Campaign;5.08.2019;1835;;;;;;; |
Apparently the data is in a single comma-separated column
In [61]:
control_df = pd.read_csv('control_group.csv', sep =";")
test_df = pd.read_csv('test_group.csv', sep =";")
In [62]:
control_df.head()
Out[62]:
| Campaign Name | Date | Spend [USD] | # of Impressions | Reach | # of Website Clicks | # of Searches | # of View Content | # of Add to Cart | # of Purchase | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Control Campaign | 1.08.2019 | 2280 | 82702.0 | 56930.0 | 7016.0 | 2290.0 | 2159.0 | 1819.0 | 618.0 |
| 1 | Control Campaign | 2.08.2019 | 1757 | 121040.0 | 102513.0 | 8110.0 | 2033.0 | 1841.0 | 1219.0 | 511.0 |
| 2 | Control Campaign | 3.08.2019 | 2343 | 131711.0 | 110862.0 | 6508.0 | 1737.0 | 1549.0 | 1134.0 | 372.0 |
| 3 | Control Campaign | 4.08.2019 | 1940 | 72878.0 | 61235.0 | 3065.0 | 1042.0 | 982.0 | 1183.0 | 340.0 |
| 4 | Control Campaign | 5.08.2019 | 1835 | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
In [63]:
control_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 30 entries, 0 to 29 Data columns (total 10 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Campaign Name 30 non-null object 1 Date 30 non-null object 2 Spend [USD] 30 non-null int64 3 # of Impressions 29 non-null float64 4 Reach 29 non-null float64 5 # of Website Clicks 29 non-null float64 6 # of Searches 29 non-null float64 7 # of View Content 29 non-null float64 8 # of Add to Cart 29 non-null float64 9 # of Purchase 29 non-null float64 dtypes: float64(7), int64(1), object(2) memory usage: 2.5+ KB
In [64]:
test_df.head()
Out[64]:
| Campaign Name | Date | Spend [USD] | # of Impressions | Reach | # of Website Clicks | # of Searches | # of View Content | # of Add to Cart | # of Purchase | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Test Campaign | 1.08.2019 | 3008 | 39550 | 35820 | 3038 | 1946 | 1069 | 894 | 255 |
| 1 | Test Campaign | 2.08.2019 | 2542 | 100719 | 91236 | 4657 | 2359 | 1548 | 879 | 677 |
| 2 | Test Campaign | 3.08.2019 | 2365 | 70263 | 45198 | 7885 | 2572 | 2367 | 1268 | 578 |
| 3 | Test Campaign | 4.08.2019 | 2710 | 78451 | 25937 | 4216 | 2216 | 1437 | 566 | 340 |
| 4 | Test Campaign | 5.08.2019 | 2297 | 114295 | 95138 | 5863 | 2106 | 858 | 956 | 768 |
In [65]:
test_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 30 entries, 0 to 29 Data columns (total 10 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Campaign Name 30 non-null object 1 Date 30 non-null object 2 Spend [USD] 30 non-null int64 3 # of Impressions 30 non-null int64 4 Reach 30 non-null int64 5 # of Website Clicks 30 non-null int64 6 # of Searches 30 non-null int64 7 # of View Content 30 non-null int64 8 # of Add to Cart 30 non-null int64 9 # of Purchase 30 non-null int64 dtypes: int64(8), object(2) memory usage: 2.5+ KB
In [66]:
# Drop not needed columns
control_df.drop(columns=['Campaign Name'], inplace=True)
test_df.drop(columns=['Campaign Name'], inplace=True)
In [67]:
column_names = ['Date', 'Spent', 'Impressions',
'Reach', 'Clicks', 'Searches',
'Views', 'Add to Cart', 'Purchases']
control_df.columns = column_names
test_df.columns = column_names
In [68]:
control_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 30 entries, 0 to 29 Data columns (total 9 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Date 30 non-null object 1 Spent 30 non-null int64 2 Impressions 29 non-null float64 3 Reach 29 non-null float64 4 Clicks 29 non-null float64 5 Searches 29 non-null float64 6 Views 29 non-null float64 7 Add to Cart 29 non-null float64 8 Purchases 29 non-null float64 dtypes: float64(7), int64(1), object(1) memory usage: 2.2+ KB
In [69]:
# Impute the missing values in columns 2-8
control_df.iloc[:, 2:] = control_df.iloc[:, 2:].fillna(control_df.iloc[:, 2:].median())
control_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 30 entries, 0 to 29 Data columns (total 9 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Date 30 non-null object 1 Spent 30 non-null int64 2 Impressions 30 non-null float64 3 Reach 30 non-null float64 4 Clicks 30 non-null float64 5 Searches 30 non-null float64 6 Views 30 non-null float64 7 Add to Cart 30 non-null float64 8 Purchases 30 non-null float64 dtypes: float64(7), int64(1), object(1) memory usage: 2.2+ KB
In [ ]:
2. Metrics¶
Cost Per Purchase (CPP)¶
Definition:
The average amount of money spent on a campaign to generate one purchase.
Formula:
It is calculated as:
$ \begin{align} \text{CPP} &= \frac{\text{Total Campaign Spend}}{\text{Total Number of Purchases}} \end{align} $
Why CPP is Important¶
Efficiency Metric:¶
- CPP measures how efficiently your campaign budget is being used to drive purchases.
- A lower CPP indicates better campaign performance.
ROI Calculation:¶
- CPP is often used to calculate the Return on Investment (ROI) of a campaign.
- For example, if the average revenue per purchase is higher than the CPP, the campaign is profitable.
Campaign Optimization:¶
- By tracking CPP, marketers can identify which campaigns or channels are most cost-effective and allocate budgets accordingly.
In [88]:
# Calculate CPA for each day
control_df['CPP'] = control_df['Spent'].astype(float) / control_df['Purchases']
test_df['CPP'] = test_df['Spent'].astype(float) / test_df['Purchases']
In [89]:
# Visualize distributions
plt.figure(figsize=(10, 6))
sns.histplot(test_df['CPP'], kde=True, label='Test', color='orange', bins=12)
sns.histplot(control_df['CPP'], kde=True, label='Control', color='blue', bins=12, alpha=0.5)
plt.title('Distribution of CPP')
plt.xlabel('CPP (USD per Purchase)')
plt.ylabel('Frequency')
plt.legend()
plt.show()
In [90]:
# Check Distribution of CPP using
# Shapiro-Wilk Test for Normality
_, p_control = shapiro(control_df['CPP'])
_, p_test = shapiro(test_df['CPP'])
print(f"Shapiro-Wilk p-value (Control): {p_control}")
print(f"Shapiro-Wilk p-value (Test): {p_test}")
Shapiro-Wilk p-value (Control): 0.011892481707036495 Shapiro-Wilk p-value (Test): 0.006934247445315123
Since the Shapiro-Wilk p-values for both the Control and Test groups are less than 0.05, we can conclude that the CPA (Cost Per Acquisition) data for both groups is not normally distributed. This means we should use a non-parametric test (e.g., Mann-Whitney U Test) instead of a parametric test like the T-Test.
In [91]:
# Perform the test
stat, p_value = mannwhitneyu(control_df['CPP'].dropna(), test_df['CPP'].dropna(), alternative='two-sided')
print(f"Mann-Whitney U Statistic: {stat}")
print(f"P-value: {p_value}")
# Interpret the results
if p_value < 0.05:
print("Reject the null hypothesis: There is a statistically significant difference in CPA between the two groups.")
else:
print("Fail to reject the null hypothesis: There is no statistically significant difference in CPA between the two groups.")
# Step 2: Visualize CPA Comparison
plt.figure(figsize=(8, 6))
sns.boxplot(x='Group', y='CPP', data=pd.concat([
control_df.assign(Group='Control'),
test_df.assign(Group='Test')
]))
plt.title('Comparison of CPP: Control vs. Test')
plt.ylabel('CPA (USD per Purchase)')
plt.show()
Mann-Whitney U Statistic: 369.0 P-value: 0.2339889162810581 Fail to reject the null hypothesis: There is no statistically significant difference in CPA between the two groups.
Explanation of Results¶
Mann-Whitney U Test:
- The Mann-Whitney U Test is a non-parametric test used to compare two independent groups when the data is not normally distributed.
- Null Hypothesis (H₀): The distributions of the two groups are equal.
- Alternative Hypothesis (H₁): The distributions of the two groups are not equal.
Interpretation:
- If the p-value is less than 0.05, we reject the null hypothesis and conclude that there is a statistically significant difference in CPA between the Control and Test groups.
- If the p-value is greater than 0.05, we fail to reject the null hypothesis and conclude that there is no statistically significant difference in CPA between the two groups.
The results of the Mann-Whitney U Test indicate that the p-value is 0.234, which is greater than 0.05. This means we fail to reject the null hypothesis, and we conclude that there is no statistically significant difference in CPA (Cost Per Acquisition) between the Control and Test groups.
Interpretation of Results¶
No Significant Difference:
- The CPA for the Control and Test groups is not significantly different.
- This suggests that both campaigns performed similarly in terms of cost efficiency for generating purchases.
Implications:
- If the goal was to determine which campaign is more cost-effective, the results indicate that neither campaign outperformed the other significantly.
- You may need to explore other metrics (e.g., conversion rate, ROI) or run the experiment for a longer duration to detect smaller differences.
Explore Other Metrics¶
Analyze other key performance indicators (KPIs) such as:
Conversion Rate: $ \begin{align} \text{Conversion Rate} = \frac{\text{Number of Purchases}}{\text{Number of Website Clicks}} \end{align} $
Click-Through Rate (CTR): $ \begin{align} \text{CTR} = \frac{\text{Number of Clicks}}{\text{Number of Impressions}} \end{align} $
Return on Investment (ROI): $ \begin{align} \text{ROI} = \frac{\text{Revenue - Campaign Spend}}{\text{Campaign Spend}} \end{align} $
Customer Lifetime Value (CLV): $ \begin{align} \text{CLV} = \text{Average Purchase Value} \times \text{Purchase Frequency} \times \text{Customer Lifespan} \end{align} $
There is currentry data only for CR and CTR
CR and CTR¶
In [75]:
# Calculate CR and CTR for each day
control_df['CR'] = control_df['Purchases']/ control_df['Clicks']
test_df['CR'] = test_df['Purchases']/ test_df['Clicks']
control_df['CTR'] = control_df['Clicks'] / control_df['Impressions']
test_df['CTR'] = test_df['Clicks'] / test_df['Impressions']
CR and CTR: Hypothesis Formulation¶
For Conversion Rate (CR):¶
Null Hypothesis (H₀):
- The population mean Conversion Rate (CR) of the Control group is equal to the population mean Conversion Rate of the Test group.
Alternative Hypothesis (H₁):
- The population mean Conversion Rate (CR) of the Control group is not equal to the population mean Conversion Rate of the Test group.
For Click-Through Rate (CTR):¶
Null Hypothesis (H₀):
- The population mean Click-Through Rate (CTR) of the Control group is equal to the population mean Click-Through Rate of the Test group.
Alternative Hypothesis (H₁):
- The population mean Click-Through Rate (CTR) of the Control group is not equal to the population mean Click-Through Rate of the Test group.
In [76]:
# Shapiro-Wilk Test for Normality
_, p_cr_control = shapiro(control_df['CR'])
_, p_cr_test = shapiro(test_df['CR'])
_, p_ctr_control = shapiro(control_df['CTR'])
_, p_ctr_test = shapiro(test_df['CTR'])
print(f"Shapiro-Wilk p-value (CR Control): {p_cr_control}")
print(f"Shapiro-Wilk p-value (CR Test): {p_cr_test}")
print(f"Shapiro-Wilk p-value (CTR Control): {p_ctr_control}")
print(f"Shapiro-Wilk p-value (CTR Test): {p_ctr_test}")
Shapiro-Wilk p-value (CR Control): 0.005544630810618401 Shapiro-Wilk p-value (CR Test): 0.037268370389938354 Shapiro-Wilk p-value (CTR Control): 0.25710153579711914 Shapiro-Wilk p-value (CTR Test): 0.00040253173210658133
In [107]:
combined_df=pd.concat([
control_df.assign(Group='Control Campaign'),
test_df.assign(Group='Test Campaign')
])
# Reset the index of combined_df to avoid duplicate labels
combined_df = combined_df.reset_index(drop=True)
Interpretation of Shapiro-Wilk p-values¶
CR Control:
- p-value = 0.0055
- Since the p-value is less than 0.05, we reject the null hypothesis of the Shapiro-Wilk test.
- Conclusion: The CR data for the Control group is not normally distributed.
CR Test:
- p-value = 0.0373
- Since the p-value is less than 0.05, we reject the null hypothesis of the Shapiro-Wilk test.
- Conclusion: The CR data for the Test group is not normally distributed.
CTR Control:
- p-value = 0.2571
- Since the p-value is greater than 0.05, we fail to reject the null hypothesis of the Shapiro-Wilk test.
- Conclusion: The CTR data for the Control group is normally distributed.
CTR Test:
- p-value = 0.0004
- Since the p-value is less than 0.05, we reject the null hypothesis of the Shapiro-Wilk test.
- Conclusion: The CTR data for the Test group is not normally distributed.
Implications for Statistical Testing¶
Conversion Rate (CR):
- Both the Control and Test groups have non-normally distributed CR data.
- Recommended Test: Use a non-parametric test like the Mann-Whitney U Test to compare the CR between the two groups.
Click-Through Rate (CTR):
- The Control group has normally distributed CTR data, but the Test group has non-normally distributed CTR data.
- Recommended Test: Use a non-parametric test like the Mann-Whitney U Test to compare the CTR between the two groups, as the assumptions for parametric tests (e.g., T-Test) are not fully met.
In [95]:
# Perform Mann-Whitney U Test for CR
stat_cr, p_value_cr = mannwhitneyu(control_df['CR'], test_df['CR'], alternative='two-sided')
print(f"Mann-Whitney U Statistic (CR): {stat_cr}")
print(f"P-value (CR): {p_value_cr}")
# Perform Mann-Whitney U Test for CTR
stat_ctr, p_value_ctr = mannwhitneyu(control_df['CTR'], test_df['CTR'], alternative='two-sided')
print(f"Mann-Whitney U Statistic (CTR): {stat_ctr}")
print(f"P-value (CTR): {p_value_ctr}")
# Create subplots
fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 4))
fig.suptitle('Violin Plots of CR and CTR',fontweight='bold')
# Violin plot for CR
sns.violinplot(x='Group', y='CR', data=pd.concat([
control_df.assign(Group='Control'),
test_df.assign(Group='Test')
]), ax=ax1)
ax1.set_title('Conversion Rate (CR): Control vs. Test')
ax1.set_ylabel('Conversion Rate (%)')
# Violin plot for CTR
sns.violinplot(x='Group', y='CTR', data=pd.concat([
control_df.assign(Group='Control'),
test_df.assign(Group='Test')
]), ax=ax2)
ax2.set_title('Click-Through Rate (CTR): Control vs. Test')
ax2.set_ylabel('Click-Through Rate (%)')
# Adjust layout
plt.tight_layout()
plt.show()
Mann-Whitney U Statistic (CR): 523.0 P-value (CR): 0.2837780479456242 Mann-Whitney U Statistic (CTR): 197.0 P-value (CTR): 0.00018916193602108462
In [119]:
plt.figure(figsize=(14, 6))
# Subplot 1: Distribution of CR
plt.subplot(1, 2, 1)
sns.histplot(data=combined_df, x='CR', hue='Group', kde=True, palette=['orange','blue'], bins=18, alpha=0.6)
plt.title('Distribution of Conversion Rate (CR)', fontsize=14)
plt.xlabel('Conversion Rate (CR)', fontsize=12)
plt.ylabel('Frequency', fontsize=12)
plt.legend(title='Campaign', fontsize=10, labels=['Test','Control'])
# Subplot 2: Distribution of CTR
plt.subplot(1, 2, 2)
sns.histplot(data=combined_df, x='CTR', hue='Group', kde=True, palette=['orange','blue'], bins=18, alpha=0.6)
plt.title('Distribution of Click-Through Rate (CTR)', fontsize=14)
plt.xlabel('Click-Through Rate (CTR)', fontsize=12)
plt.ylabel('Frequency', fontsize=12)
plt.legend(title='Campaign', fontsize=10,labels=['Test','Control'])
# Adjust layout
plt.tight_layout()
plt.show()
| Conversion Rate (CR) | Click-Through Rate (CTR) |
|---|---|
| Results: | Results: |
| - Mann-Whitney U Statistic (CR): 523.0 | - Mann-Whitney U Statistic (CTR): 197.0 |
| - P-value (CR): 0.2838 | - P-value (CTR): 0.00019 |
| Interpretation: | Interpretation: |
| - The p-value (0.2838) is greater than 0.05. | - The p-value (0.00019) is less than 0.05. |
| - This means we fail to reject the null hypothesis. | - This means we reject the null hypothesis. |
| Conclusion: | Conclusion: |
| - There is no statistically significant difference in the Conversion Rate (CR) between the Control and Test groups. | - There is a statistically significant difference in the Click-Through Rate (CTR) between the Control and Test groups. |
Key Insights¶
Conversion Rate (CR):
- The lack of a statistically significant difference suggests that the two campaigns performed similarly in terms of converting users who clicked on the ad into purchasers.
- This could mean that the changes made in the Test campaign did not have a meaningful impact on the conversion rate.
Click-Through Rate (CTR):
- The statistically significant difference in CTR suggests that the Test campaign performed differently from the Control campaign in terms of attracting clicks.
- To determine which campaign performed better, you can compare the median CTR or mean CTR of the two groups:
- If the Test campaign has a higher median/mean CTR, it performed better.
- If the Control campaign has a higher median/mean CTR, it performed better.
Next Steps¶
For CTR:
- Investigate why the Test campaign had a significantly different CTR. For example:
- Was the ad copy or design more engaging in the Test campaign?
- Were there differences in targeting or ad placement?
- If the Test campaign had a higher CTR, consider scaling it up. If it had a lower CTR, analyze what went wrong and refine the strategy.
- Investigate why the Test campaign had a significantly different CTR. For example:
For CR:
- Since there was no significant difference in CR, focus on other aspects of the campaign (e.g., ad spend efficiency, ROI) to determine which campaign is more effective overall.
Further Analysis:
- Perform additional statistical tests or exploratory data analysis (EDA) to understand other factors that might have influenced the results.
- Consider running the experiment for a longer duration to see if the results hold over time.
In [ ]: