Designing and analyzing an experiment on personalized vs. generic onboarding emails for a B2B SaaS free trial.
Background
Final project for Applied Analytics. Acting as a data scientist at a B2B SaaS company selling project-management software to small and medium businesses, I evaluated whether a personalized onboarding email sequence — tailored to the features an account used in its first session — should replace the generic three-email sequence every new trial account receives. The hypothesis: relevant tips drive engagement, which drives conversion. The experiment ran as a 50/50 split for two weeks across account, subscription-event, daily-activity, and experiment-assignment tables.
My role
Defined the unit of randomization, the primary metric, secondary metrics (including ratio metrics), and guardrail metrics.
Ran a power analysis to determine the minimum sample size per group, then verified actual group sizes against it.
Wrote five Google BigQuery SQL queries using CTEs, window functions, joins, and aggregations to profile historical baselines and compute primary, secondary, and guardrail metrics.
Interpreted results and made a launch recommendation.
Results
The personalized sequence produced a statistically significant 10 percentage-point lift in trial-to-paid conversion.
Setting up power analysis and guardrail metrics before reading results is what makes an experiment readout trustworthy.