Most churn models fail for a boring reason: they're built to be accurate, not useful. A model that's 95% accurate but misses most of the actual churners is worthless to a retention team, because the customers it misses are exactly the ones someone should have called last week.
I ran into this early while building a churn prediction model for an e-commerce dataset, 5,630 customers, 9 predictor features, and a target variable marking whether each customer had left or stayed. The class imbalance was the first real problem. Most customers don't churn in any given period, so a lazy model can hit high accuracy just by predicting "stays" for everyone and being right most of the time. That's a useless model wearing a good scorecard.
Fixing the imbalance
I used SMOTE to rebalance the training data, generating synthetic examples of the minority class instead of just duplicating existing churn cases. Paired with a Random Forest classifier, this pushed the model toward actually learning what separates a churner from a loyal customer, rather than defaulting to the safe guess.
The results held up: 85.88% training accuracy, 85.35% on the test set, an AUC of 0.87. But the number that mattered more was recall on the churn class, 78%. That means the model catches roughly 8 out of every 10 customers who are actually about to leave. Precision on that class was lower, around 62% average precision, which is the tradeoff you accept when the cost of missing a churner is higher than the cost of a false alarm. I also adjusted the classification threshold down to 0.47, deliberately trading a bit of precision for more recall, because in retention, a wasted retention email to a loyal customer costs almost nothing, but a churner who slips through unnoticed costs a customer.
What actually predicted churn
The feature importance results were more useful than the accuracy numbers, honestly, because they tell a retention team where to focus.
Tenure mattered most. Newer customers churned more, which says the danger window isn't six months in, it's the first few weeks, before habits and loyalty have formed. Complaints were the second strongest signal, customers who filed a complaint were meaningfully more likely to leave, which is less a surprising insight and more a direct challenge to whoever owns customer service response times. Days since last order came in close behind, a customer who's gone quiet is already halfway out the door, often before they'd tell you so themselves. Cashback amount and satisfaction score both mattered too, just less dramatically, financial incentives helped retention, but they weren't the main lever.
The part that's easy to miss
A churn model isn't really the deliverable. The deliverable is a ranked list of at-risk customers and a reason why each one is on it, so a retention team can act before the customer decides to leave rather than after. Accuracy numbers are how you convince a stakeholder the model is sound. Recall and feature importance are what actually change what the business does on a Monday morning.