Meaningful Outliers

TABLE OF CONTENTS1. Introduction: Gladwell’s Focus on the Exceptional

2. Reasons to Focus on Outliers

3. Frequent UX Outlier

4. Pick Your Battles - Meaningful Outlier Flowchart

5. Conclusion

Inspiration: Gladwell’s Focus & & Hidden Exceptionality

In Malcolm Gladwell's "Outliers: The Story of Success," the author dives deep into the stories of extraordinary individuals who have achieved remarkable success. Gladwell's central thesis revolves around the idea that success is not merely a result of innate talent but a complex interplay of various factors, including cultural background, timing, and unique opportunities. By examining the outliers surrounding generational talent or moments of prestige, Gladwell illuminates how exceptional cases can provide profound insights into the nature of success and failure. 

[ADD OUTLIERS BOOK IMAGE]

When Malcolm Gladwell's book “Outliers” was published, I was an undergraduate studying statistics. Five years later, I had an interesting discussion with a client at eBay about the differences between Customer Insights and UX/Usability. Her title as “CX” framed her approach in a more variable strategy: During the analysis phase of a project, she guided me to focus on the unique and exceptional viewpoints of customers, rather than the typical or the most common frustrations and delights. She wanted me to elucidate “factors and circumstances I won’t be aware of; find me the odd but inspiring elements that make a difference in their usage of eBay’s emails and the updates we provide to sellers.” This approach isn't a typical dividing line between Usability, UX methods, and the broader work of “Insights.” Still, for this client, it was clear that this was how they differentiated their technical craft at times. Searching for the diamond in the rough. Insights for her were not simply the “why”, but edge-case strategies by power-users that could be leveraged wider and further. 

It has stuck with me since, and often I believe one of the most fundamental challenges as a researcher in the non-academic world is to decide: when does unique, awkward, or extreme feedback matter? When do the furious customer rants tied to a 0/5 rating follow-up matter, even if the average is 4.4 / 5? 

This is not so different from how many foundational scholars in statistics, psychometrics, data science, and economics have spoken regarding outliers and the attention they deserve.

[SCHOLARS IMAGE]

So why has contemporary User Experience Research, Consumer Insights, Machine Learning, and other data oriented fields generally turned their back on outliers? Why is this conversation buried among the public-square of thought leaders who often shy away from deeper analysis topics? 

Generally, “outliers” are considered “bad” or in a more neutral mindset, useless due to irrelevancy to “norms”.

• Outliers can have a big impact on your statistical analyses and skew the results if they are inaccurate.

• These extreme values can impact your statistical power as well, making it hard to detect a true effect.

• We clean our data of them; they’re outside our purview. They lie separate from expected deviation. 

The implication is that measures of central tendency are more accurate without the extremes. According to my rough sampling of 47 researchers from 30 different companies, only 8 researchers were aware of an organizational policy on how to treat outliers. Most remove the data from the crux of their analysis. 

[INDUSTRY TENDENCIES CHART]

Since they are extreme enough, it is acceptable not to include outliers in analyses. 

There are often good reasons to exclude outliers, yet according to my sampling and colloquial discourse on professional forums and social media, rarely due to deep conversations about detection, context, and inclusion.

This article is not about proper outlier detection and the boundaries of what is considered an outlier. To read up on those rationales and calculations, I highly recommend one of the below guides or research papers:

1. Number Analytics: The Ultimate Guide to Outlier Detection

2. How to Find Outliers | 4 Ways with Examples & Explanation

3. "Finding Outliers In Excel: A Detailed Guide" by Acuity Training

4. "Spotting the Odd Ones Out: Identifying Outliers in SPSS" by Statistics Solutions

But incredibly quickly, three methods of outlier detection from the above links: 

The Interquartile Range (IQR) method is a popular statistical technique for detecting outliers in a dataset. This method relies on calculating the spread of the middle 50% of the data, which is the difference between the first quartile (Q1) and the third quartile (Q3).  The IQR method is a common approach where outliers are identified as data points falling below (Q1-1.5 IQR) or above (Q3 + 1.5 IQR), where Q1 and Q3 are the first and third quartiles. This method is effective for data with a roughly normal distribution. There is an alternative Boxplot Adjustment method (D-k-NN), which identifies outliers based on distances rather than direct data values and is particularly useful in datasets with mixed distributions (SpringerOpen)​. You can also Adjust IQR Limits instead of the typical 1.5 multiplier; values can be adjusted depending on the data distribution, enhancing the sensitivity to outliers​ (Statology)​.  [IQR IMAGE]

Variance-Based Thresholds: Another advanced method involves calculating thresholds based on the variance of data points relative to a central value. This method helps in datasets where variance around the mean is more indicative of outliers than fixed IQR limits​ (SpringerOpen)​.

Z-Scores are also a method of outlier exclusion. Z-scores measure how many standard deviations a data point is from the mean. Typically, a Z-score beyond ±3 indicates an outlier. This method is particularly useful for data following a Gaussian distribution (Chandola, Banerjee, & Kumar, 2009). [VARIENCE IMAGE]

And then there is Grubbs' Test. Grubbs' test is designed to detect a single outlier in a univariate data set. It compares the deviation of the suspected outlier to the sample's standard deviation, identifying outliers with high precision (Grubbs, 1969).

Tukey himself introduced the Interquartile Range (IQR) method to detect outliers, but emphasized that the detection ≠ deletion. In Exploratory Data Analysis (1977), Tukey invites analysts to “listen to the data” rather than impose rigid thresholds.  I have felt that most contemporary outlier detection and deviation-based exile of data is emotionless, purely mathematical, and all the guides almost always deflect away from conversations regarding context. 

Yet data points more than 2 standard deviations from the mean can be incredibly valuable. Aside from recording errors, some outliers deserve attention: they often align closely with what stakeholders seeking to elevate a product’s experience truly care about.

One compelling example Gladwell presents in his book is that of Canadian hockey players. Originally a trend cited by psychologist Roger Barnesly (1985), he notes that an overwhelming number of elite hockey players are born in the first few months of the year. This is due to the January 1st cutoff date for age-class hockey leagues in Canada. Players born in January, February, or March are more physically mature than their younger peers, giving them a significant advantage. This advantage leads to better coaching, more practice opportunities, and a higher likelihood of success (Gladwell, 2008, pp. 23-24).  By diving deeper into the exceptional, in this case the NHL, which represents the top 1% of the 1% of all hockey players, we can learn more about all the nuances of success. And yes, when contextualized to the entire market of hockey players, NHL players are outliers. 

[HOCKEY PLAYER CHART]

One may argue that this isn’t an outlier - heightened instances of Q1 births among hockey players are a skew within the NHL. What often makes an outlier in one context, as Andrew Gelman points out, can be an expected cluster in a better fit model.   While Gladwell's focus is on human achievement, his insights extend beyond personal success stories to broader applications, such as User Experience design. In UX, understanding outliers can be instrumental in enhancing product performance. If we attune to exceptional cases, outliers can elevate overall product performance for everyone when their potential moment as an outlier occurs.

“Outliers are not a problem to be removed, but a treasure to be examined.”

– John W. Tukey (The father of exploratory data analysis)

Reasons to examine outliers:

• They may represent accessibility gaps, especially for users with disabilities or non-standard devices.

• Outliers can often be indicative of technical issues - help create tickets!

• Outliers, if repetitive, can be a crucial way to understand customer CHURN. 

• They might predict churn or drive innovation (edge cases often point to unmet needs).

• They may reveal onboarding friction that surveys or averages obscure.

• Some outliers may be your most loyal users, frustrated dropouts, or unexpected evangelists.
  

Outliers May Indicate Churn

Research shows that extreme negative behaviors, like very low satisfaction ratings or high complaint volumes, are strong predictors of future churn. For instance, Khan et al. (2015) developed a behavioral churn model that assigns each customer a “Churn Score” based on early warning signals, accurately predicting churn with nearly 90% accuracy. Similarly, banking industry studies have found that complaints and dissatisfaction closely correlate with customers defecting and switching banks, reinforcing the idea that large deviations from the “norm” in user feedback often signal high-risk events.

Outliers may reveal Accessibility Gaps

Abnormal data can frequently be due to a participant who struggles with product settings, screen-readers, or perceptual difficulties related to their unique situation. According to the World Health Organization, around 15–20% of the global population (1+ billion people) experience significant disabilities, which translates directly to a sizable market (Guardian). For example, in the U.S. alone, people with disabilities represent over 18% of the population, wielding around $175 billion in discretionary spending annually (archive.ada.gov). Globally, businesses are tapping into a market estimated at $6–13 trillion in spending power from individuals with disabilities and their networks (W3C.org). Essentially, Outliers in user behavior tied to accessibility aren't niche - they represent a major customer segment with substantial economic impact. (Return on Disability Group, 2021). Outliers, in these cases, don’t just represent ethical imperatives; they represent major business opportunities.

Edge Cases can raise the standard for all 

In UX, edge cases represent those rare, extreme user scenarios that fall outside the typical user behavior. While it might seem counterintuitive to prioritize these rare cases, addressing them can lead to significant improvements in product performance and user satisfaction.

Gladwell's analysis of outliers, such as the success of Bill Gates and The Beatles, emphasizes the extraordinary opportunities and circumstances that propelled their success (Gladwell, 2008, pp. 50-70). Similarly, in UX, focusing on edge cases can reveal critical insights and opportunities to enhance the overall user experience. Gladwell's discussion of the "Matthew Effect", where "the rich get richer and the poor get poorer", highlights how initial advantages can lead to cumulative success (Gladwell, 2008, pp. 30-31). In UX, addressing outliers can create a similar virtuous cycle, where improving the experience for edge cases elevates the entire product ecosystem.Additionally, if the same customer across 6 months is responding to every survey in the extreme negative, their behavioral metrics are diving off a cliff, and they keep writing to Customer Support - they're far more likely to churn. This is why I highly recommend keeping a close eye on outliers, NOT just tossing them into the trash. They can often be a leading indicator of technical or lingering concerns that most people merely tolerate. 

Despite the prior warning not to sacrifice the larger experience for a small edge case, there is a false dichotomy that catering to certain exceptions means sacrificing the experience of the majority. Or that serving edge-cases will even introduce that much more effort and features. In my experience, I have found that teams are often apprehensive about making “low-hanging fruit” tweaks to UI purely out of principle, not engineering cost. Collecting outliers over time is a great way to eventually demonstrate how the exceptional or unique can become a mounting, lingering threat to diminish or an opportunity to broaden.

“A data point can be an outlier because the model is wrong, not because the data are wrong.”

— Andrew Gelman, StatModeling Blog

Which metrics in User Experience would most often have outliers? 

• Time-on-Task metrics such as time-per-page and total-task time are not confined to tight scales.

• Segmentation analysis across cohorts that include extreme purchasers - include or exclude in research?

• In within-subjects concept testing, a respondent rates all 1/7 on concepts A and B, but 7/7 on concept C.

• Qualitative sessions with an incredibly upset or confused customer - outlier-to-public-review pipeline. 

• If 3 out of 12 participants encounter a task barrier, is that an outlier? Or a minor usability issue?  

Expandable tables of UX Metrics prone to outliers… [WIP, refine…shrink?]

Quantitative UX Metrics

Metric	How Outliers Might Appear	Why It Matters
Task Completion Time	A user takes 10x longer than average	Could signal confusion, assistive tech use, or an accessibility issue
Click Count / Tap Count	Excessive or too few interactions	May reflect exploratory behavior, misclicks, or interface traps
System Usability Scale (SUS)	One user rates everything as 100 or 0	Could be a signal of extreme satisfaction or dissatisfaction
Error Rate	A user makes far more mistakes than others	May highlight a broken flow or misinterpretation of UI
Net Promoter Score (NPS)	A lone “0” promoter among a cohort segment of all “9s”	That detractor might reveal pain points others overlook
Time on Page / Screen	Very high or very low dwell time	Could indicate distraction, skimming, or deep engagement
Scroll Depth	Scrolled much farther or less than expected	May reflect skimming behavior or failure to find CTA

Behavioral / Interaction Data

Data Type	Example of Outlier	Potential Insight
Heatmaps	One user clicks in a totally unexpected place	May indicate a misleading affordance or poor visual hierarchy
Search Logs	Rare queries not searched by others	Could reveal underserved needs or misalignment in IA
Navigation Paths	A unique, inefficient path through a task	Might reflect an alternative mental model or unfamiliarity
Form Analytics	One user submits the form in 3 seconds	Could suggest automation, error, or power user behavior
Conversion Funnels	Drop-off at unusual stage	May identify an overlooked barrier or mobile-specific bug

Qualitative or Mixed-Methods Data

Data Type	Outlier Behavior	Why It’s Valuable
User Interviews	One user describes a unique workaround	May highlight unmet needs or new feature ideas
Diary Studies	One participant logs far more interactions	Could signal super-user status or pain point saturation
Surveys (open-text)	Extremely emotional or detailed response	Can enrich personas or uncover corner cases
A/B Tests	A segment behaves very differently than expected	May signal a key audience or technical segmentation error

It may be a bit easier to embrace the exceptional on the positive side: What makes a power-user so adept at flowing through the interface? While a statistical outlier, maybe that completed task in 11 seconds when it took everyone else an average of 48 seconds wasn’t because they cheated or had a lucky guess. 

The reality is the dial of business innovations often move due to outliers. Video games make massive profit from the 5% of players that contribute to 70%+ of revenue. Customer Support queries and data comes from the 5% of customers frustrated or confused enough to contact support, and accessibility standards, safety regulations, and HR policies stem from circumstances that are generally not the norm. And sadly, there are frequent biases where product owners, developers, or designers might spotlight positive outliers while claiming that negative outliers are “cherry-picking” among the ‘infrequent’ frustrated segments. So let’s dive into the consideration and ultimately the decision to include or exclude outliers in UX. 

Pick Your Battles - When to Spotlight the Outlier

Not all outliers are created equal. In UX research and design, it is crucial to discern which outliers warrant attention. Prioritizing outliers should be based on their impact on the user experience and the product's goals. For example, if an outlier represents a user group with significant influence or potential growth, addressing their needs can lead to substantial benefits. Conversely, if an outlier's needs are so unique that addressing them would detract from the core user experience, it might be prudent to deprioritize them.

Below is a context, metrics, and design-element dependant flow-chart to help you decide when to elevate abnormal data and call-up it in your reporting. This comes from 12+ years of my experience running paid research for 40+ companies. 

[ADD FLOWCHART IMAGE!]

Meaningful Outliers Flowchart: Download the PDF File here! 

“The greatest value of a picture is when it forces us to notice what we never expected to see.” — John Tukey, referring to the role of data visualization in exposing unexpected (often outlier) patterns

Conclusion

Drawing inspiration from Malcolm Gladwell's exploration of exceptional cases in "Outliers," this article underscores the importance of understanding and addressing outliers in UX design. 

Integrating outliers into UX research is not just about recognizing statistical anomalies but understanding the broader implications of unique user experiences. These insights can drive innovation, inclusivity, and overall product improvement, making outliers an essential component of comprehensive UX research.

Just as Gladwell's outliers reveal deeper truths about success, UX outliers can provide critical insights to drive product innovation and user satisfaction.

1. Understanding User Behavior: Outliers can offer a deeper understanding of diverse user behaviors and preferences. While typical user data provides a general overview, outliers highlight unique interactions that can reveal unforeseen user needs or issues. This perspective is crucial for designing inclusive and adaptable products.
   

2. Identifying Unique Pain Points and Innovations: Outliers often represent edge cases that standard usability testing might overlook. These cases can expose critical pain points or innovative ways users interact with a product. According to the Interaction Design Foundation, defining clear objectives for UX surveys, including the exploration of outliers, can lead to discovering unique pain points and unanticipated opportunities for innovation​ (The Interaction Design Foundation)​.
   

3. Improving Product Accessibility and Inclusivity: By focusing on outliers, UX researchers can enhance product accessibility and inclusivity. Products designed with edge cases in mind tend to be more robust and user-friendly for a broader audience. User Interviews discuss how different research methods, like focus groups, can reveal outliers' perceptions and preferences, contributing to a more inclusive design process​ (User Interviews)​.
   

4. Enhancing User Retention and Satisfaction: Addressing the needs of outliers can significantly improve overall user satisfaction and retention. Products that accommodate a wide range of user needs tend to have higher user loyalty. Understanding and resolving the unique challenges faced by outliers can lead to improvements that benefit all users, not just the majority.

References:

• Gladwell, M. (2008). Outliers: The Story of Success. Little, Brown and Company.

• “The Ultimate Guide to Outlier Detection”https://www.numberanalytics.com/blog/the-ultimate-guide-to-outlier-detection 

• “How to Identify and Manage Outliers in Data Analysis: Tips, Techniques, and Tools” https://www.dataskillacademy.com/post-detail/how-to-identify-and-manage-outliers-in-data-analysis-tips-techniques-and-tools/

• “Spotting the Odd Ones Out: Identifying Outliers in SPSS”  https://www.statisticssolutions.com/identifying-outliers-in-spss/ 

• “Finding Outliers In Excel: A Detailed Guide” https://www.acuitytraining.co.uk/news-tips/finding-outliers-in-excel/ 

• Rehman, A., Belhaouari, S.B. Unsupervised outlier detection in multidimensional data. J Big Data 8, 80 (2021). https://doi.org/10.1186/s40537-021-00469-z

• Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys (CSUR), 41(3), 1-58.

• https://ijirt.org/publishedpaper/IJIRT174604_PAPER.pdf 

• Barnsley, R.H, Thompson, A.H., & Barnsley, P.B. (1985). Hockey success and birthdate: The Relative age effect. Journal of the Canadian Association of Health, Physical Education and Recreation, Vol. 51.

• Grubbs, F. E. (1969). Procedures for detecting outlying observations in samples. Technometrics, 11(1), 1–21. https://www.jstor.org/stable/1266761

• Expanding Your Market: Customers with Disabilities Mean Business

Outliers & Churn:extreme negative feedback correlates with churn. E.g., “Extreme dissatisfaction signals long-term attrition risk” (e.g., Gupta & Harris, 2010)Power Users & Revenue:SaaS and gaming research confirm that the top 5% of users often generate 70–80% of revenue (e.g., Hamari et al., 2017).