I started researching/writing this post about a month ago. It took a strange arc. It began with me wondering about “up vol” vs “down vol” or how vol acts differently in rallies vs selloffs. Then it ran straight into a topic I read about this summer (the title is a clue). It will awaken both seasoned and novice option traders with both inspiration and discomfort. Which is to say, I’m really happy I wrote it, but also feel like there’s a lot more to this than what I can cover today (and sparring with LLMs about it is definitely affirming this feeling).
Before we start unfolding, one more meta thought.
While working on this I benefited from a pedagogical technique that I didn’t plan, but believe you can engineer. I mentioned it in one of my “learning science” articles, myelination:
The “hypercorrection effect” is the phenomenon where you remember corrections to wrong answers better than when you give a correct answer off-the-bat when the question is difficult. Generating a prior makes you own a prediction. When it breaks, surprise becomes the teacher.
I’ll walk you through the same steps I took, which reinforced, even with all my years, just how nebulous the concept of volatility can be and how it touches trading and investing in practice.
Before pulling any data, I wanted to test my market intuition. I start with some guesses about how the S&P 500 behaves off the top of my head:
- S&P 500 volatility hovers around 16% annually. I heuristically think of this as some blend of “volatility when the market is up” and “volatility when the market is down”.
- 2/3 of the months are positive
- Risk reversals suggest upside vol is about 10% below some “base” vol
- Downside vol is about 30% above this “base” vol
If the full market vol is 16%, and I have asymmetric volatility in up/down months, what’s the “base” volatility?
Let x = base volatility
Up month vol = 0.9x (10% lower)
Down month vol = 1.3x (30% higher)
Full variance = 2/3 × (0.9x)² + 1/3 × (1.3x)² = 16²
Full variance = 2/3 × 0.81x² + 1/3 × 1.69x² = 256
Full variance = 0.540x² + 0.563x² = 1.103x² = 256
Therefore: x = √(256/1.103) = 15.23%
So my base vol would be about 15.23%, giving me:
- Up month vol: 0.9 × 15.23% = 13.71%
- Down month vol: 1.3 × 15.23% = 19.80%
For monthly returns, I figured the standard deviation would be roughly 16%/√12 = 4.62% per month.
As for expected returns, I guessed the market delivers about 80 basis points per month (~10% annually).
If 2/3 of the months are up and 1/3 are down, and the average is +0.8%, what are the typical up and down returns?
Let’s call up months +U% and down months -D%:
2/3 × U - 1/3 × D = 0.8
2U - D = 2.4
If monthly volatility is about 4.6%, what would typical up and down returns be?
Assuming monthly returns are normally distributed with a mean of 0.80% and standard deviation 4.62%, the probability of a positive return is 57% (leaving 43% negative).
Probability of market down
Z-score = (0 - .8)/4.62 = -.173
P(Z ≤ -0.173) ~ 0.431
[Wait a minute...for N(.8, 4.62) P≤0 ~43% but I assumed the probability of a negative month is only 1/3. This is a clue some of my estimates are wrong OR the distribution is not normal. We're going to bring the real data in soon and the appendix will expand the discussion. I won't bury the lede -- my estimate of p≤0 is correct! But I get some other estimates wrong and, well, the returns aren't normally distributed. We're going to make sense of all of this.]
Again, my unconditioned estimate of monthly return is .80%.
Now I want to estimate the monthly return given that the market is up. Let’s try translating to math language:
I want the return at the midpoint of the positive portion of the distribution.
That’s at the 43.1% + 56.9%/2 = 71.5% cumulative probability point.
P(Z ≤ X) ~ .715
Solve for X using Excel:
NORM.INV(0.715,0.8,4.62) = 3.42
For a N(0.8%, 4.62%) distribution, the 71.5th percentile gives us +3.42%.
If 2/3 of the months are up, and the expected return in an up month is+3.42% but the overall mean is 0.8%, the down months must average -4.44% to balance the equation above.
Validate: 2/3(3.42%) – 1/3(4.44%) = 2.28% – 1.48% = 0.80%.
Time to test these intuitions against actual data. I pulled daily S&P 500 returns from January 2016 through October 2025—nearly a decade covering COVID, Fed policy shifts, and retail investing mania.
Market batting average:
- Up months: 81 out of 118 (68.6%) ✅ Pretty close to my 2/3 guess!
Returns:
- Average up month: +3.43% ✅ I estimated 3.42% —boom!
- Average down month: -3.93% ❌ I estimated 4.44%.
- Overall monthly average: 1.13% ❌Higher than my 80bps estimate
Volatility:
- Full sample annual vol: 18.23% ❌Higher than my 16% guess.
- Mean vol in up months: 12.47% ✅ I estimated 13.71%— so-so.
- Mean vol in down months: 20.43% ✅ I estimated 19.80%—not bad!
All of these were calculated from daily returns, whether it was the full sample or if they were then grouped into months.
This is where things got interesting. My intuitions were pretty decent about up and down vol. I decided to check if the weighted average of monthly volatilities would recover the full sample volatility:
Weighted variance = 0.686 × (12.47%)² + 0.314 × (20.43%)²
= 0.686 × 0.01556 + 0.314 × 0.04175
= 0.02377
Weighted vol = √0.02377 = 15.42%
Wait. The full sample vol using daily returns is 18.23%, but the weighted average of monthly vols is only 15.42%.
That’s an 18% gap in volatility, which is large, if we consider typical vol risk premiums of ~10% just to give a sense of proportion.
In variance terms:
- Full sample: 332.33 basis points (ie .1823
²)
- Weighted average: 237.70 basis points (ie .1542
²)
Missing: 94.74 basis points
Where did ~30% of the variance go?
Let’s take a detour before we go into the arithmetic.
I’ve been reading Geoffrey West’s book “Scale” and this anomaly reminded me of the coastline paradox—the closer you look at a coastline, the longer it becomes. These excerpts tell the story of Lewis Richardson’s discovery in the early 1950s when he discovered that various maps indicated different lengths for coastlines:
Richardson found that when he carried out this standard iterative procedure using calipers on detailed maps, this simply wasn’t the case. In fact, he discovered that the finer the resolution, and therefore the greater the expected accuracy, the longer the border got, rather than converging to some specific value!
This was a profound observation because it violated basic assumptions about measurement, which we hold to be objective to some underlying reality. But Richardson’s discovery is intuitive once you think about it:
Unlike your living room, most borders and coastlines are not straight lines. Rather, they are squiggly meandering lines… If you lay a straight ruler of length 100 miles between two points on a coastline or border… then you will obviously miss all of the many meanderings and wiggles in between. Unlike lengths of living rooms, the lengths of borders and coastlines continually get longer rather than converging to some fixed number, violating the basic laws of measurement that had implicitly been presumed for several thousand years.
When you use a finer resolution (shorter ruler), you capture more of these wiggles, leading to a longer measured length.
This gets better. (Also, you should read this friggin’ book!)
The increase follows a pattern:
When he plotted the length of various borders and coastlines versus the resolution used to make the measurements on a logarithmic scale, it revealed a straight line indicative of the power law scaling.
The practical implication:
The take-home message is clear. In general, it is meaningless to quote the value of a measured length without stating the scale of the resolution used to make it.
Risk exhibits the same property. It depends on the resolution at which you measure it and forms the link to the question: where did those 95 bps of variance go?
While I’ve pointed this out before in these articles:
Volatility Depends On The Resolution
Risk Depends On The Resolution
…I didn’t drill down to the mathematical decomposition for why this is true. We will do that in a moment but in words:
When we calculate monthly volatilities and average them, we’re essentially “sampling” risk at a monthly resolution. But when we calculate volatility from all daily returns, we’re capturing additional variation that exists between months—variation that gets smoothed away in monthly aggregation.
Let’s illustrate with a tangible example. Imagine three classes taking the same test:
Class A (Morning class): Scores: 75, 80, 85 (mean = 80)
Class B (Afternoon class): Scores: 65, 70, 75 (mean = 70)
Class C (Evening class): Scores: 85, 90, 95 (mean = 90)
If we calculate the variance two ways:
Method 1: Pool all scores together
All scores: 75, 80, 85, 65, 70, 75, 85, 90, 95
- Mean = 80
- Variance = 83.3 (average of squared deviations)
Method 2: Average the within-class variances
- Class A variance = 16.7 (sum of squared deviations is 50, then divide by 3 samples)
- Class B variance = 16.7
- Class C variance = 16.7
Average variance = 16.7
The gap: 83.3 – 16.7 = 66.7
This missing 66.7 is the variance that comes from classes having different average scores (80, 70, 90).
The Law of Total Variance captures this precisely:
Total Variance = E[Var(Score|Class)] + Var(E[Score|Class])
83.3 = 16.7 + 66.7
Circling back to our example:
- The “Full Sample Volatility” (18.23%) or 332 bps is the Total Variance
- The “Weighted Average Volatility” (15.42%) or 238 represents only the first term: the Within-Group Variance
- The “Missing Gap” (95 basis points) is the second term: the Variance of the Means
Intuitively:
The market doesn’t just wiggle around a static zero line every month. Some months the whole market shifts up (+3.43%), and some months it shifts down (-3.93%). If you only look at volatility within the month, you ignore the risk of the market shifting levels entirely. Simply averaging monthly volatilities ignores this “Between-Month” risk.
There’s another subtle effect at play: Jensen’s Inequality. This mathematical principle states that for a convex function (like squaring for variance), the average of the function is not equal to the function of the average.
💡See Jensen’s Inequality As An Intuition Tool
In this context:
- Variance is proportional to volatility squared (convex function)
- The average of squared volatilities ≠ the square of averaged volatilities
First of all, in our data, each month has a different number of trading days (19-23). When we calculated monthly volatilities, we essentially gave equal weight to each month regardless of how many observations it contained.
But even in months with equal days, averaging volatility is dangerous
The March 2020 Example:
- March 2020: 22 trading days, 91.53% annualized volatility
- October 2017: 22 trading days, 5.01% annualized volatility
In our “average of monthly vols” calculation, these months contribute equally. But their contribution to the full sample variance is vastly different:
March 2020’s contribution = (91.53%)² × 22/2473 = 74.54 basis points of variance
October 2017’s contribution = (5.01%)² × 22/2473 = 0.22 basis points of variance
March 2020 contributes 334 times more to total variance despite being weighted equally in the monthly average!
For Option Traders
The difference between realized vol at different sampling frequencies directly impacts estimates of volatility. The shorter the sampling period the higher the volatility on average. When computing realized vols based on tick data, a method sometimes known as “integrated vol”, there is a minimum sampling frequency that, if you dip below, causes the vol to explode because it is simply capturing “bid-ask bounce”. The minimum threshold can vary by asset, so by using a volatility signature plot (a plot of vol vs sampling frequency) you can see where this threshold lives.
Conversely, it’s reasonable to expect that estimating long-term vols by sqrt(time) scaling from shorter dated vols may overshoot. See the appendix on the discussion of power law scaling in the context of the coastline paradox, keeping in mind that term structure scaling takes a power law shape, but the exponent needn’t be 1/2.
[Even if you conclude that upward sloping term structures are unjustified or at least reflecting a risk premium, do you understand why it’s weakly, if at all, arbitrageable? I think this would make a good interview question for an option trader to demonstrate how they think about risk-taking and capital (and business generally). I’ll withhold my answer because I like the question too much.]
For Portfolio Construction
When combining assets with different measurement frequencies (daily equities, monthly real estate, quarterly private equity), be aware that risk measured at different resolutions isn’t directly comparable. This is not a perfectly overlapping reformulation of the “volatility laundering” criticism of slow-to-mark assets.
This little jaunt from napkin math to data analysis shows how risk, like coastlines, is fractal. The closer you look, the more you find.
When reconstructing measures of risk from lower resolution assumptions that were quite strong, I found gaps which point to my oft-repeated:
Risk depends on the resolution at which you measure it.
The resolution at which you measure risk affects three things:
- Aggregation effects: Higher frequency captures more granular variation
- Weighting effects: Different time periods get different implicit weights which can be decomposed by the Law of Total Variance
- Jensen effects: The non-linearity of variance creates gaps when averaging
The market’s full 18.23% volatility tells one story. The 15.42% average of monthly volatilities tells another.
Technical Note: This analysis used realized volatility calculated as √(Σ(X²)/n) × √252, treating daily returns as having zero mean. This approach, common in high-frequency finance, effectively assumes the drift is negligible compared to volatility at daily frequencies—a reasonable assumption given that daily expected returns are typically 0.04% while daily standard deviation is over 1%.
When measuring at daily resolution across all data:
Var(returns) = E[X²] - E[X]²
When measuring at monthly resolution, then averaging:
E[Var(returns|month)] = E[E[X²|month] - E[X|month]²]
The difference between these is:
Var(returns) - E[Var(returns|month)] = Var(E[X|month])
which implies The Law of Total Variance.
The law states that the total variance of a dataset can be broken into two parts:
- The average of the variances within each group (Within-Group Variance)
- The variance of the means of the groups (Between-Group Variance)
Var(X) = E[Var(X|Group)] + Var(E[X|Group])
The algebra used to solve for the “base volatility” x is known as a mixture model:
Total Variance = (Prob_up × Var_up) + (Prob_down × Var_down)
It’s only valid if the means of the up/down months are close enough that the “Variance of Means” component is negligible for a rough guess.
Actual Monthly Statistics (S&P 500, Jan 2016 – Oct 2025)
- Mean: 1.13%
- Std Dev: 4.39%
- Median: 1.80% (notably higher than mean)
- Up months: 68.6% (81 out of 118)
If monthly returns were truly N(1.13%, 4.39%), we’d expect only 60.1% up months.
But we actually get 68.6%—an 8.5 percentage point gap. This gap, as well as the difference between mean and median demonstrate negative skew. The left tail is longer, meaning occasional large down moves.
It’s classic equity pattern: stairs up, elevator down. The bad months are worse than the good months are good, but the good months happen more often than a normal distribution predicts, even net of a positive mean return. Both the higher mean and the skewness.
If I ran through my same logic above using actual data:
Probability of market down
Z-score = (0 - 1.13)/4.39 = -.257
P(Z ≤ -0.257) ~ 0.399
I want to estimate the monthly return given that the market is up. Let’s try translating to math language:
I want the return at the midpoint of the positive portion of the distribution.
That’s at the 39.9% + 60.1%/2 = 70% cumulative probability point.
P(Z ≤ X) ~ .70
Solve for X using Excel:
NORM.INV(0.70,1.13,4.39) = 3.43
Market return given that it’s up: +3.43% (coincidentally matching reality)
We go back to this identity with the true mean and volatility to solve for the down move:
.601 × U - .399 × D = 1.13
.601*(3.43) -.399D = 1.13
D = -2.33
If the distribution was normal N(1.13%, 4.39%), we expect the down moves to be -2.33% on average with 40% down months, but the actual data shows the down moves occurred only 31.4% of the time, but were -3.93%!
The generic power-law relationship:
y = A · xⁿ
Where n is the exponent that determines how drastically y responds to changes in x.
You can see the sensitivity by comparing different exponents:
- If n = 1/2:
To double y, you must increase x by a factor of 4 (because 4^(1/2) = 2).
- If n = 1/4:
To double y, you must increase x by a factor of 16 (because 16^(1/4) = 2).
West writes:
“To appreciate what these numbers mean in English, imagine increasing the resolution of the measurement by a factor of two; then, for instance, the measured length of the west coast of Britain would increase by about 25 percent and that of Norway by over 50 percent.”
In the British case, doubling the resolution increases the coastline by 1.25x, therefore, the exponent, n, must be ~ 1/3
2ⁿ = 1.25
n log 2 = log 1.25
n = log 1.25 / log 2 = .32
