The S&P 500 has some aggressive days. Black Monday in 1987 was a 25% drop; way off the 1.2% standard deviation of the index. If returns were Normal/Gaussian, this type of drop should only happen every 10^98 days. The universe is younger. Other violent crashes exist with deviations just as unexpected for the Great Depression, the Dot-com bust, the Great Recession, the COVID crash, and the “Liberation Day” crash. There are also unexpected upswings, although they are rarer and usually happen after violent downswings.

The chart above shows the distribution of daily returns of the S&P 500 index from 1928 to 2025. If we fit a Gaussian to these daily returns, we can match the body but completely undershoot on the tails of the distribution. I plotted the count axis on a logarithmic scale; days that are “normally” not expected show up tens to hundreds of times more often. For example, the Gaussian expects a five-sigma day once every fourteen thousand years; in reality it is closer to one every few years. This extra amount of events in the tails (excess kurtosis) is a well known fact of asset returns, as well as the fact that down days are bigger than up days (skewness) [3].

Various models exists to compensate for the non-normal behavior of asset returns. Mandelbrot fit cotton prices to stable (Lévy) distributions, which have heavy tails (but also infinite variance) [1]. You can also fit returns to the Student-t distribution which has finite variance (closer to what the data shows), and controls the fatness of the tails through a single parameter. But neither of these distributions account for the assymetry of returns. The skew-t distribution combines separate left and right scales to add asymmetry (Hansen 1994, Fernández-Steel 1998) [7, 8]. The Johnson’s SU distributions instead passes the Gaussian through a sinh transformation (using a total of four parameters) to achieve arbitrary skewness and kurtosis [4]. Other reasonable distributions to use for fitting assets returns include the Normal Inverse, Variance Gamma, and sinh-arcsinh distribution [5, 6]. The chart below shows some of these distributions fit to the S&P 500 index returns.

To compare the different distributions more rigorously to each other, I performed a simple time-series cross-validation (5-fold): fit on the past and test on the held-out future. The Normal, as expected, and the Skew Normal performe worst. The top three (Johnson SU, Student-t, Laplace) are nearly tied, which suggests the standard families have converged on roughly the same ceiling. On a typical day, the observed move in the S&P is about 24% more probably under Johnson SU than under the Normal.

Can we generate other distributions that fit the returns better (with AI)?

All the mentioned distributions were hand-designed by people (experts). In the the next post. I ask if we can generate some new novel distributions that fit the returns of the S&P 500 better. I handed the problem to an LLM-driven evolutionary search, gave it the cross-validated log-likelihood as the thing to optimize, and told it not to reinvent any distribution that already has a name.

References

  1. Mandelbrot, B. (1963). “The Variation of Certain Speculative Prices.” Journal of Business, 36(4). doi:10.1086/294632
  2. Fama, E.F. (1965). “The Behavior of Stock-Market Prices.” Journal of Business, 38(1).
  3. Cont, R. (2001). “Empirical properties of asset returns: stylized facts and statistical issues.” Quantitative Finance, 1(2), 223-236. doi:10.1080/713665670
  4. Johnson, N.L. (1949). “Systems of Frequency Curves Generated by Methods of Translation.” Biometrika, 36(1/2), 149-176.
  5. Barndorff-Nielsen, O.E. (1997). “Normal Inverse Gaussian Distributions and Stochastic Volatility Modelling.” Scandinavian Journal of Statistics, 24(1), 1-13.
  6. Jones, M.C. & Pewsey, A. (2009). “Sinh-arcsinh distributions.” Biometrika, 96(4), 761-780. doi:10.1093/biomet/asp053
  7. Hansen, B.E. (1994). “Autoregressive Conditional Density Estimation.” International Economic Review, 35(3), 705-730. doi:10.2307/2527081
  8. Fernández, C. & Steel, M.F.J. (1998). “On Bayesian Modeling of Fat Tails and Skewness.” Journal of the American Statistical Association, 93(441), 359-371. doi:10.1080/01621459.1998.10474117