Here is a histogram of the daily changes of the S&P 500 from 1950 to 2009. The x-axis is the daily log difference* and the y-axis is the number of times that difference occurred.
It turns out that a Laplace distribution is a pretty good model of the stock market. The Laplace distribution is parametrized by its median u and its average absolute difference from median b. I computed these two parameters for the S&P 500 data and plugged them into a Laplace distribution in Mathematica, then used that to generate 59 years worth of random simulated S&P 500 prices. Here is the resulting histogram.
This is not a best fit: this is simply the same u and b computed on the data and then plugged into the distribution. Here are the two histograms on top of each other.
At the aggregate level the stock market is well behaved: it’s randomness is remarkably predictable. It’s amazing that this social construct — created by people for people, and itself often personified — behaves so much like a physical process, more so than any other man-made entity I can think of.
The graphs also hint at the futility of attaching reasons to price movements every single day. Today, “the prospects for continued low borrowing costs buoyed investors’ hopes for the U.S. economy”. Tomorrow, there may be “profit taking”.** Why do reporters feel obligated to explain why, or more to the point why do readers demand to be told? Why must we infer reasons, see momentum where there may be none, and assign labels like bull and bear? Why can’t we accept randomness?***
Update: Anthony Towns conducted a fascinating and puzzling follow-up analysis showing that the volatility of the stock market has been going up steadily over the years, even though the mean return has not.
__________
* The log difference is the log of the price on day d minus the log of the price on day d-1, or equivalently ln(d/(d-1)).
** I’d love to see a controlled experiment where financial reporters are given randomized reports about the Dow and watch them manufacture explanations, I imagine occasionally invoking the same cause for both ups and downs… (a) The prospect of higher interest rates spooked investors today; or (b) the Fed’s willingness to raise rates signals a recovering economy — investors rejoiced.
*** Let me clarify. The efficient market hypothesis implies both randomness and rationality: Randomness in the sense of statistical independence (no momentum), and rationality in the sense that price changes reflect new information. So I’m not saying that prices change for no reason, just that there is rarely an easily-stated single explanation for the overall market. (BTW, “profit taking” is never a valid explanation under EMH.)
** I actually heard a story exactly about this on NPR a few years ago. I don’t remember the details, but a few reporters went into bars in NY where exchange specialists hung out and did unofficial interviews about the reason for the daily rise/drop in the stocks. Each day they asked some reasonable number of people (10-20ish, if I recall) and rarely got the same answer twice. They did this on multiple days. I tried to dig up a link but sadly couldn’t find anything.
“Why do reporters feel obligated to explain why?” — because they write stories, and stories need a narrative and a narrative needs reasons, themes and linkages, even if they have to be invented.
And people prefer to read stories than lists of facts because they’re easier to remember, and thus easier to take action on.
And talking about intentions rather than just events both makes you sound deeper, and gives you an excuse to gossip about the people involved, which is easier than technical analysis if nothing else…
Thanks Hal, I’d love to hear that.
aj: I’m sure you’re right — and hopefully most people realize it is mostly about storytelling and entertainment.
Hey, I posted a followup that looks at how a Laplace fit of the S&P 500 changes over time. Anyone who’s already read this far might find it interesting… 🙂
May I reverse the question?
I would argue that for many of the movements (especially the ones with the higher magnitudes) there is a reason that the market moved.
So, the question becomes: what aspects of human behavior caused the returns to have such a predictable distribution?
Of course, as we move to the era of high frequency trading (i.e., only algorithms trade), the question becomes: what is the set of trading models, used by the individual machines/trading agents, that generates this predictable distribution of returns?
On a second thought, the question remains the same, no matter if trading is done by a human or a machine…
Nice work. Interesting analysis
To be honest I am just a newbie in the stock market and these graphs usually go over my head but the way you have simplifies and presented is really good…
This is interesting and is something that I have experienced looking back on my trading. Short term hugely volatile performance but when you look over a longer period performance smonths out and centres around the mean.
A one-dimensional histogram of daily market price changes in no way illustrates the non-existence of correlations that would enable decision-making behavior.
Under forms of EMH that permit arbitrage, profit-taking is a plausible explanation for some declines.
Nice-looking graphs though.
Regards.
Well, it’s amazing. The miracle has been done. Hat’s off. Well done, as
we know that “hard work always pays offâ€, after a long struggle with
sincere effort it’s done.
——–
mgibs
Stocks and Shares ISA
Geoffrey: you’re right — a histogram does not imply unpredictability
What form of the EMH permits arbitrage? I thought even the weakest form doesn’t permit arbitrage.
Still a little surprised by the relative slimness of the tails given the number of perceived ‘Black Swan’ events that have taken place in the last 23 years. Nice analysis though.
I have always believed as you so aptly put “futility of attaching reasons to price movements every single day” …good stuff.
Its sure beautiful. I love the way you have simplifies and presented it. For newbie like me it’s a lot of help 🙂
Hi David,
Arbitrage is required under any form of EMH in the presence of irrational price behavior. Under strong EMH, arbitrage would not be profitable. I was taught that it would be profitable over the long term under weak EMH, and it seems reasonable to suppose that it could be profitable over the long term under semi-strong EMH with the right proprietary information.
EMH is really a very interesting hypothesis, and not all economists subscribe to it. Is it not reasonable to suppose that as market participants gain in knowledge and sophistication, evidence for stronger forms of EMH increases?
I am a big fan of efficient markets, and I would be interested in seeing further articles on EMH in the future if you have the opportunity to address the topic.
Regards.
Hi,
nice charts and analysis. But what have you done with the heavy tails in the s&p returns distribution? Where’s the great drop on 19 october 1987?
Thanks.
Hey David,
Thanks for writing the article. I am trying to randomize the market for two years (Daily SP500 prices) and get data for each day for a model I am building for my portfolio. Could you send me exact instructions on how to do this?
Thanks,
Anik
great article – I have a question though. wouldn’t it have been more appropriate to use simple returns instead of log returns? I was always under the impression that log returns are used because they fit so well in the framework of the normal distribution. since we are approaching the problem without a distribution in mind a priori, simple returns would seem to be the default choice.