(First in a series of “random thoughts on science”)
A mind boggling number of academic research conferences and workshops take place every year. Each fills a thick proceedings with publications, some containing hundreds of papers. High-profile conferences can attract five times that many submissions, often of low average quality. Smaller venues can seem absurdly specialized (unless it happens to be your specialty). Every year, new venues emerge. Once established, rarely do they “retire” (there is still an ACM Special Interest Group on the Ada programming language, in addition to a SIG on programming languages). It’s impossible for all or even most of the papers published in a given year to be impactful. Most of them, including plenty of my own, will never be cited or even read by more than the authors and reviewers.
No one can deny that incredible breakthroughs emerge from the scientific process — from Einstein to Shannon to Turing to von Neumann — but scientific output seems to have a (very) long tail.
Is this a good thing, a bad thing, or just a thing?
Is the tail…
- Good?
- Is the tail actually crucial to the scientific process? Are some breakthroughs the result of ideas that percolate through long chains — person to person, paper to paper — from the bottom up? Is science less dwarfs standing on the shoulders of giants than giants standing on the shoulders of dwarfs? I published a fairly straightforward paper that applies results in social choice theory to collaborative filtering. Then a smarter scientist wrote a better paper on a more widely applicable subject, apparently partially inspired by our approach. Could such virtuous chains actually lead, eventually, to the truly revolutionary discoveries? Is the tail wagging the dog?
- Bad?
- Are the papers in the tail a waste of time, energy, and taxpayer dollars? Do they have virtually no impact, at least compared to their cost? Should we try hard to find objective measures that identify good science and good scientists and target our funding to them, starving out the rest?
- Ugly?
- Is the tail simply a messy but necessary byproduct (I can’t resist: a “messessity”) of the scientific process? Under this scenario, breakthroughs are fundamentally rare and unpredictable hits among an enormous sea of misses. To get more and better breakthroughs, we need more people trying and mostly failing — more monkeys at typewriters trying to bang out Shakespeare. Every social system, indeed almost every natural system, has a long tail. Maybe it’s simply unavoidable, even if it isn’t pretty. Was the dog simply born with its (long and scraggly) tail attached?
I am forced to vote “bad” on this.
The vast majority of peer-reviewed technical papers I’ve read are hopeless: they are solutions to non-existent problems or trivial variations on previous work. While the academic and research communities perform an important function in filling out the edges of very large innovations, true innovation (Einstein, Shannon, etc., as you mention) is not a product of the publish-or-perish environment. What benefit, then, does that ocean of paper provide?
I’m voting for “ugly.”
Yes, the cost of production for most academic papers is probably well in excess of the value produced.
Maybe one in ten is useful for some small purpose, one in hundred more generally interesting, and one in a million provides a significant breakthrough. Only, it is impossible in advance to know which possible research articles will turn into the one useful paper in ten rather than the nine.
By your definition it’s just ugly. But most human behavior is going to be governed by an extreme distribution with a long tail. What we can do is create a better system which awards good scientists (trustworthy hardworking people) as well as good science (experiments are reproducible) (There surely many more metrics we can use here)
I think at a minimum, we need to distribute some money randomly, to try and hit these people lost in the long tail. That is, take some money away from the most popular ideas, and fund more new, off-the-wall, and unpopular ones. Yes, breakthroughs are rare and unpredictable, but their impact may be of paramount significance, as opposed to piling more money onto popular ideas to make incremental advances.
Just came across this article
http://www.technologyreview.com/blog/post.aspx?bid=354&bpid=23289
and from the original source (physics world)
http://physicsworld.com/cws/article/print/38468
“More generally, Taleb’s basic investing strategy — which could easily be translated into research terms — is to put a fair fraction of funds into very conservative processes that will not lose their value, even if they have little chance of producing big gains; and to put a small but reasonable fraction into high-risk, high-reward settings, thereby gaining exposure to the potentially enormous gains from these investments. These may be unpredictable in detail, but the statistics makes the expected long-term pay-off very high. “
Manual trackback:
“A few publications not only dominate the attention of the relevant discipline but an equally few seem to be the most profound in their practical application. What does this mean about the scientific program? How much waste is there in science? Pennock offers several different views on the “(very) long tail†of science, one of which is more or less similar to my own view.”