It started with a search for trends on PubMed. I am not sure what I expected to find, but it was nothing like the “CISCOM meta-analyses”. Here is the story of how my colleague Lucas Carey (from Universitat Pompeu Fabra) and myself discovered a collection of disturbingly similar scientific papers, and how we got to the bottom of it.
CISCOM is the medical publication database of the Research Council for Complementary Medicine. Available since 1995, it used to be mentioned in 2 to 3 papers per year, until Feburary 2014 when the number of hits started to skyrocket. Since then, “CISCOM” surfs a tsunami of one new hit per week.
But this is not what drew my attention, such waves are not unheard of on PubMed. For instance, the progression of CRISPR/Cas9, is more impressive. It was the titles of the hits that convinced me that something fishy was going on: all of them are on the model “something and something else: a meta-analysis”.
The strange pattern caught my attention, but I somehow missed its significance and put this in the back of my mind. It was only later that Lucas convinced me...
A colleague of mine (let’s call him John) recently put me in a difficult situation. John is a very good immunologist who, as nearly everybody in the field, had to embrace the “omics” revolution. Spirited and curious, he has taken the time to look more closely into statistics and he now has an understanding of most popular parametric and nonparametric tests. One day, he came to me with the following situation.
“I have this gene expression data, you see... I know that a gene is up-regulated, but it is just not significant...
In Jorge Luis Borges’s short fiction Tlön, Uqbar, Orbis Tertius, the narrator discovers by accident the existence of a secret encyclopaedia. In this story more than in the others, Borges added many details that are actually true, in such a way that it is hard to tell the reality from the fiction. Needless to say, most would think that the secret encyclopaedia is pure fiction... and they would be wrong. Secret encyclopaedias do exist and they go by the name of “shadow libraries”.
In the course of developing PubCron (a personalized academic literature watch), I learned that PubMed references more than 2,000 new articles daily. For the vast majority of those papers, the authors pay about $1,000 as publication fees. In the bio-medical field alone, this is a $2 million gift to the publishing industry. Every day.
Gift is not the proper term. This money goes to scientific editors and in such amount, it should be sufficient to sustain about 20,000 professional editors. An editor working full time would publish 3 papers per month... not an unreasonable estimate considering that only a fraction of the manuscripts are published. This means...
On June 28, 1914, Archduke Franz Ferdinand was assassinated in Sarajevo. One month later, Austria-Hungary declared war on Serbia, to which Russia responded by declaring war on Austria-Hungary, forcing its allies France and Great Britain into the war. In the aftermath, Germany honoured its defensive pact with Austria-Hungary and declared war on France, plunging Europe in a chaos that nobody had predicted.
Cliodynamics, the mathematical approach to History, still has a long way to go to reach the accuracy of Isaac Asimov’s fictive psychohistory. Its closest non science-fiction relative, culturomics, relies on the idea that historical trends are accessible through the digital literature. As Jean-Baptiste Michel explains on TED, the course of History leaves a strong mark on the things we write about, and on the way we write about them.
But historical events are not the only thing we write about. The digital records are mostly about anything we find interesting. Knowing what is talked about is not science-fiction, it is actually fairly easy. More challenging is to know whether a topic is currently on the rise or merely fluctuating, which is a changepoint detection problem. Research on changepoint problems...
The story of this post begins a few weeks ago when I received a surprising email. I have never read a scientific article giving a credible account of a research process. Only the successful hypotheses and the successful experiments are mentioned in the text — a small minority — and the painful intellectual labor behind discoveries is omitted altogether. Time is precious, and who wants to read endless failure stories? Point well taken. But this unspoken academic pact has sealed what I call the curse of research. In simple words, the curse is that by putting all the emphasis on the results, researchers become blind to the research process because they never discuss it. How to carry out good research? How to discover things? These are the questions that nobody raises (well, almost nobody).
Where did I leave off? Oh, yes... in my mailbox lies an email from David Lipman. For those who don’t know him, David Lipman is the director of the NCBI (the bio-informatics spearhead of the NIH), of which PubMed and GenBank are the most famous children. Incidentally, David is also the creator of BLAST. After a brief exchange on the topic of my previous...
In July 1982, paleontologist Steven Jay Gould was diagnosed with cancer. Facing a median prognosis of only 8 months survival, he used his knowledge of statistics to prepare for the future. As he explains in The Median Isn’t the Message, if half of the patients died of this rare case of mesothelioma within 8 months, those who did not had much better survival. Evaluating his own chances of being in the “survivor” group as high, he planned for long term survival and opted out of the standard treatment. He died 20 years later, from an unrelated disease.
If not the median, then what is the message? Statistics put a disproportionate emphasis on the typical or average behavior, when what matters is sometimes in the extremes. This general blindness to the extremes is responsible for a dreadful lot of confusion in the bio-medical field. One of my all time favorite traps is the extreme value fallacy. Nothing better than an example will explain what it is about.
June babies and anorexia
In the first call for post-docs in my lab, I realized that most applicants did not have a clear idea of what to write in a cover letter. My technicians and I read them all with great frustration. Retrospectively, this is not so surprising, given that nobody teaches you how to write a cover letter in academia. Since it may be useful for the community, I am going to explain how I read a cover letter and what I would like to see there.
What is a cover letter for?
From the principal investigator’s perspective, a cover letter is to get some insight into your motivations.
Several candidates think that the cover letter is the place to argue why you should hire them. Others use it as a chance to translate their CV into English prose. None of this will bring you very far. All you need to do is give enough information about yourself to let the employer know whether you have the profile they are looking for. At the same time, you want to demonstrate a proactive attitude.
Before you write, ask yourself what drives you, why you started this career. Is that ambition...
Optical illusions come in two flavors. The one shown on the left is a classical example of a “false” perception. It looks like the grey lines are curved, but they are actually perfectly horizontal. Even if we know that the lines are horizontal, we cannot force ourselves to see them this way. I find it somewhat of a miracle that reason is strong enough to tell that the eyes are lying, but still, however long you stare at the picture, you will never see it the right way. The brain cannot learn to see this picture.
The second flavor of optical illusions is shown on the right. This picture may either represent the face of a woman, or a saxophone player. Most people immediately see one of those, and it may take them some time to see the other. But once they see it, there is no way to “unsee” it. The brain has completely forgotten how it feels to not see it and cannot unlearn to see the picture.
Vision is not the only sense that can be fooled. As a matter of fact, what is so special about optical illusions is that we realize that something...
People often ask me whether it is difficult to be a group leader. I usually answer “No that’s easy. Being a good group leader... that’s another story.” It is one of these jobs for which there is no training. It is also one of these jobs about which there is no information. As the leader of a team of scientists, you will have to face your problems alone, because scientific leadership is...
The best kept secret in the world
The need for solid leadership skills is fully acknowledged in the civilized world, but in academic research this culture is lagging behind. So far behind that we have to admit the plain truth: leadership in science is a taboo. Understandably, academic researchers are reluctant to share their experience because they have little authority to do so, and because the information is often confidential. I could tell how I solved “the case of a post-doc of mine who put his music too loud in the lab”, but since all my colleagues would identify this person, I would end up with a serious issue in my team.
There are also systemic reasons for this lack of culture. Academic...
In the first days of my PhD, I sincerely believed that there was a chance I would find a cure against cancer. As this possibility became more and more remote, and as it became obvious that my work would not mark a paradigm shift, I became envious of those few people who did change the face of science during their PhD. One of them is Andrey Kolmogorov, whose PhD work was nothing less than the modern theory of probability. His most famous result was the strong law of large numbers, which essentially says that random fluctuations become infinitesimal on average. Simply put, if you flip a fair coin a large number of times, the frequency that ‘tails’ turn up will be very close to the expected value 1/2.
The chaos of large numbers
Most fascinating about the strong law of large numbers is that it is a theorem, which means that it comes with hypotheses that do not always hold. There are cases that repeating a random experiment a very large number of times does not guarantee that you will get closer to the expected value — I wrote the gory detail on Cross Validated, for those interested...