Since last year I have a monthly column in the Dutch magazine “de Psycholoog“, called “de Gereedschapskist” [The Toolbox]. As the title of the column suggests, my aim is to disseminate psychological methods and my reflections on what such methods can and cannot do to a wide audience of psychologists – ranging from clinical psychologists to organization psychologists – who work in practice. It is my firm belief that also psychologists working in practice can benefit from knowing how to evaluate the claims of papers in terms of their methodology: was the correct method chosen given the research question, and to what extent do the results provide evidence in favor (or not) of a hypothesis? In clinical psychology, for example, therapists may benefit from some additional methodological background knowledge in order to evaluate the extent to which a new intervention truly performs better than existing interventions.

Besides disseminating methodological knowledge, I have written pieces about slow science (and how much I love it!), methodological terrorism and open science combined with some guidelines, or checklists, as a means to optimize the probability of finding real effects.

These columns are written in Dutch but I’m planning to translate them into English somewhere in the coming months. I’ll keep you posted!

The columns:

• Simpson’s paradox
• Within- versus between-subjects effects
• Bureaucracy in the scientific enterprise (inspired by this insightful paper by Jelte Wicherts et al.)
• Slow science
• Methodological terrorism
• Explanation vs. prediction (inspired by this brilliant paper by Yarkoni and Westfall)
• Factor analysis
• Confirmatory versus exploratory hypothesis testing (based on my own paper about the multiple comparisons problem in the multiway ANOVA)

It was with great pleasure that I listened to the wonderful podcast by The Black Goat (; Twitter: @blackgoatpod) about impostor syndrome in academia. I commend the contributors to this podcast (, Alexa Tullett and Sanjay Srivastava) for having the courage to discuss this much-shied-away-from topic as well as to be honest about their own personal experiences with trying to meet the high expectations of academia while at the same time – we seem to forget about this – trying to have a non-work life. This blog post is a reflection on what they said during the podcast as well as a call for considering additional reasons for why this impostor syndrome may be particularly pervasive in academia.

What is impostor syndrome? First coined in 1978 (Clance & Imes, 1978), the term refers to high-achieving individuals who have difficulty to internalize accomplishments (e.g., I got that grant because I’m a good researcher) and a persistent fear of being exposed as a “fraud”. Personally, I’m somewhat hesitant to call it a syndrome – calling it that brings in a host of questions for which we do not really have answers (e.g., Is it real or a social construction? Is it pathological? Should we treat it?) – but for now, let’s call it the presence of impostor feelings. So, in academia, the definition is, I think, mostly used to describe that eerie feeling that other people in academia are not aware of your grave imperfections. One day, one way or another, they will find out how incompetent you really are.

As one can imagine this is not a pleasurable feeling. I can vividly recall the day of my dissertation defense. I could not sleep all night: not because I was so excited at the prospect of obtaining my PhD, or at the prospect of throwing a big party that would end with giving people ibuprofen so they could treat their hungover symptom networks the next day. No, I could not sleep because I was convinced that this would be the day that I would be found out. In front of my colleagues, friends and family, the questions of the defense committee, and my inability to answer them, would make crystal clear, once and for all, that I was an impostor. In real life this did not happen: not only did I answer all questions in a thoughtful way, I even managed to crack a joke here and there and set forth a research agenda that I’m still pursuing this very day. I obtained my PhD with honors. So, one may ask, surely the impostor syndrome is gone now, buried in my past and only relived when I give students advice about life in academia? No. I still have these bouts of impostor feelings and they will probably not go away – as the speakers during the podcast also conceded. That is, not only is feeling like an impostor not pleasurable, it is also probably long-lasting irrespective of the successes one amasses throughout an academic career.

The podcast discussed several reasons for why impostor syndrome is a thing in academia and I related to all these reasons. For example, the contributors discussed a trend of over-exaggerating the number of work hours per week. In one study that was mentioned, academics reported way more hours of work per week than could be distilled from a diary they kept for a few weeks. This “I’m super duper productive, busy and online all the time” attitude feeds impostor feelings, probably most notably in researchers who do not devote the same number of hours to work – either due to child care, caring for a family member, or other (personal) issues. Because: how can one ever be as good as the colleague in the next room who works 20 hours per week more than you? A second reason is being silent about periods in one’s working life that were not overly productive. For example, Simine Vazire shared an unproductive period in her life because she suffered a breakup. The podcast members agreed that these unproductive periods in life are completely normal yet somehow, we do not readily talk about it.

A third reason, not discussed at length during the podcast yet related to the other two, is the fact that academia has a strict hierarchy – despite so many academics claiming its non-hierarchical nature, with which I wholeheartedly disagree – that is highly visible. Take, for example, the average conference, which comes with a visible pecking order: poster, giving a talk, giving a talk in a symposium, giving a talk in an invited symposium, keynote speech. That is, it is not particularly nurturing for the insecure aspects of self that it is so darn visible that you are not ‘ready’ or ‘important enough’ yet to give a keynote.

Pecking orders do not cause impostor feelings, I do think that hierarchies, exaggerating work hours and not talking about the unproductive phases in academic life set the perfect stage for impostor feelings. A fourth, final, reason for impostor feelings is, I think, an interesting feature of academia that is notably not present to the same extent as in commercial business working environments: it is all so darn personal. While this is great when you have success, it is not that great when you lose. It is YOU who does not get that fantastic grant, it is YOU who gets horrible reviews on a paper you worked on for two years, it is YOU who is not even mentioned as an expert during a meeting about the topic you have been working on for almost a decade, etc. I could go on and on. To the contrary, as I have distilled from conservations with quite a few non-academic yet high-achieving people: in modern business, it is more about the teams and their collective performance and not so much about the individuals. Sure, individuals, not teams, get a promotion or not, but developing a successful prototype after a Sprint session is a team win with no single hero who claims the victory.

Synthesizing these observations about academia: we have created a pervasive myth, a perfected, idealized version of an academic; the 80-hour-a-week-working, prize-winning, grant-awarded, keynote-giving human (often, still, a white male). A human, do note, without a personal life: there are no depressive episodes, no elderly parents that need care, no daughter who is arrested for drug use. The academic that features in this myth is not a person, it is a persona, a vignette, one that is completely unattainable for those of us who – and I suspect (hope!) they are the majority – are devoted to not only academia but also to all other things that make life worth living.

I think impostor feelings have a lot to do with accepting, owning and showing one’s imperfections: feeling an impostor may in fact be an exaggeration of the extremely natural yet uncomfortable realization that one is not perfect. Not to say that someone who feels like an impostor has every right to feel that way because he or she truly is not good enough. But I do think, speaking from experience, that impostor feelings and feeling uncomfortable in one’s own imperfect skin, are related. If so, then being imperfect, and owning up to it, is really really hard in the academic world I have just described. Think about the following behaviors that would be considered signs of being comfortably imperfect: 1) during a meeting, acknowledging that you are not familiar with that shiny new method that the PI was just discussing and everyone else at the table seems to know; 2) owning publicly, by means of a blogpost, that you made an honest mistake when analyzing your data; 3) sharing with your close colleagues that you have suffered from episodes of major depression. All three behaviors convey the same message: I am imperfect. Would we do it, any one of these behaviors? I don’t think so, and yet, we should.

So: what can and should we do about impostor feelings in academia?

• Stop buying into the perfection myth, just don’t believe it: researchers are humans too, and we all experience fear, grief, loss, physical and mental health issues, just as much as non-academic people do.
• In order to debunk the myth I: start being honest about yourself, embrace your imperfect self. No, you don’t have to share your entire life story, but we need to start presenting ourselves as actual persons instead of persona, perfectly crafted vignettes that only contain work-related achievements. A CV of failures is a good idea in that sense. Additionally, as I have done in this blog post, sharing a personal story now and then, showing your imperfections, will hopefully help by encouraging other researchers to start doing the same.
• In order to debunk the myth II: besides working towards owning our imperfections, our working environment needs to change as well. Impostor feelings are also a product of this environment as I have argued in this post. So: for example, we could do with a little less hierarchy, from the “superstar rockstar professor” to “merely the PhD student”. I think one of the advantages of social media is that institutional hierarchies are less visible online. Additionally, we need to start learning how modern businesses work. In particular, I’m thinking about non-hierarchical structures in which agile teams, without a real leader and thus no single hero, together work on projects.
• We need to start extending the conversation to people who may not otherwise recognize that they suffer from impostor feelings too. In my experience impostor feelings are quite exclusively discussed as being experienced by specific groups, most notably women. Maybe women are somewhat more insecure than men, and women certainly face additional hurdles when trying to get ahead and build a productive career; but showing our imperfect selves as a means of creating an environment where it is OK to be unproductive for a while, to not know something, is just as beneficial to men as it is to women.

I’m done with confessing for now. This imperfect researcher is getting back to her imperfect work!

Note: I thanks Simine Vazire, Alexa Tullett and Sanjay Srivastava for their helpful feedback on an earlier version of this post.

A commentary by Eiko Fried, Sophie van der Sluis and myself in response to an editorial in Nature about the Cai et al. CONVERGE results on the genetics of major depression. 

Last week I’ve submitted a paper (written with ) entitled “Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies”. In a nutshell, we argue that without a-priori hypotheses, using a multiway ANOVA becomes an exploratory expedition in which the family of hypotheses comprises all hypotheses subject to test; and as such, a multiway ANOVA harbors a multiple comparison problem. For example, in the case of two factors, three separate null hypotheses are tested (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14% rather than 5% if the three tests are independent. Statisticians are well aware of this problem.

However, many psychology researchers do not realize this lurking multiplicity problem and as a result, almost never correct their alpha’s when using multiway ANOVA in an exploratory fashion. We show this for 819 papers in six widely read and cited psychology journals: Journal of Experimental Pyschology General, Psychological Science, Journal of Abnormal Psychology, Journal of Consulting and Clinical Psychology, Journal of Experimental and Social Psychology and Journal of Personality and Social Psychology. In almost 50% of these papers, a multiway ANOVA was the main statistical analysis, underscoring the popularity of this testing procedure. Unfortunately, of these papers, only around 1% used a correction procedure (i.e., the omnibus test).

Fortunately, there are quite some ways to remedy this multiplicity problem as we outline in the paper. First, one could use an omnibus F-test: in such a test, one pools the sums of squares and degrees of freedom for all main effects and interactions into a single F statistic. The individual F tests should only be conducted if this omnibus null hypothesis is rejected. A major drawback of this method is that is does not control the familywise Type I error under partial null conditions; and as such, the method offers only weak protection against the multiplicity problem. Second, one could opt for controlling the family-wise error rate (FWER), for example by using the sequential Bonferroni correction method. While adequately controlling the Type I error, the downside is that this method reduces power. Third, one could choose to control the false discovery rate (FDR) instead, for example with the Benjamini-Hochberg correction method. This method results in more power compared to sequential Bonferroni but at the expense of less control of the Type I error. Finally, preregistering hypotheses (for example at the ) forces the researcher to specify the specific hypotheses of interest beforehand. In that case, using the multiway ANOVA becomes a confirmatory expedition and this potentially mitigates the multiple comparison problem. For example, consider experimental data to be analyzed with a two-way ANOVA: if the researcher stipulates in advance that the interest lies solely in the interaction, this reduces the number of tested hypotheses in the family from three to one, thereby diminishing the multiplicity problem.

The latest version of this paper can be found here.

Were the comments of Tim Hunt about women in science labs appalling? Most certainly so. Was it justified to make him resign from his position at UCL and dismiss him from various committees? I tend to think so although these consequences would have, and should have in my opinion, been avoidable if only the man had offered his sincere apologies and had shown some level of understanding of to what extent these kinds of ‘jokes’ are offensive to women.

Some people come to Tim Hunt’s defense by claiming it was ‘just a joke’. I can tell you, being a woman, this is not funny, not in the least. I’m sadly familiar with these types of ‘jokes’, for example the Dutch saying “the only right of a woman is to stand behind the kitchen counter” (het enige recht van de vrouw is het aanrecht). It’s fortunately less and less common to say this but still, to this day, some people say this out loud; purportedly meant to be a joke, but I’m fairly certain that this saying holds at least a small grain of truth for these people.

Yet others admit that Tim Hunt’s comments are appalling but they do not see how his remarks make him a sexist (for example ). Although I can see what Athene Donald and others mean – there could be a discrepancy between someone’s words and his/her actions – I do wonder if we would have had this discussion at all if the comments were made about, say, people of color. Imagine for a moment that Tim Hunt would have said: “I think segregated labs are better: people of color always want to dance, hereby distracting the white scientists. And when I confront them with their behavior, they run away”. I’m absolutely sure that there would have been no civilized discussion about whether Tim Hunt is a racist; and rightly so. Because such remarks are stereotyping an entire group: it is not true, it hurts and is therefore unnecessarily offensive. Such comments are racist – and in the case of Hunt’s actual remarks – sexist, regardless of whether someone acts upon the stereotypes that secretly linger behind so-called jokes.

 

Here’s a piece in the Dutch paper Volkskrant about our just online published paper in PNAS. Using time series of autorecorded mood, we show that indicators of slowing down are also predictive of future transitions in depression. Specifically, in persons who are more likely to have a future transition, mood dynamics are slower and different aspects of mood are more correlated. This supports the view that the mood system may have tipping points where reinforcing feedbacks among a web of symptoms can propagate a person into a disorder. Our findings suggest the possibility of early warning systems for psychiatric disorders, using smartphone-based mood monitoring.

Here’s a one page interview in the Dutch newspaper ‘het Parool‘ about my work!

I just submitted a paper on hidden multiplicity (i.e., a problem of multiple comparisons) in multiway ANOVA. In brief, when a researcher uses a multiway ANOVA without specifying hypotheses a priori, we argue that this researcher is in what Adriaan de Groot (1969) called the “guess” phase of the empirical cycle in which hypotheses are formed rather than tested. And in that case, for a design with, say, two factors, we argue that the family of hypotheses encompasses all hypotheses implied by the design (in this case three: two main effects and one interaction). This inflates the family-wise Type I error to 14% instead of 5% as a result of which the alpha level should be corrected accordingly (multiple methods exist to do this, for example the sequential Bonferroni procedure). By reviewing the 2010 volumes of six flagship journals in psychology, we show that applied researchers are generally not aware of this problem; that they, as a result, do not correct the alpha level; and that applying the sequential Bonferroni correction to a random subsample of articles alters at least one of the substantive conclusions in 45 out of 60 articles considered. We conclude that researchers either have to correct alpha levels when using multiway ANOVA to “see what we can find” (de Groot, 1969), or should preregister their hypotheses. In the latter case, one enters the “predict” phase of the empirical cycle in which hypotheses are tested. In such a confirmative setting, each hypothesis could be argued to constitute its separate family; hence obviating the need to correct the alpha level.

Here’s an interview (in Dutch) on Radio 1 about my dissertation.

Here is an article (in Dutch) about my dissertation, on Kennislink: a site dedicated to disseminating scientific knowledge to a broad audience.