Getting Your Research Published: Insights on Academic Publishing with Simine Vazire

Under the Cortex typically hosts authors of peer-reviewed articles. But in this week’s episode, we do things a little differently, take a step back, and explore what happens on the editorial side of scientific publishing. 

Simine Vazire, the incoming Editor-in-Chief of APS’s journal Psychological Science, joined Özge Gürcanlı Fischer Baum to discuss her plans to further advance the practices of inclusivity in APS’s flagship journal, she highlighted the current disadvantages in academic publishing in general and said that APS is a leader in  supporting psychological scientists. The conversation evolved into topics of writing in English as a borrowed language, hidden curriculum in publishing and constructive practices such as pre-registration and reporting conflict of interest. 

Unedited Transcript

[00:00:11.050] – APS Özge Gürcanlı Fischer Baum 

How does scientific publishing work? What is the process of writing a journal article? What is peer review? This is Under the Cortex. I am Özge Gürcanlı Fischer Baum with the Association for Psychological Science. Today we are taking you to the kitchen of scientific publishing to tell you more about what happens behind the scenes at APS to publish rigorous scientific content. I’m extremely lucky to have with me Dr. Simine Vazire from University of Melbourne. In addition to a long list of affiliations, Dr. Vazire is the incoming editor in chief of APS’s journal Psychological Science. Simine, thank you for joining me today. Welcome to Under the Cortex. 

[00:00:54.940] – Simine Vazire 

Thanks. I’m happy to be here. 

[00:00:57.410] – APS Özge Gürcanlı Fischer Baum 

So you are the incoming editor in chief of APS’s academic journal, Psychological Science. It is very exciting for us. Before I ask you more questions about your new role at APS, please tell our audience how you became a psychological researcher. 

[00:01:14.380] – Simine Vazire 

Sure. Well, I think it started quite early for me. I found this note that I had written my best friend in 10th grade. So I was about 14 or so, and in it I wrote to her, no, sorry, she wrote this to me. She said, Miriam’s party. I got the idea that you wanted an honest evaluation from me about you. I think you’re very good at understanding people, but sometimes a little too harsh in your judgment. You are very athletic, a good leader, and you take the initiative to do something, blah, blah, blah. You’re trustworthy and honest. If I had to rate your personality on a one to ten scale, ten being best, you would be a nine. Obviously, she’s very biased, but apparently I was asking people to rate my personality when I was in high school, so I guess it started a long time ago. And then at university, I majored in psychology and minored in women’s studies, now called gender studies. But really, I would say kind of the beginning of my psychology research career was graduate school at University of Texas at Austin, where Sam Gosling was my advisor and kind of my first big, important mentor and really shaped a lot of my values in science. 

[00:02:18.730] – APS Özge Gürcanlı Fischer Baum 

Yeah, that anecdote is lovely. I know that you have a diverse set of research interests, and one of the things you study is self knowledge, whether people know things about themselves. How did you get interested in this topic? Like, was it this incident that happened when you were 14? And also, how does one study self knowledge? 

[00:02:40.710] – Simine Vazire 

Yeah, I mean, I think that I probably was interested in it before this incident. I’m guessing that I was just an annoying person who was always kind of talking about how people are different and how people see each other and stuff like that, and probably what a lot of teenagers are thinking about. But even as adults, I think that it’s just a fascinating topic independent of academia. But I think that for me, it was both just the inherent interest. We can call it theoretical if we want to be very intellectual about it. I think it is a deep philosophical question about how well people know themselves. But what I liked about it is that it also has clear practical implications and is kind of a methodological question about what’s the best way to measure what someone is like, their personality, or even their mood or other states. So it was always an interest to me. I mean, I started my research in personality and kind of behavioral manifestations of personality. And so one of the decisions we had to make in our research with my advisor Sam Gosling, was how we would define the gold standard, the ground truth about what a person is like. 

[00:03:41.610] – Simine Vazire 

And in my very first paper, for reasons that were not really that deep or theoretical, we ended up using peer reports like our participants friends ratings of them as our kind of ground truth measure of personality, partly because we were using the self reports for something else. But we also felt like that was justifiable that those are just as valid probably as self reports for a number of reasons. But we got a reviewer saying, surely the best measure of a person’s personality is their self report. Otherwise the whole field of personality assessment seriously needs to rethink itself. They were taking issue with us measuring personality with peer reports, and I thought, oh, that’s fascinating. I think it’s interesting how people respond to reviewer reports. For me, it just kind of lit a fire under me. And I thought, well, that’s an empirical question. Do we know that self reports are obviously the best measure for all traits and all kinds of populations, and regardless of who else we could get reports from, that seems implausible to me. So I wanted to start studying when are self reports better and when can we get equally good or maybe even better ratings from other people who know the person? 

[00:04:44.160] – APS Özge Gürcanlı Fischer Baum 

Well, your friend Miriam, who was giving. 

[00:04:48.090] – Simine Vazire 

My friend Geraldine, who was talking about Miriam’s party. Yeah, it was mean, obviously. I think that actually shows one of the big limitations of friend reports is that they can be extremely overly positive or biased, but actually, some other kinds of informants are worse than friends. So like parents or romantic partners. Parents are almost useless as informants, at least of their college age kids. They’re just way too positive. There’s not much valid information in there. 

[00:05:17.670] – APS Özge Gürcanlı Fischer Baum 

Yeah. And you have been doing this research for a while, I know, but one thing maybe people don’t know about you, in addition to your research, you are a champion of service in the academic community. Can you tell us a little bit about what type of service roles you have engaged with since you started your academic career? 

[00:05:36.980] – Simine Vazire 

Yeah, I mean, I think that probably I just do more visible service than some other people, so I do a lot of service to the field and to the kind of organizations at the field level. Instead, I have colleagues who do lots and lots of service to the university or to the community. Honestly, those are less likely to be visible and recognized, I think, by the field. So I’ve been lucky that the kind of service that I enjoy doing and that I’ve had the opportunity to do are ones that are recognized more, I think, than other kinds. But yeah, so most of my service has been a lot of journal editing, which I’m sure we’ll talk about more. I’ve gotten to sit on boards or committees like the board of APS, boards of other societies or of the publisher. Plus I’ve gotten to help organize conferences in social and personality psychology and now in meta research kind of societies. I got to sit on the social psych panel of the National Science Foundation in the US. For three years. And I was on a National Academy of Sciences study committee on replicability and reproducibility in science. 

[00:06:40.280] – Simine Vazire 

Those were all really fascinating experiences, kind of giving me a different window into the scientific world and ecosystem. And then I co founded with Brian Nosek, the Society for the Improvement of Psychological Science and sat on the board for that for a while and helped kind of start it up. 

[00:06:57.270] – APS Özge Gürcanlı Fischer Baum 

And like you mentioned this a little bit, you were reviewing people’s work. So peer review is a type of service in academia, and you have done this in the past, like many academic researchers. In addition, you have studied it. So could you tell our audience what peer review is and why is it important for the scientific community? 

[00:07:18.110] – Simine Vazire 

Yeah, so peer review mostly refers to peer review at journals. So that’s kind of before an article is published, although that’s changing these days with preprints. But typically or traditionally it would be authors submit a manuscript to a journal. The journal oversees and coordinates the peer review, so sends it to reviewers or sometimes evaluates it without reviewer input and rejects it before external review or yeah, then sends it out to reviewers and then makes a decision or asks for revisions. If you think of kind of the lifecycle of a scientific claim from when it’s born to let’s way oversimplify and say when it becomes an established fact. And so at the beginning, there’s like a huge firehose of scientific claims being made, and we need some way to filter them and to decide which ones are going to keep going on the road to becoming an established fact. So journal based review is kind of that first filter or one of the early filters, but we need more than that. There’s really a lot of role in the scientific community for more scrutiny, more review, more poking and prodding at findings before they would get to something like a textbook claim or a really established fact. 

[00:08:24.100] – Simine Vazire 

But another reason that the journal part is crucial is that in principle at least, it should give everyone an opportunity to get an evaluation of their work. So the journals help to make sure that every submission gets looked at at least by one, usually two editors if it’s going to be disrejected, or by a few reviewers if it goes out for full review. So that helps a little bit to level the playing field. I think that’s one value of journal peer review. But I do think that as we have more preprints, so there’s less of this bright line between pre publication peer review and post publication peer review because a lot of submissions that go to journals are actually already published as preprints and may already even have gotten critiques and comments on the preprint. I think this really raises questions that we probably should have been asking all along about what it means to have the peer reviewed label both because I think preprints could have been vetted quite carefully and because journal articles sometimes haven’t been vetted carefully or they were vetted carefully, but we still missed something. And I think that the existence of preprints and also honestly of journals that don’t have much integrity, like propaganda journals, or sometimes they’re like special issues of journals where there’s a lot of corruption in how things are handled. 

[00:09:36.770] – Simine Vazire 

So that means that getting the badge or the label, the seal of like this has been peer reviewed by a journal doesn’t mean that much if we include all journals, right? So then it starts to become important, well, which journal and is it a legitimate journal and how good is its peer review process? And I think the fact that those questions are coming up is actually a really good thing because it’ll mean that we can get a better handle on how do we know if a journal has a really rigorous peer review process that passing through that process should mean something. Because it’s not the case that we can just treat any journal peer review as like a really strong signal that the work has been vetted carefully. So I think there’s going to be more and more pressure on journals to be more accountable and transparent about what makes them their process. Kind of a valid signal that there’s been at least some quality control, some attention to rigor and so on. 

[00:10:29.290] – APS Özge Gürcanlı Fischer Baum 

Yeah, I like what you say. It is kind of like a good shortcut, but just like any shortcut, sometimes there is noise, sometimes people use it not for what it means, they sometimes use it for propaganda. So I think yeah, excellent point. That’s why it is important to know what journal we are talking about, what institutions those journals belong to. Excellent summary of this overall discussion we are constantly having in the academia. So you also do research about peer review, right? So what are your main findings? 

[00:11:07.090] – Simine Vazire 

So I’m going to summarize not just my own findings because I’ve only done a tiny bit of the research on peer review, but there’s a whole field now, emerging field of meta research or metascience, that just does kind of the science of scientific practice, including journals and peer review. And I think that there’s some really interesting findings out of this literature. And it’s actually, I would say most active in psychology or psychology is at least one of the top kind of fields where people are doing this kind of research about our published literature, our processes, our research practices, our journal practices. And some of the findings include some really exciting findings and promising findings. Like, for example, some practices are growing really fast, like transparency related practices. You’re seeing more and more authors choose to post their data, their code, their materials, choosing to preregister registered reports are on the rise. A lot of different transparency related practices are going up quite a bit. There’s also some research, including some of my own, showing that sample sizes are going up, which there was some meta research in earlier decades that was quite depressing, showing after all these calls for more statistical power, more precision, that sample sizes were not changing. 

[00:12:12.180] – Simine Vazire 

And then finally, probably because of, in part because of the access to Internet samples, sample sizes are finally going up. So that’s a good sign. Again, if we take it in isolation, it might also have some negative side effects, some trade offs that are not ideal. So meta researchers are also studying that, like, sure, our sample sizes are going up, but does that mean we’re using more convenient samples, but not populations that we should be drawing from, or we’re using methods that are easier to administer online, but not necessarily the methods that are best suited to the research question. So those are the kinds of questions we’re still investigating in our research. But there’s also some findings from meta research that are maybe less rosy, I guess I would say. So. Some of the meta research, for example, on replication rates and reproducibility rates suggest there’s a lot of room for improvement. And I think there’s really good debates to be had about what’s an ideal replication rate maybe it’s okay if there’s stuff that fails to replicate. I think what the lesson for me is in all the research on replications is that it’s fine to publish stuff that may be kind of uncertain at the time of publication and some of that we should expect to not replicate. 

[00:13:17.610] – Simine Vazire 

But I think we have to have better kind of truth in advertising. So when we publish things, it should be pretty clear to readers how confident and certain how definitive this finding is versus was it published more because it’s exciting and groundbreaking but not definitive. And that way we’ll be less surprised or shocked about which things are likely to replicate or not. And I think it would also help with policymakers and other kind of end users to interpret what does an acceptance in a prestigious journal mean? Was it accepted because we’re totally confident in it and ready to start applying it in the real world? Or was it accepted because it’s like the first to use this method or the first to make this discovery? And that really needs to be followed up before we apply it or generalize from it. And I think the replication work, I think, is a little bit more nuanced in terms of what it means for the state of the field, the computational reproducibility work, which is not collecting new data but reanalyzing the author’s own data. And there’s been a couple of groups like Sophia Cruell and her co authors and Tom Hardwick and his co authors, who’ve looked at computational reproducibility rates, specifically in psych science. 

[00:14:21.950] – Simine Vazire 

Actually, there’s been other projects looking at other journals too, and showing that when authors do share their data, it’s not always easy or even possible to reproduce the results in their own papers from their data, which is not surprising because data sharing is a relatively new thing. We haven’t really developed strong norms and standards around what code we should share, how we should comment our code, et cetera. But I think that shows a really great avenue for potential improvement. That’s an area where we could do better, we could develop better resources and training and norms around those things. There’s also other areas, I think, where we could improve. So preregistration is another, I think, practice that people are engaging in, but we’re still kind of in the early stages of learning how to do it best. And so there’s research, for example, by Olmo van Denakar showing that there’s quite often deviations from pre registrations that are not reported as clearly as they could be. So I think that’s something else we could improve. And then, of course, in terms of the diversity of our samples and the representativeness of our samples, or actually, I would say, the appropriateness of the sample for the research question, I think there’s still a lot of room for improvement there. 

[00:15:28.950] – Simine Vazire 

We’re often relying on convenient samples rather than trying to draw from the population that’s most relevant, or trying to draw from multiple different populations if we want to make more broad and generalizable claims. There’s research showing that our measurement practices still have a lot of room for improvement. And then in my own lab, we’ve done some research looking at things like how often do authors report limitations and what kinds of limitations do they report, or how often do they make claims about the real world applicability of their research. We’ve looked at how common are replication articles and other kinds of post publication critiques of published work, finding that those are still quite rare, despite a lot of movement in that direction in terms of journal policies and attitudes towards replication. But it’s still pretty rare for journals to publish those kinds of papers. We’ve looked at how often editors publish in their own journals. We haven’t published that paper yet, but we’ll have it ready soon. 

[00:16:22.700] – APS Özge Gürcanlı Fischer Baum 

Oh, that’s an interesting one. I haven’t thought about that. 

[00:16:26.510] – Simine Vazire 

Yeah, that’s one that I became interested in actually partly from being an editor and having well, one thing that was shocking to me when I became an editor was that it’s pretty rare for people to declare conflicts of interest in psychology for authors to declare conflicts of interest. And maybe that’s because compared to medicine, we don’t often have the more traditional kind of conflict of interest where we’re partnering with industry or we’re funded by a group that has a direct interest in our research. But we do have other kinds of conflicts of interest that we’re not used to thinking about. And one is when editors publish, submit an article to the journal that they’re an editor of, that’s a clear potential conflict of interest to my mind. And we found that it’s never disclosed. So we’d looked at a handful of journals over a handful of years. So just within our sample, I don’t think we found a single instance of an editor disclosing that as a potential conflict of interest when they published in their own journal. So it’s something I also thought about when I was on the board of APS. I pushed for a change that was implemented at APS journals, where if an editor submits a paper to their own journal, it’s actually handled by an editor outside of that journal, not by one of their co editors on their team. 

[00:17:33.890] – APS Özge Gürcanlı Fischer Baum 

How does one become an editor of a journal? You are very patient about this, very clearly, and I’m sure the young academics out there would like to know about the steps. 

[00:17:44.320] – Simine Vazire 

Yeah, I think that one step is to accept review requests for journals that you think highly of, that you would like to be more involved with. So when those review requests come in, prioritize those over other review requests, maybe. 

[00:18:01.850] – APS Özge Gürcanlı Fischer Baum 

I’m sure there is research out there about the accessibility of these communities, et cetera. Would you recommend young researchers to contact the journals to be reviewers themselves? 

[00:18:12.190] – Simine Vazire 

Yeah, I think they could definitely contact editors and just say that they’re interested in reviewing opportunities. They could also ask their kind of mentor, the mentors in their network, their supervisors, or anyone else who might be in a good position, who might get review requests themselves and not always accept them. Those people could recommend them as a reviewer. So that’s a very common way to get into the reviewer pool. But of course, that also requires connections to people who themselves get review requests. One of the things I’m planning to do when we kind of build up the editorial board, the new editorial board. So not the editors, but the people who are on the editorial board, who are frequent reviewers, is to do an open call for nominations and really reach out to different groups that are underrepresented in the kind of journal system currently to try to get nominations of people who would make good reviewers who are interested in doing it. And so looking out, I don’t know how many journals do these kinds of open calls, but looking out for those kinds of calls. But yeah, I think even just a direct email to an editor of a journal that you would like to be involved with, I think that would be welcome. 

[00:19:19.320] – Simine Vazire 

I mean, you may not get a response, but they may actually just still add you into the system because it’s so hard to find reviewers. I think there’s a decent chance that that would work. I would mention some of your maybe link to a few of your relevant papers so that they could check out your work and make sure that the quality of your work matches the quality and nature of your work matches what they’re looking for in reviewers. But yeah, I think that’s one way. 

[00:19:45.290] – APS Özge Gürcanlı Fischer Baum 

And you could have chosen other journals, obviously. Why is our journal Psychological Science important to you? 

[00:19:54.270] – Simine Vazire 

I think that it’s just one of the most well respected journals in the field and I think that makes it such an important player in communicating psychological science within psychology to other neighboring fields, to the public, to policymakers, all of that. It’s one of the journals that people look to. And so I’m really excited to play a role and work with others to try to do as much as we can to keep improving the kind of quality and importance and trustworthiness. And also because it’s so important within and outside of the field, it means that it plays a big role in people’s careers. Getting an article accepted at Psychological Science can have an impact on people’s chances of getting grants or jobs or awards and things like that. That puts a lot of responsibility on the editor. So it’s not necessarily a reason to want the job. But I guess if you’re going to be spending a lot of your time in this kind of service role anyway, it’s nice for it to be in a position where your changes and your actions have a chance to kind of shape things for the better. Of course, that also means you have the responsibility to really think through those changes and those actions because they will have big impacts on people. 

[00:21:10.250] – APS Özge Gürcanlı Fischer Baum 

So yeah, it’s an opportunity for impact. And you mentioned some of your goals briefly, but I will ask you what are your goals for the upcoming year for Psychological Science? Like, should we expect any changes? What are you thinking of? 

[00:21:27.040] – Simine Vazire 

Yeah, so I’m going to kind of follow in the footsteps of previous editors on a few things like increasing transparency. So there have been kind of incremental changes over the last ten years on that and I’ll keep going in that direction. So one thing we’ll do that was already signaled by the current editor in chief. Patricia Bauer is moving to the level two of the transparency and Openness promotion guidelines, which means requiring data and code and materials from all. Authors with exceptions in cases of ethical challenges, for example, and also valuing different kinds of research and recognizing that there are many different ways for research to be high quality or important or strong and that almost no research is going to be strong on all the dimensions. And so really appreciating that there can be different strengths, we should value different strengths. We shouldn’t have a one size fits all kind of expectation about what kind of paper we’ll publish. And the important thing is that the research is presented in a calibrated and accurate way. So the strengths are clear, but also the trade offs or any limitations are also clear. So I want to kind of not just focus on transparency and really think about more holistically about the quality and accuracy and calibration of the research, which means not making exaggerated claims, not overselling the research, being really clear about the caveats and the kind of boundary conditions or limitations. 

[00:22:49.910] – Simine Vazire 

And that means that we as editors have to hold up our end of the bargain, which is to not punish authors for being honest and transparent about those limitations or those trade offs and really reward the qualities without. Punishing authors for the kind of trade offs they have to make sometimes to achieve those qualities. Sometimes in order to get something like a sample that’s underrepresented in the literature, that means that you won’t be able to get as large of a sample size. And so taking each paper on its own merits and really seeing the value of what’s strong about it and rewarding authors who are honest and upfront about what’s not strong about it or what they didn’t do, what they couldn’t do with their design or their methods. I also would like to try to think about what Psych science can do to encourage more high quality post publication critiques and corrections and things like that. I know Patricia Bauer did some stuff in that direction too, introducing a new submission type. I want to revisit that, see how well it’s working and what we could do to kind of reward and recognize really good corrective and verification work that people are doing. 

[00:23:59.450] – Simine Vazire 

And then I want to continue doing more to increase the equity of our practices, the fairness of our practices, and also the accessibility of the journal to researchers from many different not just geographical regions, but also different traditions in terms of methods and approaches, different languages, different there’s a lot of different groups. That our system, as it currently is disadvantages. So for example, people for whom English is a borrowed language, publishing in an English language journal is a lot more work than people who either have always worked in English or they’re very comfortable in that language. And so I’m trying to think through what we could do. I mean, the problem is that journals kind of come into the picture pretty late in the research process. So I think journals maybe don’t have as much of a role to play in providing resources in the early stages of the research process. But is there something we could do to help support researchers who maybe are writing in a borrowed language in English and need extra support in the writing stage? Maybe that’s a place where journals do have a role to play. I’m thinking through those things, but also kind of trying to demystify the whole journal and peer review process. 

[00:25:09.830] – Simine Vazire 

And there’s kind of a hidden curriculum of things that happen behind the scenes that aren’t explicitly talked about or aren’t printed in author guidelines. And so I’d like to hold office hours at different time zones and so on to try to make it more accessible for people to drop in and just ask questions not about their specific paper, but about how does this process work? Is it okay to email an editor if you haven’t heard after so many months or other kinds of questions like that, that I get a lot informally at conferences and things like that, but I think there needs to be a more open, equitable, kind of accessible way to share that information. 

[00:25:47.470] – APS Özge Gürcanlı Fischer Baum 

Well, Simine, thank you very much. This is like a very long list of changes, but every each of them is very important. Thank you very much for thinking about them. And also thank you for joining to our podcast today. 

[00:26:03.520] – Simine Vazire 

Thank you. 

[00:26:04.340] – APS Özge Gürcanlı Fischer Baum 

I am personally looking forward to the changes in Psychological Science and I’m looking forward to your leadership. This is Özge Gürcanlı Fischer Baum with APS and I have been speaking to Dr. Simine Vazire from University of Melbourne. If you want to know more about this research, visit psychologicalscience.org.