SPEAKER_01 0:02

Welcome back to the Thought Broadcast, the trainee-led podcast from Australasian Psychiatry. We have with us myself, Dr. Ed Miller, Dr. Fiona Wilkes, Dr. Andrew Amos, and Dr. Michael Waiteman. And we're joining you from Congress 2024 in lovely Canberra.

SPEAKER_00 0:18

G'day Ed, how's it going?

SPEAKER_03 0:20

Morning.

SPEAKER_01 0:20

Hi Ed, nice to be here. Morning, morning everyone. So in this podcast, we're going to be talking about the peer review process in biomedical science publishing and really just try to demystify this process a little bit. So the peer review process is a fundamental aspect of biomedical science publishing, including psychiatry. And academic peer review refers to the processes by which submissions that are made to scientific journals are examined and altered, which occurs mostly prior to their publication. The most effective method for peer review remains enigmatic, and despite some advancements in the field, its necessity, its quality, and its validity have been questioned. And it's a complex process and it can also be confusing to understand. So participating in peer review can actually be an intimidating prospect for junior researchers or junior psychiatrists who can lack a clear entry point into this field. So what we're going to try to do today is give an overview of peer review and try to demystify the process. So we're going to discuss a little bit about the history of peer review, highlight some advantages and limitations of peer review, look at some potential improvements to the peer review process and ways that we can try to control the quality of published articles. And finally, we're just going to give some tips and tricks for new reviewers.

SPEAKER_03 1:49

I'll start a little with a bit of the history of the peer review for context. Peer review processes have evolved over time in a fairly indiscriminate way. From what we can tell, probably the first organized peer review process, certainly from our Eurocentric point of view, began in the UK in the early 18th century. The Royal Societies of Edinburgh and London began asking their members to evaluate submitted manuscripts. And that's because as they were getting up in these collections of largely men who were talking about the things that interested them in their gentleman-scholar way, then got more submissions than they could actually assess themselves. So they wanted to try and start and make a timely and straightforward method to determine which of their submissions should be published in their journals and discussed in their meetings. Articles were selected for publication based on their novelty, ingenuity, or importance. And the societies issued disclaimers explaining that they took no responsibility for the truth of facts, soundness of reasoning, or accuracy of the calculations found within, which is a bit different to the peer review now, where we're very definitely looking and trying to make sure that those things are accurate.

SPEAKER_01 3:05

I find that absolutely fascinating. The fact that they they didn't take any responsibility for you know apparently for what was actually published in their journal, which I can see that um sort of for and against it in the in the fact that you know the journal could potentially be just a uh sort of portal for information rather than us, the journal editors and reviewers, actually participating in making sure it meets some sort of standard, whatever that standard you know may be. But I just find it interesting it so bluntly says truth of facts, but perhaps it comes back down to perspectives in terms of what actually is you know fact or truth.

SPEAKER_02 3:39

Sounds like it was written by a lawyer.

SPEAKER_00 3:42

In modern times, we're realizing that peer review probably gives you the illusion of certainty rather than reality. There are all sorts of problems with peer review. So it may have actually been a more honest approach to say this is what we know, or this is what we would believe we know. We're looking for alternative points of view.

SPEAKER_03 4:00

And I was doing a bit more reading around this and the history, fast forward another hundred or so years. But apparently uh Albert Einstein wrote to an American journal being really annoyed when they sent it out to peer review because in Germany it wasn't happening, and he was like, What are you doing questioning my work? Yeah, interesting.

SPEAKER_00 4:16

Well, the the preprint process now is a little bit like what they used to do. So before it gets published, you put out a manuscript and you get everybody to look at it.

SPEAKER_01 4:24

And just going on that point, I find it really interesting as well. All of the early psychoanalytic authors, including Freud, basically used to just write down their ideas and their theories and then just publish it. And, you know, that's amazing. And when you read Freud's work, partly you'll see a line of genius, which is like this is amazing, and then the very next line you'll see something which is really um sort of unscientific, you know, potentially. So but sometimes I think you know, those were the days where you just literally just write whatever you wanted and sort of get it out there.

SPEAKER_00 4:54

Well, they say that he was essentially writing the map. So if you don't know where the Virgin Islands are and you approximate as the first person to write the map, somebody else then can come along and say, Well, no, you got it wrong. It's over here.

SPEAKER_03 5:06

You do read some things where you're like, gee, you really needed a good editor. They just didn't do that at the time, did they? But anyway, we've we've gone off track. I'll pull us back. I guess then moving on from that, eventually, we did start getting the biomedical journals and started making things a bit more formal. Um, and that followed then more of the general journalism model, and and the journal editors viewed themselves primarily as educators, which I think is a bit where we're heading towards now. Though different journals also developed different review methods and selection criteria for articles. You can no longer just plonk in what you want to certain journals. Though these are often subject to bias, um, such as personal preference of the editors. And then finally, following World War II, biomedical publishing became increasingly specialized and our modern peer review process materialized. We've I don't do any of you guys know the number of different journals that are around at the moment? There seem to just be hundreds and thousands.

SPEAKER_00 6:05

Well, I know that PubMed has 31 million individual articles, so that's the main repository of peer-reviewed journal articles. So that gives you an indication. The problem is that many of the journals that aren't in PubMed are just complete rubbish. So the paper mills. I mean, as Ed was saying, you can actually, in modern times, just start your own journal. There's no way of uh preventing it. But things like PubMed and and other web of sites and so on, they determine what the evidence is that you've got good results from your peer review.

SPEAKER_01 6:37

I think the other way is by looking at the impact factor. So I think just off the top of my head, like the median impact factor is I think even less than two. So that tells you that that the number of journals above two is really, really small, and the majority are in that have like zero point.

SPEAKER_00 6:53

Yeah, I think it's actually less than one. Yeah. Yeah. So it's the ratio of how many times you cite versus the number of times you are cited the articles in your journal.

SPEAKER_03 7:02

I remember when I was doing my PhD, one of my supervisors was very keen that I only publish in journals that were above a certain impact factor. And that they were a reputable one. But yeah, whereas I get all of these predatory journals emailing me six times a day asking for submissions. I guess now that we're talking about these journals, so there are predatory ones that send me emails many times a day asking me to submit things to things that I am absolutely not in any way, shape, or form in their field. I think I had one from journals of surgery asking for things or opsalmology. So the bigger versions of the journals. And the way I think depending on the level of submissions that a journal gets depends on their peer review types. For the really big ones that get lots and lots of submissions, then they will select in to publish. And so they get so many, they'll end up getting rid of 90% of the applications, and it's really difficult to get a publication in there, but it's good if you can do it because they're bigger in the impact factor, you're more likely to be read, you're more likely to be cited, and the wheel of academia turns. However, reviews for them tend to be pretty very thorough but also quite time consuming. Whereas for some of the smaller journals, it can be more of a screen-out approach for submissions. And so most submissions really are accepted as long as they meet minimum ethical and scientific standards. And that can be good because there's a quick turnaround. And it can be useful for publishing repeat or routine kind of studies, for example, pharmaceutical trials. Peer review also differs based on the type of submission. Empirical research papers and systematic reviews typically qualify for peer review, they really should. And opinion pieces, editorials, letters, correspondence, book, media reviews, bereavements, and sometimes more creative work such as poems, stories, and personal reflections may not be peer-reviewed, depending on the journal. But they can be screened, edited, and reviewed by the chief editor or by other specialty or deputy editors.

SPEAKER_00 9:11

We should probably point out that all of the articles in Australasian psychiatry are peer-reviewed except for the editorials at the beginning of the uh journal. Thank you, Fiona. Look, I'll discuss now the the uh different models of peer review. And there are in general three types of peer review models. The simplest model is the one where each submitted manuscript is managed by a single editor, either an editor-in-chief, so that's the uh man or woman at the head of the pyramid, or a deputy or specialty editor, so the people who work under that level of the editor-in-chief. So, what editors do, they they scan for excessive spelling and grammar mistakes, they confirm that submissions match the journal's scope, and they address issues of significance to the journal's readership. So Australasian Psychiatry has a focus, for example, on clinical and organisational aspects of psychiatry, whereas our sister journal, the ANZJP, Australia-New Zealand Journal of Psychiatry, probably has more of an academic focus. So we may get a very academic paper and suggest that they try and publish it in our sister journal. Editors will then either reject a submission or select it for review and send it to multiple reviewers. Final decision is always made by the editor-in-chief or a proxy for the editor-in-chief, but editors will make recommendations. The process is traditionally blinded, meaning reviewers are usually not told the identity of submitting authors. This is not foolproof, and um I only recently found out that Australia and New Zealand Journal of Psychiatry, for example, has blinding for the people submitting the manuscripts, but not for the reviewers. So there's a difference there. Australasian psychiatry blinds both reviewers and the authors. Now reviewers assess manuscripts against standardized criteria. These are different for each journal, and reviewers will then provide detailed feedback aimed at eliminating errors and improving qualities, such as the clarity, concision, and accuracy of the data reported and conclusions reached. The process can take several months. One of the things that you judge a journal by as an author is how quickly you can get through that editorial process while maintaining standards. One of the ways you judge a journal is how long it takes to get uh your article submitted through the whole process of editorial, uh that all of the editorial processes to a final decision and then I mean hopefully of course being published. So uh uh as an author, uh as we all are here, we want our stuff out and to the public as quickly as possible, basically. And and so that that is an element of quality of a journal that it can do things in a short period of time. But again, the trade-off is is it still a high quality journal?

SPEAKER_01 11:51

Just going back to the reviewing process of the manuscript, which I think we'll talk in more detail about shortly, but one of the things I find as a reviewer is I often critique the methodology of a paper and whether that fits the author's own aim within the paper. That's one of the more simple ways of critiquing a paper, I think, which is potentially more objective because you're measuring what the authors are stating that they're doing by the methods that the authors have chosen themselves. And that can be sort of more or less independent from the knowledge of the actual subject area, so whether it logically flows from what they're doing, because often people will have a particular aim or an outcome that they want to achieve, but they don't actually answer that question or they don't go about answering it in a way that makes sort of any sense, and they often end up answering a different question, and you don't necessarily need to have expertise in that subject area to see that it doesn't actually fit with what they're trying to do.

SPEAKER_00 12:44

Yeah. I I look at the article as telling a story, and what you want ideally is an introduction that introduces the background, then a clear statement of why you're doing the study. So that hypothesis usually, my hypothesis is treatment X causes outcome Y. If that's clearly stated, then as you say, the methodology really should be fairly easy to match to testing that hypothesis. So it's a really good way, as you say, a quick way of determining how uh high quality the article is in general, even if you don't understand the particular treatment or whatever it happens to be. From the article again, a general outline of the review criteria. So um we've adapted this from a from another paper. There are a number of different domains that we use for the review. What is the importance of the research question? So particularly do the authors contextualize and frame their research within a pre-established and valid research context? Have the authors addressed issues such as cost-benefit analyses and other ethical justifications? So is it reasonable for them to be doing this particular study to work out the data that they want to get from the study?

SPEAKER_01 13:51

And also, does that research question match the scope of the particular journal that you're you're reviewing for?

SPEAKER_00 13:56

Well, that's right. So Australasian Psychiatry, for example, has some fairly strict word limits, 2,000 in particular for opinions. We've now increased uh to 3,000 for things like health policy research, but other journals have sort of up to 5,000 or even open word limits, and you can fit a lot more in.

SPEAKER_01 14:12

Many many people review for more than one journal, and so one article in one journal you could review, but the same article in another journal you might not review, or you might review potentially differently based on the sort of aims and objectives of the journal as well.

SPEAKER_00 14:25

Look, another domain determining the uh the review criteria is the originality of the work. So have the reviewers conducted a brief systematic review of other literature confirm originality? That's generally in the introduction uh to the work. Have they established this is something we should be doing? It's of general interest to people, or it's of interest to a specific group of specialists. What are the strengths and weaknesses, particularly methodological and ethical? So can the methodology be appraised using an established standardized criteria? And the consort guidelines for reporting randomized control trials are an example of that type of formal framework. One of the ones we struggle with, particularly when we get in oscillation psychiatry, we do draw quite a lot of uh authors from the Asia Pacific region and from Africa increasingly. And of course, English isn't always the first language. So we sometimes get people who have a really good research idea, but it's not all that clear, they haven't expressed it clearly, and that can sometimes be a bit of a barrier. Um, what we'll often do in those cases is to say, great idea, but but we recommend that you get a copy editor to have a look at it to put it in the right format. So presentation and and clarity is uh very important. Do the spelling, grammar, and formatting mistakes actually affect the interpretation of the data and the argument?

SPEAKER_01 15:41

Sorry, and there's nothing more frustrating as a reviewer where someone's someone has a like a nugget of a really good sort of research question, or there's some legitimately good findings, but the formatting um or the layout of the paper or the the English or the grammar is sh is not up to publishable uh sort of standards. It's like you really feel for the authors that you want to help them sort of get it to a level where it's the complete package.

SPEAKER_00 16:05

Exactly, yeah. It can be quite a fun process. So you can suggest phrasing for people that will improve the clarity, and um what's most helpful often is reducing the clutter, the excess verbiage that doesn't need to be in the article. Um that's quite good fun. I uh even as an author, I quite like uh wielding the red pen and cutting out all of the rubbish from the article. So interpretation of results then is another important criterion. So have the authors justified the way in which results are interpreted and reported, including by addressing biases. That can be bias in the data, but it um also can be conflicts of interest that they have their own previous research, their positions on particular issues and so on.

SPEAKER_01 16:43

I think to point out as well that you know most, if not all, authors have their own biases or their own you know perspectives or or limitations. So it's not about not having a bias or not having a limitation, it's about declaring it and and verbalizing. I think that's the most important thing.

SPEAKER_00 16:59

Yeah. Look, impossible to be unbiased, but as you say, you need to recognise it and state it. This is this is actually a very good indication of quality. If people are identifying their conflicts or their biases and telling you about them up front, that's a good indication they've thought about the process and are likely to be doing things in a high quality way. So another criterion, then have the authors address the limitations of the study and uh considered how the limitations may affect the results or their interpretation of the results. Again, this can be the sorts of evidence available, it can be limitations of finance and that sort of thing. And then finally, does the submission require minor or major changes? So we have a number of different options as reviewers and as editors. We can outright reject an article and say not appropriate for our journal or the standard is not high enough and we don't think it's worth trying again. We can say we're going to reject it now, but we recommend that you change it in some way and resubmit it. We can suggest we think this is a good article, but we need we think these uh revisions are needed, and that can be minor revisions to spelling or whatever, but also could be more major revisions, that is, you need to change it or report something more about your methodology. And then finally acceptance. Usually we go through at least a couple of uh iterations where the authors will send in their manuscript, we'll send back some suggested revisions, they'll send back a revised version, and it might go out uh a couple of times to the reviewers, and eventually then you say yes, we accept it. It's at the standard, both in terms of the content, the research that you're reporting or the opinion that you're stating, but also in the presentation and the grammar and the spelling and those things.

SPEAKER_01 18:39

And the other thing to point out is that the submitting authors don't have to agree with what the perspective of the reviewers are. So a reviewer can have a point of view and the authors can either rebut that, and that can be either successful or unsuccessful, and at the end of the day, if the publishing authors really don't accept or agree with the perspective of the reviewer or of the journal, and it's been sort of attempted to be mediated through other reviewers, etc., then there's always the option of another journal as well for the reviewers. So it doesn't necessarily mean that for the submitting author, so it doesn't necessarily mean that what they've said is wrong, it just might mean that another journal might be appropriate.

SPEAKER_00 19:20

Yeah, well, and look, it is it is a bit of a conversation. Many authors will just accept whatever the reviewers say because they perceive that that's the easiest way to get to publication. But it's often better if there is something that you, as an author that you disagree with from the reviewer, you just say, Well, I'm not changing it in the way that you've suggested for these reasons. Well, editors have the final say, and they are taking advice both from the authors and from the reviewer. So uh if the author has a compelling case, the editor can say, Well, I think you actually have answered the question of the reviewer, and I agree you don't need to change that in the way that they've suggested. So, yeah, it's a bit of a conversation. It can be quite good fun.

SPEAKER_02 19:57

A related situation is often papers have multiple reviewers, and sometimes there can be quite different perspectives from each of the reviews as well. So the author then needs to sort of find a middle ground and sort of work out which perspective is most helpful to them and navigate that situation as well.

SPEAKER_00 20:12

Exactly right. Integrating different views, it's almost a psychoanalytic, a psychodynamic situation.

SPEAKER_01 20:16

I think it's much more conversational than sort of people realise, particularly junior researchers that that might be submitting authors. There's actually much more scope for conversation and justification of what you're arguing rather than just sort of accepting de novo what a reviewer says because they're a reviewer or they might be more senior. I think that's the whole point of this sort of conversation.

SPEAKER_00 20:38

One of the coolest things that's happened for me as a reviewer is being acknowledged in the the article that you've been the reviewer of because something you've suggested has triggered something that they've put into the article.

SPEAKER_03 20:48

It's a learning process, isn't it? Learning how to deal with reviewers. I remember as a junior researcher with my first kind of publications, my supervisor would be the one dealing with it, and then when I got more senior than I got allowed to deal with it with them kind of checking in over my shoulder that I was doing the right thing, and now as I'm doing it, it's just me just responding to reviewers, and that's quite satisfying when you go, No, I'm not doing this for X reason, and then they go, Okay, well, that's fine.

SPEAKER_00 21:11

Well, and this is how you gain confidence as a researcher. You realise, well, I'm an expert in this area, this is another expert, we have a difference of opinion, but my view is as valid as theirs if I can come up with the evidence for it. Yeah, so it's a good process. All right. Look, while peer review necessarily involves a degree of intersubjectivity and intuition, which is informed by reviewers' opinions and their personal experience, the process ideally should aim to be as objective as possible. So you don't want personal comments and so on in in uh reviews. This does sometimes happen. That then is the job of the editor to say, well, that's not appropriate. You usually that will come from very hypercritical reviewers. It's fairly rare, but it's something that can happen.

SPEAKER_01 21:53

And you want good rigor in in your argument, so you don't want ad hominem attacks or straw straw man attacks and those.

SPEAKER_00 22:00

And feedback in reviews should typically justify the reasoning behind the suggested changes, including referencing other papers, signposting areas of uncertainty, and acknowledging the subjectivity or opinion within the response. This doesn't always happen, but but uh certainly can be appropriate in certain circumstances.

SPEAKER_02 22:17

Carrying on from what you just talked about there, Andrew, I'm going to focus on the selection process for reviewers for this first model of peer review. So reviewers typically come from a pool of professionals with a high degree of knowledge about the field as relevant to that particular journal. Peer reviewers are volunteers, so they're not paid for their time, they're doing it out of a sense of contributing to the scientific process and also a sort of a moral obligation of when we submit papers, we want them to be peer-reviewed for ourselves, so we then do that for other authors as a sort of pay it-back kind of process. So reviewers are often recruited over time through invitation by the editor or other reviewers as well, either through word of mouth or identified through their relevant publications, past submissions to that journal, or even sometimes through internet searches. So each journal will have different criteria about what they expect their reviewers to be in terms of qualifications and experience. Often the expectations are holding a relevant qualification such as a PhD or medical degree. Sometimes it's also about having particular expertise in a different area, such as with clinical expertise, particularly as relevant for journals such as Australasian Psychiatry that have a bit more of that clinical focus. The editors will then send out some review requests to a group of people and then keep doing so until sufficient reviewers accept. And sometimes if the pool contains no available volunteers with the required expertise for a particular article, it might be necessary for editors to invite reviewers with less expertise in that specific topic. And certainly this is one of the biggest bans of the existence of editors is finding appropriate reviewers who have expertise and who are available and willing to review that particular paper. It's a big challenge. It's not always a bad thing to involve a reviewer who has less expertise in the specific topic of the paper, particularly if there are multiple reviewers involved. Sometimes a more generalist reviewer might be able to provide a useful perspective for the average reader of the journal for that particular paper, so that that can be quite useful. Most papers will only require the attention of an interested professional with general knowledge about the domain. They would, of course, need to have familiarity with the standard organization of a scientific article and a reasonable ability with scientific writing as well. And certainly coming back to our sort of focus on trainees, junior doctors can provide a really specific type of expertise in reviewing articles about the experience of training. So certainly there is a role for trainees to be involved in this process, and it's a great time of your career to start, a bit like what Ferni were talking about before about working with someone more senior and then building up those skills as time goes on.

SPEAKER_01 24:55

I think junior doctors or junior researchers, junior reviewers tend to be more thorough in their review as well because they're learning the process, so they want to get it right in their own mind as well. So the reviews actually tend to be much more rigorous than senior reviewers who might be operating off pattern recognition or similar sorts of processes.

SPEAKER_03 25:14

I think that's probably related. There's a huge imposter syndrome about it as well, isn't there? Like the first time I started being asked to review things, I was like, oh my goodness, what do you mean? No, I can't possibly do that. Who am I to review these papers? And then you start doing it and you realise, oh, actually, no, this is fine, I can do this, but until you start doing it, it's it's this big rarefied ivory tower that you don't know how to get into.

SPEAKER_02 25:38

Yeah, absolutely. So Andrew talked through the primary model of peer review earlier, but there are a couple of other models that might be worth covering as well that are less common but also useful to be aware about. A second model of peer review is done in House. So this is where the editorial board reviews all submissions that come to the journal. So this method therefore selects from the same set of board members to conduct all reviews and would only rarely outsource to external reviewers if there's a particular area of expertise that's being sought. So this process is much more time consuming because there's a smaller group of people doing the work. Sometimes the reviews would therefore have less detail and it requires board members to have a high level of expertise and sufficient breadth amongst the board member group to be able to review all the papers that will be submitted to that journal. So one example of this is the Cardiac Journal Circulation, which has used this method in the past. A third model of peer review is available to some of the larger and better resourced journals, such as the Lancet. This is where the journal has a professional editorial team who are employed on a full-time basis to review all articles that come through to that journal. This system can still utilize external reviewers as well and may act as a bit of a hybrid between some of these other models of peer review. But yes, they're able to have the staff to do that in-house. So talked about the Lancet being one example of this category. Other journals such as the British Medical Journal, the BMJ, employ this method but also require final review by the editorial board before publication.

SPEAKER_01 27:13

And I think you touched on this just earlier, but for smaller journals that uh don't have an employed salaried staff like Australasian psychiatry, it's actually really hard to get reviewers because reviewing it's fun, but it's it it is time consuming, particularly when you're still learning the ropes. And to do a good review takes time, and you you don't get a lot of you know credit necessarily for doing it. Um you don't get paid to do it. So you do it really because you're interested in the work and you're interested in the research process. So finding volunteers who will contribute in that manner is actually really difficult. Yeah.

SPEAKER_00 27:47

It's one of the most valuable things you can do as a prospective researcher, seeing how other people do research, but particularly the mistakes that they make along the way, or not even necessarily mistakes, but how you can improve upon the process and uh do your own research in a more professional or more smooth way.

SPEAKER_02 28:04

So, regardless of the model used, the editor plays a really crucial role in resolving disagreements between the reviewers and the authors. Sometimes this is by uh soliciting an additional reviewer, particularly if there's quite a difference in opinion between the original reviewers. Sometimes it's about um collating all the information and making an executive decision. Most journals would specify how disputes will be mediated, which can be helpful for authors to have transparency. And sometimes, particularly if a paper is of a high technical standard or has really complicated statistics, sometimes an expert review might need to be involved, and particularly the better resource journals often have access to statisticians who are able to do a review prior to publication.

SPEAKER_03 28:45

Which is fantastic that that's a thing, and we have our own statistical editor at Mass Relations Psychiatry, which is great that he's around. Yeah, we're very lucky in that regard. I might uh swing through and and talk a bit more about the limitations and the advantages of peer review, but we although we've been talking about that as we've been going anyway. I think we can all acknowledge that the the peer review process is imperfect, but at the moment we don't have any better alternatives. It's peer reviews often criticized as slow, and I think a lot of the slow can be in trying to find appropriate reviewers. Certainly I've found that. And then waiting for the reviewer to get back to you because we're all busy academics and in at our level for the reviews in in you're not getting paid, and so people are doing it in their own time often for for the love of academia rather than any other more concrete uh reimbursement. That way it's slow for other journals that have the money, it's an expensive process to go quicker. It can, yeah, time consuming, it's it can be pretty subjective. We try not to be, and as a reviewer, I try not to be. But we all do have our own biases, and if you don't have a good idea of them, it can be much easier to fall into those patterns of going, well, I know I don't agree with this, so no, which is not scientific and not appropriate, but can happen.

SPEAKER_01 30:03

I was just thinking the the quickest review I've ever gotten was within 24 hours, and it was a rejection, and it was from the Lancet Um Digital Health, which also happens to have an impact factor of 30, which is by far the highest impact factor that I've tried. So that was the quickest review, and then the second quickest review was a journal, I think, with an impact factor of 10 or 15, and that was that was within about two working days, so which was also a rejection. So I'm sure there's a correlation there between impact factor and speed of rejection.

SPEAKER_03 30:31

Well, if they're rejecting 90% of their papers and if they're paying people to just sort it out, yeah, absolutely. There's another thing which some people worry about sometimes with sending things out to peer review, is that unscrupulous reviewers could use the opportunity to go, hmm, that's a good idea. Sit on the paper for a while and then reject it and then start their own research. It's something that you hear about apocryphal. I don't know of anyone that it's happened to, but it is it is one of the worries of peer review. And then I think when we're talking about the different ways of doing it in some journals who actually will list the reviewers, or that can be less likely to happen if people know who's doing it, but in blinded ones.

SPEAKER_01 31:15

I think that is an established fact that does happen, unfortunately, particularly for career academics that have a whole other set of um you know motivations and pressures. That is something that unfortunately does does happen, is idea stealing, and that's something that's in the literature as well, written about.

SPEAKER_03 31:30

And then sometimes even though it does go through peer review, it doesn't mean that necessarily the material that comes out at the end is going to actually meet adequate levels of reliability and and relevance. And you spend lots of time on things and it still won't get through, or it it does get through, but it possibly shouldn't get through. We're also and where we talk about our ivory towers and me being well aware of how privileged I am in many, many different ways. Peer reviewers are are also overrepresentative by people from high income countries. You've got to have enough income to be able to afford a roof over your head, and food on the table and have a bit of spare time to be able to do it to review papers. There's a lot of people who can't. And so it we then can end up perpetuating our own views rather than widening it and making it a more equal process and getting other ideas from other people. Yeah, along those lines of gazumping ideas, a lot of authors argue for that process to be more openly accessible and that's the bad stuff. The good stuff. It improves the overall quality of the published work. There is nothing that I have submitted that's gone through peer review where people have told me to change it, where it's made the paper worse. They've it's universally made the paper better in many ways, even though it's a can sometimes be a painful process from an author.

SPEAKER_01 32:48

I I agree with that. And I think from my experience, often you know you work on a paper for a reasonable amount of time, you know, at least two or three months, you know, you know, often longer than that. And by the time you complete a paper, you've kind of been immersed in it for so long and you read it in a particular way because you've you've been doing it. When you give it to someone else, they aren't seeing it quite from that perspective, and they can look at it and spot mistakes that you just haven't been able to see. And so it's really frustrating when they say, actually, change this, this, this, and this, and you think, ah, I've been working on it for so long, I just want it to get out there.

SPEAKER_03 33:21

I'm so sick of looking at it. I'm so sick of it, yeah.

SPEAKER_01 33:23

But then when you step back and actually make the changes, I agree. I think that the overall result is for the betterment of the paper.

SPEAKER_02 33:29

And also then when the peer reviewers endorse the paper for publication, that's quite a good thing as well. It means that there's been that endorsement of your work and it's been recognised by peers as being worthwhile to be out there. So it's nice to have that um acknowledgement.

SPEAKER_03 33:42

And there's quite a bit of literature behind that in big ones like the Lancet and Annales of Internal Medicine, all of saying the same thing with it been through peer review. They're readers think they're better, authors think it's better, and the ones that the peer review responses often help with too much or too little information, inaccurate or misplaced information, and with structural problems. It really does help with quality of papers.

SPEAKER_02 34:07

It's worth talking a little bit more about blinding because that's an important part of the process and something that different journals do in uh in different ways. So blinding facilitates honest and critical reviews, but as we've just talked about, it may make idea theft more likely as the submitting authors don't know who's actually reading their work. Other journals prefer a more open review process, sometimes where both the reviewers and authors know each other's identities, and some journals go further still and they even publish the names of the reviewers on the paper. One to sort of acknowledge the reviewer's time and work, but also for complete transparency. This has been argued to reduce bias throughout the process.

SPEAKER_01 34:44

Personally, I think that's really good because when I read a paper and I can see who'd reviewed it, I immediately understand the perspective of the reviewer because I know I can look up what that person has published on it, and it really makes the paper a much more richer kind of experience.

SPEAKER_02 34:58

Yeah, absolutely. And and some journals go a step further even and they go for full transparency and publish the full transcript of the peer review process as well. So then you can actually see what the peer reviewers were thinking about, what concerns they had, what suggestions they had, and how that shaped the final paper.

SPEAKER_03 35:13

That would make reviewer number two much kinder, I imagine, if that was published.

SPEAKER_02 35:18

Absolutely. Yeah. No, there's an RCT by the British Journal of Psychiatry that found that compared to anonymised reviews, transparent reviews were higher quality, more courteous, but they also took uh longer time to complete, and that named reviewers were more likely to recommend publication rather than rejection.

SPEAKER_01 35:35

And in papers that I've read that where you can read the comments, it's I I found it really humbling to see just even you know well-established authors and researchers to see where they've fallen down. And actually some of the comments are are similar to feedback that I've got. You know, you haven't put that there, you've missed that out, should you get rid of that? And so it is it's actually quite humbling to be able to see the process.

SPEAKER_02 35:56

One interesting thing about that though is it's actually harder to find reviewers to participate in that process. I guess there's something about having your name attached to an article that you didn't write that some people might feel a little bit uncomfortable about, because obviously that's seen as an endorsement if the paper gets published.

SPEAKER_03 36:09

So And maybe a little bit of the imposter syndrome as well, hey, like if my name was published as a reviewer, then will people think, what the hell is she doing reviewing this article? What expertise does she have?

SPEAKER_02 36:19

Yeah, absolutely. Yeah, and it's a sort of a fascinating area, and it's yeah, interesting that so many different journals take different approaches, which kind of speaks to the next point as well. There are a few novel methods that are out there that are trying to address some of these limitations of the traditional peer review process. So one is a collaborative review, so this is where different reviewers collaborate on a single report. I think one example of this is the Frontiers group of journals. They have sort of an interactive collaborative review process on their website. Another one that's becoming increasingly common is uh pre-print publications. So papers are submitted to an online repository before formal review, which allows early access for other researchers to read them, and then also opportunity for academic scrutiny and for comments to be made to hopefully improve the eventual published version of the paper.

SPEAKER_01 37:07

And having published a preprint, I think that's actually a really good system because it also lets other people see the work that you've done so that they don't duplicate work and also they can start to build on some of your findings which might be salient before, even without having to go through that whole sort of review process.

SPEAKER_02 37:25

Yeah, absolutely. And then the sort of final novel method to mention is post-publication review. So this is where the paper is published as is at submission, and then experts then subsequently survey the paper after it's been published and add comments, and there's sometimes opportunity for authors to then resubmit a new version, building on those comments as well. So there's no perfect way of doing this. There's different sets of limitations to quality and transparency from all these kinds of approaches, but it's just really positive, I think, that people are thinking about how to do this process in different ways and trying to make it as useful as possible.

SPEAKER_00 37:59

Thanks, Michael. As a final section, we'll talk about tips and tricks for new reviewers. So as as in most areas of life, the golden rule is to be courteous, constructive, and to do as you would be done unto. So we've talked a little bit about being a new author and the impact that reviews can have. But it's probably true of all people who put in their manuscripts for critical review. You want people to be focused on the quality of the argument that you're making, the technical things like spelling and grammar and so on, and and not making attacks or implications about your personal characteristics or biases and so on. So as a reviewer, what you want to do is to read the article a few times, to note surface level reflections before attempting a detailed critique, and then to revisit your response over several days to refine what you've written before submitting your review. As with everything, the uh more time you allow to understand the article, the better the review and the better the feedback that you'll give. So a short summary at the start of your review will help reassure authors that you've understood the paper and have a good understanding of their argument, and that may make it more likely that they'll follow your suggestions.

SPEAKER_01 39:07

So, just some other tips and tricks for newer um reviewers. So try to give yourself a lot of time to do the review. So only accept do a review if you've got available time, if you feel uh familiar with the content, and try to give yourself at least 8 to 12 hours per review with setting yourself deadlines and reminders to be able to protect time and complete it in time. Try to follow a systematic process, which you'll probably develop yourself over time as you review more papers, but following the process of the journal that you're reviewing for is a really good way to start. Try to recognise your own individual limitations and acknowledge your subjective judgments or opinions where you make them. And don't be afraid to ask for help when reviewing manuscripts. So it's really helpful to get a senior colleague to provide feedback on your reviews or to compare your review with the review of the same article by a senior colleague to compare what each of you have done. Try to engage with online resources and training, and practicing critiquing articles in a journal club format is another really helpful way to improve your critical analysis of skills. And just remember that junior reviewers often do provide the highest quality work. So have confidence in what you're doing as much as possible, and don't feel like you're an imposter if that's if that's possible, but recognise that junior reviewers do actually provide really high quality work, and that's based on published data as well.

SPEAKER_00 40:36

Well, as you mentioned before, they tend to be very conscientious and want to do a very good job. I might also mention one of the reasons that we've set up the uh trainee editor program at the Australasian Psychiatry is to introduce registrars to this type of process, to do it in a high quality way, and then to provide a bit of a pipeline to the later stages of editorial involvement. So Michael Waitman, who's with us today, uh, has graduated from our trainee program and is now part of the editorial committee of the journal. I think that's a really nice pathway that we'd like other registrars to take.

SPEAKER_01 41:07

So that concludes our topic on discussing peer review. So just to summarise what we've talked about, it's a process that's evolved relatively haphazardly over time, but it's now itself increasingly subject to scrutiny and a demand for higher standards. So we've just talked a little bit about an overview of the process, the types of peer review, and had a discussion about their advantages and limitations. And really just to finish by saying that peer reviewers are recruited from a pool of professionals and everyone's got a different sort of degree of knowledge and experience. But ultimately, the aim is to provide a balance of feedback and knowledge with acknowledging differences in methodological expertise, diverse viewpoints, and also that junior colleagues have a really valuable part to play in this process.

SPEAKER_02 41:56

And just to encourage anyone who has an interest to learn more about this subject to check out our article in Australasian psychiatry that was lead-authored by Ed. Title is An Overview of the Peer Review Process in Biomedical Science.

SPEAKER_01 42:07

So that concludes the podcast on peer review process. Many thanks to Fiona, Michael, and Andrew for your input. Thanks also to our producers David Bill and Nishda Kuma, Shady Dave for our intro music, Sidoni Prentice for our artwork, and of course Australia and psychiatry for the opportunity.