Welcome back to the Thought Broadcast, the trainee-led podcast from Australasian Psychiatry. And we're joining you from the Congress in Perth. Today we've got with us Andrew Amos and Michael Waiteman. Guys, how are you both going? Great to be in the same room with you guys for the first time ever. Yeah, no, definitely fantastic to be in person and excited for the discussion that we're about to have. That's great. So the podcast today is going to focus on I suppose understanding the importance of journal clubs in medical education and then sort of more broadly kind of examining the sort of editorial and publishing process and how journal clubs might help us to have an independent view on that process and critical view on that process. So there's a few things that journal clubs can assist with in medical education. The first is that they provide a valuable forum for clinicians to update themselves on recent medical and scientific knowledge, and they also help to practice and teach important critical appraisal skills like research, statistics, and decision making and epidemiology. They also help to give a forum to respectfully debate different opinions, which should be grounded in evidence amongst professional colleagues. But I suppose there's an issue that journal clubs can differ in their rigour and also their consistency in their attendance. And I suppose some trainees might notice that maybe they have varying levels of attendance in their journal clubs or varying quality of presentations in their journal clubs. And this is something that's also reflected in the literature. So there's a real modern relevance for the journal club and particularly knowing how to critique an article in a journal club forum and context. So we're going to talk a little bit more around this, and then we we're also going to provide a framework for how to effectively run and present a paper in a journal club. So, Andy, I was wondering to kick us off, could you just talk, I suppose, a little bit more about why we actually conduct research and a little bit more about the scientific and academic process?

Yeah, thanks. Look, Journal Club and the scientific process behind journal club really is aimed at improving our knowledge and understanding. And generally we'll try and develop an understanding, of course, of the areas as clinicians that we're most interested in. And of course, that's going to be patient benefit. What are the things that we can do that will most help patients? And then what are the side effects and other things that we want to avoid? So the scientific method is uh the way that we've developed to do this in the most efficient, effective way. And what we're looking to do then with the scientific method is to submit research or or in other ways disseminate the knowledge that we get from that scientific method. So journals, of course, like Australasian psychiatry, are the most systematic way of getting that information out to people. Generally, in the college, for example, Australasian psychiatry probably is a little bit more focused on clinical information materials, and the Australia New Zealand Journal of Psychiatry is more interested in the academic side of things. What's the underlying basic science to practice of psychiatry and medicine? So having that sort of structure in place provides a structure and a framework for the advancement and the dissemination of that knowledge. People know which journals they want to read, it'll have the information that's most relevant to them. And then things like having an abstract is a really nice way that you can look at an article and think this is going to be of great relevance to me, or I can completely ignore it and move on to the next thing. Look, for a number of reasons, the academic process isn't perfect, and there is a perception that there is declining research integrity. I I'm a little bit skeptical about whether we um are declining in quality, whether our standards are rising. So I think there's always been bad practice in scientific research, but we're probably now a little bit more aware of it, we're monitoring it, and it's therefore a bit more apparent that this is going on. So if researchers and the people publishing that research aren't doing the right thing, that can obviously impact the information as it's going out and therefore mislead people in terms of the value of that information. So it has been argued that uh current models of academic advancement may encourage questionable research practices for various reasons. So people might try and further their career rather than further knowledge. There are a number of different ways that that might occur: selective reporting, switching outcomes, reciprocal gift uh authorships where people basically trade papers with others. And that's why, incidentally, you often see a cast of thousands involved in a paper. In genetic research, actually, you do see a large number of authors. That's actually probably a little bit more uh justified, but people trading authors' authorships on their papers can be a bit of a negative p-hacking. So trying to reverse engineer the statistics that you provide. An example of that might be doing a hundred tests but only reporting one. Statistically that's very dodgy, but it might then make for a paper that looks exciting. And hopefully, in a rare uh number of instances, the outright fabrication of data. So certainly most journals do retract papers for for various reasons, but certainly if people have been caught fabricating data, they'll be forced to retract those articles.

I think there are even some more reputable journals where you have to actually declare what contribution each author has made to the paper. So you have to be explicit with who's done what so you can actually keep track of each person's contribution.

Yeah. Well, I think we're really tightening up a lot of those things and um even having to register your trials is an attempt to get people to state the hypotheses before they report their results. That is intended to prevent the sort of omnibus testing of absolutely everything, but only then reporting a subset.

Michael, I was wondering if you could tell us a little bit more about some of the potential flaws in the research process.

Yeah, thanks, Ed. So I mean I guess that the quality of the research output depends a lot on the decisions that are made earlier in the process with the planning, conducting, and writing up of the research. And unfortunately, you know, it's thought that many research findings are potentially incorrect or misleading due to, in particular, things like low pretest probability leading to spurious positive findings and also biases that are included in the research design, data analysis and presentation as well. And there are a number of other important factors to consider as well in terms of the quality of research. Lots of studies have small sample size leading to underpowered results. There can be issues around prospective tangibility in the study design, which allows more routes for bias to jump in and be part of the of the study. There's quite significant financial and prejudicial biases, particularly with a lot of research being done in high-income countries as well. And areas that the research community focuses on introduces a bias as well. There are some particularly topical areas, such as COVID-19 in recent times as well, which draws a lot of funding and research interest where other areas might be more useful to focus on as well.

And I'm also aware of, I think potentially through exams as a trainee, something called publication bias, which is a particular bias I've heard of a lot. And I was wondering if you could tell us a little bit more about that.

Yeah, so this is also called the desk draw effect. So it's the phenomenon where both researchers and journals are much more likely to pursue and publish research that has a positive or significant finding, and results that are equivocal or negative are less likely to be written up in the first place, and then also more likely to be rejected, particularly by the more prestigious journals out there. So that's an issue because it skews the available data that people can find when they're doing reviews of the literature. Another particular problem in the medical field is that statistical significance is often less important compared to effect size of the result, and the focus on statistical significance tends to ignore that as well. I mean, effect size can be problematic as well, that journals are often much more likely to focus on and publish articles that have higher effect sizes compared to less prominent ones as well. So again, there's there's bias there. As with a lot of things, there's there's a lot of financial factors at play in this as well. So a lot of journals are uh run by for-profit companies. So there's a big, you know, there's some of the most profitable companies in the world, more so than Google and a lot of the tech companies, even. About a month ago, there was quite a prominent incident where the whole board of one of nature's journals, Neuro Image, resigned in outrage at this situation. They have a plan to set up their own journal in competition, which is a positive kind of thing that will hopefully signal a bit of a shift in the wider field. But there's certainly a lot of issues. The other really important one to mention is the issue around the replication crisis in science as well. So many positive findings are difficult to reproduce, and then even the impetus for researchers to go and reproduce findings is is lessened because people want to be the first person to discover something, they don't want to look at something that's already been looked at. So there is a huge issue around findings of literature not being replicated and proven by other researchers in independent situations as well.

Can I jump in? So one of the other things that's become apparent recently is that there are more crusades going on, not just in society but in science. So a number of editors of very large journals have said we're really interested in X, Y, and Z, whether it's climate change, the difficulties to developing nations of catching up to the developed world, those sorts of things. And there is a suspicion that there are some similar biases going on there. Not necessarily that they would be reporting false research, but certainly that they're directing research in a particular direction, and they might be more likely, for example, to reject certain research. I think we saw a bit of that with COVID-19 where things that were going against vaccines, for example, weren't thought to be particularly useful for the public health response. So that would be a most recent example.

That's a good point. I'm just reflecting that it's very hard to, in fact, it's impossible to get for anyone to get zero levels of bias. And I'm just thinking that that kind of highlights how, for example, in a journal club context, which also I suppose should mirror in some ways kind of a journal review editorial process, how it's important just to kind of be mindful of all the different potential factors, you know, as best as we can when weighing up evidence and discerning evidence.

Look, I I think there's a growing realization that, as you say, zero bias is impossible. Being human means that you have certain biases. One of the advantages of scientific research and the scientific method, of course, is that you get, you systematically seek feedback from a variety of sources, and that should overcome your own biases. So I think we're coming to a point where you don't necessarily have to present yourself as having no biases, but be open about what you are trying to do. So certainly, I mean, the editors that I was talking about have been writing editorials about the things that they feel are important in their approach. And that would be a good way of preventing people from reading the journal and not realizing that they have that sort of editorial position.

The other phenomenon that's sort of happening as well is people are less likely to write responses and letters about studies that they find problematic, particularly with the growth of publications that are out there, it's much harder for people to find the time to sit down and write a reply to a study that they find concerning and problematic, and that's an important part of the scientific discourse that's falling behind a bit as well, unfortunately.

Well, and this is where journal club comes in, of course, because it's a local area where you can test your ideas against the knowledge of the people that are in your local area. And then if you have a particularly strong argument, that might be then a good reason to submit that to a journal.

And yeah, I was I was also going to add, do you know why that might be, Michael? Like is it is confidence one kind of aspect of that and people not being confident enough in their own ability to adequately respond to a piece of science, or do you think there are other factors involved in that?

Yeah, I think like you said at the start, with around sort of you know journal club being a good way to build scientific literacy, maybe that is an area that is perhaps needing to be enhanced. I I think also there's a time aspect as well, like we've talked about the publish and perish kind of culture, like people are more interested in pursuing their next high-impact journal rather than writing a letter about a paper that may not get cited or widely reviewed, and and just people having trouble to keep up with the the wider literature as well. It's hard enough to keep reading broadly and let alone finding the time to put concerns to paper and and then go through the whole peer review process about getting that published as well.

It's certainly difficult to keep up with the amount of research that's available at the moment and even mindful of that sort of journal metric factors like the H index, uh which is a way to discern between journals which are more impactful or less impactful, and therefore potentially make your reading more efficient or time effective, even that in itself is is biased. Yeah, it has certain biases associated with it.

There's an eponymous law about that. I think someone has observed that the moment a metric gets invented to measure something, people will game it and it no longer becomes useful. And I think impact factor H index are good examples of that. People can strategize to maximize their scores.

So it's uh sort of mindful that it kind of sounds like we're not viewing the sort of scientific or academic process in kind of good light at the moment. And it's not necessarily the intention, but we've kind of highlighted that it's not a perfect process like any kind of human endeavor, that there are flaws within that. But I'm aware there are some uh ways that these flaws can be mitigated. Andy, do you want to talk a little bit about some of the ways that the flaws are are kind of addressed?

Well, look, as I was saying before, I really think it's actually a strength. It used to be thought that we had this pristine scientific method, and that in itself leads uh people to misunderstanding everyone does have biases. It often is the case in in all walks of life that once you become aware of a problem and you start measuring it, it looks like you've got this rapid increase in the problem, but it's really only because you're aware of it, and it's only when you're aware of it you can do something about it. So within the scientific community itself, there's there's lots of different approaches to this. I mentioned registries, for example, before. Uh there are organizations such as Retraction Watch, which keep track of uh data on article retractions and the reasons given for those retractions. Now, sometimes that can just be a mistake. It's not always malfeasance, but of course it's more serious if someone's done something wrong. So um organisations led by the Royal College of Psychiatrists in the UK have created the Research Integrity Group, uh which advises the editorial boards of the British Journal of Psychiatry, BJ Psych Open and BG Psych Bulletin. So similar to Australia New Zealand College, they've got their own stable of publications. And it's useful to have that sort of oversight to say, well, we've noticed this is a pattern of problem in the reporting or in the uh the biases that we noticed, and this is what we're going to do about it. So they'll oversee the investigations after allegations of poor research in those journals published output. And the journals themselves have also tightened checks on things like the reporting of ethics, the trial methodology, and screening for plagiarism. We initiated this podcast before ChatGPT uh was announced to the world. But obviously, this type of generative AI might lead to a lot of this sort of problem. So there's lots of software that can do things like check to see if any of the text in an article that's been submitted has been submitted elsewhere for other purposes and hasn't been properly attributed. So we're a lot better at looking at those things fairly automated in a fairly automated fashion. But look, it's more important than ever for individual researchers and for clinicians to uh to value and to develop expertise in the critical appraisement of the scientific literature. Clinicians, of course, will have their patterns of work behaviour, but very frequently you'll come up against a patient that you haven't seen before, or let's say a side effect, a novel side effect to a medication. You'll then need to have that ability to appraise the literature to enable you to go and to or to ask a question that can be answered by the research and then to go and find research that will answer that question in a convincing way. And over time you you develop your knowledge by actively engaging with the literature in that sense. So it's not just reading one paper, it's a process of continually refining your knowledge about the nature of psychiatry and the nature of clinical practice. So, look, since the advent of uh evidence-based medicine, the scientific method and academic process can be seen alongside your personal experience as central to the high quality clinical practice and to decision making in your clinical practice, and therefore will contribute to high clinical standards and professional excellence. Learning these skills as a junior doctor is an important part of medical education. It's really the time where time has been allocated to that process. If you don't start early, you probably won't get into the habit of doing it. So I think it's very important for junior doctors to be developing, and therefore journal clubs should be seen as an indispensable tool, both for students but also for junior and senior clinicians to practice and refine their critical appraisal and research skills in the context of discussing new and emerging evidence and uh scientific results about clinical practice.

Thanks, Andy. Michael, can you tell us a little bit more about journal clubs, maybe the origin and sort of formats of journal clubs?

Sure things. So, yeah, I mean journal clubs have been a part of medical culture for a long time. So the first recorded journal club was documented to have occurred in 1835 when a group of students began reading articles in a room above a baker's shop in London. And then since then there have been many further journal clubs in many different settings, and contemporary journal clubs have evolved to typically last around an hour a week or at a regular interval at least in a clinical or professional setting. They include a number of participants, including clinicians, but also the wider multidisciplinary teams in some contexts as well. There can be a range of seniority of members, right, from medical students up to consultants or professors. The topic of discussion in journal clubs can be quite broad as well, so they can include discussions of papers from peer-reviewed journals. These can be either contemporary or historical, seminal kind of papers, and they can also involve relevant books or case studies as well. Ideally, the material could be shared in advance of the journal club, so all the participants can have a chance to read the paper, become familiar with its content, and then bring their thoughts and ideas to the meeting to discuss with the group. Historically, a lot of journal club proceedings had also been published in scientific journals, and I think that goes back to the comment that Andy made earlier as well about certainly if there are flaws or issues that are identified, then that's a good opportunity to, you know, complete the scientific process and potentially write a letter summarising those to a journal as well. And when they're done well, journal clubs can be highly intellectually and professionally stimulating for the participants involved.

I know that there are multiple factors that are associated with maximising the effectiveness of journal clubs. So I was wondering if we could talk about these and introduce them to our listeners.

Before we do that, can I just say, following on uh from what Michael has said, I think there's been some technological innovations that have been driven by COVID, which have actually led to some developments in journal club. So doing it online and using things like a whiteboard that everybody has access to can be a really useful way of recording what happened and then making that available to everybody. So I think one of the tricks to making journal club interesting is making sure that everybody's engaged and then linking it to what happens after the journal club. So if you're looking at a new treatment, for example, so neurostimulation, for example, that sort of thing, or psychedelics, which we were talking about before the podcast, you want to take the information that you gain in that journal club and then apply it into clinical practice. And having a record of what was discussed is really useful for that. Second point I'd make is that this is a really good way of generating scholarly project topics. So a journal club that identifies uh a gap in the literature or a flaw in past research leads really naturally on to a scholarly project. And it's usually manageable because it'll have the right scope for a scholarly project. But look, in terms of the then the factors involved with maximizing effectiveness, probably the most important is to have a very clear purpose for that journal club and for that to be at least nominally accepted by all the people that are involved in the journal club, and then to have reasonably regular or the opportunity to reasonably regularly review those goals and objectives. And that's particularly important to link that to the chosen paper for each of the meetings.

I think that's really important because I think one issue in journal clubs that I've been to in the past is the purpose of the journal club to review a paper sort of scientifically, or is it to present a case and how much time is spent on sort of going over the sort of details of of a case versus actually looking at literature? And so if the purpose of what the time is to be used for is made clear for everybody there, everybody can make sure that they're getting the the most yield out of that time. And Michael, what about talking about the sort of membership of the journal club?

So it's important to have a critical mass of members to make the journal club viable, so enough people to come regularly to generate discussion as well. And also avoiding having too many people because it's important that everybody who comes along has the opportunity to have their voice heard and to contribute. It helps a lot to have a particular person who is appointed as the coordinator for the group, responsible for uh the administrative running of that journal club. That's things like booking the venues, communicating with members, keeping a you know attendance list, maybe disseminating information around what was discussed as well afterwards. And a key thing as well is making sure that there are some people in the journal club with particular expertise in medical and scientific. Literature, so understanding the statistical methods, the research process, someone who's able to help make sure the discussion is accurate and on track as well.

Which can be really hard. So at least having, you know, a senior colleague there, like a consultant, for instance, or or a senior registrar, so that you know accurate knowledge is actually imparted on more junior clinicians, but that can be hard. I've been in journal clubs where just due to staffing and sickness and there's there's only been one sort of senior person, you know, for the whole cohort of students. So that can be difficult. Andy, do you want to talk a little bit about the format of journal clubs?

Yeah, well, I think everyone benefits from having a fairly regular schedule, so monthly, fortnightly, whatever it happens to be. Um, it has to be an appropriate time to maximize attendance. This can be particularly difficult in rural areas. Uh so I've worked a lot in different rural regions, but with a bit of planning generally in the middle of the day, you tend to avoid Mondays and Fridays, but uh, in the middle of the day is is usually something that can be facilitated for most people to the area that you work. You want to encourage regular attendance and ensure that an attendance record is maintained. This is something that can be a little bit tricky. There are always clinical pressures, and I think you have to be realistic about what's possible, but you really do want to be encouraging people. It's better to have pool factors rather than to be forcing people to do it if they don't want to do it. And having an engaging journal club is the best way of drawing people into that. And each meeting generally then will require a leader that will present the article and then lead the discussion. So that particular individual can change from meeting to meeting. Sometimes there are people that uh take that role. I've often found that people will pair a more senior and a more junior clinician, and the senior clinician might define the area, and the junior clinician will actually deliver the material, for example.

There's also some evidence to suggest that catered journal clubs have a higher attendance rate than once that are so bringing food can be an important strategy as well.

Well, um that's a bit of a controversial topic within the uh the college. And in this particular Congress they've uh announced is we're gonna this is gonna be the last one where we've got the sponsored booths because there is such a um resistance to that. But uh yes. Look, if the hospital and health service itself can provide catering, for example, that that's a really great way of people getting uh involved.

You know, I was once in a general club where it rotated around the members of the general club to bring food each week as well. So the burden was sort of shared amongst the group, and there was a roster for the presenter and a roster for the uh caterer.

Just thinking back to when I was an intern and they'd give us uh subway every week to attend the intern teaching. And I think after about a month I got absolutely sick of subway and I'd just probably take them away from it.

Well, I'm a bit older than you guys, and in in my early days we had uh grain-fed chicken breast and various other things uh for from week to week. And there was quite good attendance to those journal clubs.

It's very fancy. I'm just thinking back to the format and just what you're talking about, Zoom or online and how in the service I work at at the moment, and it's probably a common experience, people have to come from multiple different kinds of buildings and locations within a service to a central location to attend a journal club or even rural locations. I'm sure there's lots of scenarios where moving things online, even permanently online, can actually be really, really helpful.

You do have to be careful with that editing. We found in Queensland when COVID first hit, we were doing the online training, and there are a lot of blank screens. So we eventually got to a point where we had to say, everybody, you have to have your screens on all the time so we can see you. And you had to be the presenter generally had to be fairly organised in asking people to be involved in the conversation. Some people will naturally jump in, but you do want to engage people by asking direct questions.

And I'm just mindful, where I trained previously in the UK, attendance was mandatory and was part of the training requirement, which I suppose did have pros and cons, but it did kind of just make it generally more accepted that this was a kind of valued part of your actual training and for psychiatrists in Australia can get CPD points for attending journal clubs as well.

So there's an incentive from that point of view as well.

Michael, do you want to talk a little bit about how articles might be sort of selected and potentially circulated before the journal club?

Yeah, absolutely. So as we mentioned a bit before, the selected articles must be relevant to the group's stated purpose. And it's helpful to get input from the membership for article selection through some kind of system, either a consensus system or using a rotating roster where the presenter will select an article to bring forward to the group. As we've touched on before, a key thing is giving the membership of the group sufficient time to read through the article beforehand so that they can familiarise themselves with it and start to think about discussion points. And as a rule of thumb, around a week in advance is a reasonable amount. Any less than that, then it becomes hard for people to find the time to necessarily read the article. And as we've touched on before, clinical relevance is key. So ensuring that the chosen article addresses a clinical practice question is important. You've talked before about whether that relates to a case or not, but there's certainly important considerations there about why are we discussing this article, why is it useful.

I think it's helpful to have a mix of different articles, which is why having somebody who can coordinate the group is really helpful, because I guess it doesn't always have to be an RCT or even a qualitative piece of research. It's sometimes it might be appropriate to do a book review, sometimes it might be appropriate to examine a therapy sort of case or write-up. Sometimes you can review, you know, a systematic or narrative review of other articles or just an opinion piece. So it doesn't always have to be heavy-hitting statistical analysis or an RCM RCT or meta-analysis, but as long as there's a sort of egalitarian spread of the different types of articles that people don't always just pick a book review, for example. I think that can actually be really robust and helpful for people to have a mixture of things.

I think it's also really useful to have a spectrum of the methodological rigour of the studies. It's useful to see a really well-done study analysed and a really poorly done study analysed. And then the different statistical tests that are used for different types of study is a good if if you do have that oversight mechanism, it's useful to have those different statistical methods. There are some that are not very frequently looked at but um are quite important.

Thanks very much, guys. That will conclude the first part of this podcast where we've discussed the importance and relevance of journal clubs and of having a critical mind when it comes to reviewing literature. Just for our viewers, we're hoping to uh write this up and publish it as well. So stay tuned for more information, hopefully, on this in the future when it gets published. Thanks very much for your for your time listening, and we'll be back with part two of this podcast on how to actually critique a paper where we'll be uh looking at the the original Kane et al. Thanks very much.