Welcome back to the Thoughts Broadcast, the trainee-led podcast from Australasian psychiatry. This is part two of the podcast series that we're doing on the importance of journal clubs and how to critically appraise an article, which is the latter of which is will be the um content of this podcast. We're also joining you from the RENZCP Congress 2023 in Perth. And here we have um Andrew Amos and Michael Waitman joining us for this podcast. Good to be with you. Hi everyone, pleased to be here too. Good to have you guys. So in this episode, we're going to be critically appraising a paper and in in the effort to sort of demonstrate how we might go about it, particularly in a journal club context. And the paper that we will be looking at is the Kane et al. original clozepine paper from 1988. So the first part of critiquing a paper is actually discussing the rationale for why you have chosen to review this paper or this article in this context. So for instance, maybe Michael, would you like to expand a little bit on the rationale?

Absolutely. Ed. So yeah, with the rationale for choosing a paper, the first thing to articulate to the group is why did you choose this particular paper in front of you? For example, is there a recent clinical scenario or case which led to you to go to the literature for more information? If so, it can be helpful at this point to do a short case presentation to uh orientate the members to the clinical context for the paper. Another reason for choosing it might be that it helps you refresh or learn about a particular area in psychiatry, you know, for example, before an exam, such as statistics, which is obviously examined in the multiple choice exam, so it may be relevant from that point of view. Or the article might reflect a particular intellectual interest or passion that you have and you want to learn more about it and share with the group. So focusing on that rationale is important, and then it's important to discuss how you identify this article, including the search strategy used, whether you'd already known the article prior to reading or whether it was selected specifically for the journal club, and if it had been recommended by someone or referenced elsewhere to give some context about how it was that you came to locate this particular paper. For the purposes of the podcast today, we've chosen the K Natal paper, as you've discussed before, Ed. So we chose this article, you know, as a good example to demonstrate how to critically appraise a journal paper. It has significant historical importance. It's a paper that every registrar will need to be familiar with, would have needed to have read at some point in their training. So if it's not a familiar paper to you as a listener at the moment, then hopefully this is a good chance to learn more about it. And then to answer the second part of that question about how we located it, well, due to its significance, we already knew the name and the authors, and we found it by a simple PubMed search, and it was readily available to download.

Yeah, thanks, Michael. So the next part of critiquing an article is to discuss or to give information about the article. And Andy, do you want to talk a little bit more about that?

Yeah, of course. Well, you start with the basics, of course. You want the title and the name of the journal. So name of the journal, there are journals with different reputations, and that can tell you something about the nature of the article as we were talking about in our previous podcast. Some journals will have fairly obvious biases. Generally, you'll know that before you go in. You want to know when it was published. So we've chosen one from 1988 for historical reasons. But generally, the the more recent a paper, the more it will incorporate uh current knowledge. Then who are the authors? Where do they work? Are they at uh large research institutions? Are they uh in Australia, New Zealand, are they overseas? And what's their background? So you probably wouldn't go into quite this level of detail if you're just doing it if you're in clinical practice, but certainly for a journal club you want to have a look at those pieces of information that can tell you something about the quality of literature that you're reviewing. Is there a funding statement? If so, who's funding that work? Are there any biases that are potentially present because of that? Pharmaceutical companies, of course, have funded a lot of research, but there is a perception that they may have impacted on some of the research that has been published under their rubric. Are there any conflicts of interest? So that might include the funders, but have the authors, in addition, have the authors declared any other conflicts of interest? Are they involved in licensing bodies? Do they have a business interest? Do they have patents? Those sorts of things. And then you can comment on article metrics, which are available through www.altmetric.com. Now the the cane paper that we're discussing today was uh published in 1988 in the archives of General Psychiatry, which is now JAMA Psychiatry, went through a change of name a few years ago as a branding exercise. And primary author was John Kane and uh the Clause Rule Collaborative Study Group. The institutions that are listed under the names of the authors include the Department of Psychiatry, Hillside Hospital, which is the Long Island Jewish Medical Center in New York. The funding was provided by uh Sandor's Research Institute, East Hanover, and the principal investigator John Kane had a grant from the Public Health Service, and then Meltzer provided grants from the Public Health Service as well. And in terms of our list of things that you'd look at, there were no declared conflicts of interest in this paper. So back in 1988, it wasn't realized as much back then that you really had to report conflicts of interest, and so it was less systematically done.

That's the other thing with this paper is that the statistical methods aren't reported either, which is probably more historical thing than the Trevor Burrus.

Well, weirdly, they report Fisher exact tests and nothing else.

Yeah.

Yeah.

Yeah.

And it's interesting that this study was funded both publicly and privately.

Well, not that uncommon. So, particularly in the US, that there's a perception that the government basically funds everything. Some of it's direct to private, some of it's through the universities.

Michael, do you want to talk a little bit about how we would then move on to talk about the paper's background and the sort of context for the aim of that paper?

Yeah, so the next step is then coming to the paper itself. And firstly, seeing how the authors introduce the general background and context to the research questions and aims in usually the introduction section of the paper. So what we're looking for are things like have they conducted a brief literature review around the relevant background literature? Of course, for a literature review article, that will be the focus of the methods as well, but there would normally be an introductory spill summarising some of the important background. We want to try and see if the context is clear. Have they justified the basis and the value of the research question and explained what the importance is of their particular research question? You can use particular ways of structuring this section of a critique. So one common format that's used is the PICO format, which is an acronym that stands for patient or population or problem for the first P, then intervention, comparison, and outcome. So moving to the Kane paper to look at the context here. So in terms of the patient population or problem, this paper is looking at patients with schizophrenia, particularly those who have treatment resistance. So as part of the literature review for this paper, Kane and Tal have established that the efficacy of antipsychotic drugs in short term and maintenance treatment of schizophrenia is proven by double blind placebo trials, but there is a significant subgroup of patients, up to 10 to 20% who do not respond. And additionally, up to 30% on maintenance medication may relapse within a year. So this clearly identifies a problem that needs to be looked at is, you know, there's a significant portion of people who are not getting better with first and second line treatment of schizophrenia. This presents a major public health challenge due to, firstly, the the burden of untreated symptoms for patients and impact on quality of life, intensive use of inpatient and outpatient psychiatric resources, and also the wider economic impact of unemployment and social care costs as well. So there is a significant value in identifying and addressing any factors that lead to non-responsiveness and also to explore different treatment options. So the intervention in this case is that at the time clozapine had been studied earlier, it had been identified as an antipsychotic medication that suggested increased efficacy for treating severely ill or treatment refractory patients. However, there were significant safety issues at the time when granularcytosis was identified, particularly in a Finnish population, and leading to a number of fatalities, and that worldwide there was a progressive curtailing or withdrawing of clozapine from many different markets around the world because it was thought to be unsafe. However, some strict monitoring remained in some countries that enable clozapine to still be used for a particular subset of patients, and this paper was looking at whether that could be, I guess, more safely rolled out in a treatment-resistant population. So the comparison here is looking at treatment responders who are either, if they don't respond to haloperidol treatment, then they're randomized to either clozapine or to chlorpromazine, and it's done in a double-blinded, controlled fashion to determine efficacy as a comparison.

I really like this paper for a number of reasons. One of them, one of them is the historical interest of it, and one of the reasons this became so important is because after deinstitutionalization, there were a large number of people essentially transferred from institutions like asylums, the old asylums with uh large number of people with mental illness, either onto the streets or into prisons and other areas of particularly the US but other countries as well. And so that big social problem then led to the need to develop more effective ways of treating these people. What we realize is yes, there may be some problems with clozapine, but it's extremely effective for this group of people. And then what are the uh control mechanisms we can put in place that mean that we have the benefits of the treatment but don't pay the costs of the side effects? So yeah, I think historically it's a great paper for those reasons.

Yeah. I think what I like about it is that they had a really clear rationale which had value. So they knew that antipsychotics worked for most people, except for this one subgroup, and this subgroup was really difficult to treat, cost a lot of money, and these were the ones actually that stayed in the asylums because they didn't actually get better. So it was a huge public health cost. And at the same time, they knew that no single antipsychotic was better than any other, so they didn't quite know what to do. Then there was this kind of peripheral one called clozapine, which had shown some promise, but because it had such severe side effects, people had stopped using it and researching it as well as they could have. So then they thought, okay, well, well, could clospine actually be the answer to this problem if we actually research it correctly, particularly by dosing, as Michael said, dosing the control drug adequately, which was one of the main issues with knowing whether or not clospine actually worked. So it was a really clear rationale, really clear aim, really well designed, and it had huge value and public health interest. So it was kind of like the perfect storm.

Did did your eyes pop out at the level of the drugs that they were using? The dosages. Yeah, they just kept on going up.

They high dosages. Um let's talk a little bit more about the aims. So that's this is what we would do next when critiquing papers. So I'm thinking maybe Michael, do you want to talk about this and then Andy can talk about the methodology, which is quite substantial? So, Michael, do you want to talk a little bit about the aims?

Yeah, so just to focus on the aims briefly, there are a few things that we need to consider for this section of the paper. First and foremost, are the aim or aims clearly stated in the manuscript? Do they match the background and context? So is there progression from the background that they've provided to what they're looking to achieve? And are they of clinical value or significance as defined by the background of the paper? So coming to the K-Netal article that we've been talking about, this study was designed to test the comparative efficacy of clozapine in patients with schizophrenia who were inpatients who had established treatment resistance, both historically and at the time of the study. The treatment was up to six weeks due to the risk of agranulocytosis being greatest during that six to eighteen week initial period, and previous studies showing clozapine effectiveness up to this point.

So the next part of an article critique would be looking at the methodology use, which is incredibly important. And Andy, do you want to talk a little bit more around methodology?

Yeah, so the first question: what is the methodology? There are all sorts of different designs that you can use, particularly for clinical trials, and those have changed over time. Another reason this is an interesting paper for the uh historical methodology. And do the authors justify their use of that? Very frequently, of course, the justification is this is a standard protocol for doing a clinical trial. We've used the standard approach. But if there are problems, this is a particular population, for example, we might make a variation on that, or we might might try and avoid a problem by using a variation on that methodology. And is that justified by the authors? So if the article uses a qualitative approach, does it address the specific theoretical perspective use, such as interpretative phenomenological analysis or grounded theory? So these can have some fairly philosophical bases and you want them to be very clearly explained in the context of why this particular methodology has been chosen. There are drawbacks to the qualitative approach to research, but that doesn't mean you can't use them, you just need to be aware of them. You also would be interested in whether there have been appropriate quality checks involved in the reporting of the paper. So, for example, has there been a bracketing interview? Are there comments on the coding process? Were there reports of the number of coders involved and how disagreements may have been mediated? These are particularly important, of course, for qualitative studies, but can also be important for quantitative approaches where you want to know things like variations in the reporting of people who've done the assessments for the severity of the symptoms and so on. So, if on the other hand the article is using a quantitative approach, has it described the type of data examined, which might be parametric or non-parametric data? So you've got the hierarchy of the information, the data that you're reporting, and have they justified the statistical test or tests that they have chosen? We have got a lot more systematic with that over time. There are books basically out there that will show you if you have a particular question about diagnosis versus prognosis, what are the tests that you might go to? And then has the report included the various quality and bias checks, such as describing the process of randomization, was the study blinded? Do the data themselves indicate forms of bias? So you can look at the pattern and see if it's reasonably normal in its distribution. And if it's not, you might think there's been something either fishy or just non-standard about the data's being collected, or there might be confounds and so on. So the paper by Kane used a double-blind comparator controlled trial. That was its design. There was one flaw that they didn't really describe the type of statistical analyses used, other than the Fisher exact test, but we probably would give them a little bit of benefit of the doubt given that this is a few decades ago and wasn't necessarily as systematic in its reporting back then. It was a very large study for the time. So there were 16 participating centers, they had 319 patients across those centers. They selected patients based on DSM3 criteria for schizophrenia. So we've had a couple of iterations of the DSM since then, of course. But the DSM-3 really was the first iteration of the DSM that was very much focused on an empirical look at the symptomatology rather than older, less reliable techniques. Now they did rely on some measures that we probably wouldn't rely on quite so much anymore, the the um the BPRS and the minimum clinical global impression scale. So they had to have a total BPRS brief psychiatric rating scale score of at least 45 and a CGI scale of four, and they needed item scores on at least four recognized BPRS scales criteria. So conceptual disorganization, suspiciousness, hallucinatory behaviour, and unusual thought content. So BPRS covers a lot more than just the psychotic phenomena. The criteria that they used to define someone as a treatment resistant or a patient with refractory schizophrenia was that they had at least three periods of treatment in the preceding five years, and that they had had multiple neuroleptic agents from at least two different chemical classes. This is one of the areas that I found interesting. The dosage is equivalent to or greater than 1,000 milligrams per day of chlorpromazine. With the atypical antipsychotics, we have very much moved away from that equivalent dose idea because the mechanisms are very different. They're not necessarily as equivalent. And there needed to be, for each of the previous periods of treatment that they had had on different neuroleptics, they had to have been on those for at least six weeks and not had symptomatic relief from those treatments. In addition, you were classified as treatment resistant if you'd had no period of good functioning within the preceding five years. So all patients that met those criteria were given six weeks treatment of haloperidol, up to 60 milligrams a day or higher, and benstrupine six milligrams per day, in order to confirm that there was a lack of drug responsiveness. So basically they reported they'd had uh three previous treatments of neuroleptics. They were then given an additional six weeks with haloperidol, benstrupine to treat the uh extrapromatal side effects. And if anyone improved by at least 20% drop on their BPRS scale scores and improvement in the CGI down to less than three, they were then dropped from the study. This was thought to be a demonstration that they were not treatment resistant. They'd had a response to haloperidol. So patients who had gone through that haloperidol phase then were randomly assigned to six weeks of either clozepine up to 900 milligrams a day, or chlorpromazine plus benzropine, up to six milligrams. So the reason I'm putting the benstropine in there was they thought that because of the actual parameter symptoms that you get from typical antipsychotics, you would actually eliminate the blinding because you'd know you'd had these symptoms. So the patients were randomly assigned to those two different conditions. All of the medications were coded and administered under double blind, so neither patient nor people administering the medication knew what they were being given. The closopene patients received a placebo tablet which um replaced benzrapine for the uh chlorpromazine candidates.

That was actually one of the logical choices that I quite liked about this study, how they, in order to avoid the bias of having a higher dropout rate with chlorpromazine, they added in the benzotropine. And I guess that comes back to what we were saying about bias in the previous podcast. Like it would have been very easy for them not to have done that and potentially artificially inflated the efficacy of clozepine by ignoring that step. So that's a really important choice that they made.

Yeah, oh look, um, I think as Ed pointed out, having a good design actually improves the power of the study, that is, the probability of getting a positive result, because you improve the likelihood of getting a positive statistical result by reducing the noise, essentially. So the more people that stay in the trial, the more likely it is you're going to demonstrate an effect if there actually is an effect of treatment. Okay, so chlorpromisine was chosen due to having perceived similar side effect profile to clozepine when it is given with benstropine. And a successful trial outcome was determined to be statistical superiority in all three of the outcome domains, that is the CGI scale, the BPRS scale, and in two of the four subscales of the BPRS. So psychotic symptomatology. The evaluation of efficacy was determined in interviews by physicians or psychologists on a weekly basis. So doing all of those scales. And patients were also regularly evaluated in terms of their ward behaviour by the nursing staff using the uh the nosy 30 scale, which I'm not actually familiar with.

Not to say that nursing staff are nosy either.

At the same time, um there was an evaluation of safety done. So this was evaluated by systematic patient inquiry and observation by medical and nursing personnel, and there were regular lab tests, physical examinations, ECGs, and so on. They looked at uh the Ames and the Simpson Angus scale, and at the end of the randomization, 126 were randomized to clozepine, 142 to chlorpromazine.

Thanks, Andy. So after methodology comes results. And Michael, would you like to just talk a little bit more about looking at results, analysing results?

Sure. So the key thing to start off with is what are the key results, what are the most important findings of this study. The results section may be quite lengthy. There might be primary analyses, secondary analyses, post hoc tests. You need to go through and distill what are the most important findings. Often the authors will identify these themselves, and that might be in the abstract or the focus of the discussion section, but it's important not to necessarily just go with what the authors have said. It's important to identify what you think you may disagree with them in terms of what the key findings are, or you might actually be interested in one of the sub analyses based on your clinical question. So finding out what the key results for your purposes are is important. And then based on whether the study is a quality Quantitative study or a quantitative study can then comment further on the various features and presentations of the results. So in a qualitative paper, you might look at whether the themes are presented clearly, you might look at some of the key quotes that are included and reflect on some of these. In a quantitative study, you want to look at whether results statistically significant, and in particular, you'd be looking at whether p-values are reported and also whether effect sizes are provided as well. And you'd want to check to make sure that the results are presented in a clear way. So, you know, this can be aided by tables and figures. So, for example, the use of a forest plot in a meta-analysis is a really good way to get a sense of the overall findings. So there are a number of key findings from the K and paper. So, firstly, 80% of patients finished the six-week prospective haloperidol phase with treatment resistance. So there were 80% non-responders and only 2% responders at this point of the study. In the remaining percentage of the participants, the haloperidol had to be terminated for various intolerances. However, some of these were included in the second part of the trial as they met the retrospective criteria for treatment analysis as well. In terms of efficacy, the intent was to treat analysis for dropouts using last observation carried forward, yielding essentially equal numbers in each group. As we've talked about before, the statistical analysis that they used was not documented in great detail. And the results also focused on evaluation of safety as well. So there was a statistically significant elevated risk of having hypersalivation in the clozepine group, whereas in the chlorpromazine group, hypotension and dry mouth were the more common side effects for that particular arm of the trial. And there were other side effects that were more prevalent but not to the level of statistical significance. In this particular study, there were no reports of serious side effects like agranular cytosis, and there was no difference between the two groups on white cell count drops. And in fact, this occurred in 13% of cloncipine patients and 20% of choromasine patients.

Yep. So there's lots of graphs that were included in this paper, which and the tables of results which we'dn't we won't sort of talk about in detail. But in a journal club format, you know, we might put these up on a slide, for instance, and people can see the tables and see the graphs, and then we can sort of interpret it in that context. But suffice to say that clozapine proves statistically superior to chlopromazine in the measures. And you can see that in the graphs because the confidence intervals don't overlap. And there's a really nice table as well, which looks at the negative symptoms in the BPRS scale. And this shows that clozpine is effective and chlopromazine is not for treating negative symptoms. That's why we're using clozapine to this day, because essentially of those four lines in that in that table, which is really poignant. So following the results, there's normally a discussion. And Andy, do you want to talk a little bit more around discussion?

I will indeed. I just want to make a point about um, again, the historical context is interesting because, of course, as we became more ambitious in our treatments and our goals for patients, that difference between treating the negative and the positive symptoms became much more important because it's the positive symptoms of schizophrenia that other people really notice. And it's the negative symptoms that tend to have the biggest impact on people's actual level of function. So if you're living in an asylum situation, a large institution, all of your needs are taken care of, a severely negatively affected patient actually is relatively easy to take care of. Once people started to have more ambition and say, well, we don't actually just want to reduce symptoms, we want people to be actually engaged in their communities and those things. Negative symptoms then become much more important. And so this paper and clozepine in particular then becomes much more important to improve the quality of life of patients. Look, in terms of the discussion, then, when you're considering a journal club paper, this is the guts, as far as I can see. You do want to be accurate in your reporting of the statistics, but you want to make a good argument based on what you've reported. So the discussion is where all of the interest is. Do they provide a discussion? Do they provide a convincing interpretation? And do they provide a good explanation of the results? Do they then link it with the prior literature? So none of these studies occur in a vacuum, and none of them are definitive. So you might have a positive result after a series of negative results. You would then have less confidence in that paper. You would need to have further replication in order to say, well, we had a lot of negative results, we've had one positive result, we actually need a bit more of uh information. So you need to consider previous literature. Do they go to the level of explaining any anomalous or unexpected results? So this article reports a clinical trial with a very clear purpose and got a very clear answer. We did see a much better improvement in patients with this treatment versus other forms of treatment. But if something unexpected had happened, which might, for example, have been something like a really high level of uh granular cytopine or whatever it happens to be, do they acknowledge that and do they then say, well, this is a big problem or this is not really a big problem? What are the consequences of that? And then finally, do they address limitations within the study? Do they acknowledge, for example, this is a trial with 319 people, largely male, largely veterans, apparently, according to the report? It might not generalize to other groups of uh patients, that sort of thing. So do they acknowledge limitations? Do they then address those limitations? Turning then to the Cain paper, a key feature of the discussion then is they state we believe this is the first time any specific antipsychotic drug has been shown to be superior to another in a well-defined group of treatment-resistant patients who are unresponsive to haloperidol and other traditional neuroleptics. Additionally, given the use of prophylactic benztropine, this cannot be put down to reduction in EPS. So I think that's a very nice, clear, direct statement, addresses their original hypothesis and frames the question very well.

That means that they showed that treatment resistance is a valid construct because um less than 2% of the patients in the in the um uh pretrial phase improved after the six weeks of haloperidol? Yep.

So it confirmed to them that treatment resistance is actually Well, within the context of this um article, of course, this is this is a bone of contention. Can you actually demonstrate that within six weeks? So one of the arguments has been made is do you require a longer period of treatment for some classes of patients? And with the more recent paradigm of early intervention in psychosis, does the the point in the trajectory of the illness determine what the outcome is going to be? So the argument of people like Pat McGorrie is if you can pick it up really early in its course, do something about it, you then have a much better outcome. So that's that would be something that's not directly addressed by this article, but certainly could be discussed as further research. Look, the study does suggest that there is validity to the concept of treatment resistance. However, at several of the sites, many patients who were thought to be treatment resistant had in fact not received the adequate trials of previous treatment as they did respond to the change in medication and thus became ineligible to the trial. So statistically, you could have a look at those patients, add them to the analysis and see if that changes your results.

And finally, we'll look at conclusions that are made by the paper. So, Michael, do you want to round us out by um talking about conclusions?

Yeah, so what we're looking at here, firstly, are the conclusions valid and in keeping with the evidence provided within the study, sometimes authors can be prone to over-emphasizing or talking up the findings, and sometimes that's something you need to pick up with in the conclusion. Also thinking about future questions or directions for the research that that flow on from the study in question, what we're to next in terms of further research endeavours. Also suggesting to the group further readings that might help contextualise the uh discussion in the paper that um that you've talked about today. And then thinking about a bit like how Andy was just describing some of the um criticisms of the Kane paper, have other authors and experts provide a responses to this article in the wider literature that might help us interpret the results, various resources that we can use for that, including the Science Media Centre.

So the the Joinham Moncrieff article on whether antidepressants essentially work or not, or whether depression is caused by serotonin deficiency or not, is a good example of that because there were uh on face value that paper made certain conclusions, but there are a lot of experts in the field which knew a lot of really specific details about that paper more than the average sort of clinician would. That's really helpful to making sense of that argument.

Yeah, absolutely. No, that's a good example. Coming back to the Kane paper, in terms of the conclusions here, I can read from the paper because the authors summarise it well. So they concluded that for individuals suffering from treatment-resistant schizophrenia, the availability of clozapine is, in our view, a useful therapeutic advance. If even a small proportion of these patients can go on to adjust to life in the community with the associated reduced need for long-term institutionalization, this has significance for public health and health financing.

I think as you develop a taste for scientific literature and psychiatric research, one of the things is a sense of completion. And it does depend on having a good understanding of the statistics and those sorts of things. But in the conclusions, you get the feeling that they have anticipated research that they've described, they've got had a good plan for investigating the questions they've asked, they've then answered those questions and they've come to the conclusions that you would have come to based on the previous literature. And over time, once you've read a hundred papers, let's say, you get this feeling, yes, they've answered these questions in a professional way. I understand what they've said. I know that there are some problems and things to follow on from that. And that's really, I think, what we're trying to develop in journal clubs over a period of time, is by having engaged conversations with more and less of uh senior colleagues, you develop that intuition about what is good and what is not so good research. And then hopefully you'll go on and do some of that same research yourself.

Great. So that's essentially the end of the podcast. We really hope that that's that's been helpful, and and we hope that we've been able to demonstrate in a really robust way, you know, how we would go about critiquing an article. And we hope that's been helpful, and you can take some of that into your own practice and practice it in a journal club, not be afraid to try something like this for yourself. So thanks very much to Michael and Andy for joining me on this podcast. We'd also like to thank our producers David Beal and Nushta Kuma, also Shady Dave for our intro music, Sidoni Prentice for our artwork, and of course Australasian psychiatry for the opportunity. Thanks very much.