Welcome back to the Thought Broadcast, the trainee-led podcast from Australasian Psychiatry. My name is Dr. Ed Miller. I'm the training editor for Australasian Psychiatry and joining you here from Perth Congress 2023. Today we have Nick O'Connor joining us. Nick has been a board director of the college since 2017 and is chair of the college's education committee. In his day job, Nick is clinical lead of the mental health patient safety program at the NSW Clinical Excellence Commission. Great for you to join us, Nick. Thanks very much for being with us.

It's great to be here.

So today we're going to be talking a little bit about the recent changes to the College Centralised Assessment Program, particularly the permanent removal of the OSCE exam. And I thought, Nick, to start with, it might be helpful just to talk a little bit about the history of the college examinations, just to sort of give the changes a little bit of context.

Look, I'd be very happy to do that because I think it's the history is very informative. And in some ways, I think most fellows, having gained their fellowship, carry with them forever their experience of the examination system. And to some extent that plays out in how they conceptualise things should continue in the future. So if we were to go back to my own experience, I sat the clinicals in 1986. And at that time it was a two-day exam. And on day one, there were two clinical vibers, one in the morning, one in the afternoon, one on general psychiatry and one on a low prevalence case, so you know one of the psychoses. And then day two, if you got through day one, and you only found out if you got through day one by assembling at 9 pm at Royal Park Hospital, we were in Melbourne at the time, to receive envelopes, wherein was your fate. And uh it was a you know cold, it was late, and there were some very upset people who didn't make it through to day two. If you did make it through, the envelope contained the hospital where you had to present at 7:30 the next morning on somewhere else in Melbourne that you didn't know previously, and you'd front up for either your medical case or your consultancy viber, and you'd wrote, you know, you'd do one of each morning or afternoon on day two. And I think also in that experience, um probably relevant to mention, there was a cohort of my study group who went, um, and one of us did not get through, and I think there was a general consensus that person was probably the best of us. Um and so there was a sort of puzzlement and a sort of an unfairness, I think, often in in those days as well, that you know who got through and who didn't. In 1992, the regulations were changed. It was still a two-day event. Uh, day one was again examining two patients. Uh in those exams that I sat and also in the 92 regulations, there were two examiners. Day two was a VIBA uh examination on psychiatry consultancy topics, and trainees were required to pass all three to get through. In 2003, we moved to the trainee clinical examination. Trainees were required to complete one observed clinical interview and six OSCE stations. That was when the OSCIS were introduced for the first time. So the OCI examination replaced the day one clinical vibers, and the standard at that point was that of a senior registrar. The exams were held on two consecutive days, and candidates were required to pass both the OCI and the OSCI components, although later on in 2010, candidates who obtained an OSCI score above the cut score were then not required to resit the OSCI. If they had just failed the OCI, they were just required to resit an OCI. 2011-12, the OCI and OSCI components were separated, and candidates had to pass both the OSCE examination and OSCI components of the trainee clinical examination, and they were given three possible attempts at the OCIs, of which they had to pass two. Up till 2015 that was the case, but then in 2015 candidates had to pass uh two out of three OCE. And after that time, after 2015, the OCE became an observable clinical assessment and was only a formative assessment. The OSCIS had a bit of evolution. So from 2016 there were three long stations and eight short stations. 2018 three long stations and eight short stations again, but an additional biostation was added, and there were the concept of active and non-active uh biostations. And then, of course, in COVID, the number of stations was reduced to six in order to sort of make that feasible. So that that's the history.

That's really interesting. And so I guess now could you talk a little bit about, I suppose, the reasoning and the process behind now removing the OSCI examination?

Yeah, sure. So there's really two main reasons that we came at this. The main reasons were the educational reasons, but there were also some logistical issues that were at really starting to knock on our door prior to the pandemic. I'll come back to those logistical issues at the end. So, look, uh when we did the initial ACE, so ACE is the Australian Council of Educational Research, and we got them to have a look at a number of our exams, including the OSCI, back in 2019, and they provided their report in 2020. And they were generally, I think, quite complementary about the OSCE. However, they did note, and also in the feedback to that original ACE report, there were concerns that the OSCE wasn't testing core clinical competencies. And that in fact, since really the OCI had been made a formative assessment, that seemed to be a gap in our fellowship training and preparation. There was also a concern that the OSCI, it's really an exam that can examine relatively narrow areas of competence, although it can do multiple of those, and not necessarily in a truly authentic setting. And finally, the exam's a compensatory one, which means that you can actually fail some very important competency stations, but on average exceed the cut score and therefore pass. And I think that a number of medical educational authorities and regulators, but in particular the Australian Medical Council, have been, I think, really working behind the scenes to be suggesting that consistent with modern educationalist research, colleges should be moving away from high-stake single-event summative sort of barrier assessments. I might add that more recently, Professor Lambert Shuarth, in his evaluation of our alternative assessment pathway, and then with some subsequent work using de-identified college data on our OSCIS from 2016 to 2019, so it's about nine cohorts, I think, demonstrated that the pass-fail sort of discrimination is not as good as everybody assumes, and that that is a criticism that can be, it's not exclusively for our OSCI, it's a property of high-stakes single-event summative examinations. We go back then to the logistical matters that were concerning us about the OSCI. As you know, there is a projected gap in the specialist psychiatrist workforce, and the Commonwealth Medical Workforce Plan has really been emphasizing that we need to be generating and producing more psychiatrists, particularly with generalist psychiatrist skills. So, therefore, we need to be increasing our intake. And in fact, in Victoria, there's a really quite a systematic attempt and funding to try and increase the number of trainees coming into our fellowship program. But we were already pre-pandemic, I think we'd reached the limits of growth. So we were up to offering a third OSCI each year, often with 200 or just above 200 candidates sitting each time. And that's an incredible logistical exercise. It's a costly exercise, and we can come to that later if you like. And it was starting to be beyond feasibility in that we were just starting before the pandemic to get teaching hospitals who had previously been quite happy to give up their outpatient departments for the whole of the weekend saying no. So we had to scrap around and try and find places, which introduces the idea of you know, we may have to pay for a different sort of venue, not a hospital, which would be a further expense. And we were at times scratching to find enough examiners, including reserve examiners for such a big exam.

And I think you mentioned that sort of postgraduate medical education has changed over time, and there's, I suppose, a move away from the high-stakes kind of singular event. Could you talk a little bit more about, I suppose, more broadly about how how medical education has changed and the reasons why it's changing?

Yeah. I think that there's a couple of things here. So I won't speak further about the concerns about the high-stakes single event exams, except to say that the more modern approach, instead of that, to be doing a more programmatic set of assessments, ideally based in the workplace, because that provides an authentic environment in which assessment of real patients and the complexity and unpredictability of real patients is sort of the authentic environment. I think it's really suited to our specialty psychiatry because we have such a central emphasis on the apprenticeship model, and much of our craft, I think, is complex relational interactions between the patient and the doctor, often relying on emotional intelligence and use of emotional intelligence skills, often relying on the doctor attuning to the patient's tone, body language, and sometimes even working with the person in a sort of third inter-subjective space. And I think that's the sort of environment that those skills can be taught, assessed, discussed, reflected on. The other principle I think is that assessment needs to have an element of expert judgment in it. And again, I think that's what our supervisors can bring. There's no doubt that we need to develop better supervisor resources, training and calibration. But I think by and large, our supervisors do provide that expert judgment, and they have been using that, for example, and that's I think demonstrated in our portfolio review in the way that they have scored the ITAS and various EPAs, etc. And I think the final principle of the more modern approach to well, there's two things really. One is that assessment should mainly be for the purpose of learning rather than trying to assess what has been learnt. So that relies on rich and detailed feedback as a sort of constant dialogue between the supervisor and not just your single supervisor, but seniors in the workplace to trainees. And finally, that the modern view of assessment is that you want to take as much rich data from as many work-based types assessments and other types of assessments, including summative ones, so that you've got a really rich and extensive set of data points on which you would make things like a progression decision, progression to attain fellowship, but before that progression from various stages to the next one within our training program.

And you mentioned that sort of a workplace-based approach can potentially incorporate more sort of emotional into subjective sort of data. And I know this is a debate in the literature as well that I've seen about sort of objectivity and subjectivity in assessments.

Yes.

And I suppose could you just talk a little bit more around that and potentially, you know, concerns that some people might have, I suppose, about workplace-based assessments being potentially less objective, say, than a traditional OSCE or a centrally administered examination.

Yeah, look, I think that's a really central question. So I think that we have had a belief till recently that high-stakes summative assessments like OSCI, but not limiting this is not a campaign against the OSCI as such. But those types of assessments are inverted commas objective. And it's often because they produce a number, they produce a score. But in fact, all assessments, including the OSCE and other high-stake summative assessments, involve multiple levels of subjectivity, which I think often we overlook. For example, it's a subjective choice, usually by a committee, which competencies we choose to test in any particular OSCI or MEQ exam, for example. It's a subjective choice what questions we ask. It's a subjective choice, although it's done by a committee of experts, it's still an arbitrary subjective choice. And similarly, the sort of criteria for marking. Similarly, it's an arbitrary choice where you put the cut score. At the end of the day, you get a number or a score, but actually there's a lot of subjectivity behind it. So too in work-based assessments, where we rely on expert subjective judgment. And so in both cases, the OSCI or a summative high-stakes assessment, and in the workplace type assessments, the supervisor or the assessor, their preparation, their calibration is really critical. That is somewhat easier to do in an OSCI, and I think we calibrated our OSCI examiners very well. In my day, when I sat in my OCI's back in the mid-1980s, you know, John Condon, who was uh chair of examinations back then, used to describe the examiner's magical mystery tour, where the examiner could go off track and would clearly have something in mind that he wanted from the or she wanted from the candidate. And the it's sort of like the candidate had to read the mind of the examiner or play a guessing game. That was the magical mystery tour. So coming back, work-based assessment, its strength, I think, is that you would bring together a really extensive and rich set of data points, mainly in the workplace, but also you could add in other summative assessments over a period of time, looking at the same competencies or same content done in different formats in different environments at different times. Often we will have even different supervisors. And that provides the richest information, both in terms of making a critical decision about progression of a candidate, but it also starts to build a narrative both for the trainee and for the supervisors and directors of training or various committees.

But are we losing anything by getting rid of the OSCI, or are there any sort of pros and cons? I suppose there are there any pros to the OSCI that the workplace-based assessment scheme might not have, for instance?

That's a really good question. And we are yet to determine the precise details of what will replace the OSCI. I think as people who attended the symposium yesterday and the stakeholder forum on the 28th of April, we are narrowing down to a couple of options. Both of them are going to, I think, bring in a summative observed clinical interview. It may be that there might be more than one summative observed clinical interview, depending on, you know, one in stage two, possibly one or more in stage three. And I think that that is going to replace reasonably well what we assessed in the OCI, and in particular, address the issue that I think many of us have been concerned about, and that is training and supporting trainees to do comprehensive formulative clinical assessments and you know initial management planning.

So that concludes the first part of this two-part series on the changes to the college's central examinations. Join us in the next episode of the Thought Broadcast for the second part of this interview. Thanks to our producers, David Bill and Mr. Cooman, Shady Dave from Intro Music, to Tony Printers from, of course, Mr. Lations Country for the opportunity.