Welcome back to the board called the Community Little Centre from French Asian 63. Joining you from that Congress in 2023. Today we have Nico Connelly, an oriented CPA board director and chair of the College's Education Committee, who is joining us to discuss the recent changes to the College Central Examinations. This is the second plan for the two-plan episode, and we'll pick up from where we left off in the first episode. I know in the petition that was signed, some of the concerns were around, I suppose, super pressures on supervisors to do more work, the impact that the new sort of role relationship might have on the relationship, and that it might be harder to standardise a larger number of supervisors than it would for a smaller number for an OSCE. Do you have any comments or reflections on that?

Yeah, I mean I think there's two issues there, and I think they're very real issues. So I think there's the workload issue, and then there's the issue of how can we ensure some standardization of assessor behaviour across our many jurisdictions. Just to take the first one, we're very concerned, and we absolutely hear the widespread concern about the workforce crisis that I think we are currently in. Just when we thought it was safe to come out of the water with COVID, and I don't know that we've seen the last of COVID. Hopefully we have, but the history of pandemics is that they tend to come back in cycles. But I think we're now in a completely new crisis and one that is, I think, affecting all jurisdictions in both our countries. And it's a general medical and actually nursing workforce crisis. And I know that in my own state that's placing immense pressure on services, and I think that is going to be really challenging our supervisor continuity and our supervisor sort of availability. The Commonwealth is aware of the issue of a dialogue with the various jurisdictional health ministries about the importance of quarantining and protecting and giving clear permission within health services for roles such as supervision, training, assessment, clinical governance. And we as the college, and I and I think the Commonwealth is completely on side about this, need to be really stressing that that's important firstly for patient safety. It's important secondly for uh developing the next generation of medical specialists, including our own. And thirdly, it's important because if those things aren't supported and valued in the workplace, that's a real problem for retention because nobody is going to want to work in a health system where it's just trying to put the numbers through and it's all just about seeing patients and doing procedures, and there's no time for the things that I think should be. I think they're very important things, but I also think they're the things that we also really enjoy in our roles. So that was to do with the feasibility issue. Coming back to some standardization or some quality assurance about the performance of our supervisors and our fellows as assessors. This is really important. I think it becomes more important when you go to this type of work-based assessment as being a major arm of your training and your examination process. It's definitely mentioned in our AMC accreditation report. I think it's a challenge for us, but we prepared to put the resources in to develop the resources supervisors require, the training and the calibration exercises. Part of this way of doing things is that I think we need to build in evaluation and quality assurance mechanisms whereby in any jurisdiction and across jurisdictions, there's some cross-checking, there's some networking and learning about how our system is functioning. And if there are tweaks or extra supports or things that we need to change in order to strengthen that, then we'll be able to detect them and appropriately resource them.

And what about the concerns that there might be a change to the supervisor and trainee sort of relationship, given that the assessments that are now being done might potentially have more weight than they did previously?

Yep, great. And I know that's on many people's minds as well. And it's a fair question, but there's sort of a another element to it that I'd like to come back to. But we absolutely acknowledge that it's not a fair thing to require a supervisor to make a progression decision based on a summative assessment in the workplace on your own trainee with whom you know you've built the relationship, or indeed perhaps anybody belonging to your little cohort in your hospital, because they're all going to rotate round and it's a system in which you're invested, like the trainee is. And so that will be one of our guiding principles, that it will not be your supervisor who does the workplace summative assessment. Now, the details of how that will occur, and we've got a number of ideas about that, but that's the work that we've come to now. That's the next step. That you know, there's several ways that you could do that, and with more or less central college support. The thing I wanted to come back to is, and it's a bit of an allied or an associated issue, which is this leniency bias. I think part of our training and development and calibration of supervisors is that supervisors do need to be able to give trainees the bad news that they're not performing on a particular task at the level that I as a supervisor would expect. But it's got to be rich and useful information, like this is what I mean. And here's what I mean. When you said that or when you did that, etc., let's have me do this task with the next patient and just watch. And I think the other thing is that we also I think need to get really smart in the college. I think there's some technological solutions that might help us out. This will take probably some lead-in time, and I think we'll need an external partner for this. In that if we can have some real-time voice-to-text recording of that feedback in the workplace, you know, so we go into a private room off the patient corridor, we have a face-to-face conversation, supervisor and trainee, we just dictate the supervisor dictates, that gets loaded up, goes to the trainee, and the trainee is expected to reflect on that and then think about what needs to happen next time they're asked to do that sort of task. So I think that's all in the future, but I think they're the sorts of directions we need to go in.

That's really interesting. Just going back to the sort of discussion about how exams have changed over time, and that I suppose there are sort of workforce issues at the moment, what's your understanding of the skill set required from a modern psychiatric workforce and how that may or may not be different to what it was, you know, at times in the past?

I think that, you know, and I'm probably old enough to be able to say this, I just think that the role of being a doctor and also being a psychiatrist now is sort of immeasurably more complex than it was 30 or 40 years ago, certainly 50 years ago. I think patients have become more complex, particularly the demographic of um the baby boomers means that we've got a lot of older people with multiple conditions. I think our understanding and definition of disorders and psychiatric disorders and the contributing factors is far more sophisticated. I think we have a more complex understanding of things like harm. In my day it was, you know, real emphasis on physical safety, but we now understand all sorts of psychological and cultural harms that can occur, institutional, structural harms in the system. And I think the expectations on psychiatrists and doctors is far more complex with a general greatly raised literacy in the general public, and with that a set of expectations that's quite different, and with the universal access to almost limitless information through internet, etc. So I think our understanding of what it is to be a fellow of our college has changed, and I think it's changed from some old mental models that you still do see being acted out amongst our peers. So we've gone beyond the knowledge model, you know, the London cabbie. I've just got to learn this immense amount of information and you know these roadmaps, and then I'll breach it. I think we have got a different understanding of competence. It's not a stable thing, it's dynamic and it's multifaceted, it changes over time, and therefore our old ideas of a gold standard that we can set as an examination, once you've got that, you're right. I think that the mental model that we need to have safe practitioners does apply. I think the community really, that would be the understanding of what fellowship means that it's a safe and good quality care that you're going to get if you've got FRA and ZCP. But of course, safety, harm, and even good care is dynamic. It's changing all the time. So my idea of what we should be producing is a doctor who is sophisticated and is a lifelong learner. And so I would borrow the words of one of the ancients and say that the mind of a psychiatrist is not a vessel to be filled, but a range of sensibilities to be continually developed and refined. And by sensibility I mean being able to appreciate and respond to complex emotional, social, cultural, medical, scientific, and ethical predicaments that will face a psychiatrist. I think this stuff is best taught in the workplace. And I think the product that we want is somebody that is going to develop that set of sensibilities with a commitment to continually honing and developing them.

That's really interesting. Thanks, Nick. I suppose just one more question before we sort of talk about what might be coming next in terms of assessments, is just to understand a little bit more about the cost and logistics and the amount of time that goes into delivering a traditional OSCI.

Yeah, okay. So a traditional OSCI, and you we're talking here about 200 candidates, could be anything up to $290,000. You're usually running five streams in the morning, five streams in the afternoon, uh, often in multiple centres. That means you need 55 rooms for stations plus some spaces to do other things like register, hold lunch, etc. So the venue's got to be big. We usually need 70 active examiners and we need some backup examiners because there's always life. And when you run an OSCI on a Saturday, there will be examiners that don't turn up, so you need people that can slot in. You need 50 to 60 role players with 10 backups, and you need catering for 300 people. Up till now we have generally used outpatient departments of major teaching hospitals. However, if we needed to use an exhibition centre or a university venue, if that no longer is available to us, that would add another $110,000, $111,000 to that bill.

And that's per sitting.

Yeah. That's per OSCI, yeah, yeah.

So we've um touched on it a bit, and you've mentioned that you've given a bit of information, but can you talk any more about what could I just add to that?

I mean, yeah. So we're not discontinuing the OSCIS because of cost, but we do run it on a cost recovery basis. So the cost gets passed on to the trainee who applies for the examination and is you know subsidized somewhat, but it's it's a cost recovery event. You see, you're saying the candidate pays for some of it, but most of it is okay. And that doesn't include if you're flying people in from the interstate and accommodating them.

So just looking at ahead to the future now and yeah, what what are the plans for the future assessments that you've sort of touched on a bit, but can you talk about that any more?

Yeah, absolutely. So we've worked really hard at ensuring that there's all the stakeholders have been represented and actively represented at our stakeholder forums. We've worked hard to involve our committees and to understand deeply and to have a record and to have themed the feedback and concerns. I think now we're in a pretty good place. We've got some agreed principles, we've got some really well documented and I think well-understood constraints and challenges, and we've got two agreed options, and the next step is to bring together some of our own expertise, including trainees, but mainly our fellows who've been involved in training and education and examination for a long time with some specialist medical educationalists, uh, some of whom are fellows of the college, and to try and co-develop the option that will take us through. I think we're looking at a transitional option. I think that option four, the more complex, integrated, more programmatic assessment, is probably going to be very difficult to do in the workforce environment that we're going to have for the next couple of years at least.

Are there any plans to introduce, I suppose, AI or virtual technology? You mentioned the voice-to-text kind of idea. Are there any more?

Yeah, I mean I think a lot of technology, I think that could be potentially very helpful for us. And so I did mention the voice-to-text. We also may use cameras for OSCIDs. We may observe clinical interviews, we may be able to do them remotely, or they don't necessarily all have to be face to face. We'll we'll have to work that out. I think chat GPT is a worry. I think it's a threat to written examinations, less so if you're in an exam centre, but the take-home assignment's got a whole new sort of flavour to it with in the in the age of chat GPT.

Yeah, it's very scary how quickly it can write an essay.

Yeah. And even provide fictional but highly convincing references.

Yeah. Yeah. Do you have a time frame for what the next kind of stages are in the assessment programme?

Yeah, I do have a rough time frame and I realise how important developing and publishing for everybody a clear time frame that we can commit to. So I think in the next so after the uh the Congress, when we get all get back to work, I think our next task is to bring together the stakeholders and work up some detailed and be able to test and model ways that we might bring in the new assessment, perhaps around option three. I think that's what's looking likely. I think we need to determine that assessment certainly by the end of this year and hopefully well before December. But it is complex work and the requisite consultation and making sure everybody understands that this is and agrees this is the best option. That does take time, but we need to do that work this year. And that would then give us 2024 to develop things like the technology, things like our training, our calibration, our resources for the trainees to know what the assessment and supervise, to know what the assessment will be and to have a good 12-month lead-in before we launch it. And we have committed to providing a 12-month preparation from the time that we clearly define what's going to happen until the time that it goes live. And that's important. So I would hope all going well. We're clear about what the assessment will be in 2025 by the end of this year, and then in 2024, that's a year for people, including the college staff, to be working on regulations, rules, appeal methods, resources to support uh supervisors, assessors, training resources, that sort of stuff.

Could you address any concerns that the alternative assessment pathway as it is at the moment may not be as good as I suppose the OSCI was, or what the new assessments might be, particularly I think given the relatively high pass rate?

I think when we workshopped the uh principals on the 28th of April, there were some additional principles that came in. And one of them was that we need to be monitoring, evaluating, and have quality assurance and quality improvement built into whatever we do in terms of training and the assessments. And so we're really committed to that. So if we find that there are deficiencies or or gaps or things that need to be tweaked, we will be addressing that as we go. You usually need to run something like this for a couple of years to have uh enough data to make the sort of judgment when comparing with other systems. And we're very interested in trainee and supervisor satisfaction and well-being. So that will be the other set of metrics that I think is going to be really important.

Well, that's probably a good place to finish.

Fantastic. Thanks, Ed.

No worries. Thanks very much for your time today, Nick. And thanks also to our producers, David Bill and Nish Takuma, uh, Shady Dave for our intro music, to Doni Prentice for our artwork, and of course, Australasian psychiatry for the opportunity.