Babylon's AI 'outperforms average doctor' in MRCGP exam

Artificial intelligence (AI) developed by technology company Babylon performs better than the average candidate in an MRCGP assessment, the company has said.


Additional tests had also shown that the AI system was able to provide health advice to patients ‘on par with practising clincians’, Babylon claimed.

Babylon assessed its AI technology using elements from both the AKT and CSA MRCGP exams that related to diagnostics. The company took a representative sample of questions from publicly available sources, some of which had been published by the RCGP.

Presenting the results at the Royal College of Physicians in London on Wednesday evening, Babylon’s medical director Dr Mobasher Butt said that the average pass mark for both exams was 72% over a five-year period. Babylon’s AI system achieved a score of 81% on its first attempt, he said.

‘It takes most doctors around 12 years before they have the knowledge and skills to sit this exam and even then some fail on their first attempt and even second, or third attempt,’ Dr Butt said. ‘It’s taken our scientists two years to create AI that can achieve 81% on its first attempt.’

However, the RCGP said that the results of the experiment were ‘dubious’ and that no app or algorithm was ‘able to do what a GP does’.

RCGP vice chair Professor Martin Marshall said: ‘The exam-preparation materials, used by Babylon in this research, will have been compiled for revision purposes and are not necessarily representative of the full-range of questions and standard used in the actual MRCGP exam, so to say that Babylon’s algorithm has performed better than the average MRCGP candidate is dubious.’

Read more
No app or algorithm can ever replace a GP, says RCGP

Babylon also conducted a series of tests to assess how its AI performed when dealing with 'real-life clinical scenarios'.

Dr Megan Mahoney, chief of general primary care at Stanford University’s division of primary care and population health, and doctors from the Royal College of Physicians devised 100 scenarios or ‘vignettes’, which were then tested with the AI system and 12 experienced UK GPs who had no connection with Babylon. In the tests patients were played by practising GPs, some of whom were Babylon employees.

Artificial intelligence

The doctors’ correct diagnosis rate ranged from 64% to 94%, with an average of 80%. Babylon’s AI system also correctly diagnosed 80% of cases, Dr Butt said. When the results were assessed against conditions seen most often in primary care, the GPs' diagnostic accuracy ranged from 52% to 99% and Babylon's AI accuracy was 98%.

Dr Butt said the results showed that AI had the potential to 'significantly improve as it learns more about conditions it sees less frequently'.

The appropriateness of triage decisions was also independently assessed, although only by one doctor. GPs made the correct decision in 90.5% of cases, while the AI system did so in 90% of cases.

Babylon founder and chief executive Dr Ali Parsa said: ‘Even in the richest nations, primary care is becoming increasingly unaffordable and inconvenient, often with waiting times that make it not readily accessible. Babylon’s latest artificial intelligence capabilities show that it is possible for anyone, irrespective of their geography, wealth or circumstances, to have free access to health advice that is on-par with top-rated practicing clinicians. Tonight’s results clearly illustrate how AI-augmented health services can reduce the burden on healthcare systems around the world.’

Professor Marshall said that no app or algorithm could replace a GP. 'An app might be able to pass an automated clinical knowledge test but the answer to a clinical scenario isn’t always cut and dried,’ he said.

‘The potential of technology to support doctors to deliver the best possible patient care is fantastic, but at the end of the day, computers are computers, and GPs are highly-trained medical professionals: the two can’t be compared and the former may support but will never replace the latter.’

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins


Already registered?

Sign in

Follow Us:

Just published

£20 notes spread out

VAT trap for PCNs could strip millions of pounds from general practice

Tens of millions of pounds could be stripped from general practice because work carried...

Talking General Practice logo

Podcast: Is the BMA representing GPs effectively, why GPs face a pension tax hit, and views on the workload crisis

In our regular news review the team discusses representation of GPs, a new survey...

Man sleeping

NICE guidance on insomnia backs app to replace sleeping pills

Hundreds of thousands of people with insomnia could be offered treatment via a mobile...

Health worker prepares a dose of COVID-19 vaccine

JCVI backs autumn COVID-19 booster campaign for high-risk adults and NHS staff

Frontline health and social care staff and adults at increased risk of severe illness...

GP consultation

Government accused of 'misleading' claims on general practice workforce

GP leaders have accused the government of making misleading claims about the general...

Consulting room door

LMC calls for enhanced access to be scrapped after abuse forces practice to close reception

A Midlands LMC has backed a practice forced to close its reception desk after abuse...