Doctors are using AI software that does not meet minimum standards to record and transcribe patient meetings, according to a Sky News investigation.
NHS bosses have demanded GPs and hospitals stop using artificial intelligence software that could breach data protection rules and put patients at risk.
A warning sent out by NHS England this month came just weeks after the same body wrote to doctors about the benefits of using AI for notetaking – to allow them more time to concentrate on patients – using software known as Ambient Voice Technology, or “AVT”.
Health Secretary Wes Streeting will next week put AI at the heart of the reform plan to save the NHS in the 10-year plan for the health service in England.
But there is growing controversy around software that records, transcribes and summarises patient conversations using AI.
In April, NHS England wrote to doctors to sell the benefits of AVT and set out minimum national standards.
However, in a letter seen by Sky News, NHS bosses wrote to doctors to warn that unapproved software that breached minimum standards could harm patients.
More on Artificial Intelligence
Related Topics:
The 9 June letter, from the national chief clinical information officer of NHS England, said: “We are now aware of a number of AVT solutions which, despite being non-compliant … are still being widely used in clinical practice.
“Several AVT suppliers are approaching NHS organisations … many of these vendors have not complied with basic NHS governance standards.
“Proceeding with non-compliant solutions risks clinical safety, data protection breaches, financial exposure, and fragmentation of broader NHS digital strategy.”
Sky News has previously revealed the danger of AI “hallucinations”, where the technology makes up answers then lies about them, which could prove dangerous in a healthcare setting.
Please use Chrome browser for a more accessible video player
1:59
Is ChatGPT reliable despite its ‘hallucinations’?
NHS England sets minimum standards but does not tell NHS trusts and healthcare providers which software providers to use.
Sky News can now reveal there is growing pressure on NHS England and similar bodies to be more proactive.
Dr David Wrigley, deputy chair of the British Medical Association’s GP committee, said: “Undoubtedly, as a GP myself and my 35,000 colleagues, we’ve got responsibilities here – but in such a rapidly developing market when we haven’t got the technical knowledge to look into this.
“We need that help and support from those who can check that the products are safe, check they’re secure, that they’re suitable for use in the consulting room, and NHS England should do that and help and support us.”
Dr Wrigley continued: “We’re absolutely in favour of tech and in favour of taking that forward to help NHS patients, help my colleagues in their surgeries.
“But it’s got to be done in a safe and secure way because otherwise we could have a free for all – and then data could be lost, it could be leaking out, and that just isn’t acceptable.
“So we are not dinosaurs, we’re very pro-AI, but it has to be a safe, secure way.”
Image: The head of the NHS Confederation says the letter is ‘a really significant moment’
The spectre of dozens of little-known but ambitious AI companies lobbying hospitals and surgeries to get their listening products installed worries some healthcare professionals.
There are huge profits to be made in this technological arms race, but the question being asked is whether hundreds of different NHS organisations can really be expected to sift out the sharks.
Matthew Taylor, chief executive of the NHS Confederation, said the letter was “a really significant moment”.
He said it was right for the NHS to experiment, but that it needed to be clearer what technology does and does not work safely.
“My own view is that the government should help in terms of the procurement decisions that trusts make and should advise on which AI systems – as we do with other forms of technology that we use in medicine – which ones are safe,” Mr Taylor said.
“We’ll need [government] to do a bit more to guide the NHS in the best way to use this.”
When pressed whether in the short term that actually makes it sound like it could be quite dangerous, Mr Taylor replied: “What you’ve seen with ambient voice technology is that kind of ‘let a thousand flowers bloom’ approach has got its limits.”
Please use Chrome browser for a more accessible video player
0:45
Godfather of AI warns of its dangers
Earlier this year, the health secretary appeared to suggest unapproved technology was being used – but celebrated it as a sign doctors were enthusiastic for change.
Mr Streeting said: “I’ve heard anecdotally down the pub, genuinely down the pub, that some clinicians are getting ahead of the game and are already using ambient AI to kind of record notes and things, even where their practice or their trust haven’t yet caught up with them.
“Now, lots of issues there – not encouraging it – but it does tell me that contrary to this, ‘Oh, people don’t want to change, staff are very happy and they are really resistant to change’, it’s the opposite. People are crying out for this stuff.”
Image: GP Anil Mehta says the AI software helps cut paperwork and patients are ‘extremely reassured’
Doctors who use AI that complies with national standards already say there are big benefits.
Anil Mehta, a doctor in the health secretary’s Ilford constituency, told Sky News he backed his MP’s drive for more AI technology in healthcare.
“I spend 30% of my week doing paperwork,” he said. “So I think once I’ve explained all of those features of what we’re doing, patients are extremely reassured. And I haven’t faced anybody that’s not wanted to have me do this.
He added: “(I) think that consultation with your doctor is extremely confidential, so that’s not changed at all.
“That remains confidential – so whether it’s a vulnerable adult, a vulnerable child, teenager, young child with a parent, I think the concept of that confidentiality remains.”
An NHS spokesperson said: “Ambient Voice Technology has the potential to transform care and improve efficiency and in April, the NHS issued guidance to support its use in a safe and secure way.
“We are working with NHS organisations and suppliers to ensure that all Ambient Voice Technology products used across the health service continue to be compliant with NHS standards on clinical safety and data security.”
The mother of murdered teenager Brianna Ghey is calling on the government to introduce a ban on mobile phones in schools – a move she says will not only safeguard children, but also improve their behaviour and engagement in class.
In the lead-up to the attack, her killers had spent time on the dark web. At the same time, Brianna was also trapped online, struggling with a phone addiction.
Her mother Esther Ghey’s Phone Free Education campaign is driven by her personal experiences as a parent and the impact Brianna’s phone use had on her education.
Image: Brianna Ghey struggled with a mobile phone addiction, according to her mother
“All the arguments that me and Brianna had were down to her phone use,” Esther said.
“But even in school, she had issues and I used to have phone calls from the school saying that Brianna wouldn’t put her phone away.”
Brianna, who was transgender, struggled with an eating disorder and also self-harmed.
Her mother says the constant time she spent online exacerbated those issues, while impacting her behaviour at school, where she had 120 safeguarding logs and 116 behaviour incidents recorded by her teachers.
Image: Esther Ghey said she had calls from her daughter’s school saying that ‘Brianna wouldn’t put her phone away’
“It was so difficult as a parent, because I felt in one way that I was failing and then in another way, and this is really difficult for me to speak about, I was so annoyed with Brianna,” she recalled.
“I thought, why can’t you just go to school, get your head down and just focus on your education, because this is important.
“Only now, after two years of being immersed in this world, do I realise that actually, it’s so much harder than that.”
Research by the Children’s Commission has shown that 79% of secondary schools are still allowing pupils to bring their mobile phones into school, and even into classrooms.
Image: Brianna’s school introduced a ban on mobile phones in September last year
How phone ban is working at Brianna’s old school
Esther is campaigning for government guidance on phones to become statutory, with funding also set aside for the equipment to help schools implement the ban, arguing the lack of legislation is “setting children up to fail”.
At Birchwood Community High School in Warrington, where Brianna was a pupil, they introduced a ban on phones last September.
At the beginning of the day, pupils turn off their phones and place them in pouches, which are locked. At the end of the school day, the pouches are then unlocked.
Image: Pupils at Birchwood Community High School in Warrington place their phones in pouches, which are then locked
The headteacher, Emma Mills, said introducing these measures has come with several benefits.
“It’s had an impact in all areas of school, and it’s actually had a really positive impact in ways that I didn’t foresee,” said Ms Mills.
“Attendance has improved this year. In terms of behaviour, behaviour has improved. We’ve had no permanent exclusions this year in school, which is actually the first time since I’ve been headteacher in six years, there’s been no permanent exclusion.”
This summer, the school also saw its best-ever GCSE results in the core subjects of Science, maths, and English.
Image: Emma Mills, headteacher at Birchwood Community High School in Warrington
‘They can live without their phones’
For Ms Mills, another significant change has been the atmosphere in the school.
“They’re not as worried, they’re not as distracted,” Ms Mills said.
“They’ve realised that they can live without their phones. Something else we’ve really noticed is that it’s a bit louder in school at breaks and lunch times. It’s because they’re talking more, they’re interacting more, and they’re communicating more.”
The positive impact of a ban at Brianna’s old school has served as encouragement to Esther, who has written an open letter addressed to Sir Keir Starmer and Education Secretary Bridget Phillipson, asking for government support.
Police across the UK dealt with more than 3,000 protests over three months this summer – more than three times as many as just two years ago.
There were 3,081 protests this June, July, and August across England, Wales, Scotland and Northern Ireland, figures from the National Police Chiefs’ Council have revealed.
Last summer, when riots were raging across the country following the Southport murders, police dealt with 2,942 protests. In 2023, it was 928.
The summer months this year have been dominated by widespread demonstrations, some against the ban on Palestine Action and others against housing asylum seekers in hotels.
Image: Counter-protesters with police as people take part in a Stand Up To Racism rally in Orpington in August. Pic: PA
‘Increasing tension’
Gavin Stephens, chairman of the NPCC, said it was clear that there has been “more community tension and more division”, adding that “we all have a responsibility, policing included, to set the tone”.
“Anybody in a leadership position should think about how we can reduce and defuse tensions and not sow division,” Mr Stephens said.
The senior official said protests this year were a “chronic pressure” for police compared to last year’s disorder, which was acute.
“This is not talking about the volume of protest, and this is not a commentary from policing on people’s right to protest peacefully,” he said.
“We absolutely support that in a democracy, but we do know that there is a climate of increasing tension and polarity in what we’re seeing.”
He is convinced communities will be able to reunite and “reset”, and said claims that the UK is on the verge of civil disobedience are “exaggerated”.
The group of Thames Water lenders aiming to rescue the company have set out plans for £20.5bn of investment to bolster performance.
The proposals, submitted to the regulator for consideration, include commitments to spending £9.4bn on sewage and water assets over the next five years, up 45% on current levels, to prevent spills and leaks respectively.
Of this, £3.9bn would go towards the worst performing sewage treatment sites following a series of fines against Thames Water, and other major operators, over substandard storm overflow systems.
It said this would be achieved at the 2025-30 bill levels already in place, so no further increases would be needed, but it continued to argue that leniency over poor performance will be needed to effect the turnaround.
The creditors have named their consortium London & Valley Water.
It effectively already owns Thames Water under the terms of a financial restructuring agreed early in the summer but Ofwat is yet to give its verdict on whether the consortium can run the company, averting the prospect of it being placed in a special administration regime.
Please use Chrome browser for a more accessible video player
1:32
Is Thames Water a step closer to nationalisation?
Thames is on the brink of nationalisation because of the scale of its financial troubles, with debts above £17bn.
Without a deal the consortium, which includes investment heavyweights Elliott Management and BlackRock, would be wiped out.
Ofwat, which is to be scrapped under a shake-up of oversight, is looking at the operational plan separately to its proposed capital structure.
The latter is expected to be revealed later this month.
Sky News revealed on Monday that the consortium was to offer an additional £1bn-plus sweetener in a bid to persuade Ofwat and the government to back the rescue.
Please use Chrome browser for a more accessible video player
2:35
Thames Water handed record fine
Mike McTighe, the chairman designate of London & Valley Water, said: “Over the next 10 years the investment we will channel into Thames Water’s network will make it one of the biggest infrastructure projects in the country.
“Our core focus will be on improving performance for customers, maintaining the highest standards of drinking water, reducing pollution and overcoming the many other challenges Thames Water faces.
“This turnaround has the opportunity to transform essential services for 16 million customers, clean up our waterways and rebuild public trust.”
The government has clearly signalled its preference that a market-based solution is secured for Thames Water, though it has lined up a restructuring firm to advise on planning in the event the proposed rescue deal fails.
A major challenge for the consortium is convincing officials that it has the experience and people behind it to meet the demands of running a water company of Thames Water’s size, serving about a quarter of the country’s population.