Hoisted from my inbox, a troubling discussion of the way AI and tech dependence generally are severely degrading the competence of advanced degree studies among medical and bioscience programs at elite universities. This is a big red flag as far as relying on the opinion of a medical professional under 35, since they won’t have one. They are just mouthpieces for what passes for Internet wisdom. So why should they be paid big bucks? This devolution makes the case for fobbing all sorts of work off to nurse technicians…except IM Doc has warned that they at least recognize they are not competent to diagnose in most cases, so they bring in specialists. And in the end, that only increases treatment costs. What’s not to like?
And what happens when the specialists who actually know how to interpret symptoms, data, and images retire? Medical tourism will be in your future if you can afford the cost and effort of making the trip.
Recall KLG is a professor of medicine who among other things did a lot of bench work in his day. GM works at an elite school you most assuredly heard of.
To their exchange:
KLG: Medical students love their AI. Or some of them do. Scary.
GM: Not just medical students, PhDs too.
I see it everywhere now. Thinking is increasingly offloaded to the AI.
KLG: I can’t remember the last time I asked a medical student in tutorial group a question that was answered without looking at a screen.
GM: Something to add — right now two very active areas of biomedical AI research are literature review and experimental design.
The former is already widely in use by people in the form of plain ChatGPT, both by trainees and PIs, but there are more advanced and specialized tools being developed too. The idea being that we don’t need to read papers, AI will do it for us and will provide a summary. The problem here should be obvious given, first, the well-known deficiencies of current AI, and second, the fact that a lot of key information in papers is kind of buried in between the lines or in more explicit statements deep in Methods sections and supplements that a “summary” will miss. And in mundane human things such as knowing how to interpret what was written by knowing the people who wrote it, background that AI does not have.
Also, people already mostly can’t be bothered to read papers, and just skim them, imagine what this will do to the ability to read a scientific paper (which is a skill of its own).
Still, let’s say there is some utility to AI speeding up the quick look up of things.
Experimental design is where its get really problematic.
I have seen several of these presentations now and the idea is that you want to do some experiment, but don’t want to bother putting together the protocol, so you tell AI to design it all for you, then you follow it. Of course, this is accompanied by some wet lab then actually following the AI prescriptions and showing that it indeed designed the experiment correctly.
And again the catastrophic downstream effects should be obvious. Once students get in the habit of trusting the AI, then how much training will those students actually get? Being really good at the bench means having all those mundane details about buffers and volumes in your head. Troubleshooting failed experiments usually depends on paying attention to such little details. And developing new methods depends on someone knowing the existing ones inside out and then building on top of that. Offload the mundane basics to AI, what is left of that very important expertise then? Yet this is the path that will likely be followed soon. It is still early days, but that is the clear trajectory.
I’ve literally already witnessed situations in which a grad student is giving a presentation in group meeting, then got to the point where she ran into an issue, asked ChatGPT for help, ChatGPT didn’t quite help her, so then she just gave up as in “ChatGPT doesn’t know the answer, what can I do?”.
Note that wet lab work is probably one of the absolute last things to be replaced by AI, because it requires very fine motor control of hands and fingers and is something that has always been based on “feel”. Prostitution is the only AI-proof profession, but experimental scientists will also last quite a long time. So you don’t want to corrupt them mentally.
However, that process has already been ongoing for a while even before AI. A precursor of all this was the inability to do mental math. That must have disappeared already before my time because I have run into this problem literally every single time I have had to train students. At the most elite institutions no less. Again, literal real life examples, and that situation has repeated itself many times, e.g.:
“We have a 1M master solution, we need to make a 50 mL of buffer that has that component at 10 mM, how much do we need to add?”
Blank stares, followed by reaching for calculators (and in more recent times phones). I stop them immediately and tell them:
“Do it in your head”.
They can’t. I tell them the answer (500 microliters), then we move to the next component, which also has to be at 10 mM, but the master solution is 5M.
“How much do we need to add”?
Blank stares again. Nobody can figure out to just divide the already arrived at answer for the 1M master solution of 500 by 5, no that is too hard. They start doing the calculation from scratch…
That was the already preexisting condition to which AI is being added now…
I find the mental math deficiencies to be quite worrisome. Without at least a sense of what is reasonable, large errors can go un-noticed, for a while.
At a visit decades ago to friends, I was asked to help one of their children with maths homework. The child was having difficulty implementing the procedure to solve a problem. I don’t recall the details, but I remember that I found the procedure to be more complex than needed. I think that maths pedagogy may have been needlessly revised from the simple, robust, workable methods I was taught decades before. What was the point of that?
Dumbing down of mathematics to unacceptable levels was observed by me back in 2000. During my PhD I taught health economics and medical statistics to med students. Decided they were beyond awful.
I’ve said this before but it bears repeating: dumbing down had happened to worrying degree even by 1991 when I took UK A levels (for university entry). Two of my subjects were mathematics and further mathematics. When we did past papers to practise for the exams in June 1991 we realised that papers from pre-1980 for practising FURTHER mathematics had to be real exam papers from the SINGLE mathematics subject. Ergo, much of the single subject stuff was considered too difficult and got put into further mathematics….. which virtually NO medic studies.
I read Economics at Cambridge and my Mathematics friends were compelled to attend a “catch up” (remedial) 8 week course over the summer before term officially began in October. Go figure.
I have an undergraduate degree in engineering and a grad degree in economics, both from the 1980’s. I remember being told as an undergrad that it was important to develop one’s math intuition and to always check the dimensions on quantities precisely because it would help in identifying when a calculation error had led to an absurd result.
My first boss—whose degree was in physics—could glance at a page of numbers and instantly point to one, saying “This is wrong.” I was in awe of him. Many years later, when I joined a team in the middle of a project, I saw that the current draft report had consistently left off an “M” in the units describing numbers of metric tons of carbon emissions. They were off by a factor of a million! I certainly earned my pay that day—and the undying gratitude of my new teammates—by pointing that out.
Indeed. My (tor)mentor had the ability to just look at a dataset of choice modelling data and make statements. I, over 15 years, gradually learnt this. It doesn’t matter what the log-likelihood of the logit regression says, that solution does not make sense in terms of real world behaviour (because there are an infinite number of solutions and all those models normalise the variance to one in order to give you a solution).
I got to be able to instantly look at data and go “big fat nope”. I tried to help juniors to develop this “art” but they generally were too indoctrinated with mainstream economics backgrounds and took the solution from the stats program as gospel. Plus they never got into the habit of doing “smell tests”: quick and dirty cross-tabs of things to see if it looks “odd”. And thus we end up with a bunch of complete idiots.
This will get WORSE under AI because the field I worked in dealt with “what if” scenarios from statistical designs, NOT existing data. So people will find themselves even more “stuck to the production possibility curve” and never even given a possible solution beyond it thanks to AI. But that’s not my problem anymore.
Just downloaded ChatGPT and asked series of questions about Best-Worst Scaling. Cue LULZ.
It more or less got the order of importance of people in field right (yours truly is number 2 and number 1 if you count people still alive). However it very quickly started making erroneous statements regarding experimental design (which clinicians, academic marketers, social scientists etc) will no doubt take in.
At least I could tell that it hadn’t yet surreptitiously scraped our textbook and ignored my instructions to the publisher. I sensed much of the content had been scraped from Wikipedia along with marketing stuff (which is BLATANTLY wrong) from the biggest company providing black box solutions.
EPIC has AI embedded. Currently being sold as transcription into form of traditional progress notes, yet labs and meds are already tied to reimbursable indications. Just need a robot to listen to chest and check reflexes and Evidence Based evaluation and management no longer requires thought. It’s like that was an ulterior motive for electronic records all along…
Christ, I’m pretty sure the hospital systems whose data I manage use EPIC (I manage output data, not anything internal, so I’ve never used it)…I wonder how much of the information I’m responsible for shepherding is actually hallucination. Probably not a ton yet if just because AI is fairly new and in my region anything new takes tremendous hemming and hawing before actual implementation (this is a good thing in my view). Still, I dread the day when I will have outlived the doctors and epidemiologists who didn’t come up on asking the machine for the answer to the question every time.
On other news Klarna is going to bring back humans, they say, because of AI fails. The withered shell of the IRS quickly found that the AI chatbots start out limited and get worse and worse over time.
The Neverending techschmoe and C-suit search for unpaid workers who cannot complain is a tragic story.
I have learned that the best way to educate in medicine is to use real-life examples. So I will briefly discuss yesterday’s fiasco – the details are all fresh in my mind. But trust me, similar and much worse things happen every single day.
32 year old athletic male was doing an aggressive hike and fell down a hillside about 3 weeks ago. Instantly had severe pain in this right heel. Had to hobble back to the car. The next day things seemed to be getting better. About two weeks later, however, there was still lingering pain in the right heel, but now he noticed his entire right calf was a bit red, and a bit warm and a bit tender. This continued on for a week ( 32 year old jocks tend to ignore pain). He showed up in the local urgent care to be seen by the Nurse practitioner du jour. This particular NP graduated from nursing school ( no BSN) – and within 6 months had already started her 2 year correspondence course to get her NP degree and to become a medical “provider”. Our EMR now is so handy that it looks over all the symptoms and labs and X-ray findings etc – and gives you a list of “suggestions”. I could write a 4000 page comedy novel of all the suggestions provided to me the past little while. Just use AI for anything in your life for 5 minutes – you will see what I am talking about. Well, the NP and the AI decided that he had a deep venous thrombosis in that leg. So, she immediately ordered an ultrasound of the leg – the clot was deep in the calf muscle and did not even get close to the knee – nothing in the thigh. So, either she or the AI – it is hard sometimes to tell – made the diagnosis of DVT – and started him on Xarelto – at an out of pocket cost of 900 dollars a month. Over the next 2 months, every 2 weeks the patient underwent a repeat ultrasound to check on the “progress” of the clot. This surveillance is not even closely medically indicated in any way shape or form. I am his PCP. I was notified that a FIFTH ultrasound had been ordered, the radiologist had become wary – and “Gee, maybe someone with a little experience ought to look at this.” I did not know the first thing of this story until he called.
I do have experience. I did Gross Anatomy and carefully dissected every vein, nerve, muscle, tendon and artery of the lower extremity under the careful tutelage of an absolutely sadistic retired orthopedic surgeon. My training was intense, it was ugly, it was a hazing, it was THANK YOU SIR MAY I HAVE ANOTHER SIR 10 times a day. But I have come to realize that is the only way. I learned all about a very old medical concept called Virchow’s Triad – that is still very important to go through in your mind EVERY SINGLE TIME you see a blood clot. I have taken care of innumerable blood clots in my life in any and all locations in the body. As a resident, I saw these all the way through from clinic to hospital and have continued to do do this for 35 years. I do not need AI or UpToDate to guide me through.
So, what may be missing in nurse practitioner correspondence school? Well, they dissect nothing. So they are blissfully unaware of the nexus of veins in each knee that prevent clots in the calf from propagating up into the thigh. This is important – when a clot forms in the thigh – it can easily dislodge and go right to the heart and lungs. Not so the calf – the knee vein nexus keeps it right in place. Had this provider attended medical school, she would have known about Virchow’s triad years before she ever laid eyes or hands on a real patient – again the triad should be right in the front of the internist’s brain when looking at a fresh clot. Basically it is WHAT CAUSED the clot? A triad of three things – 1) Recent or distant leg trauma 2) Medical conditions that make people clot – like all kinds of bleeding disorders, pregnancy etc 3) Cancer. The END. You must account for this in every diagnosis. So, the first thing I did with the patient is a careful examination of his still painful foot – oh, on a simple x-ray of the foot – there is a very large crack – a fracture – right through the heel bone. Neither the NP or the AI had even touched or examined his heel in any way. Now that we are approaching 6 weeks out – it may have already started to heal – and heal wrong – so much misery may be in store for this young man. The NP and the AI however did decide that he needed a vascular surgery consult because of this issue ( I MEAN REALLY?) – and he was scheduled to see a vascular surgeon at the end of June. We did not even have an orthopedic consult go through brain cells or circuits however. And this is the nexus of a whole new problem in medicine – the non-trained providers and the AI are just flooding the specialists with all kinds of consults that have been historically handled by internists or more likely just absolutely lunatic like this vascular surgery consult would have been until I cancelled it yesterday. That is what we were trained to do – an internist will only send out the most complicated of things. When I look through NP charts today – there are at least 1 and sometimes 3 referrals on almost every one.
But the most important issue – is that the patient’s calf blood clot needs a heating pad for several days and an aspirin a day and very close observation by a physician who knows what they are doing. No 800 dollar a month Xarelto – no ultrasounds twice a month – just good evidence based and experience based care. I looked at this 32 year old’s account yesterday – 32,000 bucks and counting. AGAIN, most importantly, he needed someone to diagnose the underlying issue – the fractured heel.
Just FYI – when you accept care from an NP or PA in most states – you are accepting that the level of expertise is going to be much lower than an MD. You then must realize that in the medicolegal world, your ability to be compensated for malpractice is greatly diminished and may even be zero. And honestly, I am not even sure what the rules are for suing AI – do we get to sue EPIC? UpToDate? I have no idea.
It is all I can do every day not to despair. I do not know what else to say. I am just often looked at by the MBA crowd – and you can just tell they are in way over their heads. The system is just cratering – and I am pretty sure it will be the same as the old line about “Going broke gradually, and then all of a sudden”. God help us – and God help this young man. The ortho docs are going to have to refracture his heel in the OR – and then weeks of painful therapy.
More patient visits, more expensive tests, more expensive consults; I think the system is working as “intended”. Sigh.
You mean there are other types? Sorry but having had the top UK rheumatologist as my ultimate boss, his dislike of orthopods rubbed off on me somewhat.
To be a bit more constructive, my experience with NPs here in UK is just as variable as with GPs. Some are just awful. One was brilliant in taking the bull by the horns and getting 3 consultants at the hospital to agree to go by majority vote as to whether I could be discharged after a 24 hour in-patient stay for suspected sepsis. The infection was gone but one consultant held out but was out-voted so I was let out. She knew that I had made a perfectly sound decision that the risk of staying in a hospital ward with COVID floating around any longer far outweighed any other risk which the one holdout was unable to put into words anyway. I wanted out of there and she managed it by playing umpire. She was a rarity IMO.
Since you triggered this… Way back when I suffered a sports injury, dislocated knee (stepped on a spool while retreating in fencing). Luckily the closest ER was actually a specialized trauma ER and basically across the street so I was helped there in minutes (one of my mates even took care that my bicycle was stored in the sport-center’s carage as not to be stolen).
Within 12 hours I met three orthopedic surgeons. The one on duty in the evening told me I’d better start thinking of becoming a trainer, that the knee would never be the same. The surgeon on the morning rounds figured that the knee will be ok and allow sports.
And finally the afternoon surgeon, who was the one to operate my knee promised to make it better than it ever was. I must say I was happy he was the one to operate. I guess his attitude rubbed off on me, because I recovered really fast. I did more exercises than my physio demanded, I learned to listen to my body and push it to the healthy limits, and a year and a half later I won the championship, basically because my legs were better than anybody. And you fence with your legs.
Very concerning. I was a lab assistant in a research lab as a HS student – paid better than mowing lawns or packing groceries, plus looked great on my college applications. Mental math, following protocols, the how and why of experimental design – all of that was repeatedly beaten into my brain and the uptake constantly rechecked. Made it just a bit easier when going through two semesters of Research Methods combined with two semesters of Statistics in college, making junior year the stuff of nighmares for every psych major. The flip side is that even though I went a different career route, I can still be woken in my sleep to check a study design and will still catch most design mistakes, even without AI. That future medical professionals are being dumbed down in critical skill sets is my new stuff of nightmares.