Hoisted from my inbox, a troubling discussion of the way AI and tech dependence generally are severely degrading the competence of advanced degree studies among medical and bioscience programs at elite universities. This is a big red flag as far as relying on the opinion of a medical professional under 35, since they won’t have one. They are just mouthpieces for what passes for Internet wisdom. So why should they be paid big bucks? This devolution makes the case for fobbing all sorts of work off to nurse technicians…except IM Doc has warned that they at least recognize they are not competent to diagnose in most cases, so they bring in specialists. And in the end, that only increases treatment costs. What’s not to like?
And what happens when the specialists who actually know how to interpret symptoms, data, and images retire? Medical tourism will be in your future if you can afford the cost and effort of making the trip.
Recall KLG is a professor of medicine who among other things did a lot of bench work in his day. GM works at an elite school you most assuredly heard of.
To their exchange:
KLG: Medical students love their AI. Or some of them do. Scary.
GM: Not just medical students, PhDs too.
I see it everywhere now. Thinking is increasingly offloaded to the AI.
KLG: I can’t remember the last time I asked a medical student in tutorial group a question that was answered without looking at a screen.
GM: Something to add — right now two very active areas of biomedical AI research are literature review and experimental design.
The former is already widely in use by people in the form of plain ChatGPT, both by trainees and PIs, but there are more advanced and specialized tools being developed too. The idea being that we don’t need to read papers, AI will do it for us and will provide a summary. The problem here should be obvious given, first, the well-known deficiencies of current AI, and second, the fact that a lot of key information in papers is kind of buried in between the lines or in more explicit statements deep in Methods sections and supplements that a “summary” will miss. And in mundane human things such as knowing how to interpret what was written by knowing the people who wrote it, background that AI does not have.
Also, people already mostly can’t be bothered to read papers, and just skim them, imagine what this will do to the ability to read a scientific paper (which is a skill of its own).
Still, let’s say there is some utility to AI speeding up the quick look up of things.
Experimental design is where its get really problematic.
I have seen several of these presentations now and the idea is that you want to do some experiment, but don’t want to bother putting together the protocol, so you tell AI to design it all for you, then you follow it. Of course, this is accompanied by some wet lab then actually following the AI prescriptions and showing that it indeed designed the experiment correctly.
And again the catastrophic downstream effects should be obvious. Once students get in the habit of trusting the AI, then how much training will those students actually get? Being really good at the bench means having all those mundane details about buffers and volumes in your head. Troubleshooting failed experiments usually depends on paying attention to such little details. And developing new methods depends on someone knowing the existing ones inside out and then building on top of that. Offload the mundane basics to AI, what is left of that very important expertise then? Yet this is the path that will likely be followed soon. It is still early days, but that is the clear trajectory.
I’ve literally already witnessed situations in which a grad student is giving a presentation in group meeting, then got to the point where she ran into an issue, asked ChatGPT for help, ChatGPT didn’t quite help her, so then she just gave up as in “ChatGPT doesn’t know the answer, what can I do?”.
Note that wet lab work is probably one of the absolute last things to be replaced by AI, because it requires very fine motor control of hands and fingers and is something that has always been based on “feel”. Prostitution is the only AI-proof profession, but experimental scientists will also last quite a long time. So you don’t want to corrupt them mentally.
However, that process has already been ongoing for a while even before AI. A precursor of all this was the inability to do mental math. That must have disappeared already before my time because I have run into this problem literally every single time I have had to train students. At the most elite institutions no less. Again, literal real life examples, and that situation has repeated itself many times, e.g.:
“We have a 1M master solution, we need to make a 50 mL of buffer that has that component at 10 mM, how much do we need to add?”
Blank stares, followed by reaching for calculators (and in more recent times phones). I stop them immediately and tell them:
“Do it in your head”.
They can’t. I tell them the answer (500 microliters), then we move to the next component, which also has to be at 10 mM, but the master solution is 5M.
“How much do we need to add”?
Blank stares again. Nobody can figure out to just divide the already arrived at answer for the 1M master solution of 500 by 5, no that is too hard. They start doing the calculation from scratch…
That was the already preexisting condition to which AI is being added now…
I find the mental math deficiencies to be quite worrisome. Without at least a sense of what is reasonable, large errors can go un-noticed, for a while.
At a visit decades ago to friends, I was asked to help one of their children with maths homework. The child was having difficulty implementing the procedure to solve a problem. I don’t recall the details, but I remember that I found the procedure to be more complex than needed. I think that maths pedagogy may have been needlessly revised from the simple, robust, workable methods I was taught decades before. What was the point of that?
Dumbing down of mathematics to unacceptable levels was observed by me back in 2000. During my PhD I taught health economics and medical statistics to med students. Decided they were beyond awful.
I’ve said this before but it bears repeating: dumbing down had happened to worrying degree even by 1991 when I took UK A levels (for university entry). Two of my subjects were mathematics and further mathematics. When we did past papers to practise for the exams in June 1991 we realised that papers from pre-1980 for practising FURTHER mathematics had to be real exam papers from the SINGLE mathematics subject. Ergo, much of the single subject stuff was considered too difficult and got put into further mathematics….. which virtually NO medic studies.
I read Economics at Cambridge and my Mathematics friends were compelled to attend a “catch up” (remedial) 8 week course over the summer before term officially began in October. Go figure.
I have an undergraduate degree in engineering and a grad degree in economics, both from the 1980’s. I remember being told as an undergrad that it was important to develop one’s math intuition and to always check the dimensions on quantities precisely because it would help in identifying when a calculation error had led to an absurd result.
My first boss—whose degree was in physics—could glance at a page of numbers and instantly point to one, saying “This is wrong.” I was in awe of him. Many years later, when I joined a team in the middle of a project, I saw that the current draft report had consistently left off an “M” in the units describing numbers of metric tons of carbon emissions. They were off by a factor of a million! I certainly earned my pay that day—and the undying gratitude of my new teammates—by pointing that out.
Indeed. My (tor)mentor had the ability to just look at a dataset of choice modelling data and make statements. I, over 15 years, gradually learnt this. It doesn’t matter what the log-likelihood of the logit regression says, that solution does not make sense in terms of real world behaviour (because there are an infinite number of solutions and all those models normalise the variance to one in order to give you a solution).
I got to be able to instantly look at data and go “big fat nope”. I tried to help juniors to develop this “art” but they generally were too indoctrinated with mainstream economics backgrounds and took the solution from the stats program as gospel. Plus they never got into the habit of doing “smell tests”: quick and dirty cross-tabs of things to see if it looks “odd”. And thus we end up with a bunch of complete idiots.
This will get WORSE under AI because the field I worked in dealt with “what if” scenarios from statistical designs, NOT existing data. So people will find themselves even more “stuck to the production possibility curve” and never even given a possible solution beyond it thanks to AI. But that’s not my problem anymore.
I don’t know about other countries. All I remember is that we had no teachers actually know how to teach math and physics. Biology was easier since it´s more relateable for the children. With pure numbers it was different. But how do you trigger fascination and understanding and purpose (!) within young people? It cannot be coincidence that these subjects were mostly if not dreaded but then disliked.
I do assume that little children are interested in anything. So something goes really wrong on the path from when they are 4 or 5 up to the point when they turn 10, 11. (at least in my time that was start of grammar-school). Pressure and competition are no solution. I loathe both concepts frankly.
Late to the party because reasons. When my now very successful son was in the 4th grade (9 years old) he was having trouble with a math problem. OK, says I, what can be that hard about a 4th-grade math problem. You won’t believe what I found. Well, yes, you will. I do not remember the details more than 20 years later, but it was a simple algebra problem, though it wasn’t called that in his math book. The problem was it was an unsolvable algebra problem: 2 variables, 1 equation. Yes, I flipped out but remained calm and told him not to worry about it. I was secretly proud that he knew intuitively that something was wrong with the problem and not with him. I did talk to his teacher, who was outstanding. During one conference she started in on his standardized test scores, but I told her I only wanted to know if he was a good student in class. She asked me to run for public office. In the reddest of red states. Ha! According to the curriculum, this was a problem that required “estimation.” The teacher was stuck. Because the curriculum was not to be violated, ever, she had to teach something called “estimation” to her students. I cannot imagine what the math curriculum looks like now.
The irony is that anyone who has ever run a limited dependent variable regression (logit/probit) has encountered that exact same problem but 99% of the time doesn’t realise what the stats program does in the background. It normalizes one of the variables to 1 (a function of the variance)!
The likelihood function underpinning these models has the mean and variance on the latent scale (utility or whatever) as multiplicative so you’d need a second equation to split them.
This is quite probably the most serious problem in all of statistics since it means most RCTs and other trials/surveys could be wrong.
Just downloaded ChatGPT and asked series of questions about Best-Worst Scaling. Cue LULZ.
It more or less got the order of importance of people in field right (yours truly is number 2 and number 1 if you count people still alive). However it very quickly started making erroneous statements regarding experimental design (which clinicians, academic marketers, social scientists etc) will no doubt take in.
At least I could tell that it hadn’t yet surreptitiously scraped our textbook and ignored my instructions to the publisher. I sensed much of the content had been scraped from Wikipedia along with marketing stuff (which is BLATANTLY wrong) from the biggest company providing black box solutions.
On a personal note. I asked Google, “Percentage of salt to make sauerkraut?”
Google AI replied, “2%”
One week later, the sauerkraut was rotten and had to be thrown out.
I went back to human beings, on youtube (I check the comments for veracity) and the correct answer is 3% to 4%.
In my naivety, I assumed that AI could answer simple/simple/simpleton, basic questions.
nope
A few weeks back I was with some mates, and I was only half-listening as they talked about George Foreman. When I got home I just idly thought – why the random discussion about GF? Oh, maybe he died.
I googled it (as you do), and Google AI cheerfully he told me, and I remember it very well so I will quote it “George Foreman is alive and well.”. Right above a news article detailing his passing….
Given the above anecdote, I can think of literally nothing that could go wrong with AI doing medical diagnoses….
How about bomb disposals?
‘AI – do I cut the red or the blue cable?’
2% is plenty for sauerkraut. Worked for me many times and is the ratio recommended by Sandor Katz in his various fermentation books.
Don’t blame google AI there. 2g salt for every 100g of cabbage.
3-4% will work but will be too salty for my taste.
maybe something else in your method went wrong?
EPIC has AI embedded. Currently being sold as transcription into form of traditional progress notes, yet labs and meds are already tied to reimbursable indications. Just need a robot to listen to chest and check reflexes and Evidence Based evaluation and management no longer requires thought. It’s like that was an ulterior motive for electronic records all along…
Christ, I’m pretty sure the hospital systems whose data I manage use EPIC (I manage output data, not anything internal, so I’ve never used it)…I wonder how much of the information I’m responsible for shepherding is actually hallucination. Probably not a ton yet if just because AI is fairly new and in my region anything new takes tremendous hemming and hawing before actual implementation (this is a good thing in my view). Still, I dread the day when I will have outlived the doctors and epidemiologists who didn’t come up on asking the machine for the answer to the question every time.
On other news Klarna is going to bring back humans, they say, because of AI fails. The withered shell of the IRS quickly found that the AI chatbots start out limited and get worse and worse over time.
The Neverending techschmoe and C-suit search for unpaid workers who cannot complain is a tragic story.
I have learned that the best way to educate in medicine is to use real-life examples. So I will briefly discuss yesterday’s fiasco – the details are all fresh in my mind. But trust me, similar and much worse things happen every single day.
32 year old athletic male was doing an aggressive hike and fell down a hillside about 3 weeks ago. Instantly had severe pain in this right heel. Had to hobble back to the car. The next day things seemed to be getting better. About two weeks later, however, there was still lingering pain in the right heel, but now he noticed his entire right calf was a bit red, and a bit warm and a bit tender. This continued on for a week ( 32 year old jocks tend to ignore pain). He showed up in the local urgent care to be seen by the Nurse practitioner du jour. This particular NP graduated from nursing school ( no BSN) – and within 6 months had already started her 2 year correspondence course to get her NP degree and to become a medical “provider”. Our EMR now is so handy that it looks over all the symptoms and labs and X-ray findings etc – and gives you a list of “suggestions”. I could write a 4000 page comedy novel of all the suggestions provided to me the past little while. Just use AI for anything in your life for 5 minutes – you will see what I am talking about. Well, the NP and the AI decided that he had a deep venous thrombosis in that leg. So, she immediately ordered an ultrasound of the leg – the clot was deep in the calf muscle and did not even get close to the knee – nothing in the thigh. So, either she or the AI – it is hard sometimes to tell – made the diagnosis of DVT – and started him on Xarelto – at an out of pocket cost of 900 dollars a month. Over the next 2 months, every 2 weeks the patient underwent a repeat ultrasound to check on the “progress” of the clot. This surveillance is not even closely medically indicated in any way shape or form. I am his PCP. I was notified that a FIFTH ultrasound had been ordered, the radiologist had become wary – and “Gee, maybe someone with a little experience ought to look at this.” I did not know the first thing of this story until he called.
I do have experience. I did Gross Anatomy and carefully dissected every vein, nerve, muscle, tendon and artery of the lower extremity under the careful tutelage of an absolutely sadistic retired orthopedic surgeon. My training was intense, it was ugly, it was a hazing, it was THANK YOU SIR MAY I HAVE ANOTHER SIR 10 times a day. But I have come to realize that is the only way. I learned all about a very old medical concept called Virchow’s Triad – that is still very important to go through in your mind EVERY SINGLE TIME you see a blood clot. I have taken care of innumerable blood clots in my life in any and all locations in the body. As a resident, I saw these all the way through from clinic to hospital and have continued to do do this for 35 years. I do not need AI or UpToDate to guide me through.
So, what may be missing in nurse practitioner correspondence school? Well, they dissect nothing. So they are blissfully unaware of the nexus of veins in each knee that prevent clots in the calf from propagating up into the thigh. This is important – when a clot forms in the thigh – it can easily dislodge and go right to the heart and lungs. Not so the calf – the knee vein nexus keeps it right in place. Had this provider attended medical school, she would have known about Virchow’s triad years before she ever laid eyes or hands on a real patient – again the triad should be right in the front of the internist’s brain when looking at a fresh clot. Basically it is WHAT CAUSED the clot? A triad of three things – 1) Recent or distant leg trauma 2) Medical conditions that make people clot – like all kinds of bleeding disorders, pregnancy etc 3) Cancer. The END. You must account for this in every diagnosis. So, the first thing I did with the patient is a careful examination of his still painful foot – oh, on a simple x-ray of the foot – there is a very large crack – a fracture – right through the heel bone. Neither the NP or the AI had even touched or examined his heel in any way. Now that we are approaching 6 weeks out – it may have already started to heal – and heal wrong – so much misery may be in store for this young man. The NP and the AI however did decide that he needed a vascular surgery consult because of this issue ( I MEAN REALLY?) – and he was scheduled to see a vascular surgeon at the end of June. We did not even have an orthopedic consult go through brain cells or circuits however. And this is the nexus of a whole new problem in medicine – the non-trained providers and the AI are just flooding the specialists with all kinds of consults that have been historically handled by internists or more likely just absolutely lunatic like this vascular surgery consult would have been until I cancelled it yesterday. That is what we were trained to do – an internist will only send out the most complicated of things. When I look through NP charts today – there are at least 1 and sometimes 3 referrals on almost every one.
But the most important issue – is that the patient’s calf blood clot needs a heating pad for several days and an aspirin a day and very close observation by a physician who knows what they are doing. No 800 dollar a month Xarelto – no ultrasounds twice a month – just good evidence based and experience based care. I looked at this 32 year old’s account yesterday – 32,000 bucks and counting. AGAIN, most importantly, he needed someone to diagnose the underlying issue – the fractured heel.
Just FYI – when you accept care from an NP or PA in most states – you are accepting that the level of expertise is going to be much lower than an MD. You then must realize that in the medicolegal world, your ability to be compensated for malpractice is greatly diminished and may even be zero. And honestly, I am not even sure what the rules are for suing AI – do we get to sue EPIC? UpToDate? I have no idea.
It is all I can do every day not to despair. I do not know what else to say. I am just often looked at by the MBA crowd – and you can just tell they are in way over their heads. The system is just cratering – and I am pretty sure it will be the same as the old line about “Going broke gradually, and then all of a sudden”. God help us – and God help this young man. The ortho docs are going to have to refracture his heel in the OR – and then weeks of painful therapy.
More patient visits, more expensive tests, more expensive consults; I think the system is working as “intended”. Sigh.
I thought I would add two experiences to this very interesting account by IM Doc, as it chimes what he’s related.
1.) A year or two ago, after moving and seeking out a new primary care physician (PCP), by some infernal means I got locked into a nurse practitioner (NP). I’m still not sure how it happened. The person I talked to on the phone is the most likely culprit, as I suspect they slyly asked if “so and so” would be a suitable PCP for me–without mentioning they’re not a doctor!–and I casually assented. Trying to switch from this NP to a real doctor proved onerous. The powers that be did not make it easy. Since I’m relatively healthy, I shrugged my shoulders and forgot about it.
Then I had some issues. I go in to see my nurse practitioner PCP. She’s nice enough, but never in my life have I had so many tests and referrals. It was wild. For one issue, she sent me to a urologist. The doctor, a young man with great bedside manner (as I’m sure is important for urologists, considering), was surprised to see me, as I was unusually young. He even said something to that effect. Well, when I explained my problem, I could see that he was annoyed–not with me, with the fact that he was seeing me for this problem–and he spent the remainder of our short appointment very kindly reassuring me that this problem was no issue at all and that I did not need to worry about it. I now realize that his annoyance was obviously with the NP.
So I did what I had to do to switch PCPs (a needlessly annoying process). I get a real doctor, an internist. At that time, the local health system was short on nurses. So this internist did the usual intake stuff that the nurse usually does. She took my weight and blood pressure (BP). The BP result was so good that I commented on it, and the internist said that the nurses rarely do BP correctly and pointed out all the things that she did while taking my BP that, indeed, the nurses never do.
When I explained to this internist my experience with the NP, she agreed that they’re pushing NPs hard, and after I mentioned that she (my actual-doctor PCP) was much better than the NP, this internist quipped: “Well, I did go to medical school!”
2.) My father, who’s retired, told me over the phone that he was getting a skin cancer screening at his dermatologist’s and off-hand mentioned it was with an NP. I had recently gone through the NP saga related above. I insisted that an actual dermatologist conduct his screening. He brushed me off, as an NP had done his prior skin-cancer screenings. It turned out on the day he went in that the NP couldn’t be there–so an actual dermatologist did my father’s screening.
Dad called me immediately afterward to tell me how right I was. He said the dermatologist was amazingly thorough, laid hands on him, and discovered all kinds of other problems with his skin and was able to help my dad, who has very sensitive skin, immensely with advice and prescriptions. The NP screenings he’d had in the past were basically glances over sectors of his body, an “all clear,” and out he went. My dad said that he’d demand a doctor for such appointments from now on–the difference was, in his words, “night and day.”
You are right about the BP. Most nurses and all nursing assistants do not know how to properly take a BP. I go to the VA as my PCP and in my 20+ years of going there, only one nurse got it right. The VA is very big on NP’s too. I have an NP as my PCP and luckily, she is pretty good. Very professional and detail oriented. But still, she doesn’t even compare to my MD cancer doc. There is a huge difference between a competent doctor and an NP.
I had a very similar expertise with a talus fracture. It took three years before they took an Xray that would image that hard to image bone, and by that time it was mal-union neck and non-union lateral process. Terrible injury, recovery took years and that was after the injury was diagnosed. The MD that finally found it was an old guy with a leather notebook, whom I saw years later by chance for a physical and he had me flagged in that book for the talus, which we discussed. There’s no way a non-displaced talus fracture is going to get flagged for special (I remember holding sandbags) xrays by AI. It’s simply too rare of an injury, especially for non-car/bike/horse/ladder causes like mine.
NO ONE THOUGHT TO LOOK FOR A HEEL FRACTURE??? This is criminal.
But also, why didn’t the patient get stroppy? I would have demanded a heel X-ray (and I’m normally the type who says no to imaging unless there is a good reason for ordering them).
“absolutely sadistic retired orthopedic surgeon”
That is the type of mentor who will drill the necessary knowledge into your head in such intense way that it will remain in your head even after you die (hm?)
I had a few of those through my engineering studies and as much as I hated them then and wanted to run them over with my car, boy do I just love them now. And have loved them in the last 30 years of my professional career. For they drilled into my head such an instinctive feel for materials, stresses, defects, damage and instant recognition of negligence and botch jobs without which I feel I would not be doing my consulting job properly.
I feel you hated that guy then, but suspect you are grateful to him now.
Big respect for all of your great posts.
There were sadistic biochemists, cell biologists, and geneticists who did the same thing! Scary doesn’t begin to describe them. Now I cherish every single time they made me feel inadequate. Like one of my high school coaches recently told me 50 years after he made my football and baseball life miserable on the practice field. But as I remember it now, never during the game no matter what happened on the field:
“You didn’t need to worry until I stopped yelling at you.”
And he never stopped until the very last game, in which we won a state championship and after which there were no more practices. I asked him if he could coach now. He laughed out loud.
There is a connection there.
Homage to IM Doc….
“To pathology we owe the realization that the contrast between health and disease is not to be sought in a fundamental difference between two kinds of life, nor in an alteration of essence, but only in an alteration of conditions.”
Rudolf Virchow 1822-1902
“Disease, Life and Man”, selected essays by Rudolf Virchow, p 169
You mean there are other types? Sorry but having had the top UK rheumatologist as my ultimate boss, his dislike of orthopods rubbed off on me somewhat.
To be a bit more constructive, my experience with NPs here in UK is just as variable as with GPs. Some are just awful. One was brilliant in taking the bull by the horns and getting 3 consultants at the hospital to agree to go by majority vote as to whether I could be discharged after a 24 hour in-patient stay for suspected sepsis. The infection was gone but one consultant held out but was out-voted so I was let out. She knew that I had made a perfectly sound decision that the risk of staying in a hospital ward with COVID floating around any longer far outweighed any other risk which the one holdout was unable to put into words anyway. I wanted out of there and she managed it by playing umpire. She was a rarity IMO.
Since you triggered this… Way back when I suffered a sports injury, dislocated knee (stepped on a spool while retreating in fencing). Luckily the closest ER was actually a specialized trauma ER and basically across the street so I was helped there in minutes (one of my mates even took care that my bicycle was stored in the sport-center’s carage as not to be stolen).
Within 12 hours I met three orthopedic surgeons. The one on duty in the evening told me I’d better start thinking of becoming a trainer, that the knee would never be the same. The surgeon on the morning rounds figured that the knee will be ok and allow sports.
And finally the afternoon surgeon, who was the one to operate my knee promised to make it better than it ever was. I must say I was happy he was the one to operate. I guess his attitude rubbed off on me, because I recovered really fast. I did more exercises than my physio demanded, I learned to listen to my body and push it to the healthy limits, and a year and a half later I won the championship, basically because my legs were better than anybody. And you fence with your legs.
Yeah it has a lot to do with the traditional PHYSICAL requirements to be an orthopod: you needed to be pretty physically strong (increasingly so in the west with obese patients needing legs raising to do joint replacement etc). Thus, in contrast to many other areas of medicine (which became much closer to 50/50 in terms of gender), orthopaedics remained the realm of (in UK anyway) the “ex rugby playing private school guy who had no interest nor need to talk to patients. Patients were slabs of meat on the table to be fixed”.
As Yves said recently IIRC, with more keyhole procedures, there are more orthopods who don’t need these physical requirements and so you’re more likely to get one with a good bedside manner who understands what other things in your life you might value. Again, IIRC, she had a good experience with one.
So things are looking up. But there are (in UK anyway) still a lot of old school ones. One did both my mum’s total knee replacements. Neither of us was happy with the results, even allowing for the fact knees are well-known to be more difficult than hips.
Probably a kneecap dislocation as opposed to a knee dislocation. Knee dislocations are problematic. And the mass of primary care referrals we get generate a bigger mass of MRIs which generate a lot of billing….I loved the episode of the Sopranos with the health care exec on the overlook at the Passaic River falls in Paterson NJ. He was discussing MRIs which Tony had an interest in.
I get the idea from talking with British physicians I know that the NP pathway in Britain is much more appropriate than it is here – I mean orders of magnitude. In the USA, it is literally about 2 years of correspondence school any time after you graduate from nursing school. There are a few programs here and there that are more formal – but all of the NPs I work with here are from the correspondence school model. They have no real licensing or specialty boards – so they can just call themselves primary care or internal medicine or peds – and we are off to the races. The really fun ones are the ones who have the above level of education and then suddenly are GI docs or cardiologists. It is demoralizing to the actual cardiologist – who has done 4 years of college – 4 years of med school – 3 years of Internal Medicine residency – and another 3-5 years of cardiology fellowship. But of course to the MBA mind – that is all the same – and the NPs are MUCH MUCH CHEAPER as far as salary. And now we have AI to tell them what to do. So, who cares? It is all going to work out splendidly. It works great for everyone but the constant flow of patients as I describe above – and for the absolutely overwhelmed specialists and for people like me – who are experiencing such overwhelming moral burnout dealing with all the disasters – and on top of it all being given the task of “educating” the NPs. This task is now part of our job – the MBAs will have it no other way.
Thanks. I never thought that a NP might be so differently qualified/experienced across our countries.
Like I say, there are some around here who I think are very much “follow some simplistic rules” and I do wonder if they just “revised for the test”. But 2/3 of the ones I’ve seen have been pretty on the ball and in some cases have actually been very very good (better than the GPs) at being holistic and thinking across all my symptoms etc.
I hesitate to generalise TOO much, but my clinician (GP) friends have also thought that getting a good NP for their practice is like gold dust in terms of making everybody’s job easier. Ironically a lot of the nurses I encountered in Australia 2009-2015 were British emigres who I thought “you really should be a NP back in UK and it’s awful we lost you”.
When the NPs and PAs are used as an adjunct or to help with all the tasks and simple follow up visits – that is at times very helpful. When they are out on their own – or left to man urgent cares, not so much.
Yeah I can understand that. Our NP at our practice is pretty good as an adjunct in looking across all my long COVID/mental health stuff and alerting one of the *good* GPs “Terry has changed in area xyz – please advise if he needs to be called in….”.
The previous one was useless. The irony is that my suburb is (whispers) pretty awful in terms of many GP outcomes. But the mean values conceal some high variances…..you have to know how to work the system….which I do….but it’s really unfair that so many don’t. That’s not how the NHS is meant to work.
Very concerning. I was a lab assistant in a research lab as a HS student – paid better than mowing lawns or packing groceries, plus looked great on my college applications. Mental math, following protocols, the how and why of experimental design – all of that was repeatedly beaten into my brain and the uptake constantly rechecked. Made it just a bit easier when going through two semesters of Research Methods combined with two semesters of Statistics in college, making junior year the stuff of nighmares for every psych major. The flip side is that even though I went a different career route, I can still be woken in my sleep to check a study design and will still catch most design mistakes, even without AI. That future medical professionals are being dumbed down in critical skill sets is my new stuff of nightmares.
I’m no math genius, and I easily figured out all you had to do was divide 500 by 5. But I was educated in the 60s and 70s…
Unfortunately, I’m math-deficient, and could not get it, but I suspected it was a relatively easy answer. However, the terms were unknown to me: what does the capital “M” designate? And, when I say, “math-deficient,” I mean it’s always been tough for me once things started into algebra territory. Like most people, I can do “basic” math (and even taught it to third-graders–man oh man, division got many of them every time, just like it got me when I was their age, but I knew from bad experience what not to do), but thank God I didn’t need to know quadratic equations and other esoteria for my life’s work (house-painting).
Thanks all! Interesting comments here that provide granular insights in just how the realm of ‘professional’ experts like those in medicine is being re-engineered for the brave new world a la Kornbluth’s ‘The Marching Morons’ and ‘The Little Black Bag’ which the US is becoming.
https://en.wikipedia.org/wiki/The_Little_Black_Bag#Plot_summary
I want to personally thank you for these story recommendations.
I have never read either one – I am slowly but surely going through all the classic SCIFI stories and these are the next two on the list.
So many of them are real dazzlers – and have so much to say to us today.
I easily found them both in their original publication on Archive.org
This is the link for The Marching Morons
And this is the link for The Little Black Bag
I love to look through these old scifi magazines of the 1940s 1950s and 1960s. It is an absolute delight to look at the cover art. And furthermore, there is almost always another story or two in each one worth reading. The modern scifi stories literally pale in comparison to these old classics.
I am not sure what happened to the cover of Astounding July 1950 – from which comes The Little Black Bag – but with a little looking around, you can find the cover art easily elsewhere.
Archive.org has almost every one of the magazines of scifi in addition to earlier pulps – detectives and westerns totally free – you just have to hunt for them. Amazing.
A worrisome anecdote from a recent chat with a friend who is the senior class math teacher at an elite New England prep school (think 65-70% acceptance into Ivy).
A student in her advanced calculus class went from being middle of the cohort in the fall to the lowest this spring. The teacher has been working with her off hours to try to understand why, and the real reason came our last week with one week to go before finals: the student was gifted a paid subscription to an OpenAI reasoning engine over Christmas! She has been ignoring all class notes and exercises with answer sets and has been ‘studying’ with ChatGPT, and the logorrhea it produces has absolutely tanked this kid’s classroom ability.
It would be amusing, if the future we are headed to with this specious bullshit were not so clearly in the direction of that movie ‘idiocracy’.
It seems that as critical thinking skills atrophy, the use of “AI” (the term itself is way too broad but I’ll just stick with it) will create a positive feedback loop of enshittification. Breakthroughs in science and medicine require “thinking outside the box” and all AI can do is stay in the box.
As authentic human inputs become increasingly scarce, I suspect these models will start resembling encyclopedias written in the year 1543.
Yep, sorry if you’ve heard this before from me but AI cannot by definition model/scrape points to the north-east of the production possibilities curve because there are no data; we have no idea to what extent there are two-way, three-way and higher order interactions etc unless people like me did discrete choice experiments.
Yet another reason why it “makes stuff up”. You must set up well-designed experiments to generate those data. AI cannot do that. Hence why I laugh at it rather than feel threatened.
It is hard enough for humans to make good guesses in that area, using logic and other skills. AI? Nope. Big fat nope.
I take a swimming for fitness class at my local community college. It’s a for credit PE class for regular students. There is an ‘educational component’ to the class and a ‘final exam’ is mandatory to pass the class.
That exam was last Monday morning at the outdoor Olympic pool while sitting in the bleachers. After the written portion, the class assembled to answer group questions: ‘Whom has the fastest freestyle time for 50 meters?’; What year did the Los Banos pool open?’; etc. All the while one class member surreptitiously held his cell phone up to capture the verbal questions. ChatGPT captured the questions and answered correctly on two of the three questions.
I don’t take my cell phone to the pool.
I sterted getting more and more invites through LinkedIn for jobs to train AI in health sciences.
I decline every time.
The only time I used ChatGPT was to categorize a list of declared occupations according to the US Equal Employment Opportunity Classification system (I prefer it to the Canadian NOC because it instills a certain hierarchy in it that is fundamental in describing some latent variables like independence at work, underlying education, etc beside being more simple).
The conforting part was, after a lot of back and forth, was that it used statistical tools, but not the whole array, not being able to download some packages from R doe fuzzy detection.
Having done programatically this kind of thing before, with days and days of tedious work, this was much faster with decent results. And I would very likely integrate this methodology for future work, which is kind of operational, while building a big database of already (well) classified occupations that can be further used classify new data. But this is such a targeted aspect, with sound methodological approach underlying it that I am confident to use.
For diagnostics though, while in the past I was a proponent of potentially using AI as a help, I realized that given the sheer complexity of the information, it can become very easy for AIs to start hallucinating and, of couse, there would be the bias introduced in demanding more tests…
The description of the old training methods for docs reminded me of the mentats in Dune. Yes, we should continue to produce them and renounce the craze of AI. I am more and more convinced on the righteousness of the Butlerian Jihad (which must have a sort of Marxian ideological underpinning – after all, in the unfettered, unregulated XIX century capitalism, mothers were induced to give opium to the newborns to calm them down, so they could go to the factories to work, which reduced the life expectancy to maybe 2 years).
BTW, in the new Dune movies and the TV series associated, have anyone noticed that there are no mentats involved?! Now why is that?
The underlying problem we have is some ingrained laziness and desire for a quick profit. This is ultimately how the US economy was hollowed out for the past 40 years.
I am though curious to compare the performance of Chat GPT in the above problem with DeepSeek. The initial tests were, from a qualitative way, inclining for DeepSeek, but there were some size limitations of the data input it could handle.
Regarding invites to “help” with AI, I’m quite smug that I have such an obnoxious reputation amongst some quarters that I’m pretty sure they won’t touch me with a 10 foot bargepole.
Don’t get me wrong: biggest course c2011 for all the top choice modellers in the world to teach an executive education course and I far and away came top as best teacher: I ALWAYS had three ways to teach something. I’d watch people’s eyes after explanation one and see who was saying they understood when they didn’t, so I’d used explanation 2, then go on to explanation 3. I pretty much had everyone on board if I had to use all 3.
But if you were an annoying neoclassical who wanted to make trouble by claiming humans are deterministic then I’d say “then why are you here? It CLEARLY says what our paradigm is in course description.” I wasn’t afraid to call out the troublemakers so economists mostly hate me. A badge I wear with pride. But in case you think I’m just being objectionable for the sake of it, it is like talking about satellite orbits to somebody who turned up purely to argue flat Earth.
Suffice to say there are a bunch of (primarily North American but I could name some Swiss/Swedish people) who do their best to airbrush me out of the picture. I LOL. AI shows no inclinication to ask me for advice, which is probably good :)
One day I will buy your book.
You should be happy to know that it is not piratable…
Yeah, thanks, I am happy. Though a slightly (though still pitifully small) uptick in last royalty payment suggests either some people who realise they can’t get the designs from AI have bought it or one of the AI owners has bought the electronic copy to scrape the detail anyway and it might just be a matter of time until the designs etc are given by the AI problems.
However, it still takes a lot of on the job experience to know which of (say) 3 possible statistical designs to use in a study since each one has its own strengths and weaknesses in terms of respondent fatigue/data quality (signal to noise ratio) etc. The response I made to KLG above illustrates one of the statistical “judgment calls”: you have a perfect confound of two key variables and I often can make an educated guess as to the “right” split between the two but it is becoming increasingly difficult as the methods are used in wider applications that go beyond the experience of people like me.
Oddly enough, a week ago I was in seminar with a lot of talks on AI in medical research by doctoral students, and after some talks I honestly asked the same thing from a colleague sitting next to me: don’t these kids know common sense anymore?
I think it was after being presented a machine learning model that did what a somewhat skilled sql-query could do. It was about longitudinal data, and that is always complicated*, but you still don’t need a black box of regression analysis to tell which patients had two consecutive treshold-triggering lab measurements within 90 days of each other.
* where topology is the deep end for mathematicians, for computer scientists it’s the handling of time
Everyone visiting this portal knows that this shiny AI crutch is hammered down our throats for a reason as old as hills – to make inadequate operators look adequate by arming them with sufficient slang and braggadocio so that unsuspecting marks (tax-paying funders of public health services) regard them as real services providers. Which of course they are not, not by a long margin. This is the very definition of fraud.
A couple examples from my experience.
A person close to me was invited to have an intrusive examination by a consultant surgeon. After drugs kicked in, the surgeon disappeared and a Nurse Doctor appeared and then went on to perform the procedure on a visibly severely inflamed internal organ owned by a very stoned patient victim. The result, terminal damage to said internal organ, after which the unsuspecting patient victim had to have it excised and be condemned to the rest of life with a disability. The victim’s litigating Solicitors’ expert witness, a top top top toppest UK surgeon blandly guided the case Barrister that this is simply the way of NHS these days and therefore not carrying the case in Court; nothing wrong with bait and switch and nothing wrong with the fact that Nurse Doctor had no idea of patient victim’s anatomy or medical history. And nothing wrong with the fact that Nurse Doctor did not provide any scope photos of the near fatal procedure. Sorry, computer said no to the images.
The other one is mine – after several visits to ED with severe chest pains, and of course with consultants absent from ED wards on every occasion, at some point a puzzled Nurse Doctor “in charge” requested I should have to have probes inserted into my heart muscles so to see what is going on. This did not sound encouraging so I said no at which point ND got so upset to write a stern report saying that if I did not have this sorted right now in accordance with his guidance, I would die there and then. This was enough for me, I tore out all the cannulas out of my arms and with blood gushing all over, left the ED.
Then I went to see a Doctor. Like, a real Doctor. He ordered a MRI of the heart which showed a good ticker. Then a gastroenterologist figured that gallbladder could be the culprit and another MRI confirming this and a laparoscopic job to follow and hey presto out the old gallbladder went. I feel 10 years younger now and reassured never to trust these clowns with my body problems.
I got about enough money to go through further 3-4 big ops before I clock off but at least whatever these come to be I will have them done by consultants and surgeons I vet myself and find adequate for the job. None of this NHS life lottery for me, with or without AI, thanks.
Just a side note. Even prostitution isn’t AI proof anymore. Plenty of AI/virtual models on Onlyfans and other similar sites pulling in a lot of money. Welcome to the gooner economy.
Great bunch of comments here even if extremely disturbing. Hard to believe that all these advanced skills are being thrown away and people relying on what exactly? A computer program that is notorious for making stuff up? Seriously? This really sounds like that medically we are rushing into the “Idiocracy” future for medicine but without having to wait for all those generations to go by first-
https://www.youtube.com/watch?v=LXzJR7K0wK0 (32 seconds)
Maybe we have been getting a glimmer of the future here on NC with Covid. Right from the get-go people were on top of it and were bringing to light actual facts about what was happening and what was effective while government officials, medical authorities and the media just gave out continuous lines of bs such as the pandemic is over. So for medical problems, are we going to have to do our own research so that we can get semi-qualified practitioners to look at alternative diagnosis? Ones not found by asking ChatGTP?
I am a research scientist and have 20+ year bench works under my belt. Last summer I had an opportunity to “train” a undergrad, who literally asked me whether he could use a calculator to calculate 50/5. This was after I spent days going through the calculations on paper and pen for buffer preparation. He also froze when I asked him how to balance 9 samples in a rotor that accommodates 30 tubes. Actually, it was not just undergrads. Several grad students and post-docs who went to fancy school also made me wonder the quality of their training…. I am very depressed at the trajectory of science….
“…how to balance 9 samples in a rotor that accommodates 30 tubes…”
OMG. I’m just a recovering plumber but even I would observe here that both numbers are factors of three.
You can perfectly balance 5 or 7 in a 24-tube rotor too.
But that is even harder to see for most…
It took me a little while to figure it out, but when I did, it was an immediate “That’s so simple!” moment.
(Imagine today’s maths based “experts” having to do calculations with a slide rule. I’ve still got my Dad’s K&E Log-log slide-rule somewhere in the library.)
Thinking on human versus machine; I had a first generation Texas Chinese room-mate at university. He could beat the original Texas Instruments hand held calculators using an abacus. He won us several schooners of beer doing it at the campus side beerhall.
There are real Mentats in the world. All it takes is the right training.
Muskian Jihad anyone?
Kudos to you, ambrit. I tried to teach the same undergrad about the trick balancing 7 tubes in the same 30-sample rotor and eventually gave up…. and told him to find an extra tube to make them even sample number.
As I am running out of the runway for my own scientific career at the moment, teaching simple logic to people who should know better is not a hill I am willing to climb.
Before AI, increasing portions of academia across all disciplines lived by the dictum, for professors, “we pretend to teach,” and, for students, “we pretend to learn.”
And the name of the real game, for increasing numbers of faculty and students in the 1970s, 1980s, 1990s, 2000s, 2010s,and 2020s, was first getting the necessary certificates by any means necessary, then the salary increases, then the grants, and finally the prestige.
AI is the perfect structural end-product for this cesspool of largely pretend learning and teaching.
Now, no more pretenses about either teaching or learning actually going on–AI will take care of everything for everyone.
Western medicine is limited enough as it is.
It virtually ignores nutrition, and knows next to nothing about the immune system. It is not preventive, and deliberately ignores natural therapies, treating them all as quackery regardless of actual efficacy.
AI will only be as good as the programming and the data it is given, as well as the built-in biases – so, in short, not much better than the average doctor who is stuck at the same level of knowledge as his/her training years, and rarely progresses beyond.
Who knows, AI might even show up the many limitations of western medicine, and make all of those real ‘quacks’ redundant.
AI won’t show up the deficiencies in western medicine if that is all it knows about. It will usually get an approximately correct answer within set paradigms (no answers involving the gut biome, or immunology); and every now and again it will ‘hallucinate’ something that will use known terminology and look at first glance plausible.
Bravo for reporting the zeitgeist. A regular benefit of NC.
Every survivor of health problems is going to need a respectable reference manual on anatomy and physiology and practitioners who can explain their diagnoses.
I have long treated my health as needing to meet my caregiver intellectually half way where we can agree on the problem(s). Thus I don’t need extensive training, just the persistence to understand a narrow scientific study. I just need to understand the rationale for what I’m being told.
Then my heavily trained partner in my medical journey will likely know how to select an adequate protocol / repair person.
I have yet to give and AI a prompt. I see that as comparable to walking around with a sandwich board saying “Tell me a lie”.
I suppose all the major medical journal article are scraped into the med AI programs. I’ve read here on NC how fallen those journal articles are now from what they once were before corporate capture. Sounds like med AI will become another case of Garbage In, Garbage Out (GIGO). In a reiterative loop. Maybe AI is starting to author med journal articles. Does med AI become self-referential GIGO?
So, will AI Medicos “delete” their failures?
ChatGPT-enabled cheating is allegedly rampant at all colleges, elite and not. as many classes, of all varieties, allow unsecured laptops to be used during exams.
I imagine no one takes their exams by hand. Econometrics to Shakespeare, most of my exams were done by hand, usually written in university-issued mini-notebooks.
those were the days, lol
Yes. I remember essay based tests in High School.
One teacher explained a low score to me thus; “I want to see you explain your reasoning. Then I’ll know that you understand the subject.”
The fundamental drive for digitization and AI is a form of religious fundamentalism. Think about the man waiting on the mountain and here comes the big arm out of the clouds with what? The tablets…or think of the Protestant Reformation. Or think about Ramon Lull and Liebniz (Calculemus) leading to George Boole (The Laws of Thought) and finally Claude Shannon. Anybody see Elon Musk’s remarks about TruthGTP?? The age old western drive for an infallible authority that is non-human. You have to take humans out of the loop because we are fallen creatures. We cannot be trusted with a burnt out match. And of course this effort has always failed. And it will fail again.
Do all these “students” get to use AI when the tak IQ tests?
We (patients) need to develop a protocol for asking where PAs and NPs have been trained. I.e., how do we get people who have done more than correspondence school?