Our current AI models handle complex challenges, like predicting how a sequence of amino acids folds into a complex 3D protein structure. But it will get simple math problems wrong. And it will state the wrong answer with complete confidence.
By David Sussin
Since the 1950's we've been actively pushing for computers to do anything humans do. It's like humanity can't wait to collectively retire from all work.
For some, the dream is for computers to achieve "AGI", or Artificial General Intelligence. This is the point where a chat bot evolves from asking how it can help with your phone bill to replicating the full spectrum of human intelligence.
Reaching AGI, large language models like Gemini or ChatGPT would be able to generalize, handle problems that cross disciplines, apply common sense, see a bigger picture when tackling questions, and -- most impressively -- learn. AGI level computers would improve themselves recursively at a pace faster than humans, making breakthroughs in science and medicine and everything else.
That's the utopian version. And, with current AI models already performing mind-blowing feats, AGI seems like it's within reach.
But the current version is not there yet. It has major flaws. The big one? It hallucinates…
Why Billionaires Are Stockpiling This "Boring" Token
The world's largest financial institutions are building massive positions in a protocol most retail investors consider too "unsexy" to notice. As markets are volatile post-tariffs, this coin continues setting transaction records while flying almost completely under the radar.
Discover the "boring" financial token that's making the elite wealthy for only $3.
Prominent AI Researcher Andrej Karpathy coined the perfect term for AI's current state: "AJI", or Jagged Intelligence. Karpathy says this describes the "strange unintuitive fact that state of the art LLMs can both perform extremely impressive tasks while simultaneously struggle with dumb problems."
Our current AI models handle complex challenges, like predicting how a sequence of amino acids folds into a complex 3D protein structure. But it will get simple math problems wrong. And it will state the wrong answer with complete confidence.
You will never find an AI agent who says, "I don't know." Why? Because they don't really "know" anything -- they're just putting word tokens together. When they look for an answer, there are always words that can be assembled. More often than not, those words form the correct answer. But there's never a time when the AI agent can't find words.
This is one of the big reasons why USC's Dr. Mitchell Goldenberg says AI is not ready to replace doctors. In an article for Keck Medicine updated this year, Goldenberg admits AI can help a physician make a diagnosis, but isn't ready to replace the doctor's clinical decision making.
Based on a 2024 survey from the American Medical Association, most physicians agree: 70% have equal or more concern compared with excitement over increased use of AI in health care.
To be sure, AI is doing amazing things in the medical world. Deep learning models are trained on vast datasets of medical images, including X-rays, CT scans, MRIs, mammograms, and pathology slides. It's more than any one doctor could absorb.
AI uses this enormous dataset to identify subtle patterns, anomalies, and early signs of disease that might be difficult for the human eye to detect, even for experienced radiologists or pathologists. For example, AI can detect early signs of breast cancer in mammograms, identify polyps during colonoscopies, or spot tiny brain lesions in MRI scans more accurately and faster than humans.
In this same way, AI can absorb a patient's entire medical record, lifestyle and family history, and use this data to identify intricate patterns and correlations, helping to predict a patient's risk of disease, and forecast their response to medications. AI can even suggest the most effective and least toxic treatment options.
AI is a superpower in the doctor's toolbox, a sophisticated "second pair of eyes," flagging urgent cases the doctor might miss. Rather than, say, replacing a radiologist, it makes the radiologist mind-blowingly effective.
Given that AI makes mistakes, it seems like an easy call not to let it make medical decisions. Is AI at the point now where it can independently diagnose a patient without physician help? Goldenberg puts it bluntly: "I don't think it's ready for prime time yet."
But one person disagrees, and he has enormous influence on health care policy in the United States. Dr. Mehmet Oz is the newly confirmed Administrator of the Centers for Medicare & Medicaid Services and serves under HHS Secretary Robert F. Kennedy, Jr. and President Donald Trump.
Oz controls a $1.5 trillion budget and oversees the nation's major healthcare programs, including Medicare, Medicaid, and the Children's Health Insurance Program (CHIP), providing health coverage and setting quality standards for 160 million Americans.
In his debut all-staff meeting this April, he had a major change in mind: prioritize AI avatars over frontline health care workers. Dr. Oz emphasized the cost savings, pointing out an actual human doctor might cost $100 an hour to make a diabetes diagnosis, where an AI avatar would be as little as $2 an hour.
This may have been in response to cuts in CMS staff. Part of Secretary Kennedy's plan to eliminate 10,000 jobs in Health and Human Services includes losing 300 of Dr. Oz's employees.
But Dr. Oz put a positive spin on his AI-first initiative, insisting patients may prefer an AI avatar over an actual physician.
We could imagine this, if the comparison was waiting months for appointments with physicians vs getting an informed diagnosis in a matter of seconds with an AI agent.
Unless the diagnosis is wildly wrong and the treatment destroys our lives.
Sources:
https://www.wired.com/story/dr-oz-ai-health-care-medicare-cms-town-hall/
https://www.theguardian.com/us-news/2025/apr/09/mehmet-oz-doctors-ai
https://www.youtube.com/watch?v=IACy_jZUkiE
https://www.keckmedicine.org/physician-hub/can-artificial-intelligence-replace-doctors/
https://www.ama-assn.org/practice-management/digital-health/ai-already-reshaping-care-heres-what-it-means-doctors
https://www.ama-assn.org/system/files/future-health-augmented-intelligence-health-care.pdf
https://www.nytimes.com/2025/03/14/opinion/dr-oz-medicare-medicaid-trump.html?searchResultPosition=3
https://www.cnbc.com/2025/04/04/cms-makes-job-cuts-as-rkf-jr-guts-hhs.html