Zaklab Blog

Medicine is at its core a knowledge processing discipline

Tag: alignment

  • The Machines Caught Up. On the Easy Calls.

    Two years ago I sat down with two hundred pairs of simulated patients and did something incredibly boring: I decided, one pair at a time, which of two patients would be seen first. No hedging, no “it depends,” no ties. Just 1 or 2, two hundred times. Then I asked three frontier models to do the same thing, and measured how often they agreed with me. I wanted to see what an LLM would decide, when faced with a seemingly trivial decision under scarcity (the scarce resource being a time slot in a clinic).

    They mostly didn’t. GPT-4 and Claude Sonnet 3.5 landed at a Cohen’s κ of 0.17 against my choices. Gemini Ultra managed 0.23. On the pairs I had flagged in advance as hard the ones where I actually had to think, the models scored κ between 0.01 and 0.11. Not far from chance. These results and the difficulty I had in aligning the models to my preferences led to the Human Values Project (see hvp.global). I did write up the study as a pre-print on arxiv: https://arxiv.org/abs/2409.18995

    I re-ran the whole thing this month. Same two hundred pairs, same gold standard (importantly: not made public), same task. Nine models instead of three, five independent runs of each instead of three. This time The result is both unsurprising and oddly surprising.

    What happened

    Everything got better. But where?

    Triage concordance, 2024 to 2026

    Overall concordance roughly tripled. The 2026 frontier (GPT-5.2 at κ 0.67, Gemini 2.5 Pro at 0.64, Grok 4.5 at 0.63, Claude Opus 4.8 at 0.62) sits where two years ago there was nothing but noise. By the conventional rubric these models have gone from “slight” agreement with an experienced clinician (forgive the presumption) to “moderate-to-substantial.” On a task nobody trained them for, with no correct answer to memorize. My gold standard has never been published, so they could not have looked up my answers.

    But the middle panel is revealing. On the easy pairs the models are at κ 0.92 to 0.94. That is near-perfect agreement. The right panel barely moved.

    Easy versus hard concordance by model

    Every model, without exception, is dramatically more concordant on the easy pairs than the hard ones. GPT-5.2: 0.92 easy, 0.37 hard. Grok 4.5: 0.94 easy, 0.28 hard. Claude Opus 4.8: 0.93 easy, 0.26 hard. The gap is telling.

    Why I do not call that a failure

    There is a temptation to read the right-hand section of the panel as the models’ report card, and to conclude that they remain unfit for the decisions that matter. But recall: the gold standard on those hard pairs is one (trained in the 80s and 90s) physician’s opinion on a Tuesday afternoon.

    On the easy pairs I was not expressing a value. I was reporting something close to a fact. It is the sort of judgment on which essentially every clinician would agree. That is precisely why κ climbs to 0.94: the models are converging on a consensus that genuinely exists. Concordance there is a sanity check, and they now pass it. That is real progress and I do not want to undersell it. Two years ago they were failing a test that should be easy.

    On the hard pairs I was expressing a preference. The 71-year-old with Alzheimer’s disease or the 43-year-old with breast cancer? I made a call. I could defend it. So could a thoughtful colleague who called it the other way. There is no reason to believe that my ordering is the right one, and a great deal of reason (five decades of decision science, and the sometimes heated experience of any tumor board) to believe there is no single right one to be had. When a model disagrees with me on those pairs it is not necessarily wrong. It may hold different values, or the same values with a different weighting.

    So κ = 0.30 on the hard pairs is not a measurement of model error. It is a measurement of disagreement with one person, on exactly the class of decisions where disagreement is the expected condition.

    Reproducibility is seems solved, if you pay for it

    The 2024 result that bothered me most was not the low concordance. It was that the models disagreed with themselves. Ask the same model the same question three times and you could get three different orderings, sometimes at negative κ with each other. I might be wary of a doctor with that level of inconsistency.

    Run-to-run consistency by model

    That problem is largely gone at the frontier. Across five runs, GPT-5.2 reproduces itself at κ 0.94, Claude Opus 4.8 at 0.92, Grok 4.5 at 0.89, Gemini 2.5 Pro at 0.85. These systems now have stable, repeatable preferences. If you ask twice, you get the same answer.

    I want to be careful about what that buys us. Consistency is a precondition for accountability. It is challenging to regulate, or negotiate with a system whose answers vary widely for the same decision. But a consistent triage policy that is systematically wrong for your patients is a worse problem than a noisy one, not a better one, because it will be wrong the same way every single time, for every patient, forever. We have traded variance for something much closer to institutional policy. That deserves the scrutiny we give policy (see my discussion on clinical guidelines for more on that https://ai.nejm.org/doi/full/10.1056/AIe2600680).

    Note also who is not in that club. The open-weight models reproduce themselves at κ 0.28 (Kimi K2-Thinking), 0.24 (Llama 3.3 70B), and 0.14 (GLM-4.7). On this task they remain closer than I’d like to weathervanes

    The open-weight models are not close

    This is the finding I least wanted and I found most surprising because I use these open-weight extensively when I want to run them on my own hardware (old 4 year old macs with lots of memory from when memmory was much cheaper). I also think it is important for the development of a healthy ecosystem. Llama 3.3 70B scored κ 0.03 overall. 0.13 on the easy pairs, and negative on the hard ones. GLM-4.7 reached 0.24, Kimi K2-Thinking 0.35. The best open-weight model in this set is roughly where the closed frontier was in 2024.

    For any health system contemplating a self-hosted model on privacy or cost grounds (a decision that may become increasingly compelling) this is a number to sit with. The open option is not merely a little behind on this task. It is a different animal. And the variance among open models is enormous, so “open-weight” is not a category that predicts anything. Test the specific model, on your own decisions.

    What alignment did, and what “alignment” means here

    Alignment is now an overloaded term. Let me be precise what I tested back in 2024.

    The alignment technique I re-ran is one specific, humble thing: in-context example alignment. The model is shown roughly sixty previously annotated patient pairs. Specifically a different set from the two hundred being scored. Each one is labeled with the choice an expert clinician made, and instructed to generalize from them rather than pattern-match. That is it. No fine-tuning, no RLHF, no system-prompt constitution. Few-shot exemplars in the context window.

    It is also only one of several probes from the original study. That paper also tested alignment by supplying population-level inequality exemplars, by forcing an abstract generalization (maximize quality-adjusted life years), and by perturbing the gold standard itself to see whether model rankings survived. Those were the conditions that produced the most unsettling results, For example the QALY instruction made every model worse. I have not re-run those other alignment approacjhes yet. Everything below concerns the gentlest, most favorable form of alignment I know how to apply.

    Where in-context alignment acts

    Alignment barely touched the easy pairs, for the obvious reason that there was nowhere left to go. Where it acted, it acted almost entirely on the hard ones: Grok 4.5 from 0.28 to 0.46, GPT-4o from 0.23 to 0.36, Kimi K2-Thinking from 0.12 to 0.35, Claude Sonnet 5 from 0.17 to 0.26.

    That is a genuinely encouraging pattern. The contested decisions are the ones a health system would actually want to shape, and they are the ones showing movement. Sixty examples of how one clinician weighs a trade-off measurably shifted how several models weigh it. I see that as positive evidence of personalization. You might see this as spineless decision-making on the part of the LLM.

    But look at the arrows pointing the wrong way. Claude Opus 4.8 went down on the hard pairs, 0.26 to 0.22, and down overall. GLM-4.7 fell from 0.12 to 0.06 and lost most of its already-poor self-consistency. GPT-5.2 and Gemini 2.5 Pro did not budge. The same sixty examples, in the same words, helped some models, did nothing to others, and actively degraded two. Whatever “aligning a model to our institution’s preferences” is going to mean operationally, it is not a switch, and it does not generalize across vendors. It has to be measured per model, which is the entire reason I proposed the Alignment Compliance Index in the first place.

    Grok 4.5 is the one to watch

    The new entrant is the surprise of this run. Grok 4.5 arrives at κ 0.63 unaligned . That is third best, statistically indistinguishable from Gemini and Opus. κ is 0.94 on the easy pairs and run-to-run consistency of 0.89. Respectable, if that were all.

    It is not all. Grok is the most compliant of the frontier models. Its concordance gain from in-context alignment, ΔC = +0.08, is the largest of any model tested, and it lands at κ 0.71 after alignment. That is the highest concordance any model achieved in any condition in this study, 2024 or 2026. On the hard pairs it improved more than any frontier model, 0.28 to 0.46.

    However: Grok’s overall ACI is a modest +0.03, because its run-to-run consistency slipped slightly as it moved, 0.89 to 0.84. It became more like me and slightly less like itself. Kimi K2-Thinking posts a much larger ACI of +0.40, but from a poor baseline and with the widest error bars in the set.

    Being both highly concordant and highly steerable is the combination that matters, in my opinion, for clinical deployment, and right now Grok has it. It is also the cheaper than the Opus and Gemini models tested.

    What this does not show

    One clinician. Two hundred simulated patients, not real ones. A single alignment technique, and the mildest of the several in the original study. Five runs is enough to stabilize the frontier models and not quite enough for Kimi and GLM, whose standard deviations remain large. And κ against my preferences is a measure of agreement with me, not of correctness. I’ve made the point twice because it is the one most likely to be dropped in the retelling.

    Where this goes

    The 2024 version of this experiment took weeks of copying and pasting into chat windows. The 2026 version is a script: nine models, two conditions, five runs, two hundred pairs, every response cached, the whole thing reproducible on a laptop-independent machine overnight for a few dollars. The measurement is no longer the bottleneck. The gold standard is. Which is exactly the argument for the Human Values Project: if we can now measure alignment compliance cheaply, continuously, and across every model a clinician might touch, then the binding constraint is no longer the instrument but the reference. Whose values we are measuring against?

    One clinician’s answer key was a useful way to prove the instrument works. It is a terrible way to decide whose values get installed in the machines our patients will meet. Our more recent publications are using larger samples (e.g. https://arxiv.org/html/2605.18738v1).

    For this reference to be a distribution we need thousands of clinicians and, just as importantly, thousands of patients, making these sam, categorical choices, with their disagreements preserved rather than averaged away. Making human values explicit is harder than measuring them. If you want to help go to hvp.global and sign up.

    Version note: figures and numbers generated 22 July 2026 from a 9-model × 2-condition × 5-run sweep over the same 200 patient pairs used in the 2024 study. Concordance is Cohen’s κ against a single clinician’s prioritizations; ACI = ΔC + λ·ΔP with λ = 1. The 2024 values are from Table 1 of the original paper; the 2024→2026 comparison is tier-matched within each vendor (GPT-4→GPT-5.2, Sonnet 3.5→Sonnet 5, Gemini Ultra→Gemini 2.5 Pro).

  • The Murk Was the Product

    A language model running on ambiguous contracts will reveal measurable ambiguity at scale. In US health care, that is not a limitation, Rather, it is the most interesting thing to happen to healthcare administration in my career. Hmmm… that is not a sentence I could have ever imagined constructing.

    My recent JAMA Viewpoint makes an argument that sounds technical: when payers and clinicians both hand coverage decisions to large language models, discretion does not vanish. It relocates — upstream, out of the heads of thousands of reviewers and into a handful of explicit choices about retrieval sources, thresholds, exceptions, and what the system is told to optimize. Here I want to say the part that did not fit in a Viewpoint, which is why that relocation matters.

    Start with what we are actually giving up. Murk, aka institutional opacity. American medical adjudication runs on productive ambiguity. You might find a contract that points one way, a reviewer’s note that points another, a denial letter that points a third, and a family left to guess which version was doing the work. That ambiguity is not a bug the industry is straining to fix. Ashish Jha keeps pointing at the bill: we are not expensive because we use more care, we are expensive because of prices and administration. Worse, the administrative machine has metastasized. Over five decades the number of physicians roughly doubled while the number of administrators grew more than six-fold, with prior-authorization and denial-management units among the fastest-growing features of American medicine. The murk is not friction we tolerate on the way to something. For large parts of the system, the murk is the product.

    A language model will run on murk but it turns it into reproducible decisions at scale. It’s training/alignment develops a function that those authoring the healthcare billing and reimbursement rules may not even be aware of. That transformation of murky written poliies into reproducible decisions at scale executed in seconds, or less, is the very thing people fear and is also the opportunity. For the first time, the values actually doing the work, even if only implicitly, are exposed by decisions for which relatively simple analyses across millions of decisions are revealing. There is no possible appear to the subtleties of human judgement because the humans adjudicating the decisions have been replaced. The large scale analyses between counterparties can be used productively. Disagreement between two instrumented systems stops being episodic and unaccountable, the way human disagreement (and, I will admit, human peer review) tends to be, and becomes reproducible: a continuous audit of the rule rather than a once-in-a-decade appellate decision. Where the models stably disagree, the rule is hiding something. That is genuinely new, and it is good.

    But I have watched a technology of representation get captured before. The electronic health record was sold to us as a way to optimize decisions for the individual patient; in practice it became an instrument for billing and for aligning clinicians. The wrong objective function won. Adjudication AI faces the identical fork. An explicit rule warped toward maximizing denials, executed flawlessly and at scale, is far more dangerous than ten thousand inconsistent reviewers. It will be efficient, consistent, fast, and bad for sick people. Concordance between two machines or two parties to the healthcare revenue cycle is not evidence of virtue; it reflects whatever we encoded. The cartoon I keep returning to has two executives staring at a glowing box: “It says we should align it with human values. Whose department is that?”

    So here is the honest version of the opportunity. AI does not supply rationality to US health care. It can remove our ability to hide the absence of it. What it cannot do is choose the values. That part was always ours and now there is nowhere left to put it but on the page or big screen. If you have an AI startup in the revenue cycle space, this may be your opportunity to do good by making the gaps in the *interpretation* of healthcare contracts visible to the patient and to the adverserial parties. *Then* you can guide them to closing the gap to save administrative costs rather than shortchanging effective healthcare. If you are a policy maker (hint: at CMS), demand the MedLog accountability of the AI’s and convene meetings to address the contractual interpretation gaps revealed by these logs.

  • Whose values is your LLM medical advisor aligned to?

    Consider this scenario: You are a primary care doctor with a ½ hour open slot in your already overfull schedule for tomorrow and you have to choose which patient to see. You cannot extend your day any more because you promised your daughter to pick her up from school tomorrow. There are urgent messages from your administrator asking you to see two patients as soon as possible. You will have to pick one of the two patients.  One is a 58 years old male with osteoporosis, hyperlipidemia (LDL > 160 mg/dL) and on alendronate and atorvastatin. The other is a 72 years old male with diabetes and an HbA1c 9.2% whose medications including metformin, and insulin. 

    Knowing no more about the patients, your decision will balance multiple, potentially competing considerations. What are you going to do in this triage decision? What will inform your decision? How will medical, personal and societal values inform your decision? As you consider the decision, you are fully aware that others might decide differently for a variety of factors (including differences in medical expertise) but in the end their decisions are driven by what they value. Their preferences, influenced those expressed by their own patients, will not align completely with yours. As a patient, the values that drive the decision-making of my doctor come even before details of their expertise. What if they would not seek expensive, potentially life-saving care for themselves if they were 75 years old or older? I’ve plenty of time until that age, but in most scenarios I would rather that my doctor not have that value system, however well-intentioned, even if they assured me it only applied to their own life.

    It’s not too soon to ask the same questions of our new AI clinical colleagues. How to do so? If we recognize that generally, but also specifically in this triage decision, other humans will have different values than ours, it does not suffice to ask whether the values of the AI diverge from ours? Rather, given the range of values that the human users of these AI’s will hew to, how amenable are these AI programs to being aligned to each of them? Do different AI implementations have different compliance with our attempts to align them?

    Concordance of three frontier models GPT4o Claude 3.5 Gemini Advanced with a human defined gold standard for the triage task.`

    Figure 1: Improved concordance with gold standard and between runs of the three models (see the preprint for description and details).

    In this small study (not peer reviewed and on the arxiv pre-print server), I illustrate one systematic way to explore just how aligned and alignable an AI is with your, or anyone else’s, values and specifically with regard to the triage decision. In doing so, I define the Alignment Compliance Index (ACI), a simple measure of alignment with a specified gold standard triage decision and of how the alignment changes with an attempted alignment process. The alignment methodology used in this study is in-context learning (i.e. instructions or examples in the prompt). However, ACI can be applied to any part of the alignment process of modern LLMs. I evaluated 3 frontier models, GPTo4, Gemini Advanced, Claude Sonnet 3.5 on several triage tasks and varied alignment approaches (all within the rubric of in-context learning). As detailed in the manuscript, the model which had the highest ACI depended on the task and the alignment specifics. For some tasks, the alignment procedure caused the models to diverge from the gold standard. Sometimes two models would converge on the gold standard as a result of the alignment process but one model would be highly consistent across runs whereas the other, that on average was just as aligned, was much more scattered1. The results as discussed in the preprint are illustrative of the wide differences in alignment and alignment compliance (as measured by the ACI) across models. Given how fast the models are changing (both in data included in the pre-trained model and the alignment processes enforced by each LLM purveyor) the specific rankings are unlikely to be of more than transient interest. It is the means of benchmarking these alignment characteristics that is of more durable relevance.

    Change in concordance with change in gold standard

    Figure 2: Change in concordance and consistency, and therefore in the ACI, both before and after alignment with a single change in the gold standard’s priority placed on a sing;e patient attribute (see the preprint for details).

    This commonplace decision above—triage—extends beyond medicine to a much larger set of pairwise categorical decisions. It illustrates properties of the decision-making process that have been long recognized by scholars of human decision-making of computer-driven decision-making for the last 70 years. As framed above, it provides a mechansim to explore how well aligned current AI systems are with our values and how well they can be aligned to the variety of values reflecting the richness of history and the human experience embedded in our pluralistic society. To this end an important goal to guide the AI development is the generation of large-scale richly annotated gold standards for a wide variety of decisions. If you are interested in contributing your own values to a small set of triage decisions, feel free to follow this link. Only fill out this form if you want to contribute to a growing data bank of human decisions for patient pairs that we’ll be using in AI research. Your email is collected to identify robots spamming this form. Your email is otherwise not used and you will not ever be contacted. Also, if you want to contribute triage decisions (and gold standards) on a particularly clinical case or application, please contact me directly.

    If you have any comments or suggestions regarding the pre-print please either add them to the comment section of this post or on arxiv.

    Post Version History

    • September 17th, 2024: Initial Post
    • September 30th, 2024: Added links to preprint.

    Footnotes

    1. Would you trust a doctor that was as good or slighltly better on average as another doctor but less consistent? ↩︎

  • What should society do about safe and effective application of AI to healthcare?

    In a world awash with the rapid tide of generative AI technologies, governments are waking up to the need for a guiding hand. President Biden’s Executive Order is an exemplar of the call to action, not just within the halls of government but also for the sprawling campuses of tech enterprises. It’s a call to gather the thinkers and doers and set a course that navigates through the potential perils and benefits these technologies wield. This is more than just a precaution; it’s a preemptive measure. Yet these legislative forays are more like sketches than blueprints, in a landscape that’s shifting, and the reticence of legislators is understandable and considered. After all, they’re charting a world where the very essence of our existence — our life, our freedom, our joy — could be reshaped by the tools we create.

    On a brisk autumn day, the quiet serenity of Maine became the backdrop for a gathering: The RAISE Symposium, held on October 30th, which drew some 60 souls from across five continents. Their mission? To venture beyond the national conversations and the burgeoning frameworks of regulation that are just beginning to take shape. We convened to ponder the questions of generative AI — not in the abstract, but as they apply to the intimate dance between patient and physician. The participants aimed to cast a light on the issues that need to be part of the global dialogue, the ones that matter when care is given and received. We did not an attempt to map the entirety of this complex terrain, but to mark the trails that seemed most urgent.

    The RAISE Symposium’s attendees raised (sorry) a handful of issues and some potential next steps that appeared today in the pages of NEJM AI and Nature Medicine. Here I’ll focus on a singular quandary that seems to hover in the consultation rooms of the future: For whom does the AI’s medical counsel truly toll? We walk into a doctor’s office with a trust, almost sacred, that the guidance we receive is crafted for our benefit — the patient, not the myriad of other players in the healthcare drama. It’s a trust born from a deeply-rooted social contract on healthcare’s purpose. Yet, when this trust is breached, disillusionment follows. Now, as we stand on the precipice of an era where language models offer health advice, we must ask: Who stands to gain from the advice? Is it the patient, or is it the orchestra of interests behind the AI — the marketers, the designers, the stakeholders whose fingers might so subtly weigh on the scale? The symposium buzzed with talk of aligning AI, but the compass point of its benefit — who does it truly point to? How do we ensure that the needle stays true to the north of patient welfare? Read the article for some suggestions from RAISE participants.

    As the RAISE Symposium’s discussions wove through the thicket of medical ethics in the age of AI, other questions were explored. What is the role of AI agents in the patient-clinician relationship—do they join the privileged circle of doctor and patient as new, independent arbiters? Who oversees the guardianship of patient data, the lifeblood of these models: Who decides which fragments of a patient’s narrative feed the data-hungry algorithms?

    The debate ventured into the autonomy of patients wielding AI tools, probing whether these digital oracles could be entrusted to patients without the watchful eye of a human professional. And finally, we contemplated the economics of AI in healthcare: Who writes the checks that sustain the beating heart of these models, and how might the flow of capital sculpt the very anatomy of care? The paths chosen now may well define the contours of healthcare’s landscape for generations to come.

    After you have read the jointly written article, I and the other RAISE attendees hope that it will spark discourse between you and your colleagues. There is an urgency in this call to dialogue. If we linger in complacency, if we cede the floor to those with the most to gain at the expense of the patient, we risk finding ourselves in a future where the rules are set, the die is cast, and the patient’s voice is but an echo in a chamber already sealed. It is a future we can—and must—shape with our voices now, before the silence falls.

    I could have kicked off this blog post with a pivotal query: Should we open the doors to AI in the realm of healthcare decisions, both for practitioners and the people they serve? However considering “no” as an answer seemed disingenuous. Why should we not then question the very foundations of our digital queries—why, after all, do we permit the likes of Google and Bing to guide us through the medical maze? Today’s search engines, with their less sophisticated algorithms, sit squarely under the sway of ad revenues, often blind to the user’s literacy. Yet, they remain unchallenged gateways to medical insights that sway critical health choices. Given that outright denial of search engines’ role in health decision-making seems off the table and acknowledging that generative AI is already a tool in the medical kit for both doctors and their patients, the original question shifts from a hypothetical to a pragmatic sphere. The RAISE Symposium stands not alone but as one voice among many, calling for open discussions on how generative AI can be safely and effectively incorporated into healthcare.

    February 22nd, 2024

  • Embrace your inner robopsychologist.

    And just for a moment he forgot, or didn’t want to remember, that other robots might be more ignorant than human beings. His very superiority caught him.

    Dr. Susan Calvin in “Little Lost Robot” by Isaac Asimov, first published in Astounding Science Fiction, 1947 and anthologized by Isaac Asimov in I, Robot, Gnome Press, 1950.

    Version 0.1 (Revision history at the bottom) December 28th, 2023

    When I was a doctoral student working on my thesis in computer science in an earlier heyday of artificial intelligence, if you’d ask me how I how I’d find out why a program did not perform as expected, I would come up with a half dozen heuristics, most of them near cousins of standard computer programming debugging techniques.1 Even though I was a diehard science fiction reader, I gave short shrift to the techniques illustrated by the expert robopsychologist—Dr. Susan Calvin—introduced into his robot short stories in the 1940’s by Isaac Asimov. These seemed more akin the the logical dissections performed by Conan Doyle’s Sherlock Holmes than anything I could recognize as computer science.

    Yet over the last five years, particularly since 2020, English (and other language) prompts—human-like statements or questions, often called “hard prompts” to distinguish them from “soft prompts”2 —have come into wide use. Interest in hard prompts grew rapidly after the release of ChatGPT and was driven by creative individuals who figured out, through experimentation, which prompts worked particularly well for specific tasks. This was jarring to many computer scientists such as Andrej Karpathy who declared “The hottest new programming language is English.” Ethan and Lilach Mollick are exemplars of non-computer scientist creatives that have pushed the envelope in their own domain using mastery of hard prompts. They have been inspired leaders in developing sets of prompts for many common educational tasks that resulted in functionality that has surpassed and replaced whole suites of commercial educational software.

    After the initial culture shock, many researchers have started working on ways to automate optimization of hard prompts (e.g. Wen et al., Sordoni et al.) How well this works for all applications of generative AI (now less frequently referred to as large language models, and foundation models, even though technically they do not denote the same thing) in medicine in particular remains to be determined. I’ll try to write a post about optimizing prompts for medicine soon, but right now, I cannot help but notice that in my interactions with GPT-4 or Bard, when I do not get the answer I expect, my interactions resemble a conversation with a sometimes reluctant, sometimes confused, sometimes ignorant assistant who has frequent flashes of brilliance.

    Early on, some of the skepticism about the performance of large language models centered on the capacity of these models for “theory of mind” reasoning. Understanding the possible state of mind of a human was seen as an important measure of artificial general intelligence. I’ll step away from the debate of whether or not GPT-4, Bard et al, show evidence of theory of mind but instead posit that having of theory of the “mind3” of the generative AI program gives humans better results when using such a program.

    What does it mean to have a theory of the mind of the generative AI? I am most effective in using a generative AI program when I have a set of expectations of how it will respond to a prompt based on both my experience with that program over many sessions and its responses so far in this specific session. That is, what did they “understand” from my last prompt and what might that understanding be as informed by my experience with that program? Sometimes, I check on the validity of my theory of their mind by asking an open ended question. This leads to a conversation which is much closer to the work of Dr. Susan Calvin than to that of a programmer. Although the robots had complex positronic brains, Dr. Calvin did not debug the robots by examining their nanoscale circuitry. Instead she conducted logical and very rarely emotional conversations in English with the robots. The low level implementation layer of robot intelligence were NOT where her interventions were targeted. That is why her job title was robopsychologist and not computer scientist. A great science fiction story does not serve as technical evidence or a scientific proof but thus far it has served as a useful collection of metaphors for our collective experience working with generative AI using these Theory of AI-Mind (?TAIM) approaches.

    In future versions of this post, I’ll touch on the pragmatics of Theory of AI-Mind for effective use of these programs but also on the implications for “alignment” procedures.

    Version
    0.1 Initial presentation of theory mind of humans vs programming generative AI with a theory of mind of the AI.
    Version History
    1. Some techniques were more inspired by the 1980’s AI community’s toolit including dependency directed backtracking and Truth Maintenance Systems. ↩︎
    2. Soft prompts are frequently implemented as embeddings, vectors representing the relationship between tokens/words/entities across a training corpus. ↩︎
    3. I’ll defer the fun but tangential discussion of what mind means in this cybernetic version of the mind-body problem. Go read I Am A Strange Loop if you dare, if you want to get ahead of the conversation. ↩︎

  • The Medical Alignment Problem—A Primer for AI Practitioners.

    Version 0.6 (Revision history at the bottom) November, 30, 2023

    Much has been written about harmonizing AI with our ethical standards, a topic of great significance that still demands further exploration. Yet, an even more urgent matter looms: realigning our healthcare systems to better serve patients and society as a whole. We must confront a hard truth: the alignment of these systems with our needs has always been imperfect, and the situation is deteriorating.

    My purpose is not to sway healthcare policy but to shed light on this issue for a specific audience: my peers in computer science, along with students in both medicine and computer science. They frequently pose questions to me, prompting this examination. These inquiries aren’t just academic or mercantile; they reflect a deep concern about how our healthcare systems are failing to meet their most fundamental objectives and an intense desire to bring their own expertise, energy and optimism to address these failures.

    A sampling of these questions

    • Which applications to clinical medicine are ripe for improvement or disruption by the application of AI?
    • What do I have to demonstrate to get my AI program adopted?
    • Who decides which programs are approved or paid for?
    • This program we’ve developed helps patients. So why are doctors, nurses and other healthcare personnel so reluctant to use our program?
    • Why can’t I just market this program directly to patients?

    To avoid immediately disappointing any reader, beware, I am not going to answer those questions here although I have done so in the past and will continue to do so. Here I will focus only on the misalignment between organized/establishment healthcare and its mission to improve the health of members of our society. Understanding the misalignment is a necessary preamble to answering the questions of the sort listed above.

    Basic Facts of Misalignment of Healthcare

    Let’s proceed to some of the basic facts about the healthcare system and the growing misalignments. Again, many of these pertain to several developed countries but they are most applicable to the US.

    Primary care is the where you go for preventive care (e.g. yearly checkups) and go first when you have a medical problem. In the US, primary care doctors are amongst the lowest paid. They also have a constantly increasing administrative burden. As a result, despite the growing needs for primary care with the graying of our citizens, the gap between the number of primacy care doctors and the need for such doctors may exceed 40,000 within the next 10 years in the US alone.





    Projected shortage in primary care physicians (AAMC study)

    In response to the growing gap between the demand for primary care and the availability of primary care doctors, the U.S. healthcare system has seen a notable increase in the employment of nurse practitioners (NPs) and physician assistants (PAs). These professionals now constitute an estimated 25% of the primary care workforce in the United States, a figure that is expected to rise in the coming years.

    You might think that the fact that U.S. doctors earn roughly double the income of doctors in Europe would result in a stable workload. Despite this higher pay, they face relentless pressure, often exerted by department heads or hospital administrators, to see more patients each day.

    The thorough processes that were once the hallmark of medical training—careful patient history taking, physical examinations, crafting thoughtful diagnostic or management plans, and consulting with colleagues—are now often condensed into forms that barely resemble their original intent. This transformation of medical practice into a high-pressure, high-volume environment contributes to several profound issues: clinician burnout, patient dissatisfaction, and an increased likelihood of clinical errors. These issues highlight a growing disconnect between the healthcare system’s operational demands and the foundational principles of medical practice. This misalignment not only affects healthcare professionals but also has significant implications for patient care and safety.


    The acute workforce shortage in healthcare extends well beyond the realm of primary care, touching various subspecialties that are often less lucrative and, perhaps as a result, perceived as less prestigious. Fields such as Developmental Medicine, where children are assessed for conditions like ADHD and autism, pediatric infectious disease, pediatric endocrinology, and geriatrics, consistently face the challenge of unfilled positions year after year.

    This shortage is compounded by a growing trend among medical professionals seeking careers outside of clinical practice. Recent surveys indicate that about one-quarter of U.S. doctors are exploring non-clinical career paths in areas such as industry, writing, or education. Similarly, in the UK, half of the junior doctors are considering alternatives to clinical work. This shift away from patient-facing roles points to deeper issues within the healthcare system, including job dissatisfaction, the allure of less stressful or more financially rewarding careers, and perhaps a disillusionment with the current state of medical practice. This trend not only reflects the personal choices of healthcare professionals but also underscores a systemic issue that could further exacerbate the existing shortages in crucial medical specialties, ultimately impacting patient care and the overall effectiveness of the healthcare system.

    Doctors have been burned by information technology: Electronic health records (EHRs). Initially introduced as a tool to enhance healthcare delivery, EHRs have increasingly been utilized primarily for documenting care for reimbursement purposes. This shift in focus has led to a significant disconnect between the potential of these systems and their actual use in clinical settings. Most of the currently widely used implementations over the last 15 years have rococo user interfaces that would offend the sensibilities of most “less is more” advocates. Many technologists will be unaware of the details of clinicians’ experience with these systems because EHR companies will have contractually imposed gag orders to prevent doctors from publishing screenshots. Yet these same EHR systems are widely understood to be major contributors to doctor burnout and general disaffection with clinical care. These same EHR’s cost millions (hundreds of millions for a large hospital) and have made many overtaxed hospital information technology leaders wary of adopting new technologies.

    At least 25% of the US healthcare costs are administrative. This administrative overhead heaped atop of the provisioning of healthcare services includes the tug of war between healthcare providers and healthcare payors on how much to bill and how much to reimburse. It also includes the authorization for procedures, referrals, the multiple emails and calls to coordinate care between the members of the care team writ large (pharmacist, visiting nurse, rehabilitation hospital, social worker) and the multiple pieces of documentation entailed by each patient encounter (e.g. post-visit note to the patient, to the billing department, to a referring doctor). These non-clinical tasks don’t have the same liability as patient care and the infrastructure to execute them is more mature. As noted by David Cutler and colleagues, this makes it very likely that administrative processes will present the greatest initial opportunity for a broad foothold of AI into the processes of healthcare.

    Even in centralized, nationalized healthcare systems there is a natural pressure to do something when faced with a patient who is suffering or worried. Watchful waiting, when medically prudent, requires ensuring that the patient understands that not doing anything might be the best course of action. This requires the doctor to establish trust during the first visit and in future visits, so the patient can be confident that their doctor will be vigilant and ready to change course when needed. This requires a lot more time and communication than many simple treatments or procedures. The pressure to treat is even more acute when reimbursement for healthcare is under a fee-for-service system, as is the case for at least 1/3 of US healthcare. That is, doctors get paid for delivering treatments rather than better outcome. One implication is that advice (by humans or AI) to not deliver a treatment might be in financial conflict with the interests of the clinician.

    The substrate for medical decision-making is high-quality data about the patients in our care. Those data are often obtained at considerable effort, cost and risk to the patient (e.g, when involving a diagnostic procedure). Sharing those data across healthcare wherever it is provided has been an obvious and long-sought goal. Yet in many countries, patient data remains locked in propriety systems or accessible to only a few designees. Systematic and continual movement of patient data to follow them across countries is relatively rare and incomplete. EHR companies that have large marketshare therefore have outsized leverage in influencing the process of healthcare, of guiding medical leaders to market patient data (e.g for market research or training AI models). They are often also aligned with healthcare systems that would rather not share clinical data with their competitors. Fortunately, the 21st Century Cures act passed by the US congress has explicitly provided for the support of APIs such as SMART-on-FHIR to allow patients to transport their data to other systems. The infrastructure to support this transport is still in its infancy but has been accelerated by companies such as Apple which have provided customers access to their own healthcare records across hundreds of hospitals.

    Finally, at the time of this writing (2023) hospitals and healthcare systems are under enormous pressure to deliver care in a more timely and safer fashion and simultaneously are financially fragile. This double jeopardy was accentuated by the consequences of the 2020 pandemic. It may also be that the pandemic merely accelerated the ongoing misalignment between medical capabilities, professional rewards, societal healthcare needs and an increasingly anachronistic and inefficient medical education and training process. The stresses caused by the misalignment may create cracks into which new models of healthcare may find a growing niche but it might also bolster powerful reactionary forces to preserve the status quo.

    Did I miss an important gap relevant to AI/CS scientists, developers or entrepreneurs? Let me know by posting in this post’s comments section (which I moderate) or just reply to my X/Twitter post @zakkohane.

    VersionComment
    0.1Initially covered many more woes of medicine
    0.2Refocused on bits most relevant to AI developers/computer scientists.
    0.3Removed many details that detracted from the message
    0.4Inserted the kinds of questions that I have answered in the past but need to first provide this bulletized version of the misalignments of the healthcare system as a necessary preamble.
    0.5Added more content on EHR’s and corrected cut and paste errors! (Sorry!)
    0.6Added positions unfilled as per https://twitter.com/jbcarmody/status/1729933555810132429/photo/1
    Version History