I’ve admired the work of Aaron Cope for a very long time – since he was at the Cooper-Hewitt and I was at the Carnegie Museum of Art – which I now realize measures my admiration in decades, not years. Time flies.

Anyhoo, Aaron’s now at the SFO Museum and he recently prompted several LLMs to tell him about his place of employment. He tested both open source and proprietary models, and found great disparity between them, which highlights some big questions around AI and socio-economic equity.

No model performed well and some flat-out lied. The entire recap is a must-read, but this passage gets a chef’s kiss from me:

Which begs the question: Why is Google’s open model (gemma3) so wrong? I am going to go out on a limb and suggest that the same dynamic is at play with OpenAI’s (and everyone else’s) flagship, and subscription-based, models and their open models: Accuracy is metered toll road and everything is just a mystery-meat coleslaw of signals.

He goes on:

In a nutshell, we are on our way to replicating the same environment that the collective-we have fostered around processed foods for the last 75 years – all the problems concerning availability, cost, nutrition, consequences – but with knowledge and understanding itself.

The comparison to processed foods is apt. We are finding ourselves in quite the predicament and models getting ‘righter’ over time is not the answer. I get the sense that we are walking through a one-way door and on the other side waits a perpetual diet of mental hot dogs.