The Promise is Perception

Casey Newton in Platformer:

One of the more famous papers about artificial intelligence last year came from METR, a nonprofit that evaluates frontier AI models. In July, it published results of a randomized controlled trial studying experienced open-source developers. It found that when they use AI tools, completing tasks takes them 19 percent longer than when they go without. That was surprising enough. But the real twist is that when these same developers were asked what AI had done for them, they reported that it had sped them up by 20 percent.

This was a fascinating dive into the professional productivity that’s promised by our AI overlords. We’re starting to learn that much of this productivty is perceived. I’ve felt this in my own professional life, and have drastically reduced my use of Copilot at work because I found myself spending far too much time reviewing and correcting incorrect output from the model.

I also found many of the outputs, when accurate and correct, were just OK and simply not up to my professional standards. So much of my daily work requires communicating effectively through writing – explaining value and impact to leadership; acting as a translation layer between engineering, design and the business; and aligning stakeholders to broad, complex initiatives – all of which need to be buttoned-up to my highest standard. I’m simply not getting that quality from any AI model I’ve tried.