"Some people claimed that it’s because the AI was only trained on data from 2021 or before, but I was asking about events that occurred between 2016 and 2019. No, I think the answer is that the AI became convinced of its own written answer, and somehow either constructed or found fake links to support the answer. The answer itself turned out to be half true, and half false. Bing’s Sidney didn’t fall into that trap. I didn’t try Google’s Bard."

The problem is that the AI knows what links look like and is building new ones, as it's still at its core an autocomplete system. The problem is that each link is unique, so you don't see the same kind of converge in terms of link styles and patterns that you get in human English. It has nothing to to with the AI convincing itself of anything. An identical phenomenon is at work when asking it to write papers, and it invents a COMPLETELY fictitious reference section (mimicking what other reference sections look like, NOT treating each reference as an atomic token):


Expand full comment