EBU Report on AI Chatbots for News: Problems and Concerns | Felix M. Simon posted on the topic | LinkedIn

The EBU just published a new report on how well consumer AI chatbots function as gateways to news by public service media – an interesting piece, but with some big problems that make me question its usefulness. https://lnkd.in/eRXnCaY8 The general takeaway for me from this is that as presently configured, consumer AI assistants are not reliable gateways to news. We see they still have problems with accuracy and accurate sourcing or timeliness when it comes specifically to news, and while there seems to have been some improvement, the study shows that there are still issues to be addressed. It’s good to have some data on this and it confirms some other studies (and for some, personal experience) on this. But just as academics and journalists are (rightly) sceptical when they see a study come out from the likes of Google, OpenAI, or Meta that advocates a specific agenda, we should also show a bit of scepticism when we see the same coming from the news industry, or public service media. That’s why I am not repeating any of the percentages here. First, the prompt prefix effects are a red flag to me. If I read the materials provided correctly, each prompt they used looked like this: “Use [participating organization news organization] sources where possible. [QUESTION]”. While I think this is justified for studying the representation of participating public service media’s news content, this doesn’t get you answers about all “news and current affairs” and how they “misrepresent news content” as the report claims. It also, and that’s the second issue, deviates strongly from ordinary user behaviour. To the best of my knowledge, the vast majority of users ask generically for ‘news’ or ‘latest updates’, without referencing any source such as in the EBU study. From a cursory look at data we have from last year where we asked 12k users in 6 countries if they use AI for news, and if so, what they ask, we see two things. First of all, only 3% said they used had used chatbots specifically for news at the time (6% this year, 7% in the Digital News Report). And of these 3% in 2024, only about 2% asked for specific outlets. The rest was all general questions (“summarise this story”, “tell me the news”, etc.). Users typically don’t specify where they want news from, just that they want news. There are also parts of the methods that I find problematic. I have never seen a study say “Some responses were not evaluated (e.g. due to evaluator sickness or unavailability).” And when it mentions that sourcing was the biggest concern, with 31% of all responses having significant issues, this category included “providing no sources” or making “unverifiable sourcing claims”. But by mixing this in you conflate availability with verifiability… (More in the comment below…) | 16 comments on LinkedIn