AI chatbots unable to accurately summarise news, BBC finds
-
The problem is that the "train of the thought" is also hallucinations. It might make the model better with more compute but it's diminishing rewards.
Rpg can use the llms because they're not critical. If the llm spews out nonsense you don't like, you just ask to redo, because it's all subjective.
-
For local LLMs, this is an issue because it breaks your prompt cache and slows things down, without a specific tiny model to "categorize" text... which no one has really worked on.
I don't think the corporate APIs or UIs even do this.
You are not wrong, but it's just not done for some reason.
-
Gemini Flash Thinking from earlier this year was very good for its speed/price, but it regressed a ton.
Gemini 1.5 is literally better than the new 2.0 in some of my tests, especially long-context ones.
-
-
-
Or at least as an assistant on a field your an expert in. Love using it for boilerplate at work (tech).
-
-
Bing/chatgpt is just as bad. It loves to tell you it's doing something and then just ignores you completely.
-
It is stated as 51% problematic, so maybe your coin flip was successful this time.
-
Whoops, yeah, should have linked the blog.
I didn't want to link the individual models because I'm not sure hybrid or pure transformers is better?
-
-
Fuckin news!
-
How could I blindly trust anything in this context?
-
-
You can say Space Needle. We get it.
-
Turns out, spitting out words when you don't know what anything means or what "means" means is bad, mmmmkay.
It got journalists who were relevant experts in the subject of the article to rate the quality of answers from the AI assistants.
It found 51% of all AI answers to questions about the news were judged to have significant issues of some form.
Additionally, 19% of AI answers which cited BBC content introduced factual errors, such as incorrect factual statements, numbers and dates.
Introduced factual errors
Yeah that's . . . that's bad. As in, not good. As in - it will never be good. With a lot of work and grinding it might be "okay enough" for some tasks some day. That'll be another 200 Billion please.
-
-
-
They were actually really vague about the details. The paper itself says they used GPT-4o for ChatGPT, but apparently they didnt even note what versions of the other models were used.
-
Looks pretty interesting, thanks for sharing it
