nobe4 / Posts / Confidence is not evidence _

Am I being told the truth?

I have a tendency to enjoy grandiloquence; if someone speaks with certainty and flourished prose, I’m captivated¹. I often mistake this awe-inspiring confidence for the delivery of truth.

Like many, I used to follow online mentors. And I would, sometimes for hours, listen to their views.

There was something deeply comforting about thinking that someone, somewhere, had found something so fundamentally true that it could apply to everyone, everywhere.

Life is messy and complex; their confidence and certainty were reassuring.

And somewhen, I got lost: I started thinking about what I consumed. It became harder to find certainty as I kept questioning the information I received.

Am I being told the truth?

It opened my eyes to the idea that some mentors are actually preachers².

They use poise and wit as ground for their facts. This attitude can be due to actual expertise, but also to manipulation, incentive, or Nobelitis.

Confidence is not evidence.

Complete expertise is rarely possible. I don’t know everything, and neither do you. We all have gaps in the map of our knowledge; we bridge them with beliefs³.

The only path to expertise is to dedicate focused time to a topic. We must consume data, arguments, counter-arguments, research, etc. Eventually we might understand, become proficient, and even feel confident.

However, we must not confuse the extent of our knowledge.

Confidence is not evidence, even inside our own head.

It’s known by now that LLMs are often confidently wrong. They are not truth machines, but write as if they are.

Am I being told the truth?

Even as experts, we should ask for sources. Think critically about what LLMs tell us.

Confidence is not evidence, even at a million tokens per second.

Thanks to Thomas and Masha for reviewing this.