James's Blog

Sharing random thoughts, stories and ideas.

Surprisal

Posted: Feb 27, 2020
◷ 2 minute read

Surprisal is a formal concept in information theory. It is also called Shannon information or simply information content, and is related closely to the information theoretic entropy. It’s essentially the idea that the amount of information contained in a particular message can be directly quantified by how surprised the receiver of the message is upon receiving it. A dog walking into a veterinarian’s office will not be met with shock from the receptionist, but a clown walking in will no doubt (at least somewhat) surprise everyone working there. In this sense, the clown walking in conveyed more information (e.g. perhaps there is a clown gathering nearby) to the employees at the vet than the dog walking in, which presumably happens all the time.

In the practical real world, I find the colloquial idea of “surprise” serves as a good analog for the amount of information contained in everyday events. As a result, I find that surprisal is a decent proxy to the value of various interactions I have with people. Be it watching a video, a one-on-one conversation, or a group discussion. If I am not surprised much from an interaction, that is, if I am able to predict fairly accurately how the interaction will be like beforehand, then it should have correspondingly little informational value. Small talk, for example, is nearly devoid of all information content. People will generally say that they are “doing well”, comments about the weather are usually about knowledge that both parties already possess, etc… Of course, it has other values, such as socialization, but in terms of informational value it has nearly none, because I am almost never surprised by it.

So I like to be surprised, or caught off guard in a sense, when I talk with someone or discuss things in a group. Surprised by ideas that is; being suddenly slapped in the face in a casual chat would certainly be surprising, but not the kind I want. I have started to ask myself how well I could have predicted the way things transpired after an interaction, in order to gauge how valuable or productive it was. It may sound like a really obvious thing, especially when the information theory connection is clearly there, but I have personally never really evaluated interactions like this until recently. I found that the less I could’ve predicted a conversation (or the more surprised I was), the more I learned from it.

It certainly explains why I (like many others) find certain bureaucratic committee-style meetings morbidly uninteresting: things usually proceed exactly as expected by all participants (just like in small talk chitchats). People tend to dislike being surprised, but it’s often in conversations filled with surprises that the most value can be found.