The Bitter Lesson

The Bitter Lesson is an observation by the computer scientist Rich Sutton, set out in a short 2019 essay of the same name, that general methods which leverage computation are ultimately the most effective, and by a large margin, over approaches that build in human knowledge of a domain.

Sutton drew the lesson from 70 years of AI research across chess, Go, speech recognition, and computer vision. In each field, researchers first tried to encode expert knowledge directly into systems, and these approaches produced short-term gains and satisfied researchers' desire to feel they were contributing something clever. In each case, a simpler approach that instead scaled search and learning with more computation eventually overtook the hand-engineered systems, once enough compute became available. The lesson is "bitter" because it repeatedly undercuts the intellectual investment researchers make in domain-specific techniques, and because the deciding factor turns out to be something as unglamorous as the availability of compute rather than insight into the problem.

The essay is frequently cited as a rationale for the design philosophy behind transformer-based large language models: comparatively simple architectures trained with enormous compute and data, rather than systems built around hand-crafted linguistic rules. It relates to the scaling laws that describe how such models' capabilities improve predictably as compute, data, and parameters increase.

The lesson is not that domain knowledge is worthless, only that it is a poor long-term bet compared to methods that keep improving as more computation is applied. Sutton’s own caveat is that this pattern has held for 70 years and there is no principled reason to expect it to stop.

References