Statistical Language Models: Responding to Chomsky

Chomsky derided researchers using purely statistical methods to produce linguistic behavior without understanding its meaning. He questioned whether approximating unanalyzed data represents a valid form of scientific success. His criticisms held that such models provide no insight, simulate language generation incorrectly (consulting probability tables rather than the actual human process of mapping semantic forms to syntactic structures), fail at learning language, and achieve only engineering success unrelated to genuine science.

The author argues that engineering success, though distinct from pure science, demonstrates something works and provides evidence of scientific validity. Science requires both fact-gathering and theory-building; accumulation of facts has always dominated scientific practice. For interpretation tasks like speech recognition, probabilistic models are inherently necessary—determining what was likely meant from noisy input requires probabilistic reasoning. A comprehensive probabilistic model covering words, syntax, semantics, context, and discourse would outperform simple Markov models or tree-structure approaches lacking probability. Gold’s Theorem (1967) identified limitations in learning formal languages, but by 1969 probabilistic context-free grammar learning was proven feasible—a result inapplicable to natural language acquisition. Humans possess innate language-learning capacities whose mechanisms remain unknown and could potentially employ probabilistic representations.

The article defines mathematical models as specifying relations among variables, while probabilistic models specify probability distributions over random variables. Trained models use learning algorithms to select optimal candidates from data. Claude Shannon developed probabilistic communication models using Markov chains a decade before Chomsky. A second-order Markov model with a 100,000-word vocabulary requires quadrillion probability values; learning these necessitates statistical inference from data. The ideal gas law (P = N k T / V) exemplifies a probabilistic statistical model that, though uncertain and incomplete compared to modeling individual molecular motions, provides both reliable predictions and insight unavailable from attempting to understand true molecular movements.