There have been two distinct notions of heuristics, i.e., Kahneman and Tversky’s (1974) heuristics as biased approximations to rational inference, and Gigerenzer et al.’s (1999) idea of smart and adaptive heuristics. Despite the conceptual differences, we provide evidence that heuristics can be seen as approximations to a rational account which is at its core adaptive. In a large cross-validation, we demonstrate that a regularized regression model (from machine learning) with a penalty noise parameter could outperform both heuristics and simple linear regression. Importantly, the penalized regression with an L2-norm could be approximated by tallying, whereas the L1-norm was approximated by take-the-best. Results indicate that the penalized regression treats both heuristics and linear regression as extreme cases of the model. The research implies a common rational basis for heuristics and integrative strategies, suggesting that the relation need not be adversarial. Implications for reconciling adaptive and irrational views of heuristics are discussed.