Knowledge of color has strong individual, environmental, and cultural differences that may systematically influence performance in cognitive tasks. For example, color knowledge has been shown to influence recall of color (Persaud & Hemmer, 2014). This manifests as a systematic regression to the mean effect, where memory is biased towards the mean hue of each universal color category. What remains unclear is whether differences, such as culture and environment, might differentially influence memory. We tested recognition memory for color in the Tsimane’ of Bolivia; an indigenous population with little or no modern schooling, whose environment is very different from industrialized societies. We found that recognition regressed towards the mean of some universal color categories, but for others was systematically biased toward neighboring categories. A cluster analysis suggested that the Tsimane’ use five underlying color categories—not the standard universals. This might be shaped by education, language and the environment.