Observing the actions of others allows us to learn not only about their mental states, but also about hidden aspects of a shared environmental situation -- things we cannot see, but they can, and that influence their behavior in predictable ways. This paper presents a computational model of how people can learn about the world through social inferences, extending the Bayesian Theory of Mind (BToM) model of Baker et al. (2011), which treats agents' behavior as the output of a planning process and then reasons backwards to infer the most likely inputs to the agent's planner. We conducted a large-scale experiment comparing the world-state inferences of the model and those of human subjects, given observations of agents moving along paths in simple spatial environments. The model predicts subjects' graded beliefs about world states with high accuracy, showing the power of social learning for acquiring surprisingly fine-grained knowledge about the world.