webppl-oed: A practical optimal experiment design system
- Long Ouyang, Psychology, Stanford University, Stanford, California, United States
- Michael Tessler, Psychology, Stanford University, Stanford, California, United States
- Daniel Ly, Psychology, Stanford University, Stanford, California, United States
- Noah Goodman, Psychology, Stanford University, Stanford, California, United States
AbstractAn essential part of cognitive science is designing experiments that distinguish competing models. This requires patience and ingenuity---there is often a large space of possible experiments one could run but only a small subset that might yield informative results. But we need not comb this space by hand---if we use formal models and explicitly declare the space of experiments, we can automate the search for good experiments, looking for those with high expected information gain. Here, we present an automated system for experiment design called webppl-oed. In our system, users simply declare their models and experiment space; in return, they receive a list of experiments ranked by their expected information gain. We demonstrate our system in two case studies, where we use it to design experiments in studies of sequence prediction and categorization. We find strong empirical validation that our automatically designed experiments were indeed optimal.