webppl-oed: A practical optimal experiment design system

AbstractAn essential part of cognitive science is designing experiments that distinguish competing models. This requires patience and ingenuity---there is often a large space of possible experiments one could run but only a small subset that might yield informative results. But we need not comb this space by hand---if we use formal models and explicitly declare the space of experiments, we can automate the search for good experiments, looking for those with high expected information gain. Here, we present an automated system for experiment design called webppl-oed. In our system, users simply declare their models and experiment space; in return, they receive a list of experiments ranked by their expected information gain. We demonstrate our system in two case studies, where we use it to design experiments in studies of sequence prediction and categorization. We find strong empirical validation that our automatically designed experiments were indeed optimal.

Return to previous page