Multi-modes for Detecting Experimental Measurement Error

Experiments should be designed to facilitate the detection of experimental measurement error. To this end, the authors advocate the implementation of identical experimental protocols employing diverse experimental modes. They suggest iterative nonparametric estimation techniques for assessing the magnitude of heterogeneous treatment effects across these modes. They propose two diagnostic strategies—measurement metrics embedded in experiments, and measurement experiments—that help assess whether any observed heterogeneity reflects experimental measurement error. To illustrate their argument, first we conduct and analyze results from four identical interactive experiments: in the lab; online with subjects from the CESS lab subject pool; online with an online subject pool; and online with MTurk workers. Second, they implement a measurement experiment in India with CESS Online subjects and MTurk workers.