Select a participant id

Scoring Rules & Payoffs

The below information about scoring rules and payoffs was provided before the study and was always available during the study.

Scoring Rules:

• Interval Response

  • Description: Lower scores are better. This scoring rule rewards tighter intervals that contain the true value, and includes a penalty if the provided interval does not contain the true value.
  • Equation: $${score} = ({upper} - {lower}) + 40 \times ({lower} - {true\ answer}) \times 1[{true\ answer < {lower}}] $$ $$+ 40 \times ({true\ answer} - {upper}) \times 1[{true\ answer > {upper}}]$$

• Binary Response

  • Description: Lower scores are better. Scores on binary responses range from 0 to 2—the minimum score is obtained by assigning 1 to yes_guess if yes is the actual answer and 1 to no_guess if no is the actual answer.
  • Equation: $${score} = ({yes\ guess} - {yes\ actual})^{2} + ({no\ guess} - {no\ actual})^{2}$$
    • ${yes\ actual} = 1$ if actual answer is yes and 0 otherwise
    • ${no\ actual} = 1$ if actual answer is no and 0 otherwise

Score → Payoff ($):

Once you have answered all questions in a “block,” i.e., a set of four questions about a dataset, we normalize your scores so they are all on the same scale, and translate the normalized scores into a payoff. This process is how we translate scores → money. your scores are normalized and translated to a payoff.

First, we calculate a normalized score for each question:

• ${score}_{normalized} = \frac{X - score}{X}$, where $X$ is a constant based on the dataset & question (see bottom of doc)

Then, we calculate a normalized score for the block:

• ${score}_{normalized\ for\ block} = \sum{{score}_{normalized}}$, for each question in the block

Finally, we translate the normalized score into a dollar amount by defining a payoff function:

• ${payoff} = {min}({max}(2.5 \times {score}_{normalized\ for\ block}, 0), 10)$

$X$ constants by dataset (ACS, DIA, STU) and question:

Census (ACS)

Census survey dataset

Diabetes (DIA)

Diabetes dataset

Student (STU)

Student evaluation dataset