DRAFT: Review: “Ego Depletion”, JDW 2010

Michael Stone, April 20, 2013, , (src)

Earlier this weekend, I read a fun paper (SAGE) by Job, Dweck, and Walton questioning the “strength” model of willpower, I think as part of a much larger research program on whether and how our “self-theories” influence, e.g., our ability to perform or to persist in performing difficult, frustrating, or tiring tasks.

Some things I really liked about this paper were that:

However, there were also some parts that bothered or confused me. As a result, in the hopes that my bothers and confusions may be of some use to others, I’ve written up several “bug reports” below that may be of interest.

Okay, here we go:

  * Plot 3b: All five plots have dependent measures with labels that begin
    with the prefix "Probability of a Mistake" and four of the five figures
    have ratio scales for these measures: that is, their scales cover
    intervals ranging from $[0, 0.08]$ to $[0, 0.12]$. Unlike all the other
    plots, Plot 3b's scale is presented as an interval scale, covering the
    interval $[0.20, 0.45]$. Why? (Just to devote more ink to showing the
    measured inter-class differences? Or is there some deeper confusion
    about what scale matters for measuring effect sizes?)
  * Simmons #1: *"Authors must decide the rule for terminating data
    collection before data collection begins and report this rule in the
    article."*

      * No data collection stopping rules were included in the paper.

  * Simmons #2: *"Authors must collect at least 20 observations per cell or
    else provide a compelling cost-of-data-collection justification."*

      * Per-cell observation counts were not included in the paper.

  * Simmons #3: *"Authors must list all variables collected in a study."*

      * I believe that all variables collected may be reported in the
        supplemental material published alongside the original paper but
        only a subset were reported in the original paper.

  * Simmons #4: *"Authors must report all experimental conditions,
    including failed manipulations."*

      * I don't see a claim that all experimental conditions have been
        reported.

  * Simmons #5: *"If observations are eliminated, authors must also report
    what the statistical results are if those observations are included."

      * I see some effort here, e.g., when the authors observe on their
        longitudinal study that the 59% of participants who did not
        complete the study were demographically similar to those who
        continued.

  * Simmons #6: *"If an analysis includes a covariate, authors must report
    the statistical results of the analysis without the covariate."

      * Some effort is also made here, particularly in Note #1. (That being
        said, Note #2 and the "speed/accuracy tradeoff" covariates seem
        just like what Simmons et al. are asking about...)