Androgens act through androgen receptor (AR) to maintain muscle mass. Evidence suggests that this pathway is influenced by "the gene for speed," <i>ACTN3</i> (α-actinin-3). Given that one in five people lack α-actinin-3, it is possible that they may respond to androgens differently. Here, we show that α-actinin-3 deficiency decreases AR in muscles of mice and humans (in males and females) and that AR positively correlates with α-actinin-3 expression in a dosage-dependent manner. α-Actinin-3 deficiency exacerbates gastrocnemius mass loss with androgen deprivation in male mice and stunts the muscle growth response to dihydrotestosterone in female mice at the onset of puberty. This is mediated by differential activation of pathways regulating amino acid metabolism, intracellular transport, autophagy, mitochondrial activity, MAPK, and calcineurin signaling, likely driven by seven key genes that are both androgen sensitive and α-actinin-3-dependent in expression. Our results highlight a role for <i>ACTN3</i> as a regulator of muscle mass and a genetic modifier of androgen action in skeletal muscle.