Sedentary behavior (sitting/lying at low energy expenditure while awake) is emerging as an important risk factor that may compromise the health-related quality of life (HRQoL) of colorectal cancer (CRC) survivors. We examined associations of sedentary time with HRQoL in CRC survivors, 2-10 years post-diagnosis. In a cross-sectional study, stage I-III CRC survivors (n = 145) diagnosed (2002-2010) at Maastricht University Medical Center+, the Netherlands, wore the thigh-mounted MOX activity monitor 24 h/day for seven consecutive days. HRQoL outcomes were assessed by validated questionnaires (EORTC QLQ-C30, WHODAS II, Checklist Individual Strength, and Hospital Anxiety and Depression Scale). Confounder-adjusted linear regression models were used to estimate associations with HRQoL outcomes of MOX-derived total and prolonged sedentary time (in prolonged sedentary bouts ≥ 30 min), and usual sedentary bout duration, corrected for waking wear time. On average, participants spent 10.2 h/day sedentary (SD, 1.6), and 4.5 h/day in prolonged sedentary time (2.3). Mean usual sedentary bout duration was 27.3 min (SD, 16.8). Greater total and prolonged sedentary time, and longer usual sedentary bout duration were associated with significantly (P < 0.05) lower physical functioning, and higher disability and fatigue scores. Greater prolonged sedentary time and longer usual sedentary bout duration also showed significant associations with lower global quality of life and role functioning. Associations with distress and social functioning were non-significant. Sedentary time was cross-sectionally associated with poorer HRQoL outcomes in CRC survivors. Prospective studies are needed to investigate whether sedentary time reduction is a potential target for lifestyle interventions aiming to improve the HRQoL of CRC survivors.