We aimed to examine the associations between sleep duration and cognitive functions and memory in older Chinese adults attending the China Health and Nutrition Survey. A total of 7924 participants 55 years and older who reported their sleep duration and had a cognitive screen test in 2004, 2006, and 2015 were included in the analysis. Mixed-effects logistic regression models were used to assess the associations. A short sleep duration (≤6 h/day) and long sleep duration (≥10 h/day) were positively associated with a low global cognitive score (odds ratio-OR: 1.23, 95% CI: 1.01-1.50; OR: 1.47, 95% CI: 1.17-1.79, respectively). Both short sleepers and long sleepers had an increased risk of self-reported poor memory (OR: 1.63, 95% CI: 1.39-1.91; OR: 1.48, 95% CI: 1.25-1.74, respectively). No differences in the above associations were found for income, education, and urbanity. In conclusion, both the short and long sleep duration were associated with declined cognition and memory. Maintaining a normal sleep duration may aid in the prevention of cognitive function decline in older adults.