Exercise capacity is a spectrum which reflects an individual's functional capacity and the dynamic nature of cardiac remodeling along with respiratory and skeletal muscle systems. The relationship between increasing physical activity, increased cardiac mass and volumes, and improved cardiorespiratory fitness (CRF) is well-established in the endurance athlete. However, less emphasis has been placed on the other end of the spectrum, which includes individuals with a more sedentary lifestyle and small hearts who are at increased risk of functional disability and poor clinical outcomes. Reduced CRF is an independent predictor of all-cause mortality and cardiovascular events determined by multiple inter-related exogenous and endogenous factors. In this review, we explore the relationship between physical activity, cardiac remodeling and CRF across the exercise spectrum, emphasising the critical role of cardiac size in determining exercise capacity. In contrast to the large compliant left ventricle of the endurance athlete, an individual with a lifetime of physical inactivity is likely to have a small, stiff heart with reduced cardiac reserve. We propose that this may contribute to the development of HFpEF in certain individuals, and is key to understanding the link between low CRF and increased risk of heart failure.