Understanding how differences in brain structure relate to differences in cognition across the lifespan is essential for addressing age-related cognitive decline. Since age is strongly associated with both brain structure and cognition, predictive models often risk simply capturing age effects. To mitigate this risk, deconfounding is typically applied to remove the effects of age. Here, beyond treating age as a confound, we treat it as a moderator by estimating brain-cognition associations separately across age groups. This captures age-stratified changes in how brain structure and cognitive performance are statistically connected. For this view to hold, variations in brain structure linked to differences in cognitive performance in older subjects (eg related to disease) would differ from those in younger subjects. Using structural brain imaging data from the UK Biobank we found an asymmetry in generalisability: models trained on younger subjects successfully predicted cognition in older subjects, but models trained on older subjects failed to generalize to younger individuals. These findings reveal a trade-off between model specificity and generalisability, suggesting the optimal approach-whether age-specific or pooled-depends on the research or clinical goal for the target population.