Men and women differ in specific cognitive abilities and in the expression of several neuropsychiatric conditions. Such findings could be attributed to sex hormones, brain differences, as well as a number of environmental variables. Existing research on identifying sex-related differences in brain structure have predominantly used cross-sectional studies to investigate, for instance, differences in average gray matter volumes (GMVs). In this article we explore the potential of a recently proposed multi-view matrix factorisation (MVMF) methodology to study structural brain changes in men and women that occur from adolescence to adulthood. MVMF is a multivariate variance decomposition technique that extends principal component analysis to "multi-view" datasets, i.e. where multiple and related groups of observations are available. In this application, each view represents a different age group. MVMF identifies latent factors explaining shared and age-specific contributions to the observed overall variability in GMVs over time. These latent factors can be used to produce low-dimensional visualisations of the data that emphasise age-specific effects once the shared effects have been accounted for. The analysis of two datasets consisting of individuals born prematurely as well as healthy controls provides evidence to suggest that the separation between males and females becomes increasingly larger as the brain transitions from adolescence to adulthood. We report on specific brain regions associated to these variance effects.