Early clinical assessment of Alzheimer's disease relies on behavior scores that measure a subject's language, memory, and cognitive skills. On the medical imaging side, functional magnetic resonance imaging has provided invaluable insights into the neural pathways underlying Alzheimer's disease. While prior studies have used resting-state functional MRI by extracting functional connectivity matrices, these approaches neglect the temporal dynamics inherent in functional data. In this work, we present a deep state space modeling framework that directly leverages the blood-oxygenation-level-dependent time series to learn a sparse collection of brain regions to predict behavior scores. Our model extracts temporal features that encapsulate nuanced patterns of intrinsic brain activity, thereby enhancing predictive performance compared to traditional connectivity methods. We identify specific brain regions that are most predictive of cognitive impairment through experiments on data provided by the Michigan Alzheimer's Disease Research Center, providing new insights into the neural substrates of early Alzheimer's pathology. These findings have important implications for the possible development of risk monitoring and intervention strategies in Alzheimer's disease.