Most research in cognitive aging is based on literate participants from high-income and Western populations. The extent to which findings generalize to low-income and illiterate populations is unknown. The main aim was to examine the structure of between-person differences in cognitive functions among elderly from rural Bangladesh. We used data from the Poverty and Health in Aging (PHA) project in Bangladesh. The participants (n = 452) were in the age range 60–92 years. Structural equation modeling was used to estimate the fit of a five-factor model (episodic recall, episodic recognition, verbal fluency, semantic knowledge, processing speed) and to examine whether the model generalized across age, sex, and literacy. This study demonstrates that an established model of cognition is valid also among older persons from rural Bangladesh. The model demonstrated strong (or scalar) invariance for age, and partial strong invariance for sex and literacy. Semantic knowledge and processing speed showed weak (or metric) sex invariance, and semantic knowledge demonstrated also sensitivity to illiteracy. In general, women performed poorer on all abilities. The structure of individual cognitive differences established in Western populations also fits a population in rural Bangladesh well. This is an important prerequisite for comparisons of cognitive functioning (e.g., declarative memory) across cultures. It is also worth noting that absolute sex differences in cognitive performance among rural elderly in Bangladesh differ from those usually found in Western samples.