Background Family caregivers are important allies for healthcare providers in facilitating the recovery process among people with mental illness (PWMI). The present study examined the factors associated with quality of life (QoL) among family caregivers of PWMI. Methods A multi-center cross-sectional survey was conducted. Family caregivers of people with schizophrenia, major depressive disorder, and bipolar disorder were recruited using convenience sampling. A survey assessing their QoL, depression, anxiety, and self-esteem was completed with self-rated psychometric scales including the Rosenberg Self-Esteem Scale, Caregiver Burden Inventory, Taiwanese Depression Questionnaire, Beck Anxiety Inventory, and World Health Organization Quality of Life Instrument Short Form. A mediation model was constructed with QoL as the dependent variable, care burden as the independent variable, and psychological distress (including depression and anxiety) with self-esteem as mediating variables. Results Family caregivers of people with schizophrenia had worse QoL compared with counterparts of people with major depression and bipolar disorder. The sociodemographic of both caregivers and PWMI had less impact on QoL when psychological factors were considered. Caregivers with lower self-esteem, higher levels of psychological distress, and heavier care burdens had poorer QoL. Care burden had a significant total effect on QoL. Both self-esteem and psychological distress were significant mediators. Conclusion The findings indicated that caregivers' psychological health and care burden influenced their QoL. Interventions that target family caregivers' self-esteem and psychological distress may attenuate the effect from care burden, and further improve their QoL.