According to the Eighth National Survey Report on Students’ Constitution and Health released by the Ministry of Education in September 2021, the decline in college students’ physical fitness has not been effectively curbed. To understand the current situation and differences in physical health between Zhuang and Han college students and analyze the related influencing factors, this study collected the physical fitness test data of Zhuang and Han freshmen from 2018 to 2022 in a college in Guangxi Zhuang Autonomous Region. It analyzed the grade distribution of the body mass index (BMI) of Zhuang and Han students and compared the differences in physical fitness test indicators and physical fitness index between Zhuang and Han students. The Zhuang and Han students were divided into two groups based on their BMI, and the differences in physical fitness test indicators and physical fitness index within and between the groups were compared. Multivariate logistic regression analysis was employed to explore the influencing factors of physical health. The results indicated that the proportion of underweight, overweight, and obesity in boys was higher than that in girls, while the proportion of normal weight in girls was higher than that in boys. Zhuang students performed better than Han students in 50-meter running, standing long jump, pull-ups (for males) and sit-ups (for females), but the average values of height, weight, vital capacity, and sit-and-reach were lower than those of Han students. Generally, the physical fitness index of Zhuang students was higher than that of Han students. Within the same ethnic group, the differences in physical fitness test indicators and physical fitness index among students with different BMI levels were generally more significant in the normal weight group than in the overweight and obese groups. When comparing Zhuang and Han students with the same BMI level, the differences in physical fitness between the two ethnicities were mainly in the normal weight group. The average values of Zhuang students with a normal body mass index in standing long jump, pull-ups (for males) and sit-ups (for females) were higher than those of Han students, and the mean values of vital capacity and sit-and-reach were lower than those of the Han nationality. Multivariate regression analysis revealed that ethnicity and BMI levels were the influencing factors of students’ physical fitness and health. Compared with normal weight students, the probability of low physical fitness index scores in boys and girls decreased from obesity, overweight, to low weight. The results suggested that there were certain differences in physical fitness and health status between Zhuang and Han students. Different intervention measures should be adopted to enhance the physical fitness and health status of students in different ethnic groups and with different BMI levels.