MzansiBench

Large language model (LLM) performance on low-resource languages, including most South African languages, remains under-evaluated due to the absence of standardised benchmarks and centralised datasets. Existing African-language benchmarks provide valuable evaluation resources, but South African language and task coverage remains uneven, and the relevant datasets, prompt templates, and fine-tuning configurations have historically been scattered across separate repositories and concentrated in a small number of languages, such as English, Afrikaans, isiZulu, and isiXhosa. MzansiBench addresses these limitations by unifying a modular Python evaluation library, a centralised data repository, and a public leaderboard website into a single, transparent framework for South African NLP research, evaluating large language models across 13 tasks and 16 datasets spanning South Africa's 11 official written languages, with particular attention to the nine low-resource Niger-Congo B languages. Two complementary pipelines are provided. The prompt-based pipeline evaluates models directly through prompt templates and is validated against AfroBench, a recent African-language benchmark, closely reproducing its published results under a matched environment. The fine-tuning pipeline offers a standardised workflow for encoder-only models across sequence classification, paired-text classification, and token classification tasks, validated through functional testing, comparison with published baselines, and repeated experiments across random seeds. Across both pipelines, results reveal a substantial performance gap between high-resource languages such as English and Afrikaans and South Africa's low-resource Niger-Congo B languages, along with severe data coverage gaps for languages such as Tshivenda and isiNdebele. MzansiBench provides the research community with a publicly available, reproducible benchmark for evaluating large language models across South Africa's 11 official written languages.