Fine-Tuning Large Language Models for Low-Resource Languages

Most Large Language Models (LLMs) have been trained either on English text only or on text from multiple languages, but with a severe imbalance between data from different languages due to orders of magnitude differences in available web text per language. Despite this, some research has shown that by fine-tuning LLMs on smaller amounts of data from low-resource languages one can improve performance on downstream tasks for these languages.

The project is comprised of three subsections:

We investigated synthetic data generation and generated 3 sets of synthetic isiZulu data using machine translation and parent llm prompting to create the data. We achieved better in-language performance for Gemma3-4B by fine-tuning it on our synthetic data. 

We investigated how fine-tuning Gemma-3-4B-IT on diverse instruction-tuning datasets (AfridocMT, Aya Collection, and Inkuba-Instruct) affects accuracy across different AfriMMLU (question-answering dataset) subject areas and analysed the impact of zero-shot and few-shot prompting strategies on model performance.

We investigated and achieved substantial gains in model performance on isiZulu math word problems using fine-tuning on Chain-Of-Thought data. Furthermore, Group-Relative Policy Optimisation (GRPO) was implemented and used to further improve reasoning ability.