LearnPost
Continued Pretraining Learning Dynamics for South African Languages
Abstract
Continued Pretraining (CPT) is widely used to adapt language models to new languages. The technique involves further training pretrained language models (PLMs) on target-language text to improve their language modelling abilities in those languages. CPT is resource-efficient and effective, and is used to develop many state-of-the-art models for low-resource languages. However, the underlying adaptation process remains poorly understood, as most studies evaluate CPT-adapted models only after training is complete, which provides limited insight into how models adapt during training. To address this gap, we investigate the learning dynamics of encoder-only and encoder-decoder PLMs during CPT on South African languages. We select models with different levels of prior exposure to our target languages and analyse their downstream performance and internal representations across CPT checkpoints. Our findings suggest that model architecture, prior language exposure, and training setup all matter when adapting models to new languages through CPT.
Videos 1
Watch presentations, demos, and related content
LearnPost Project Video
Like, comment, and subscribe on YouTube to support the creator!
LearnPost Project Video
Gallery 1
Explore the visual story of this exhibit
Title Image