LLaMA 1.9B - Kazakh Causal Language Model
LLaMA 1.9B
Kazakh Causal Language Model
Model Description
This model is a Kazakh language version of the LLaMA model with 1.9 billion parameters, trained for causal language modeling. The train dataset includes some mixed Russian texts, which occasionally cause the model to generate Russian text. Despite this, the model shows promising results. Future steps may include retraining with a purer dataset, fine-tuning, or using the model for various NLP tasks with additional fine-tuning.
Training Setup
- Training Examples: Over 5.3 million examples
- Training Hardware: Two NVIDIA A100 GPUs (80GB each)
- Training Status: Ongoing, currently partway through the first epoch
- Optimizer: Cosine with restarts scheduler
- Parallelism: Distributed Data Parallel (DDP)
- Number of Warmup Steps: 8000
Model Authors
Name: Kadyrbek Nurgali
- Email: nurgaliqadyrbek@gmail.com
- LinkedIn: Kadyrbek Nurgali
@misc{
nurgali_kadyrbek_2024,
author = {NURGALI, Kadyrbek},
title = {llama-1.9B-kaz (Revision 299ebbb)},
year = 2024,
url = {https://huggingface.co/nur-dev/llama-1.9B-kaz},
doi = {10.57967/hf/3043},
publisher = {Hugging Face}
}
Citation
If you use this model, please cite the following publication:
Kadyrbek, N.; Tuimebayev, Z.; Mansurova, M.; Viegas, V. The Development of Small-Scale Language Models for Low-Resource Languages, with a Focus on Kazakh and Direct Preference Optimization. Big Data Cogn. Comput. 2025, 9, 137. https://doi.org/10.3390/bdcc9050137
- Downloads last month
- -