Migrated from saken-tukenov/kazakh-gpt2-8m

README.md

@@ -0,0 +1,77 @@
+---
+language:
+  - kk
+license: apache-2.0
+library_name: transformers
+tags:
+  - gpt2
+  - kazakh
+  - causal-lm
+  - from-scratch
+  - soz
+pipeline_tag: text-generation
+---
+
+# Kazakh GPT-2 8M (Soz)
+
+A small GPT-2 language model trained from scratch on Kazakh text. Part of the Soz project for building native Kazakh language models.
+
+## Overview
+
+| Property | Value |
+|----------|-------|
+| **Parameters** | ~8M |
+| **Architecture** | GPT-2 |
+| **Vocab size** | 50,257 |
+| **Hidden dim** | 128 |
+| **Layers** | 6 |
+| **Attention heads** | 4 |
+| **FFN inner dim** | 512 |
+| **Training data** | [kazakh-clean-pretrain](https://huggingface.co/datasets/saken-tukenov/kazakh-clean-pretrain) (~80M tokens) |
+| **Epochs** | 2 |
+| **Final train loss** | ~6.4 |
+| **Tokenizer** | [kazakh-gpt2-50k](https://huggingface.co/saken-tukenov/kazakh-gpt2-50k) |
+| **License** | Apache 2.0 |
+
+## Design
+
+This is the smallest model in the Soz GPT-2 family. At ~8M parameters with ~80M training tokens, it is approximately Chinchilla-optimal (tokens-to-params ratio of ~10:1). It serves as a baseline for scaling experiments.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("saken-tukenov/kazakh-gpt2-50k")
+model = AutoModelForCausalLM.from_pretrained("saken-tukenov/kazakh-gpt2-8m")
+
+input_ids = tokenizer("Қазақстан — ", return_tensors="pt").input_ids
+output = model.generate(input_ids, max_new_tokens=50, do_sample=True, temperature=0.8)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+## Training Details
+
+- Trained from scratch using the Soz training pipeline
+- Optimizer: AdamW
+- Precision: bfloat16
+- Hardware: NVIDIA A10 GPUs
+
+## Project
+
+Part of the [Soz — Kazakh Language Models](https://github.com/saken-tukenov) project, a research effort to build open-source language models for Kazakh.
+
+## Citation
+
+```bibtex
+@misc{tukenov2026soz,
+  title={Soz: Small Language Models for Kazakh},
+  author={Tukenov, Saken},
+  year={2026},
+  url={https://huggingface.co/saken-tukenov/kazakh-gpt2-8m}
+}
+```
+
+## License
+
+Apache 2.0

config.json

@@ -0,0 +1,34 @@
+{
+  "activation_function": "gelu_new",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 2,
+  "dtype": "float32",
+  "embd_pdrop": 0.1,
+  "eos_token_id": 0,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 128,
+  "n_head": 4,
+  "n_inner": 512,
+  "n_layer": 6,
+  "n_positions": 1024,
+  "pad_token_id": 1,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.1.0",
+  "use_cache": false,
+  "vocab_size": 50257
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": [
+    0
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 1,
+  "transformers_version": "5.1.0",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3629867ad26de1ca422c7aa68f2429cf982976b01f881ea3543b828268119d4f
+size 31022800

tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "eos_token": "<|endoftext|>",
+  "is_local": true,
+  "model_max_length": 1024,
+  "pad_token": "<|padding|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}