Llama-2-13b-best_ratio90 (90% Parameters)
This model is a pruned and finetuned version of meta-llama/Llama-2-13b-hf, retaining approximately 90% of parameters while maintaining strong performance through genetic algorithm pruning and RMSNorm fine-tuning.
Model Details
- Base Model: meta-llama/Llama-2-13b-hf
- Parameter Retention: ~90%
- Pruning Method: Genetic Algorithm
- Fine-tuning Method: RMSNorm calibration
Performance
| Metric | Value |
|---|---|
| PPL (Before Fine-tuning) | 5.80 |
| PPL (After Fine-tuning) | 5.01 |
| Improvement | 13.64% |
Performance Comparison
| Model | PPL (After FT) |
|---|---|
| 50% params | 10.03 |
| 70% params | 6.59 |
| 80% params | 5.64 |
| 90% params | 5.01 |
Files Included
- : Full model state dict
- : This documentation
License
Llama 2 Community License (inherited from base model)
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for ra225/Llama-2-13b-best_ratio90_rmsnorm_finetuned
Base model
meta-llama/Llama-2-13b-hf