itachi023/phi-2-4-bit-quantized

🧠 Phi-2 (4-bit Quantized with AutoRound)

This is a 4-bit quantized version of the microsoft/phi-2 model using Intel's AutoRound for weight-only post-training quantization (W4G128). It achieves significant compression while preserving model performance, making it ideal for resource-constrained inference.

---

🧾 Model Details

• Base model: microsoft/phi-2
• Quantization method: AutoRound (W4G128 - 4-bit, group size 128)
• Framework: 🤗 Transformers
• Precision: 4-bit weights
• Quantized size: ~1.85 GB (original: ~5.5 GB)
• Compression ratio: ~67%

---

🚀 How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("itachi023/phi-2-4-bit-quantized", torch_dtype=torch.float16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("itachi023/phi-2-4-bit-quantized")

prompt = "write a essay on deep learning"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=100)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

📦 Intended Uses

• Fast inference with low memory footprint
• Deployment on consumer GPUs or edge devices
• Offline assistants, document generation, or chatbots

---

⚠️ Limitations

• This model has not been fine-tuned post-quantization.
• Slight accuracy drop may occur vs. full-precision, especially on sensitive NLP tasks.
• Phi-2 is a pretrained model without alignment or safety tuning.

---

📈 Performance Notes

• Quantization config: W4G128 (4-bit, symmetric), 512 calibration samples, 1000 iterations
• AutoRound version: Latest (as of May 2025)
• Target device: GPU (A100/L4), float16 scale

---