File size: 3,584 Bytes
ec53ef2 15a2261 ec53ef2 15a2261 ec53ef2 15a2261 ec53ef2 15a2261 ec53ef2 15a2261 ec53ef2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | ---
license: mit
base_model: fredzzp/open-dcoder-0.5B
tags:
- code-generation
- diffusion-model
- masked-diffusion
- code-correction
- python
datasets:
- code
language:
- code
pipeline_tag: text-generation
---
# CDLM-0.5B
## Model Description
**CDLM-0.5B** is a fine-tuned version of [fredzzp/open-dcoder-0.5B](https://huggingface.co/fredzzp/open-dcoder-0.5B), trained using **error-aware training** with the mixture objective proposed in our paper on Corrective Diffusion Language Models. This model is designed to improve error-aware confidence and targeted refinement capabilities in code generation tasks.
### Key Features
- **Base Model**: [fredzzp/open-dcoder-0.5B](https://huggingface.co/fredzzp/open-dcoder-0.5B) (a masked diffusion language model based on Qwen2)
- **Training Method**: Error-aware training with mixture objective that explicitly supervises visible incorrect tokens
- **Architecture**: Masked Diffusion Language Model (MDLM)
- **Parameters**: ~0.5B
## Training Details
This model was fine-tuned from `fredzzp/open-dcoder-0.5B` using error-aware training with a mixture objective. For detailed information on the training methodology, please refer to our paper: [Corrective Diffusion Language Models](https://arxiv.org/pdf/2512.15596).
## Usage
### Installation
```bash
pip install torch transformers
```
### Code Generation
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "Shuibai12138/CDLM-0.5B"
device = "cuda" if torch.cuda.is_available() else "cpu"
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
trust_remote_code=True
).to(device)
# Generate code
prompt = "def fibonacci(n):"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
# Use diffusion generation
outputs = model.diffusion_generate(
inputs=input_ids,
max_new_tokens=100,
steps=16,
temperature=0.8
)
prompt_len = input_ids.shape[1]
generated_text = tokenizer.decode(outputs.sequences[0][prompt_len:], skip_special_tokens=True)
print("Generated Code:")
print(generated_text)
```
**Note**: This model uses a custom `diffusion_generate` method, so `trust_remote_code=True` is required when loading the model.
### Iterative Refinement
The model supports iterative refinement for code correction. See the [CDLM repository](https://github.com/zhangshuibai/CDLM) for examples of using the model for code correction tasks.
## Citation
If you use this model in your research, please cite:
```bibtex
@misc{zhang2025correctivediffusionlanguagemodels,
title={Corrective Diffusion Language Models},
author={Shuibai Zhang and Fred Zhangzhi Peng and Yiheng Zhang and Jin Pan and Grigorios G. Chrysos},
year={2025},
eprint={2512.15596},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2512.15596},
}
```
## Related Resources
- **Paper**: [Corrective Diffusion Language Models](https://arxiv.org/pdf/2512.15596)
- **Code Repository**: [zhangshuibai/CDLM](https://github.com/zhangshuibai/CDLM)
- **Collection**: [HuggingFace Collection](https://huggingface.co/collections/Shuibai12138/cdlm)
- **Base Model**: [fredzzp/open-dcoder-0.5B](https://huggingface.co/fredzzp/open-dcoder-0.5B)
## License
This model is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
## Contact
For questions and issues, please contact:
**Shuibai Zhang** <shuibai@cs.wisc.edu>
|