cnmoro/nomic-embed-text-v2-moe-distilled-high-quality
Feature Extraction • Updated • 42.4k • 4
Carlo Moro
cnmoro
AI & ML interests
None yet
Recent Activity
reacted to sergiopaniego's post with 🔥 1 day ago
Earlier this month, Apple introduced Simple Self-Distillation: a fine-tuning method that improves models on coding tasks just by sampling from the model and training on its own outputs with plain cross-entropy
And… it's already supported in TRL, built by Kashif Rasul. you can really feel the pace of development in the team 🐎
Paper by Ruixiang ZHANG, He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang at Apple 🍎
How it works: the model generates completions at a training-time temperature (T_train) with top_k/top_p truncation, then fine-tunes on them with plain cross-entropy. no labels or verifier needed
You can try it right away with this ready-to-run example (Qwen3-4B on rStar-Coder):
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd.py
or benchmark a checkpoint with the eval script:
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd_eval.py
One neat insight from the paper: T_train and T_eval compose into an effective T_eff = T_train × T_eval, so a broad band of configs works well. even very noisy samples still help
Want to dig deeper?
Paper: https://huggingface.co/papers/2604.01193
Trainer docs: https://huggingface.co/docs/trl/main/en/ssd_trainer liked a model 1 day ago
Qwen/Qwen3.6-35B-A3B updated a model 2 days ago
cnmoro/nomic-embed-text-v2-moe-distilled-high-qualityOrganizations
cnmoro/inference-free-splade-co-condenser-en-ptbr
Sentence Similarity • Updated
cnmoro/inference-free-splade-bert-tiny-en-ptbr
Sentence Similarity • Updated
cnmoro/LFM-Q4-GGUFS
3B • Updated • 340
cnmoro/static-nomic-384-pten
Feature Extraction • Updated • 34
cnmoro/RotorQuant-ModelWeights-Runtime
Updated • 3
cnmoro/Qwen2.5-0.5B-Portuguese-v1
Text Generation • 0.5B • Updated • 35 • 5
cnmoro/LFM2-PTBR-imatrix-Quants
3B • Updated • 26
cnmoro/LexicalEmbed-Base
Feature Extraction • 16.6M • Updated • 6 • 1
cnmoro/low-dimension-static-model
Updated • 5
cnmoro/custom-model2vec-tokenlearn-medium
Updated • 6 • 1
cnmoro/custom-model2vec-tokenlearn-small
Updated • 8 • 1
cnmoro/Qwen3-16B-A3B-REAP-PTBR
16B • Updated • 1 • 1
cnmoro/Qwen3-7B-A3B-REAP-PTBR
7B • Updated • 10 • 2
cnmoro/Qwen3-7B-A3B-REAP-PTBR-Q4_K_M-GGUF
7B • Updated • 45
cnmoro/Qwen3-16B-A3B-REAP-PTBR-SuperExp
16B • Updated • 2 • 1
cnmoro/distilbert-portuguese-tokenizer-lower-greedy
Updated
cnmoro/bert-hash-femto-mlm
Fill-Mask • 1.8M • Updated • 16
cnmoro/LFM2-350M-PTBR
Text Generation • 0.4B • Updated • 4
cnmoro/static-nomic-distilled-vocab-quantized
Updated • 5
cnmoro/gpt-oss-20b-tokenizer-optional-reasoning
Updated • 3
cnmoro/gliclass-edge-v3.0-onnx
Text Classification • Updated • 1
cnmoro/gliclass-large-v3.0-onnx
Text Classification • Updated • 1
cnmoro/gliclass-base-v3.0-onnx
Text Classification • Updated • 1
cnmoro/gliclass-modern-large-v3.0-onnx
Text Classification • Updated • 1
cnmoro/gliclass-x-base-onnx
Text Classification • Updated • 1
cnmoro/prompt-router
Text Classification • 0.1B • Updated • 7 • 2
cnmoro/portuguese-nomic-embed-text-v2-moe
Sentence Similarity • 0.3B • Updated • 9
cnmoro/portuguese-bge-m3
Sentence Similarity • 0.3B • Updated • 158 • 5
cnmoro/portuguese-en-bge-m3
Sentence Similarity • 0.4B • Updated • 4