Instructions to use stabilityai/StableBeluga-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use stabilityai/StableBeluga-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="stabilityai/StableBeluga-7B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("stabilityai/StableBeluga-7B")
model = AutoModelForCausalLM.from_pretrained("stabilityai/StableBeluga-7B")

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use stabilityai/StableBeluga-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "stabilityai/StableBeluga-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stabilityai/StableBeluga-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/stabilityai/StableBeluga-7B

SGLang

How to use stabilityai/StableBeluga-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "stabilityai/StableBeluga-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stabilityai/StableBeluga-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "stabilityai/StableBeluga-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stabilityai/StableBeluga-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use stabilityai/StableBeluga-7B with Docker Model Runner:
```
docker model run hf.co/stabilityai/StableBeluga-7B
```

Fine tunening model

by Mehri - opened Jul 31, 2023

Discussion

Mehri

Jul 31, 2023

Does anyone fine tuned this model with their own data set?

RoversX

Aug 2, 2023

Already tried to use Qlora fine tuned merge but the file conversion of .safetensors got me confuse 😅

breadlicker45

Aug 2, 2023

Already tried to use Qlora fine tuned merge but the file conversion of .safetensors got me confuse 😅

Qlora works for me

RoversX

Aug 3, 2023

Already tried to use Qlora fine tuned merge but the file conversion of .safetensors got me confuse 😅

Qlora works for me

Thanks, I realized the problem

arvind2626

Aug 13, 2023

Hey, I was trying to finetune this model using qLora with this config:
config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["query_key_value"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)

model = get_peft_model(model, config)
print_trainable_parameters(model)

However i ran into the following error:
ValueError: Target modules ['query_key_value'] not found in the base model. Please check the target modules and try
again.

I am a complete beginner, can someone please help me out? Thanks!

RoversX

Aug 14, 2023

Hey, I was trying to finetune this model using qLora with this config:
config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["query_key_value"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)

model = get_peft_model(model, config)
print_trainable_parameters(model)

However i ran into the following error:
ValueError: Target modules ['query_key_value'] not found in the base model. Please check the target modules and try
again.

I am a complete beginner, can someone please help me out? Thanks!

Perhaps consider checking out Maxime Labonne's Tutorial. I found the quality of writing to be superior to that of my own notebook. I attempted to train this model, and it worked. Just replace the model name to this

arvind2626

Aug 15, 2023

Hey, thanks for the tutorial, I was able to fine tune my model. However, when I load the model saved on hugging face in text gen webui, it gets loaded on the cpu ram, instead of the gpu ram. Can you pls help?
https://huggingface.co/arvind2626/Stable-Beluga-arvind this is the fine tune model

breadlicker45

Aug 15, 2023

This comment has been hidden

RoversX

Aug 15, 2023

Hey, thanks for the tutorial, I was able to fine tune my model. However, when I load the model saved on hugging face in text gen webui, it gets loaded on the cpu ram, instead of the gpu ram. Can you pls help?
https://huggingface.co/arvind2626/Stable-Beluga-arvind this is the fine tune model

Hi, I tested it on my Colab, and I think it's fine. I am also a beginner here. I'm curious about the dataset format you use. Does it look like this?

{"text": "### Human: ABABA### Assistant: ABABAB### Human: ABABA### Assistant: ABABAB"}

Because the quality of the model I fine-tuned isn't very good, and I suspect the problem lies with the format. Anyway, here's the notebook I tested. I believe it works: notebook

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment