Instructions to use togethercomputer/GPT-JT-6B-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use togethercomputer/GPT-JT-6B-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="togethercomputer/GPT-JT-6B-v1")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("togethercomputer/GPT-JT-6B-v1")
model = AutoModelForCausalLM.from_pretrained("togethercomputer/GPT-JT-6B-v1")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use togethercomputer/GPT-JT-6B-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "togethercomputer/GPT-JT-6B-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-6B-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/togethercomputer/GPT-JT-6B-v1

SGLang

How to use togethercomputer/GPT-JT-6B-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "togethercomputer/GPT-JT-6B-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-6B-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "togethercomputer/GPT-JT-6B-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-6B-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use togethercomputer/GPT-JT-6B-v1 with Docker Model Runner:
```
docker model run hf.co/togethercomputer/GPT-JT-6B-v1
```

8bit quantization

#12

by deleted - opened Dec 12, 2022

Discussion

deleted

Dec 12, 2022

•

edited Dec 12, 2022

I wonder if there are any plans to produce an 8bit (quantized) version of GPT-JT, as was done for the original GPT-J in hivemind/gpt-j-6B-8bit. Could address steep hardware requirements as in discussion #9.

hivemind provides the script they used to quantize GPT-J (convert-gpt-j.ipynb in the model repo), but my attempt was unsuccessful.

deleted changed discussion title from Quantization of GPT-JT to Quantization Dec 12, 2022

deleted changed discussion title from Quantization to 8bit quantization Dec 18, 2022

vipul

Together org Dec 24, 2022

We have one in the lab, and does quite well on benchmarks. We'll release it once we've done more performance work on it.

fractalego

Jan 1, 2023

I've used from_pretrained(... , load_in_8bit=True) and it seems to work. Haven't benchmarked it yet. Memory-wise it seems to stay under 10GB this way.

DarwinAnim8or

Jan 5, 2023

We have one in the lab, and does quite well on benchmarks. We'll release it once we've done more performance work on it.

This is exciting news! Any idea how the performance stacks up to the regular version? (if you can share that already of course)

yahma

Jan 6, 2023

Looking forward to the release of the 8-bit quantized model..

Considering the UL2 training objective used in this model, would adjustments need to be made to fine-tuning this model, or is it no different than fine-tuning regular GPT-J?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment