Instructions to use togethercomputer/GPT-JT-6B-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use togethercomputer/GPT-JT-6B-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="togethercomputer/GPT-JT-6B-v1")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("togethercomputer/GPT-JT-6B-v1")
model = AutoModelForCausalLM.from_pretrained("togethercomputer/GPT-JT-6B-v1")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use togethercomputer/GPT-JT-6B-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "togethercomputer/GPT-JT-6B-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-6B-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/togethercomputer/GPT-JT-6B-v1

SGLang

How to use togethercomputer/GPT-JT-6B-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "togethercomputer/GPT-JT-6B-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-6B-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "togethercomputer/GPT-JT-6B-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-6B-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use togethercomputer/GPT-JT-6B-v1 with Docker Model Runner:
```
docker model run hf.co/togethercomputer/GPT-JT-6B-v1
```

Successes and failures: Examples of GPT-JT doing well or poorly

by zhangce - opened Nov 30, 2022

Discussion

zhangce

Together org Nov 30, 2022

Discussion thread to share examples or areas in which GPT-JT does well or very poorly.

zhangce

Together org Nov 30, 2022

Thanks @espadrine on Twitter! (https://twitter.com/espadrine/status/1597696925071216640?s=20&t=orzxwNb0_OFPsCwM43KwOg)

GPT-JT (could improve):

Answer the following questions as an expert, explaining your reasoning in steps.

Question: How many months does it take for two women to give birth to five babies with no twins?
Answer: A human pregnancy has an average gestation period of nine months. Two women could undergo pregnancy at the same time. They would first produce two babies for nine months, then two more for another nine months, then one of them could have another baby for nine months. Overall, that would take twenty-seven months.

Question: Who was the father of the first French king?
Answer: The first French king was Clovis I. His father was Childeric I.

Question: How much time would a dolphin take to circumnavigate the Earth?
Answer: [A dolphin can swim at a speed of about 20 miles per hour. It takes the dolphin about one day to swim around the Earth.]

GPT-3:

Answer the following questions as an expert, explaining your reasoning in steps.

Question: Who was the father of the first French king?
Answer: The first French king was Clovis I. His father was Childeric I.

Question: How much time would a dolphin take to circumnavigate the Earth?
Answer: [A typical bottlenose dolphin swims at a speed of about 8 km/h. If a dolphin were to circumnavigate the Earth, it would have to swim approximately 40,075 km. At 8 km/h, it would take the dolphin approximately 5,009 hours, or 209 days, to circumnavigate the Earth.]

BigSalmon

Dec 1, 2022

•

edited Dec 1, 2022

I also noted that it does not do the best at producing factual information. For example, I asked: Who was South Korea's first president? Instead, it gave me Kim Il-sung.

Edit: I actually tried a different prompt.

Unsuccessful Prompt:

Q: Who was South Korea's first president?
A: [Kim Il-sung]

Successful Prompt:

Q: Who was South Korea's first president?
A: [South Korea's first president was Syngman Rhee].

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment