Instructions to use casperhansen/mpt-7b-8k-chat-awq with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use casperhansen/mpt-7b-8k-chat-awq with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="casperhansen/mpt-7b-8k-chat-awq", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("casperhansen/mpt-7b-8k-chat-awq", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("casperhansen/mpt-7b-8k-chat-awq", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use casperhansen/mpt-7b-8k-chat-awq with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "casperhansen/mpt-7b-8k-chat-awq"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "casperhansen/mpt-7b-8k-chat-awq",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/casperhansen/mpt-7b-8k-chat-awq

SGLang

How to use casperhansen/mpt-7b-8k-chat-awq with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "casperhansen/mpt-7b-8k-chat-awq" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "casperhansen/mpt-7b-8k-chat-awq",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "casperhansen/mpt-7b-8k-chat-awq" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "casperhansen/mpt-7b-8k-chat-awq",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use casperhansen/mpt-7b-8k-chat-awq with Docker Model Runner:
```
docker model run hf.co/casperhansen/mpt-7b-8k-chat-awq
```

MPT-7b-8k-chat

This model is originally released under CC-BY-NC-SA-4.0, and the AWQ framework is MIT licensed.

Original model can be found at https://huggingface.co/mosaicml/mpt-7b-8k-chat.

⚡ 4-bit Inference Speed

Machines rented from RunPod - speed may vary dependent on both GPU/CPU.

H100:

CUDA 12.0, Driver 525.105.17: 92 tokens/s (10.82 ms/token)

RTX 4090 + Intel i9 13900K (2 different VMs):

CUDA 12.0, Driver 525.125.06: 134 tokens/s (7.46 ms/token)
CUDA 12.0, Driver 525.125.06: 117 tokens/s (8.52 ms/token)

RTX 4090 + AMD EPYC 7-Series (3 different VMs):

CUDA 12.2, Driver 535.54.03: 53 tokens/s (18.6 ms/token)
CUDA 12.2, Driver 535.54.03: 56 tokens/s (17.71 ms/token)
CUDA 12.0, Driver 525.125.06: 55 tokens/ (18.15 ms/token)

A6000 (2 different VMs):

CUDA 12.0, Driver 525.105.17: 61 tokens/s (16.31 ms/token)
CUDA 12.1, Driver 530.30.02: 46 tokens/s (21.79 ms/token)

How to run

Install AWQ:

git clone https://github.com/mit-han-lab/llm-awq && \
cd llm-awq && \
pip3 install -e . && \
cd awq/kernels && \
python3 setup.py install && \
cd ../.. && \
pip3 install einops

Run:

hfuser="casperhansen"
model_name="mpt-7b-8k-chat-awq"
group_size=128
repo_path="$hfuser/$model_name"
model_path="/workspace/llm-awq/$model_name"
quantized_model_path="/workspace/llm-awq/$model_name/$model_name-w4-g$group_size.pt"

git clone https://huggingface.co/$repo_path

python3 tinychat/demo.py --model_type mpt \
    --model_path $model_path \
    --q_group_size $group_size \
    --load_quant $quantized_model_path \
    --precision W4A16

Citation

Please cite this model using the following format:

@online{MosaicML2023Introducing,
    author    = {MosaicML NLP Team},
    title     = {Introducing MPT-30B: Raising the bar
for open-source foundation models},
    year      = {2023},
    url       = {www.mosaicml.com/blog/mpt-30b},
    note      = {Accessed: 2023-06-22},
    urldate   = {2023-06-22}
}

Downloads last month: 31