Skip to content

Commit

Permalink
Merge pull request #22 from GeraudBourdin/main
Browse files Browse the repository at this point in the history
typo in readme
  • Loading branch information
GeraudBourdin authored Sep 7, 2024
2 parents 8db4639 + a4a5d59 commit 0fec491
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,7 @@ CHAT_MODEL=mistral-7b-instruct-v0.2.Q4_K_M.gguf
Starting to v0.6.0 vLLM is fully compatible with the tools/tool_choice from mistral's plateforme.
To get an ISO instance locally you will need to specify the template: [examples/tool_chat_template_mistral.jinja](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_mistral.jinja) . Here is a docker-compose you can use for example:


```yaml
services:
mistral:
Expand All @@ -260,7 +261,7 @@ services:
--enforce-eager
--enable-auto-tool-choice
--tool-call-parser mistral
--chat-template /tool_chat_template_mistral.jinja
--chat-template /config/tool_chat_template_mistral.jinja
environment:
- HUGGING_FACE_HUB_TOKEN=*****
- NVIDIA_VISIBLE_DEVICES=0,1
Expand Down

0 comments on commit 0fec491

Please sign in to comment.