docs: reasoning quickstart #110
Open
+206
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
👥 vLLM Semantic Team NotificationThe following members have been identified for the changed files in this PR and have been automatically assigned: 📁
|
|
can refer to it: https://docusaurus.io/zh-CN/docs/next/migration/v3#common-mdx-problems |
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
tao12345666333
force-pushed
the
docs/reasoning-quickstart
branch
from
6668536 to
927f1bc
Compare
rootfs
reviewed
Sep 10, 2025
|
|
||
| # Map concrete model names to a reasoning family | ||
| model_config: | ||
| "deepseek-v3": |
There was a problem hiding this comment.
nit: deepseek-v31 (since v3 is not a reasoning model)
rootfs
reviewed
Sep 10, 2025
| "deepseek-v3": | ||
| reasoning_family: "deepseek" | ||
| preferred_endpoints: ["endpoint1"] | ||
| "qwen3-7b": |
There was a problem hiding this comment.
nit: qwen3-30b (which is a reasoning model)
rootfs
reviewed
Sep 10, 2025
| score: 1.0 | ||
| - model: deepseek-v3 | ||
| score: 0.8 | ||
| - model: qwen3-7b |
rootfs
reviewed
Sep 10, 2025
| model_scores: | ||
| - model: openai/gpt-oss-20b | ||
| score: 1.0 | ||
| - model: deepseek-v3 |
rootfs
reviewed
Sep 10, 2025
| - model: qwen3-7b | ||
| score: 0.8 | ||
|
|
||
| - name: general |
There was a problem hiding this comment.
at the moment, all the categories must be from mmlu-pro. There is no general there. You can create an issue to support generic, free style categories and we can map the mmlu-pro categories to them.
rootfs
reviewed
Sep 10, 2025
| score: 0.8 | ||
|
|
||
| # A safe default when no category is confidently selected | ||
| default_model: qwen3-7b |
|
@tao12345666333 thanks for writing this up! Some nit but most looks good to me. |
aeft
reviewed
Sep 11, 2025
| - A model only gets reasoning fields if it has a model_config.<MODEL>.reasoning_family that maps to a reasoning_families entry. | ||
| - DeepSeek/Qwen3 (chat_template_kwargs): the router injects chat_template_kwargs only when reasoning is enabled. When disabled, no chat_template_kwargs are added. | ||
| - GPT/GPT-OSS (reasoning_effort): when reasoning is enabled, the router sets reasoning_effort based on the category (fallback to default_reasoning_effort). When reasoning is disabled, if the request already contains reasoning_effort and the model’s family type is reasoning_effort, the router preserves the original value; otherwise it is absent. | ||
| - For more stable classification, you can add category descriptions in config and keep them semantically distinctive. |
There was a problem hiding this comment.
@rootfs It seems like OpenAIRouter.CategoryDescriptions Category.ReasoningDescription Category.Description aren't used in the code (i.e., don't affect performance) currently. Will we use it in the future?
There was a problem hiding this comment.
nice catch! the reasoning description is no-op, information only.
Comment on lines
+111
to
+113
| Option B: Docker Compose | ||
| - docker compose up -d | ||
| - Exposes Envoy at http://localhost:8801 (proxying /v1/* to backends via the router) |
There was a problem hiding this comment.
Do you think #73 (review) can be satisfied here?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
docs: reasoning quickstart
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #51
Release Notes: Yes/No