[Bug]: `vllm bench serve --random-input-length=1` generates an empty prompt

[smarterclayton]

Your current environment

Since at least the last few weeks (mid August), but probably much further back.

🐛 Describe the bug

While benchmarking P/D I realized that --random-input-len=1 results in an empty prompt for some models, specifically DeepSeek-R1.

This is due to this chunk in the random dataset:

        num_special_tokens = int(tokenizer.num_special_tokens_to_add())
        real_input_len = max(0, int(input_len) - num_special_tokens)

For deepseek-r1 and several other tested models (facebook/opt-125m) tokenizer.num_special_tokens_to_add() returns 1, meaning the prompt is empty. It also means in many cases we generate fewer tokens than we expect, which throws off perf calculations.

When running a benchmark, I would expect the actual prompt to contain the number of random input tokens specified. Num special tokens should not be counted as part of the expected random input tokens (or we should specify that in bench via another parameter).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[smarterclayton]

A related issue is that for accurate decode behavior testing it should be possible to send an empty prompt and specify it via the benchmark tool - #24405 (comment) just changed this behavior to be impossible, but future versions of the API should allow empty prompts if the model will actually generate output.