Run Llava with MultimodalRunner #14250
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14250
Note: Links to docs will display an error until the docs builds have been completed. ❌ 3 New Failures, 3 Unrelated FailuresAs of commit cc4c2e5 with merge base 378c700 ( NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following jobs failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
This PR needs a
|
mergennachin
requested review from
kirklandsign,
shoumikhin and
larryliu0820
rohansjoshi
force-pushed
the
multimodal-llava
branch
from
21ed8d1 to
cc4c2e5
Compare
| def forward(self, cache_positions, embeddings): | ||
| return self.text_model(None, {"input_pos": cache_positions[:1]}, embeddings) |
There was a problem hiding this comment.
Why is this necessary?
There was a problem hiding this comment.
E.g. say embeddings is length 5 and start pos is 2, MultimodalRunner text decoder passes input like ([2,3,4,5,6], embeddings), OTOH for LlavaRunner it would be ([2], embeddings).
This makes the first argument need dynamic shape. Alternatively, the other route would be to modify MultimodalRunner to handle both cases
There was a problem hiding this comment.
We are modifying the multimodal runner to handle both cases
Summary
Run Llava model with MultimodalRunner instead of LlavaRunner