azure/bedrock api integration #1048

sameelarif · 2025-09-03T20:23:25Z

why

Need to support Azure/Bedrock and all other AI SDK providers that have more than just an apiKey parameter.

what changed

Allowed free input of modelClientOptions and added logic for authenticating with AWS for Bedrock usage.

test plan

…serialize

changeset-bot · 2025-09-03T20:23:29Z

⚠️ No Changeset found

Latest commit: adec13c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

greptile-apps

Greptile Summary

This PR refactors the LLM client architecture to support Azure OpenAI and AWS Bedrock API integrations. The changes focus on standardizing client option handling across different AI providers while maintaining backwards compatibility.

The core changes include:

Type System Refactoring: Modified ClientOptions type to include a generic Record<string, string> union alongside existing OpenAIClientOptions and AnthropicClientOptions, enabling support for diverse provider-specific configurations
Client Architecture Updates: Made clientOptions property optional across all LLM client implementations (OpenAIClient, AnthropicClient, CerebrasClient) and removed it from the abstract LLMClient base class
Provider System Enhancement: Updated LLMProvider.getAISDKLanguageModel() to accept unified ClientOptions parameter instead of separate apiKey and baseURL parameters, allowing more flexible provider configuration
Import Organization: Standardized import ordering across LLM client files, moving error type imports to the top for consistency
API Key Handling: Simplified API key resolution in the main Stagehand constructor to rely exclusively on environment variables for legacy providers (OpenAI, Anthropic, Google)

These changes integrate with the existing AI SDK framework that supports multiple providers through factory functions, enabling the codebase to accommodate cloud-based AI services that require additional configuration beyond simple API keys (such as regions, resource names, and custom endpoints).

PR Description Notes:

The PR description is empty and lacks the required "Why", "What Changed", and "Test Plan" sections as outlined in the repository's pull request template

Confidence score: 3/5

This PR introduces significant architectural changes that could impact existing integrations and require careful testing
Score reflects the complexity of the type system changes and potential for runtime issues with the loosened type constraints
Pay close attention to types/model.ts, lib/llm/LLMProvider.ts, and lib/index.ts as they contain the most critical changes

Context used:

Context - We enforce linting and prettier at the CI level, so no code style comments that aren't obvious. (link)

_{8 files reviewed, 4 comments}

_{Edit Code Review Bot Settings | Greptile}

greptile-apps · 2025-09-03T20:24:00Z

lib/llm/CerebrasClient.ts

@@ -1,5 +1,6 @@
-import OpenAI from "openai";
+import { CreateChatCompletionResponseError } from "@/types/stagehandErrors";
 import type { ClientOptions } from "openai";


style: Import ClientOptions from openai but use OpenAI.ClientOptions in constructor parameter - consider using consistent type reference

Suggested change

import type { ClientOptions } from "openai";

clientOptions?: ClientOptions;

greptile-apps · 2025-09-03T20:24:10Z

lib/llm/LLMProvider.ts

-    }
-    const provider = creator(providerConfig);
+    // Create the provider instance with the custom configuration options
+    const provider = creator(modelClientOptions);


logic: Type safety concern: ClientOptions is a union type that may not match what the creator function expects, potentially causing runtime errors

types/model.ts

…serialize

# why solves #1060 patch regression of playwright arguments being removed from agent execute response # what changed agent.execute now returns playwright arguments in its response # test plan tested locally

…ms to docs (#1065) # why reflect project id changes in docs # what changed advanced configuration comments # test plan reviewed via mintlify on localhost

# why Easier to use for Custom LLM Clients and keep users up to date with our aisdk file # what changed added export of aisdk to lib/index.ts # test plan build local stagehand, import local AISdkClient, run Azure Stagehand session

…onfigu… (#1073) …ration settings # why Updated docs to match the new fingerprint params in the Browserbase docs here: https://docs.browserbase.com/guides/stealth-customization#customization-options # what changed Update browser configuration docs to reflect the docs changes. # test plan

# why Updating docs to reflect aisdk can be imported directly # what changed The model page # test plan Reviewed page with mintlify dev locally

miguelg719 · 2025-09-16T18:38:15Z

lib/api.ts

+    // Add modelClientOptions as a header if provided
+    if (modelClientOptions) {
+      defaultHeaders["x-model-client-options"] =
+        JSON.stringify(modelClientOptions);


why send it as a header? modelClientOptions already gets sent in the payload. A stringified json in a header is probably not the move

# why # what changed # test plan

# why Currently, we do not support stagehand agent within the api # what changed When api is enabled, stagehand agent now routes through the api # test plan Tested locally

# why Currently, using playwright screenshot command is not available when the execution environment is Stagehand. A customer has indicated they would prefer to use Playwright's native screenshot command instead of CDP when using Browserbase as CDP screenshot causes unexpected behavior for their target site. # what changed - added a StagehandScreenshotOptions type with useCDP argument added - extended page type to accept custom stagehand screeenshot options - update screenshot proxy to default useCDP to true if the env is browserbase and use playwright screenshot if false - added eval for screenshot with and without cdp # test plan - tested and confirmed functionality with eval and external example script (not committed)

…1057) # why We want to build a best in class agent in stagehand. Therefore, we need more eval benchmarks. # what changed - Added Web-bench evals dataset - Added a subset of OS World evals - those that can be run in a chrome browser (desktop-based tasks omitted) - added LICENSE noticed to the copied evals tasks - Added ground truth / expected result to some WebVoyager tasks using reference_answer.json from Browser Use public evals repo. Improvements to `pnpm run evals -man` to better describe how to run evals. # test plan Evals should run locally and bb for these new benchmarks.

# why Initial instructions didn't mention uv or pip prerequisites and also didn't mention venv. Fix reduces friction on first timers. # what changed - added link to install uv - added details for initializing venv - adjusted code example respectively # test plan docs change

# why - webpage structure changed, needed to update the xpath in the expected locator

… with LanguageModelV1 + LiteLLM works for python (#1086) # why 1. aisdk not yet available through npm package 2. customLLM provider only works with LanguageModelV1 3. LiteLLM compatible providers are supported in python # what changed 1. change docs to install stagehand from git repo 2. pin versions that use LanguageModelV1 # test plan local test

# why currently we pass stagehand page to agent, this results in our page management having issues when facing new tabs # what changed the stagehand object is now passed instead of stagehandPage # test plan tested locally

# why Our existing screenshot service is a dummy time-based triggered service. It also does not trigger based on any actions of the agent. # what changed Added img hash diff algo (quick check with MSE, verify with SSIM algo) to see if there was an actual UI change and only store ss in the buffer if that is so. Added ss interceptor which copies each screenshot the agent is taking to a buffer (if different enough from the previous ss) to be later used for evals. - There's also a small refactor of the agent initialization config to enable the screenshot collector service to be attached # test plan Tests pass locally --------- Co-authored-by: Miguel <36487034+miguelg719@users.noreply.github.com> Co-authored-by: miguel <miguelg71921@gmail.com>

# why To help make sense of eval test cases and results # what changed Added metadata to eval runs, cleaned deprecated code # test plan

…serialize

# why # what changed # test plan

examples/example-bedrock.ts

# why anthropic released a new sota computer use model # what changed added claude-sonnet-4-5-20250929 as a model to the list # test plan ran evals

miguelg719 · 2025-09-29T22:06:24Z

lib/api.ts

+    this.logger({
+      category: "execute",
+      message: `Executing ${method} with args: ${JSON.stringify(args)}`,
+      level: 1,


maybe level 2

miguelg719 · 2025-09-30T17:22:59Z

lib/StagehandPage.ts

            const result = await this.api.act({
              ...observeResult,
              frameId: this.rootFrameId,
+              modelClientOptions: this.stagehand["modelClientOptions"],


is this required?

We will need this once we add self-healing

miguelg719 · 2025-09-30T17:23:10Z

lib/StagehandPage.ts

-          result = await this.api.extract<T>({ frameId: this.rootFrameId });
+          result = await this.api.extract<T>({
+            frameId: this.rootFrameId,
+            modelClientOptions: this.stagehand["modelClientOptions"],


is this required?

Yes because otherwise it doesn't get sent to the API. We need this param on all api calls now

miguelg719 · 2025-09-30T17:31:12Z

lib/utils.ts

+    message:
+      "No Amazon Bedrock authentication credentials found. Please provide credentials via modelClientOptions (accessKeyId/secretAccessKey or bearerToken) or environment variables (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY or AWS_BEARER_TOKEN_BEDROCK)",
+    level: 0,
+  });


should we throw here

It would throw once they make an LLM call. We let people use stagehand without an LLM so I figured we just print a warning for now. Down to change it tho

Why Custom AI SDK tools and MCP integrations weren't working properly with Anthropic CUA - parameters were empty {} and tools weren't tracked. What Changed - Convert Zod schemas to JSON Schema before sending to Anthropic (using zodToJsonSchema) - Track custom tool calls in the actions array - Silence "Unknown tool name" warnings for custom tools Test Plan Tested with examples file. Parameters passed correctly ({"city":"San Francisco"} instead of {}) Custom tools execute and appear in actions array No warnings

# why To improve context # what changed Added current page and url to the system prompt # test plan

# why To inform the user throughout the agent execution process # what changed Added logs to tool calls, and on the stagehand agent handler # test plan - [x] tested locally

# why # what changed # test plan

# why anthropic released a new sota computer use model # what changed added claude-sonnet-4-5-20250929 as a model to the list # test plan ran evals

Why Custom AI SDK tools and MCP integrations weren't working properly with Anthropic CUA - parameters were empty {} and tools weren't tracked. What Changed - Convert Zod schemas to JSON Schema before sending to Anthropic (using zodToJsonSchema) - Track custom tool calls in the actions array - Silence "Unknown tool name" warnings for custom tools Test Plan Tested with examples file. Parameters passed correctly ({"city":"San Francisco"} instead of {}) Custom tools execute and appear in actions array No warnings

# why To improve context # what changed Added current page and url to the system prompt # test plan

# why To inform the user throughout the agent execution process # what changed Added logs to tool calls, and on the stagehand agent handler # test plan - [x] tested locally

…serialize

sameelarif added 2 commits August 29, 2025 14:37

allow free modelclientoptions

76cc39b

Merge branch 'main' into sameel/stg-692-azurebedrock-api-integration-…

28f4b6e

…serialize

greptile-apps bot reviewed Sep 3, 2025

View reviewed changes

sameelarif and others added 8 commits September 9, 2025 14:04

Merge branch 'main' into sameel/stg-692-azurebedrock-api-integration-…

1a415bb

…serialize

change zod ver to working build

04fb315

add playwright arguments to agent (#1066)

9daa584

# why solves #1060 patch regression of playwright arguments being removed from agent execute response # what changed agent.execute now returns playwright arguments in its response # test plan tested locally

[docs] add info on not needing project id in browserbase session para…

f6f05b0

…ms to docs (#1065) # why reflect project id changes in docs # what changed advanced configuration comments # test plan reviewed via mintlify on localhost

Export aisdk (#1058)

c886544

# why Easier to use for Custom LLM Clients and keep users up to date with our aisdk file # what changed added export of aisdk to lib/index.ts # test plan build local stagehand, import local AISdkClient, run Azure Stagehand session

send client options on every request

8eccd56

[docs] export aisdk (#1074)

3c39a05

# why Updating docs to reflect aisdk can be imported directly # what changed The model page # test plan Reviewed page with mintlify dev locally

miguelg719 reviewed Sep 16, 2025

View reviewed changes

miguelg719 and others added 11 commits September 16, 2025 11:43

Fix zod peer dependency support (#1032)

bf2d0e7

# why # what changed # test plan

add stagehand agent to api (#1077)

7f38b3a

# why Currently, we do not support stagehand agent within the api # what changed When api is enabled, stagehand agent now routes through the api # test plan Tested locally

update xpath in observe_vantechjournal (#1088)

b9c8102

# why - webpage structure changed, needed to update the xpath in the expected locator

Fix session create logs on api (#1089)

536f366

Improve failed act logs (#1090)

8ff5c5a

pass stagehand, instead of stagehandPage to agent (#1082)

8c0fd01

# why currently we pass stagehand page to agent, this results in our page management having issues when facing new tabs # what changed the stagehand object is now passed instead of stagehandPage # test plan tested locally

filip-michalsky force-pushed the sameel/stg-692-azurebedrock-api-integration-serialize branch from fc0c9e5 to 8eccd56 Compare September 23, 2025 10:26

filip-michalsky and others added 5 commits September 23, 2025 13:37

test bedrock file

c6a752d

add azure test file

2931804

Eval metadata (#1092)

f89b13e

# why To help make sense of eval test cases and results # what changed Added metadata to eval runs, cleaned deprecated code # test plan

fix bedrock test

2f3b8b9

Merge branch 'main' into sameel/stg-692-azurebedrock-api-integration-…

be8b7a4

…serialize

sameelarif and others added 5 commits September 25, 2025 14:20

Update pnpm-lock.yaml

27c722c

update evals cli docs (#1096)

108de3c

# why # what changed # test plan

better modelclientoption api handling

69c3d93

dont override region

467dade

fix bedrock example

0735ca3

filip-michalsky requested changes Sep 28, 2025

View reviewed changes

examples/example-bedrock.ts Show resolved Hide resolved

filip-michalsky self-requested a review September 29, 2025 13:22

adding support for new claude 4.5 sonnet agent model (#1099)

e0e6b30

# why anthropic released a new sota computer use model # what changed added claude-sonnet-4-5-20250929 as a model to the list # test plan ran evals

miguelg719 reviewed Sep 29, 2025

View reviewed changes

sameelarif added 2 commits September 29, 2025 21:13

lint

76b44ae

read aws creds from client options obj

18937ee

miguelg719 reviewed Sep 30, 2025

View reviewed changes

tkattkat and others added 12 commits October 1, 2025 10:53

Add current date and page url to agent context (#1102)

a99aa48

# why To improve context # what changed Added current page and url to the system prompt # test plan

Additional agent logging (#1104)

a1ad06c

# why To inform the user throughout the agent execution process # what changed Added logs to tool calls, and on the stagehand agent handler # test plan - [x] tested locally

update evals cli docs (#1096)

0af4acf

# why # what changed # test plan

adding support for new claude 4.5 sonnet agent model (#1099)

c762944

# why anthropic released a new sota computer use model # what changed added claude-sonnet-4-5-20250929 as a model to the list # test plan ran evals

Add current date and page url to agent context (#1102)

ce07cfa

# why To improve context # what changed Added current page and url to the system prompt # test plan

Additional agent logging (#1104)

06ae0e6

# why To inform the user throughout the agent execution process # what changed Added logs to tool calls, and on the stagehand agent handler # test plan - [x] tested locally

fix system prompt

9fe40fd

remove dup log

938b51c

pass modelClientOptions for stagehand agent

607b4c3

Merge branch 'main' into sameel/stg-692-azurebedrock-api-integration-…

adec13c

…serialize

miguelg719 force-pushed the main branch from 4994eab to bd0a799 Compare October 29, 2025 16:15

	import type { ClientOptions } from "openai";
	clientOptions?: ClientOptions;

azure/bedrock api integration #1048

Are you sure you want to change the base?

azure/bedrock api integration #1048

Conversation

sameelarif commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

why

what changed

test plan

Uh oh!

changeset-bot bot commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Greptile Summary

Confidence score: 3/5

Context used:

Uh oh!

greptile-apps bot Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

sameelarif commented Sep 3, 2025 •

edited

Loading

changeset-bot bot commented Sep 3, 2025 •

edited

Loading