AI Hybrid Inference: extract expected inputs from prompt

[erikeldridge]

Problem Statement

Vertex doesn't require callers to pre-specify expected input types. Could we make the hybrid API do the same?

Now that Edge has added support for text inputs, but not image inputs, we need to remove the default image input type, which further motivates this change.

Solution

Since the AI SDK request object contains the intended input types, we can extract those and reformat them into the format expected by Chrome.

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: dae5fee4d82b5ccaca36bec62ef262de93816035

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions[bot]]

Vertex AI Mock Responses Check :warning:

A newer major version of the mock responses for Vertex AI unit tests is available. update_vertexai_responses.sh should be updated to clone the latest version of the responses: v11.0

[google-oss-bot]

Size Report ¹

Affected Products

• @firebase/ai

  Type	Base (58d92df)	Merge (feb1e52)	Diff
  browser	48.3 kB	49.0 kB	+750 B (+1.6%)
  main	49.5 kB	50.4 kB	+945 B (+1.9%)
  module	48.3 kB	49.0 kB	+750 B (+1.6%)

• firebase

  Type	Base (58d92df)	Merge (feb1e52)	Diff
  firebase-ai.js	38.1 kB	40.5 kB	+2.34 kB (+6.1%)
  firebase-vertexai.js	38.1 kB	40.5 kB	+2.34 kB (+6.1%)

Test Logs

• Base (58d92df): https://github.com/firebase/firebase-js-sdk/actions/runs/15311497376
• Merge (feb1e52): https://github.com/firebase/firebase-js-sdk/actions/runs/15496403066

1. https://storage.googleapis.com/firebase-sdk-metric-reports/jpfXLGnNo2.html

[google-oss-bot]

Size Analysis Report ¹

Affected Products

• @firebase/ai

  • AIError

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.58 kB	4.60 kB	+18 B (+0.4%)

  • AIModel

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	24.3 kB	24.3 kB	+18 B (+0.1%)

  • ArraySchema

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	6.37 kB	6.39 kB	+18 B (+0.3%)

  • Backend

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.58 kB	4.60 kB	+18 B (+0.4%)

  • BackendType

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • BlockReason

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • BooleanSchema

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	6.37 kB	6.39 kB	+18 B (+0.3%)

  • ChatSession

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	20.4 kB	20.5 kB	+18 B (+0.1%)

  • FinishReason

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • FunctionCallingMode

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.60 kB	4.61 kB	+18 B (+0.4%)

  • GenerativeModel

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	24.3 kB	24.3 kB	+18 B (+0.1%)

  • GoogleAIBackend

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • HarmBlockMethod

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • HarmBlockThreshold

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • HarmCategory

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • HarmProbability

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • HarmSeverity

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • ImagenAspectRatio

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • ImagenImageFormat

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.87 kB	4.89 kB	+18 B (+0.4%)

  • ImagenModel

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	26.1 kB	26.1 kB	+18 B (+0.1%)

  • ImagenPersonFilterLevel

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.60 kB	4.62 kB	+18 B (+0.4%)

  • ImagenSafetyFilterLevel

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.60 kB	4.62 kB	+18 B (+0.4%)

  • IntegerSchema

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	6.37 kB	6.39 kB	+18 B (+0.3%)

  • Modality

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.58 kB	4.60 kB	+18 B (+0.4%)

  • NumberSchema

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	6.37 kB	6.39 kB	+18 B (+0.3%)

  • ObjectSchema

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	6.37 kB	6.39 kB	+18 B (+0.3%)

  • POSSIBLE_ROLES

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.64 kB	4.65 kB	+18 B (+0.4%)

  • ResponseModality

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.63 kB	4.65 kB	+18 B (+0.4%)

  • Schema

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	6.37 kB	6.39 kB	+18 B (+0.3%)

  • SchemaType

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.60 kB	+18 B (+0.4%)

  • StringSchema

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	6.37 kB	6.39 kB	+18 B (+0.3%)

  • VertexAIBackend

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.59 kB	4.61 kB	+18 B (+0.4%)

  • VertexAIError

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.60 kB	4.62 kB	+18 B (+0.4%)

  • VertexAIModel

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	24.3 kB	24.3 kB	+18 B (+0.1%)

  • getAI

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.95 kB	4.96 kB	+18 B (+0.4%)

  • getGenerativeModel

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	28.0 kB	28.4 kB	+393 B (+1.4%)
    size-with-ext-deps	47.1 kB	49.1 kB	+2.06 kB (+4.4%)

    External Dependency

    Module	Base (58d92df)	Merge (feb1e52)	Diff
    deepmerge		default export	+ default export

  • getImagenModel

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	26.2 kB	26.2 kB	+18 B (+0.1%)

  • getVertexAI

    Size

    Type	Base (58d92df)	Merge (feb1e52)	Diff
    size	4.98 kB	5.00 kB	+18 B (+0.4%)

Test Logs

• Base (58d92df): https://github.com/firebase/firebase-js-sdk/actions/runs/15311497376
• Merge (feb1e52): https://github.com/firebase/firebase-js-sdk/actions/runs/15496403066

1. https://storage.googleapis.com/firebase-sdk-metric-reports/mM9vetpEqz.html