feat(vertexai): Gemini multimodal output #8922
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: b898f35 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Vertex AI Mock Responses Check
|
Size Report 1Affected Products
Test Logs |
Size Analysis Report 1This report is too large (443,341 characters) to be displayed here in a GitHub comment. Please use the below link to see the full report on Google Cloud Storage.Test Logs |
dlarocque
force-pushed
the
dl/gemini-image-out
branch
2 times, most recently
from
9a05c5b to
4f7f1ec
Compare
dlarocque
force-pushed
the
dl/gemini-image-out
branch
from
4f7f1ec to
1d58a06
Compare
dlarocque
changed the title
Changeset File Check ✅
|
hsubox76
reviewed
Apr 30, 2025
| * | ||
| * @beta | ||
| */ | ||
| export const ResponseModality = { |
There was a problem hiding this comment.
This is the code object we agreed we should be exporting instead of TS enums so I get that, but we've had build issues in the past mixing JS code in types files so we should probably put these in a separate file. Looks like it's not causing build issues now so maybe we can move it along with the others whenever we plan to convert all our enums to JS objects.
hsubox76
reviewed
Apr 30, 2025
hsubox76
approved these changes
Apr 30, 2025
rachelsaunders
requested changes
May 5, 2025
rachelsaunders
approved these changes
May 5, 2025
|
|
||
| Generation modalities to be returned in generation responses. | ||
|
|
||
| - Multimodal response generation is only supported in some Gemini models and versions; see [model versions](https://firebase.google.com/docs/vertex-ai/models)<!-- -->. - Only image generation (`ResponseModality.IMAGE`<!-- -->) is supported. |
There was a problem hiding this comment.
Suggested change
| - Multimodal response generation is only supported in some Gemini models and versions; see [model versions](https://firebase.google.com/docs/vertex-ai/models)<!-- -->. - Only image generation (`ResponseModality.IMAGE`<!-- -->) is supported. | |
| - Multimodal response generation is only supported by some Gemini models and versions; see [model versions](https://firebase.google.com/docs/vertex-ai/models)<!-- -->. - Only image generation (`ResponseModality.IMAGE`<!-- -->) is supported. |
| * Generation modalities to be returned in generation responses. | ||
| * | ||
| * @remarks | ||
| * - Multimodal response generation is only supported in some Gemini models and versions; see {@link https://firebase.google.com/docs/vertex-ai/models | model versions}. |
There was a problem hiding this comment.
Suggested change
| * - Multimodal response generation is only supported in some Gemini models and versions; see {@link https://firebase.google.com/docs/vertex-ai/models | model versions}. | |
| * - Multimodal response generation is only supported by some Gemini models and versions; see {@link https://firebase.google.com/docs/vertex-ai/models | model versions}. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds new
ResponseModalityenum that allows users to specify which modalities should be included in a response.Since we provide a
text()accessor, a similarinlineDataParts()accessor was added to return allInlineDataPart[]in the first candidate.API Proposal: https://goto.google.com/vinf-multimodal-output-api (internal)