Skip to content

VinF Hybrid Inference: throw if only_on_device and model is unavailable #8965

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 23, 2025
Merged

VinF Hybrid Inference: throw if only_on_device and model is unavailable #8965

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a685395
Return true unconditionally if mode is only_on_device
erikeldridge Apr 22, 2025
8979482
Remove stray log statement
erikeldridge Apr 22, 2025
69f393e
Throw Vertex error
erikeldridge Apr 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 7 additions & 59 deletions packages/vertexai/src/methods/chrome-adapter.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,19 +61,8 @@ describe('ChromeAdapter', () => {
})
).to.be.false;
});
it('returns false if AI API is undefined', async () => {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was stale, since we no longer have an ai namespace

const adapter = new ChromeAdapter(undefined, 'prefer_on_device');
expect(
await adapter.isAvailable({
contents: []
})
).to.be.false;
});
it('returns false if LanguageModel API is undefined', async () => {
const adapter = new ChromeAdapter(
{} as LanguageModel,
'prefer_on_device'
);
const adapter = new ChromeAdapter(undefined, 'prefer_on_device');
expect(
await adapter.isAvailable({
contents: []
Expand All @@ -82,7 +71,9 @@ describe('ChromeAdapter', () => {
});
it('returns false if request contents empty', async () => {
const adapter = new ChromeAdapter(
{} as LanguageModel,
{
availability: async () => Availability.available
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We now check availability first, so this was added in a couple places

} as LanguageModel,
'prefer_on_device'
);
expect(
Expand All @@ -93,7 +84,9 @@ describe('ChromeAdapter', () => {
});
it('returns false if request content has function role', async () => {
const adapter = new ChromeAdapter(
{} as LanguageModel,
{
availability: async () => Availability.available
} as LanguageModel,
'prefer_on_device'
);
expect(
Expand All @@ -107,51 +100,6 @@ describe('ChromeAdapter', () => {
})
).to.be.false;
});
it('returns false if request system instruction has function role', async () => {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to copy the systemInstruction out of the request, but removed that in favor of onDeviceParams.systemInstruction, which isn't currently validated. Added backlog item to revisit this.

const adapter = new ChromeAdapter(
{} as LanguageModel,
'prefer_on_device'
);
expect(
await adapter.isAvailable({
contents: [],
systemInstruction: {
role: 'function',
parts: []
}
})
).to.be.false;
});
it('returns false if request system instruction has multiple parts', async () => {
const adapter = new ChromeAdapter(
{} as LanguageModel,
'prefer_on_device'
);
expect(
await adapter.isAvailable({
contents: [],
systemInstruction: {
role: 'function',
parts: [{ text: 'a' }, { text: 'b' }]
}
})
).to.be.false;
});
it('returns false if request system instruction has non-text part', async () => {
const adapter = new ChromeAdapter(
{} as LanguageModel,
'prefer_on_device'
);
expect(
await adapter.isAvailable({
contents: [],
systemInstruction: {
role: 'function',
parts: [{ inlineData: { mimeType: 'a', data: 'b' } }]
}
})
).to.be.false;
});
it('returns true if model is readily available', async () => {
const languageModelProvider = {
availability: () => Promise.resolve(Availability.available)
Expand Down
30 changes: 12 additions & 18 deletions packages/vertexai/src/methods/chrome-adapter.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,29 +60,23 @@ export class ChromeAdapter {
* separation of concerns.</p>
*/
async isAvailable(request: GenerateContentRequest): Promise<boolean> {
// Returns false if we should only use in-cloud inference.
if (this.mode === 'only_in_cloud') {
return false;
}
// Returns false if the on-device inference API is undefined.;
if (!this.languageModelProvider) {
return false;
console.log(this.languageModelProvider);
const availability = await this.languageModelProvider?.availability();
if (availability === Availability.downloadable) {
// Triggers async model download.
this.download();
}
// Returns false if the request can't be run on-device.
if (!ChromeAdapter.isOnDeviceRequest(request)) {
return false;
}
const availability = await this.languageModelProvider.availability();
switch (availability) {
case Availability.available:
// Returns true only if a model is immediately available.
return true;
case Availability.downloadable:
// Triggers async download if model is downloadable.
this.download();
default:
return false;
if (this.mode === 'only_on_device') {
return true;
}
// Applies prefer_on_device logic.
return (
availability === Availability.available &&
ChromeAdapter.isOnDeviceRequest(request)
);
}

/**
Expand Down
Loading