Chat History significantly degrades Function Calling performance

[maxbeech]

Description of the bug:

Whenever I use Chat History (from the last 12 hours, I add the last 20 messages), the amount of times that Function Calling works (ie. it calls a function) drops very significantly.

After 50 tests, with ChatHistory filled with the same request each time...
With Chat History enabled: the function is called 10% of the time
With Chat History disabled: the function is called 100% of the time

In the case of Chat History being enabled, I've found it is particularly susceptible to not calling the function when optional function parameters aren't specified in the prompt (ie. it thinks it needs the parameters when it doesn't).

Actual vs expected behavior:

Expected:
Function Calling function is called as part of call to Gemini

Actual:
Gemini decides not to use any functions

Any other information you'd like to share?

Running within a NodeJS Google Cloud Function
Tried using both Gemini 1.5 Flash and Gemini 1.5 Pro but get the same results.
Using Function Calling mode "Auto" (as expected, mode "Any" fixes the issue but this mode cannot be used in my use case).

[zavocc]

Chat history essentially influences model response so it may be biased how and when it should call tools, this also happens with Python SDK... Looks like gemini side of problem

[hkt74]

+1 to @zavocc, passing the history could indeed give models more context from previous turns. It really depends on the use case to pass history or start a new session. Agree it is more of an api side problem and not a sdk issue

Hi @maxbeech, could you possibly share a code snippet to reproduce the error? I made some change based on this sample and model returned all the function calls as expected. Perhaps the existing sample isn't complex enough to trigger the problem? Thanks!

[zavocc]

fyi Gemini 2.0 is better at calling tools now compared to previous models... as I've been test driving and using 2.0 models for months now and its better at calling tools regardless of the context (apparently 1.5 models weren't mature enough to call tools properly)

Unless staying at 1.5 models due to constraints... Gemini 2.0 would be suitable for tool use dependent tasks