Async API server starts throwing errors and works fine after restart

[AshuBik]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

All the requests to the open ai end point start throwing timeouts. This changed after doing two things

• Upgrading to the new GPT_3_5_TURBO_1106 model as well as using the async client.
• I reverted the GPT_3_5_TURBO_1106 change but the issue is still happening

I am using openai = "^1.3.9"

To Reproduce

Use a combination of above + Fastify
After some number of requests we start getting timeouts

Code snippets

No response

OS

macOs

Python version

3.11.5

Library version

1.3.9

[rattrayalex]

Hmm, there's not a lot of details to go off of here. Can you share some of your code, and the timeouts that you're seeing?

[AshuBik]

@rattrayalex

So, the code that I using is straightforward

            completion = await client.chat.completions.create(
                model=model.value,
                messages=messages_json,
                functions=functions_json,
                temperature=0.3,
                stream=stream,
                seed=12345,
                timeout=6
            )
        else:
            completion = await client.chat.completions.create(
                model=model.value,
                messages=messages_json,
                temperature=0.3,
                stream=stream,
                seed=12345,
                timeout=6
            )
        return completion

After a few hundred API calls, we start seeing a lot of timeout events. i.e. the API is not able to respond within 6 seconds.

Once we restart the server the open AI end points start functioning again.

---

I feel that this is somewhat related this issue

#769 (comment)

The problem still exists in 1.3.2. After triggering the content security policies multiple times, the requests time out, and the whole solution froze.

If we restart the application using the openai library, it will work again smoothly (until it is broken with the security policies).

[rattrayalex]

Hmm. Have you tried increasing the connection pool size?

[rattrayalex]

I suspect this is the result of running out of connections; please try this:

from openai import AsyncOpenAI
import httpx

client = AsyncOpenAI(
  http_client=httpx.Client(
    limits=httpx.Limits(
      max_connections=1000,
      max_keepalive_connections=100
    )
  )
)

Let us know if that helps.

If that doesn't work, we should reopen this issue.

[krrishdholakia]

seeing the same issue here. @rattrayalex wouldn't bumping the limit, just cause it to show up later?

[rattrayalex]

If the limit is higher than your rate limits or use in practice, it would not show up later.

[krrishdholakia]

Hey @rattrayalex my understanding was that this was being caused by connections not being properly closed, and therefore increasing over time.

Is that incorrect? If so, what's causing this?

[rattrayalex]

ah, since you are passing stream=stream, and the value may be True sometimes, you will need to be sure that in the cases where stream=True, you either read the response to completion or explicitly close it with completion.close() when you are done.

If you don't do this, the connection for that request will hang open indefinitely.