Skip to content

Allow configurable connection pool size #204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
angadsingh opened this issue Mar 14, 2021 · 2 comments · Fixed by #215
Closed

Allow configurable connection pool size #204

angadsingh opened this issue Mar 14, 2021 · 2 comments · Fixed by #215
Labels
enhancement New feature or request
Milestone

Comments

@angadsingh
Copy link

angadsingh commented Mar 14, 2021

Proposal:
This client should allow configuring the underlying urllib3 connection pool size just like the older client:
influxdata/influxdb-python#349

Current behavior:
Currently this client throws the following error if there are more than 20 connections made:
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: eu-central-1-1.aws.cloud2.influxdata.com

Desired behavior:
Connection pool size should be configurable. E.g. in the old influx-python client there is a pool_size argument in the client's constructor:
https://influxdb-python.readthedocs.io/en/latest/api-documentation.html?highlight=pool#influxdb.InfluxDBClient
image

Use case:
This is necessary to avoid the above error and allow multi threaded clients to hit influx db 2.0 cloud properly

@angadsingh
Copy link
Author

angadsingh commented Mar 14, 2021

here's the hack i am currently using:

auth_header_name = "Authorization"
auth_header_value = "Token " + INFLUX_TOKEN
influx_client = InfluxDBClient(url="https://eu-central-1-1.aws.cloud2.influxdata.com", token=INFLUX_TOKEN)
conf = influx_client.api_client.configuration
conf.connection_pool_maxsize = 50
influx_client.api_client = ApiClient(configuration=conf,
                                     header_name=auth_header_name,
                                     header_value=auth_header_value)
return influx_client.write_api(write_options=SYNCHRONOUS)

the default connection_pool_maxsize is multiprocessing.cpu_count() * 5, which in my case on a raspberry pi 4 evaluates to 20. Not enough when one is making 50+ threads to send data to influx. Use case: a ping tool which pings 50 hosts parallely and sends results to influx (so its a very valid use case).

@bednar bednar added the enhancement New feature or request label Mar 15, 2021
@bednar
Copy link
Contributor

bednar commented Mar 15, 2021

@angadsingh, thanks for using our client, we will take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants