Performance

[JSchnepp]

Hello everybody,

In the current beta release of Influxdb 2, I have built a time series database for 11 zones (tag). Each zone contains a float load value (field) over the period from 01-2006 to 02-2020 in 5 minute intervals. Each zone therefore contains 1489536 load values.
With the Influx Data Explorer, the following query of the data within "Zone 1" including the time for printing the curve takes less than 3 seconds.

query = 'from (bucket: "my_bucket")'
                                    '|> range (start: -13mo, stop: -1mo)'
                                    '|> filter (fn: (r) => r._measurement == "my_measurement")'
                                    '|> filter (fn: (r) => r.Zone == "Zone1")'

For the identical query via the python influxdb interface the following program line

result = client.query_api (). query (org = org, query = query)

requires 63 seconds.

According to your experience what measures have to be taken to bring the response time to the order of 3 seconds?

best regards JS

[bednar]

Hi @JSchnepp,

Thanks for an open issue. Could you please share an information how your data looks like (cardinality, field types,...) ?

Regards

[JSchnepp]

Dear @bednar,

Attached you will find a screenshot of an Excel file with explanations and the extracts from the associated CSV files. I hope the information helps.
best regards

2006-01-01.txt
2020-29-02.txt

[bednar]

I prepared a performance test-case: performance_ny_electrical.py.

The results on the latest beta are:

python examples/performance_ny_electrical.py

Tables count: 1, results count: 88421
Query takes: 1.60405486 [sec]

Could you please run this test case?

[JSchnepp]

Thank you for preparing the performance test-case!

Here are the results:
Tables count: 1, results count: 105128
Query takes: 61.972300602 [sec]

I have attached some additional screenshots.

The python version is: python 3.8.1
and influxdb-client 1.5.0

best regrds

[bednar]

There are results with same version of client and python:

python --version
Python 3.8.1

pip install influxdb-client
Installing collected packages: influxdb-client
Successfully installed influxdb-client-1.5.0

python examples/performance_ny_electrical.py
Tables count: 1, results count: 88388
Query takes: 1.741436229 [sec]

The client uses ciso8601 to high performance date parsing... maybe it could be caused by conda that contains precompiled version of ciso8601.

Could you test the query without time fields?

query = f'from(bucket: "{bucket}")' \
        '|> range(start: -13mo, stop: -1mo)' \
        '|> filter(fn: (r) => r._measurement == "NYISO_Data")' \
        '|> filter(fn: (r) => r.Zone == "DUNWOD")' \
        '|> drop(columns: ["_time", "_start", "_stop"])'

[JSchnepp]

Dear bednar,

I have made the change and we see a light at the end of the tunnel.

Tables count: 1, results count: 105128
Query takes: 19.519807607 [sec]

best regards and thank you for your support

[bednar]

You are welcome

[JSchnepp]

Dear bednar,

I have installed now the from anaconda the following CISO8601 version:

conda install -c conda-forge/label/cf202003 ciso8601

Compared to the first result where it took 63 seconds to execute the query I am now at 1.04 seconds for the same query.

Thank you again for your support and best regards