InfluxDB silently ignores data points written to it using influxdb-client-python

[TomLXXVI]

I had an issue with this piece of code that I wrote myself:

@classmethod
    def write_power_to_db(cls, df: pd.DataFrame, measurement_name: str):
        columns = [col for col in df.columns if col != 'unit']
        for index, row in df.iterrows():
            for col in columns:
                point = Point(measurement_name)
                point.tag('powerFlow', col)
                point.field('power', row[col])
                point.time(datetime.strptime(index, DATETIME_STR_FMT), WritePrecision.S)
                print(point.to_line_protocol())
                cls.writer.write(cls.bucket, cls.org, point)

In this method I get data out of a Pandas DataFrame object, put it in a Point object and then write this object to its destination in my InfluxDB bucket. This code runs without any error or warning. But: no data in the bucket!
If I change the code to:

@classmethod
    def write_power_to_db(cls, df: pd.DataFrame, measurement_name: str):
        columns = [col for col in df.columns if col != 'unit']
        for index, row in df.iterrows():
            for col in columns:
                point = Point(measurement_name)
                point.tag('powerFlow', col)
                point.field('power', row[col])
                point.time(datetime.strptime(index, DATETIME_STR_FMT))
                print(point.to_line_protocol())
                cls.writer.write(cls.bucket, cls.org, point, WritePrecision.S)

then I get an error back from InfluxDB:

influxdb_client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json; charset=utf-8', 'X-Platform-Error-Code': 'invalid', 'Date': 'Sun, 31 May 2020 20:20:11 GMT', 'Content-Length': '188'})
HTTP response body: {"code":"invalid","message":"unable to parse 'solaredgePower,powerFlow=selfconsumption power=309.02734 1590508800000000000': time outside range -9223372036854775806 - 9223372036854775806"}

I finally could solve the issue myself just by coincidence when I tried this:

@classmethod
    def write_power_to_db(cls, df: pd.DataFrame, measurement_name: str):
        columns = [col for col in df.columns if col != 'unit']
        for index, row in df.iterrows():
            for col in columns:
                point = Point(measurement_name)
                point.tag('powerFlow', col)
                point.field('power', row[col])
                point.time(datetime.strptime(index, DATETIME_STR_FMT), WritePrecision.S)
                print(point.to_line_protocol())
                cls.writer.write(cls.bucket, cls.org, point, WritePrecision.S)

[bednar]

Hi @TomLXXVI,

thanks for using our client.

If you are use a batching then errors from InfluxDB are log into standard output. You could use synchronous way and see errors immediately.

Could you share DATETIME_STR_FMT with us?

Regards

[TomLXXVI]

DATETIME_STR_FMT refers to a datetime format string that I use throughout my program:

DATETIME_STR_FMT = "%Y-%m-%d %H:%M:%S"
"""DateTime format used by the SolarEdge Monitoring Server API"""

As you can see in the format string the smallest time unit is seconds (%S). Therefore, I use WritePrecision.S in point.time(...) to create the timestamp in the InfluxDB line protocol string.
What I didn't know, but I found out now, is that apparently it is needed to also set the same WritePrecision again in the call to WriterApi.write(...).

At this moment I run InfluxDB v2.0 inside a Docker container on Windows 10. I have almost no experience with Docker. I just followed the installation instructions on InfluxDB's website and luckily the installation went just fine. In "Docker Desktop" there is possibility to view log files and one has access to a terminal in the docker container. In none of these two I could see any error message from InfluxDB when I ran the first code snippet mentioned in my previous message (that's why I mentioned "...silently ignores...". When I ran the second snippet, I got immediately back an error message on my terminal from where I started my program (shown in my previous message) and also in the log file in Docker Desktop:

ts=2020-06-01T08:18:26.068862Z lvl=error msg="Error parsing points" log_id=0N6UJtFl000 org=TcEngineeringSoft bucket=solaredgeBucket error="unable to parse 'solaredgePower,powerFlow=selfconsumption power=309.02734 1590508800000000000': time outside range -9223372036854775806 - 9223372036854775806"

I used and still use "synchronous mode". The whole class implementation (with the working code version) for writing to InfluxDB looks like this:

class DBWriter:
    """
    Class that reads a csv-file from disk and stores the data in the InfluxDB time series database.
    """
    bucket: str = ""
    token: str = ""
    org: str = ""
    netloc: str = ""
    client: Optional[InfluxDBClient] = None
    writer: Optional[WriteApi] = None
    csv_path: str = ""

    @classmethod
    def init(cls, bucket: str, token: str, org: str, netloc: str, csv_path: str):
        """Initialize `DBWriter` to connect to the database."""
        cls.bucket = bucket
        cls.token = token
        cls.org = org
        cls.netloc = f"http://{netloc}"
        cls.client = InfluxDBClient(
            url=cls.netloc,
            token=cls.token,
            org=cls.org
        )
        cls.writer = cls.client.write_api(write_options=SYNCHRONOUS)
        cls.csv_path = csv_path

    @classmethod
    def read_power_csv(cls, start_date: date, end_date: date) -> pd.DataFrame:
        csv_file = f"power-details_{start_date}--{end_date}"
        with open(os.path.join(cls.csv_path, f"{csv_file}.csv")) as file_path:
            df = pd.read_csv(file_path, index_col=['time'])
        return df

    @classmethod
    def write_power_to_db(cls, df: pd.DataFrame, measurement_name: str):
        columns = [col for col in df.columns if col != 'unit']
        for index, row in df.iterrows():
            for col in columns:
                point = Point(measurement_name)
                point.tag('powerFlow', col)
                point.field('power', row[col])
                point.time(datetime.strptime(index, DATETIME_STR_FMT), WritePrecision.S)
                print(point.to_line_protocol())
                cls.writer.write(cls.bucket, cls.org, point, WritePrecision.S)

In the init method I set SYNCHRONOUS mode.

[bednar]

@TomLXXVI thanks for detail info.

The first snipped:

@classmethod
    def write_power_to_db(cls, df: pd.DataFrame, measurement_name: str):
        columns = [col for col in df.columns if col != 'unit']
        for index, row in df.iterrows():
            for col in columns:
                point = Point(measurement_name)
                point.tag('powerFlow', col)
                point.field('power', row[col])
                point.time(datetime.strptime(index, DATETIME_STR_FMT), WritePrecision.S)
                print(point.to_line_protocol())
                cls.writer.write(cls.bucket, cls.org, point)

should work, but I found a bug in writing Points with different precision than is specified in write_precision parameter. We should use a precision from Point not a precision from write_precision parameter.

So your data was written but with incorrect precision.

Thanks again for detail info. We will fix it ASAP.

• I've prepared a test case for that -

  influxdb-client-python/tests/test_WriteApi.py

  Line 309 in 8f20747

  def test_write_point_different_precision(self):

Regards

[Eslih]

I am experiencing the same issue. Point object seems to be formatted correctly, but the timestamp is interpreted incorrectly.

Turned on debug mode:

client = InfluxDBClient(url="http://10.10.20.62:9999", token=token, org=org, debug=True)

Created a point with seconds precision. If you look to the output, it is clear that the API is called with nanoseconds precision.

Code snippet:

p = Point("test_6") \
        .tag("method", 'test') \
        .field("value", 10.0) \
        .time(datetime.now(), write_precision=WritePrecision.S)

print(p.to_line_protocol())
write_api.write(bucket, org, p)

Output:

test_6,method=test value=10.0 1591264869
send: b'POST /api/v2/write?org=...&bucket=test_bucket&precision=ns HTTP/1.1\r\nHost: 10.10.20.62:9999\r\nAccept-Encoding: identity\r\nContent-Length: 40\r\nContent-Encoding: identity\r\nContent-Type: text/plain\r\nAccept: application/json\r\nAuthorization: Token ...\r\nUser-Agent: influxdb-client-python/1.7.0\r\n\r\n'
send: b'test_6,method=test value=10.0 1591264869'
reply: 'HTTP/1.1 204 No Content\r\n'
header: Date: Thu, 04 Jun 2020 08:01:15 GMT

Tried to change the url parameter, but that resulted in a parsing error. I guess it could be easily fixed to add a precision parameter to InfluxDBClient.

E.g.

client = InfluxDBClient(url="http://10.10.20.62:9999", precision=WritePrecision.S, token=token, org=org, debug=True)

[bednar]

Hi @Eslih,

your are right.

client = InfluxDBClient(url="http://10.10.20.62:9999", precision=WritePrecision.S, token=token, org=org, debug=True)

is correct workaround. Thx

Regards

[bednar]

Hi All,

The issue is fixed in 1.8.0 milestone.

If you would like to use a dev version then install client via:

pip install git+https://github.com/influxdata/influxdb-client-python.git@master

Regards