Floats can lose precision when loading to BigQuery

[danielchatfield]

The float precision is set here: https://github.com/pydata/pandas-gbq/blob/d251db03b159447331ac9ae63e13d295d75bad70/pandas_gbq/load.py#L22

This is insufficient to represent all 64 bit floats without losing precision. For example 26/59 should be represented as 0.4406779661016949 but under this it is represented as 0.440677966101695.

This was added intentionally here to fix a different issue but it causes us some issues as we need perfect reconciliation between systems. It seems like it should be possible to get the best of both worlds and output the correct number of digits in all cases.

The original suggestion was to use %g but this was changed to %.15g – it's not clear to me what the rationale is for that, it seems like %g is strictly better but I'm sure I'm missing something.

[danielchatfield]

@max-sixty I was wondering if you might be able to provide some help / guidance here?

[dkapitan]

@danielchatfield

Not sure if this is helpful, but I think one of the issues as explained here is that a conservative choice is made in the number of significant digits.

A possible solution if you do need to have larger precision is to use .parquet format instead, as suggested here?

[max-sixty]

Thanks @dkapitan

[tswast]

Would %.16g avoid the precision issues?

[tswast]

The original suggestion was to use %g but this was changed to %.15g – it's not clear to me what the rationale is for that, it seems like %g is strictly better but I'm sure I'm missing something.

According to https://docs.python.org/3.6/library/string.html#formatspec

A precision of 0 is treated as equivalent to a precision of 1. The default precision is 6.

I think that would result in too much rounding on some systems.

[tswast]

According to https://en.wikipedia.org/wiki/IEEE_754#Character_representation 17 digits are precision are required to preserve the original binary value. 16 digits was not enough in my testing of #336