ValidationError if column value is 1 or 2

[davicorreiajr]

I'm having a weird issue: whenever the column value is 1 or 2, I get the following error:

target_postgres.exceptions.SingerStreamError: ('Invalid records detected above threshold: 0. See `.args` for details.', [(<ValidationError: '2.0 is not valid under any of the given schemas'

It's weird that I get the error for 1 or 2, but from 3 on it seems to work.

If it helps, the generated schema for this column is:

{
   "anyOf":[
      {
         "type":"null"
      },
      {
         "type":"number",
         "multipleOf":1e-15
      },
      {
         "type":"string"
      }
   ]
}

I'm using target-postgres along with tap-google-sheets.

[AlexanderMann]

@davicorreiajr that is bizarre, but I imagine that has something to do with floating point error/binary number gaps? Our code doesn't actually do those checks, that's Python's implementation of JSONSchema (we try to do as little data munging when validating the schema against the records we ingest).

This looks very similar: python-jsonschema/jsonschema#247

[djdevin]

Another postgres target with the same issue, possible solution:

https://github.com/statsbotco/target-postgres/pull/11/files

[AlexanderMann]

@djdevin ah, interesting. I think the best way to solve this, with the smallest footprint is prolly just to use the suggestion straight from the Python JSON docs:

>>> import decimal
>>> json.loads('1.1', parse_float=decimal.Decimal)
Decimal('1.1')

With that sort of change, handling this issue might be trivial.

[AlexanderMann]

If someone wants to put up a PR for the above which adds a test for the original problem AND uses the simple json.loads trick, I'm happy to review and try to get this out early next week. If not, I'll try and get the change in myself! 😄

[davicorreiajr]

Nice, thanks guys!

@AlexanderMann I'll try and do it.

[AlexanderMann]

This just needs to be deployed now. Will close once deployed.