Skip to content

raise error indicating gzipped data #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 15, 2022
Merged

Conversation

FoamyGuy
Copy link
Contributor

This change presents a short term improvement of the error messages that arise from fetching GZipped data from an API. The message has been changed to state the exact cause of the problem and let the user know the way to access the raw data.

Tested successfully on Feather TFT ESP32-S2 and CPython on Linux using this code (with slightly modified initialization on the Feather for the circuitpython way of setup):

https = requests.Session(socket, ssl.create_default_context())

JSON_GET_URL = "https://analytics.usa.gov/data/live/realtime.json"

print("Fetching JSON data from %s" % JSON_GET_URL)
response = https.get(JSON_GET_URL)
print("-" * 40)

print("JSON Response: ", response.text)
print("-" * 40)

# print("JSON Response: ", response.json())
# print("-" * 40)

current version output (on CPython, it's very similar in CircuitPython but the error type is more general ValueError):

Fetching JSON data from https://analytics.usa.gov/data/live/realtime.json
----------------------------------------
Traceback (most recent call last):
  File "/home/timc/repos/circuitpython/Adafruit_CircuitPython_Requests/examples/gov_requests_https_cpython.py", line 19, in <module>
    print("JSON Response: ", response.text)
  File "/home/timc/repos/circuitpython/Adafruit_CircuitPython_Requests/adafruit_requests.py", line 437, in text
    self._cached = str(self.content, self.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

new output:

Fetching JSON data from https://analytics.usa.gov/data/live/realtime.json
----------------------------------------
Traceback (most recent call last):
  File "/home/timc/repos/circuitpython/Adafruit_CircuitPython_Requests/examples/gov_requests_https_cpython.py", line 19, in <module>
    print("JSON Response: ", response.text)
  File "/home/timc/repos/circuitpython/Adafruit_CircuitPython_Requests/adafruit_requests.py", line 439, in text
    raise ValueError(
ValueError: Content-encoding is gzip, data cannot be accessed as json or text. Use content property to access raw bytes.

I also verified that catching the exception does allow user code to successfully access the raw data:

try:
    print("JSON Response: ", response.json())
    print("-" * 40)
except ValueError:
    stream = io.BytesIO(response.content)
    field = pyflate.RBitfield(stream)
    magic = field.readbits(16)
    if magic == 0x1f8b: # GZip
        out = pyflate.gzip_main(field)

    json_data = json.loads(out)
    print(json_data)
    print("Active Visitors {}".format(json_data["data"][0]["active_visitors"]))

Copy link
Contributor

@askpatrickw askpatrickw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks solid, did not test.

@askpatrickw askpatrickw merged commit ea1c492 into adafruit:main Feb 15, 2022
adafruit-adabot added a commit to adafruit/Adafruit_CircuitPython_Bundle that referenced this pull request Feb 16, 2022
Updating https://github.com/adafruit/Adafruit_CircuitPython_ADT7410 to 1.3.2 from 1.3.1:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_ADT7410#20 from tammymakesthings/issue18-type-annotations
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_ADXL34x to 1.11.12 from 1.11.11:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_ADXL34x#33 from tammymakesthings/issue30-add-type-annotations
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_AS726x to 2.0.9 from 2.0.8:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_AS726x#19 from tammymakesthings/issue17-add-type-annotations
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_AS7341 to 1.2.4 from 1.2.3:
  > Merge pull request adafruit/Adafruit_CircuitPython_AS7341#24 from tekktrik/doc/add-CV-classes
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_AS7341#23 from tammymakesthings/issue20-add-type-annotations

Updating https://github.com/adafruit/Adafruit_CircuitPython_BME280 to 2.6.10 from 2.6.9:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_BME280#60 from tekktrik/doc/add-typing
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_FocalTouch to 1.3.0 from 1.2.10:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_FocalTouch#24 from kmatch98/master
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_MCP9600 to 1.2.3 from 1.2.2:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_MCP9600#18 from angerer/Issue-17_TypeAnnotations
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_PN532 to 2.3.7 from 2.3.6:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_PN532#53 from jerryneedell/jerryn_none
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_ST7789 to 1.5.5 from 1.5.4:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_ST7789#29 from mlashley/main
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_TLA202X to 1.0.5 from 1.0.4:
  > Merge pull request adafruit/Adafruit_CircuitPython_TLA202x#4 from tekktrik/doc/add-usage-example
  > Merge pull request adafruit/Adafruit_CircuitPython_TLA202x#5 from tekktrik/doc/add-typing
  > Fixed readthedocs build
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_VL6180X to 1.4.0 from 1.3.1:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_VL6180X#23 from JonasSchatz/main
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_BLE to 8.2.1 from 8.2.0:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_BLE#158 from tannewt/packet_buffer_test
  > Merge pull request adafruit/Adafruit_CircuitPython_BLE#156 from tekktrik/doc/consolidate-readme

Updating https://github.com/adafruit/Adafruit_CircuitPython_BLE_BroadcastNet to 0.12.1 from 0.12.0:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_BLE_BroadcastNet#28 from tannewt/cp_native_bridge
  > Merge pull request adafruit/Adafruit_CircuitPython_BLE_BroadcastNet#26 from tekktrik/doc/consolidate-readme

Updating https://github.com/adafruit/Adafruit_CircuitPython_BLE_Radio to 0.5.0 from 0.4.2:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_BLE_Radio#21 from tekktrik/feature/use-ble-consts-2
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_BusDevice to 5.1.5 from 5.1.4:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_BusDevice#80 from dhalbert/prune-mock-imports

Updating https://github.com/adafruit/Adafruit_CircuitPython_Debouncer to 1.5.0 from 1.4.1:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_Debouncer#35 from d-a-v/multi
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_LED_Animation to 2.5.14 from 2.5.13:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_LED_Animation#94 from tekktrik/doc/add-pause-example
  > Post-patch cleanup

Updating https://github.com/adafruit/Adafruit_CircuitPython_NeoKey to 1.0.5 from 1.0.4:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_NeoKey#6 from mattgoff/typeAnnotations
  > Consolidate Documentation sections of README

Updating https://github.com/adafruit/Adafruit_CircuitPython_Requests to 1.10.7 from 1.10.6:
  > Fixed readthedocs build
  > Merge pull request adafruit/Adafruit_CircuitPython_Requests#100 from FoamyGuy/gzip_error
  > Consolidate Documentation sections of README
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants