-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Loss of precision with scattergl
#6636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
@alexcjohnson Could you please take a moment to look into this issue? Unlike #5346, I've found that the issue of precision loss is specific to the I'm using https://github.com/plotly/Plotly.NET for plotting time-series data that requires nearly microsecond-level resolution. With Unix timestamps currently hovering around ~1.7e9, IEEE-754 doubles serve my needs well, unlike IEEE-754 singles. For my particular use case, I've found an effective workaround: I'm rendering high-frequency data streams via WebGL lines and overlaying less frequent, yet business-critical, "events" using non-WebGL markers. However, I recognize this approach may not be universally applicable. |
I'm seeing a similar issue, but my x axis values are supposed to be integers. It seems that they're being converted to single-precision floats. Is it possible to force plotly and WebGL to keep the x-axis as integers to avoid the loss of precision?
|
The issue I pointed out in my previous comment is actually M1 Mac specific. See #6820 |
Codepen link: https://codepen.io/DavidLHW/pen/MWzYXWK
In the above link, with small enough increments (in this case, at
1e-14
), the plot will suffer a loss of precision if it's plotted using thescattergl
type plot. Notice that the values are all clustered on the far left of the chart:If plot type is changed to
scatter
, the data would then be plotted correctly like:The text was updated successfully, but these errors were encountered: