Skip to content

Fix flaky test #6132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 29, 2024
Merged

Fix flaky test #6132

merged 1 commit into from
Jul 29, 2024

Conversation

mrober
Copy link
Contributor

@mrober mrober commented Jul 29, 2024

Fix flaky submitTaskFromAnotherWorkerDoesNotUseLocalThreads test. This test would fail about 2% of the time on my machine due to dealing with threads. With this change, it fails less than 0.5% of the time. Tested by running the test in a loop 1,000 times.

Copy link
Contributor

github-actions bot commented Jul 29, 2024

📝 PRs merging into main branch

Our main branch should always be in a releasable state. If you are working on a larger change, or if you don't want this change to see the light of the day just yet, consider using a feature branch first, and only merge into the main branch when the code complete and ready to be released.

1 similar comment
Copy link
Contributor

📝 PRs merging into main branch

Our main branch should always be in a releasable state. If you are working on a larger change, or if you don't want this change to see the light of the day just yet, consider using a feature branch first, and only merge into the main branch when the code complete and ready to be released.

@mrober mrober marked this pull request as ready for review July 29, 2024 18:43
@mrober mrober enabled auto-merge (squash) July 29, 2024 18:43
@google-oss-bot
Copy link
Contributor

@mrober mrober merged commit 26c8a5a into main Jul 29, 2024
47 checks passed
@mrober mrober deleted the crashlytics-fixflakytest branch July 29, 2024 18:50
@google-oss-bot
Copy link
Contributor

@google-oss-bot
Copy link
Contributor

Startup Time Report 1

Note: Layout is sometimes suboptimal due to limited formatting support on GitHub. Please check this report on GCS.

Notes

Startup Times

  • fire-cls

    DeviceStatisticsDistributions
    oriole-32
    Percentile3a4ec18d2e81ebDiffSignificant (?)
    p1010.1 ±1 ms10.5 ±2 ms+477 μs (+4.7%)NO
    p2510.7 ±2 ms11.2 ±2 ms+523 μs (+4.9%)NO
    p5011.5 ±2 ms12.2 ±2 ms+666 μs (+5.8%)NO
    p7512.8 ±2 ms13.5 ±2 ms+694 μs (+5.4%)NO
    p9014.5 ±2 ms14.9 ±2 ms+444 μs (+3.1%)NO

    20 test runs in comparison
    CommitTest Runs
    3a4ec18
    • 2024-07-29_18:10:47.155319_UWRc
    • 2024-07-29_18:10:47.155361_bmnN
    • 2024-07-29_18:10:47.155371_tPFa
    • 2024-07-29_18:10:47.155379_OXcT
    • 2024-07-29_18:10:47.155387_azlR
    • 2024-07-29_18:10:47.155403_qRmL
    • 2024-07-29_18:10:47.155411_fTsZ
    • 2024-07-29_18:10:47.155417_ClLi
    • 2024-07-29_18:10:47.155424_Dhtf
    • 2024-07-29_18:10:47.155431_MDTD
    d2e81eb
    • 2024-07-29_18:53:56.612230_TBxN
    • 2024-07-29_18:53:56.612274_lhgj
    • 2024-07-29_18:53:56.612283_GGxk
    • 2024-07-29_18:53:56.612290_EXoQ
    • 2024-07-29_18:53:56.612297_oUUP
    • 2024-07-29_18:53:56.612304_GOXu
    • 2024-07-29_18:53:56.612310_uBXo
    • 2024-07-29_18:53:56.612317_QRqn
    • 2024-07-29_18:53:56.612323_nBHv
    • 2024-07-29_18:53:56.612329_oNMV
    redfin-30
    Percentile3a4ec18d2e81ebDiffSignificant (?)
    p1016.9 ±7 ms15.4 ±6 ms-1.54 ms (-9.1%)NO
    p2518.0 ±7 ms16.2 ±7 ms-1.77 ms (-9.9%)NO
    p5019.4 ±9 ms17.4 ±8 ms-1.95 ms (-10.1%)NO
    p7521.1 ±9.6 ms18.8 ±9 ms-2.26 ms (-10.7%)NO
    p9023.3 ±11 ms21.0 ±9.8 ms-2.25 ms (-9.7%)NO

    20 test runs in comparison
    CommitTest Runs
    3a4ec18
    • 2024-07-29_18:10:47.155319_UWRc
    • 2024-07-29_18:10:47.155361_bmnN
    • 2024-07-29_18:10:47.155371_tPFa
    • 2024-07-29_18:10:47.155379_OXcT
    • 2024-07-29_18:10:47.155387_azlR
    • 2024-07-29_18:10:47.155403_qRmL
    • 2024-07-29_18:10:47.155411_fTsZ
    • 2024-07-29_18:10:47.155417_ClLi
    • 2024-07-29_18:10:47.155424_Dhtf
    • 2024-07-29_18:10:47.155431_MDTD
    d2e81eb
    • 2024-07-29_18:53:56.612230_TBxN
    • 2024-07-29_18:53:56.612274_lhgj
    • 2024-07-29_18:53:56.612283_GGxk
    • 2024-07-29_18:53:56.612290_EXoQ
    • 2024-07-29_18:53:56.612297_oUUP
    • 2024-07-29_18:53:56.612304_GOXu
    • 2024-07-29_18:53:56.612310_uBXo
    • 2024-07-29_18:53:56.612317_QRqn
    • 2024-07-29_18:53:56.612323_nBHv
    • 2024-07-29_18:53:56.612329_oNMV
  • fire-rc

    DeviceStatisticsDistributions
    oriole-32
    Percentile3a4ec18d2e81ebDiffSignificant (?)
    p10364 ±579 μs704 ±676 μs+340 μs (+93.3%)NO
    p25387 ±610 μs744 ±709 μs+358 μs (+92.5%)NO
    p50422 ±657 μs809 ±756 μs+387 μs (+91.7%)NO
    p75488 ±744 μs955 ±875 μs+467 μs (+95.7%)NO
    p90575 ±847 μs1.18 ±1 ms+602 μs (+104.6%)NO

    20 test runs in comparison
    CommitTest Runs
    3a4ec18
    • 2024-07-29_18:10:47.155319_UWRc
    • 2024-07-29_18:10:47.155361_bmnN
    • 2024-07-29_18:10:47.155371_tPFa
    • 2024-07-29_18:10:47.155379_OXcT
    • 2024-07-29_18:10:47.155387_azlR
    • 2024-07-29_18:10:47.155403_qRmL
    • 2024-07-29_18:10:47.155411_fTsZ
    • 2024-07-29_18:10:47.155417_ClLi
    • 2024-07-29_18:10:47.155424_Dhtf
    • 2024-07-29_18:10:47.155431_MDTD
    d2e81eb
    • 2024-07-29_18:53:56.612230_TBxN
    • 2024-07-29_18:53:56.612274_lhgj
    • 2024-07-29_18:53:56.612283_GGxk
    • 2024-07-29_18:53:56.612290_EXoQ
    • 2024-07-29_18:53:56.612297_oUUP
    • 2024-07-29_18:53:56.612304_GOXu
    • 2024-07-29_18:53:56.612310_uBXo
    • 2024-07-29_18:53:56.612317_QRqn
    • 2024-07-29_18:53:56.612323_nBHv
    • 2024-07-29_18:53:56.612329_oNMV
    redfin-30
    Percentile3a4ec18d2e81ebDiffSignificant (?)
    p10165 ±26 μs424 ±520 μs+258 μs (+156.4%)NO
    p25173 ±27 μs480 ±622 μs+307 μs (+177.8%)NO
    p50186 ±30 μs572 ±738 μs+386 μs (+206.8%)NO
    p75216 ±47 μs730 ±933 μs+514 μs (+237.8%)NO
    p90268 ±98 μs958 ±1246 μs+690 μs (+257.6%)NO

    20 test runs in comparison
    CommitTest Runs
    3a4ec18
    • 2024-07-29_18:10:47.155319_UWRc
    • 2024-07-29_18:10:47.155361_bmnN
    • 2024-07-29_18:10:47.155371_tPFa
    • 2024-07-29_18:10:47.155379_OXcT
    • 2024-07-29_18:10:47.155387_azlR
    • 2024-07-29_18:10:47.155403_qRmL
    • 2024-07-29_18:10:47.155411_fTsZ
    • 2024-07-29_18:10:47.155417_ClLi
    • 2024-07-29_18:10:47.155424_Dhtf
    • 2024-07-29_18:10:47.155431_MDTD
    d2e81eb
    • 2024-07-29_18:53:56.612230_TBxN
    • 2024-07-29_18:53:56.612274_lhgj
    • 2024-07-29_18:53:56.612283_GGxk
    • 2024-07-29_18:53:56.612290_EXoQ
    • 2024-07-29_18:53:56.612297_oUUP
    • 2024-07-29_18:53:56.612304_GOXu
    • 2024-07-29_18:53:56.612310_uBXo
    • 2024-07-29_18:53:56.612317_QRqn
    • 2024-07-29_18:53:56.612323_nBHv
    • 2024-07-29_18:53:56.612329_oNMV
  • fire-sessions

    DeviceStatisticsDistributions
    oriole-32
    Percentile3a4ec18d2e81ebDiffSignificant (?)
    p103.11 ±0.5 ms3.07 ±0.7 ms-38.2 μs (-1.2%)NO
    p253.34 ±0.5 ms3.29 ±0.7 ms-53.5 μs (-1.6%)NO
    p503.77 ±0.6 ms3.66 ±0.7 ms-104 μs (-2.8%)NO
    p754.37 ±0.6 ms4.39 ±0.7 ms+22.5 μs (+0.5%)NO
    p905.22 ±0.8 ms5.30 ±0.9 ms+87.3 μs (+1.7%)NO

    20 test runs in comparison
    CommitTest Runs
    3a4ec18
    • 2024-07-29_18:10:47.155319_UWRc
    • 2024-07-29_18:10:47.155361_bmnN
    • 2024-07-29_18:10:47.155371_tPFa
    • 2024-07-29_18:10:47.155379_OXcT
    • 2024-07-29_18:10:47.155387_azlR
    • 2024-07-29_18:10:47.155403_qRmL
    • 2024-07-29_18:10:47.155411_fTsZ
    • 2024-07-29_18:10:47.155417_ClLi
    • 2024-07-29_18:10:47.155424_Dhtf
    • 2024-07-29_18:10:47.155431_MDTD
    d2e81eb
    • 2024-07-29_18:53:56.612230_TBxN
    • 2024-07-29_18:53:56.612274_lhgj
    • 2024-07-29_18:53:56.612283_GGxk
    • 2024-07-29_18:53:56.612290_EXoQ
    • 2024-07-29_18:53:56.612297_oUUP
    • 2024-07-29_18:53:56.612304_GOXu
    • 2024-07-29_18:53:56.612310_uBXo
    • 2024-07-29_18:53:56.612317_QRqn
    • 2024-07-29_18:53:56.612323_nBHv
    • 2024-07-29_18:53:56.612329_oNMV
    redfin-30
    Percentile3a4ec18d2e81ebDiffSignificant (?)
    p105.45 ±0.8 ms6.08 ±2 ms+638 μs (+11.7%)NO
    p255.84 ±1 ms6.65 ±2 ms+806 μs (+13.8%)NO
    p506.21 ±1 ms7.50 ±3 ms+1.29 ms (+20.8%)NO
    p756.85 ±2 ms8.57 ±4 ms+1.72 ms (+25.1%)NO
    p907.69 ±4 ms9.80 ±6 ms+2.11 ms (+27.5%)NO

    20 test runs in comparison
    CommitTest Runs
    3a4ec18
    • 2024-07-29_18:10:47.155319_UWRc
    • 2024-07-29_18:10:47.155361_bmnN
    • 2024-07-29_18:10:47.155371_tPFa
    • 2024-07-29_18:10:47.155379_OXcT
    • 2024-07-29_18:10:47.155387_azlR
    • 2024-07-29_18:10:47.155403_qRmL
    • 2024-07-29_18:10:47.155411_fTsZ
    • 2024-07-29_18:10:47.155417_ClLi
    • 2024-07-29_18:10:47.155424_Dhtf
    • 2024-07-29_18:10:47.155431_MDTD
    d2e81eb
    • 2024-07-29_18:53:56.612230_TBxN
    • 2024-07-29_18:53:56.612274_lhgj
    • 2024-07-29_18:53:56.612283_GGxk
    • 2024-07-29_18:53:56.612290_EXoQ
    • 2024-07-29_18:53:56.612297_oUUP
    • 2024-07-29_18:53:56.612304_GOXu
    • 2024-07-29_18:53:56.612310_uBXo
    • 2024-07-29_18:53:56.612317_QRqn
    • 2024-07-29_18:53:56.612323_nBHv
    • 2024-07-29_18:53:56.612329_oNMV
  • timeToInitialDisplay

    DeviceStatisticsDistributions
    oriole-32
    Percentile3a4ec18d2e81ebDiffSignificant (?)
    p10201 ±4 ms206 ±5 ms+4.54 ms (+2.3%)NO
    p25206 ±4 ms212 ±6 ms+5.48 ms (+2.7%)NO
    p50214 ±5 ms220 ±6 ms+5.60 ms (+2.6%)NO
    p75222 ±5 ms228 ±7 ms+5.76 ms (+2.6%)NO
    p90229 ±5 ms238 ±6 ms+9.04 ms (+3.9%)NO

    20 test runs in comparison
    CommitTest Runs
    3a4ec18
    • 2024-07-29_18:10:47.155319_UWRc
    • 2024-07-29_18:10:47.155361_bmnN
    • 2024-07-29_18:10:47.155371_tPFa
    • 2024-07-29_18:10:47.155379_OXcT
    • 2024-07-29_18:10:47.155387_azlR
    • 2024-07-29_18:10:47.155403_qRmL
    • 2024-07-29_18:10:47.155411_fTsZ
    • 2024-07-29_18:10:47.155417_ClLi
    • 2024-07-29_18:10:47.155424_Dhtf
    • 2024-07-29_18:10:47.155431_MDTD
    d2e81eb
    • 2024-07-29_18:53:56.612230_TBxN
    • 2024-07-29_18:53:56.612274_lhgj
    • 2024-07-29_18:53:56.612283_GGxk
    • 2024-07-29_18:53:56.612290_EXoQ
    • 2024-07-29_18:53:56.612297_oUUP
    • 2024-07-29_18:53:56.612304_GOXu
    • 2024-07-29_18:53:56.612310_uBXo
    • 2024-07-29_18:53:56.612317_QRqn
    • 2024-07-29_18:53:56.612323_nBHv
    • 2024-07-29_18:53:56.612329_oNMV
    redfin-30
    Percentile3a4ec18d2e81ebDiffSignificant (?)
    p10242 ±3 ms265 ±3 ms+23.1 ms (+9.5%)YES
    p25247 ±3 ms270 ±4 ms+23.2 ms (+9.4%)YES
    p50254 ±5 ms278 ±4 ms+23.4 ms (+9.2%)MAYBE
    p75262 ±6 ms286 ±4 ms+24.1 ms (+9.2%)MAYBE
    p90271 ±6 ms299 ±7 ms+27.9 ms (+10.3%)MAYBE

    20 test runs in comparison
    CommitTest Runs
    3a4ec18
    • 2024-07-29_18:10:47.155319_UWRc
    • 2024-07-29_18:10:47.155361_bmnN
    • 2024-07-29_18:10:47.155371_tPFa
    • 2024-07-29_18:10:47.155379_OXcT
    • 2024-07-29_18:10:47.155387_azlR
    • 2024-07-29_18:10:47.155403_qRmL
    • 2024-07-29_18:10:47.155411_fTsZ
    • 2024-07-29_18:10:47.155417_ClLi
    • 2024-07-29_18:10:47.155424_Dhtf
    • 2024-07-29_18:10:47.155431_MDTD
    d2e81eb
    • 2024-07-29_18:53:56.612230_TBxN
    • 2024-07-29_18:53:56.612274_lhgj
    • 2024-07-29_18:53:56.612283_GGxk
    • 2024-07-29_18:53:56.612290_EXoQ
    • 2024-07-29_18:53:56.612297_oUUP
    • 2024-07-29_18:53:56.612304_GOXu
    • 2024-07-29_18:53:56.612310_uBXo
    • 2024-07-29_18:53:56.612317_QRqn
    • 2024-07-29_18:53:56.612323_nBHv
    • 2024-07-29_18:53:56.612329_oNMV

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/wGPpXQRmlH/index.html

mrober added a commit that referenced this pull request Aug 19, 2024
@firebase firebase locked and limited conversation to collaborators Aug 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants