Exceptions in Callback Handlers can lead to deadlock

[btasker]

Specifications

• Client Version: 1.3.6
• InfluxDB Version: 1.8.10 / 2.x
• Platform: Any

Code sample to reproduce problem

Attaching repro script

repro_lockup.py.gz

Expected behavior

Script should exit once processing in main() is complete, even if the callback handlers fail (for whatever reason)

Actual behavior

Script runs until killed.

Strace shows processing is stuck in sleep

clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, {tv_sec=1438366, tv_nsec=978518494}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=1438366, tv_nsec=979255844}) = 0
clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, {tv_sec=1438367, tv_nsec=79255844}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=1438367, tv_nsec=79775567}) = 0
clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, {tv_sec=1438367, tv_nsec=179775567}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=1438367, tv_nsec=180567114}) = 0

Looks to be this while loop

Additional info

No response

[btasker]

If a configured callback throws an exception, the thread used to handle batching writes dies.

As a result, the conditional used in that while will never evaluate true, leaving the script stuck in an infinite loop.

There are probably 2 changes needed for this

• Update the callback invocations (here) to trap exceptions and warn that they happened
• Add a counter or timer to the while loop so that if the thread takes too long to close, the script can still exit

I'm going to try and get a PR in to implement both shortly