Skip to content

AbstractParallelEvaluator RuntimeException in 8.2.0 #25841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
westrik opened this issue Apr 14, 2025 · 8 comments · May be fixed by coderabbit-test/bazel#1
Closed

AbstractParallelEvaluator RuntimeException in 8.2.0 #25841

westrik opened this issue Apr 14, 2025 · 8 comments · May be fixed by coderabbit-test/bazel#1
Assignees
@fmeum
Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@westrik
Copy link

westrik commented Apr 14, 2025

Description of the bug:

We're seeing the following error intermittently with 8.2.0:

ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/5c2f943d20519c831aa2980e5ea19296/external/org_golang_x_mod/internal/lazyregexp/BUILD.bazel:3:11: GoCompilePkg external/org_golang_x_mod/internal/lazyregexp/lazyregexp.a failed: I/O exception during sandboxed execution: java.util.concurrent.ExecutionException: java.lang.InterruptedException
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData0{actionLookupKey=ConfiguredTargetKey{label=@@io_bazel_rules_go//:stdlib, config=BuildConfigurationKey[240063384f9257e894b5195f75fa38c16397f9c03ae72b3bb2698fbafd6e0bb9]}, actionIndex=0}' (requested by nodes 'File:[[<execution_root>]bazel-out/k8-opt-exec-ST-d57f47055a04/bin]external/io_bazel_rules_go/stdlib_/pkg')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:547)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException: ActionInput not found for filename bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/go_sdk/builder in CacheNotFoundException
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1008)
	at com.google.devtools.build.lib.remote.common.BulkTransferException.getLostInputs(BulkTransferException.java:92)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy$SpawnExecutionContextImpl.lambda$prefetchInputs$0(AbstractSpawnStrategy.java:303)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:207)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:194)
	at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:136)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:516)
	at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:108)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
	at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:560)
	at com.google.common.util.concurrent.CombinedFuture$AsyncCallableInterruptibleTask.setValue(CombinedFuture.java:175)
	at com.google.common.util.concurrent.CombinedFuture$AsyncCallableInterruptibleTask.setValue(CombinedFuture.java:153)
	at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.afterRanInterruptiblySuccess(CombinedFuture.java:129)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:87)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.execute(CombinedFuture.java:108)
	at com.google.common.util.concurrent.CombinedFuture.handleAllCompleted(CombinedFuture.java:65)
	at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:315)
	at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:298)
	at com.google.common.util.concurrent.AggregateFuture.lambda$init$1(AggregateFuture.java:171)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:516)
	at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:54)
	at com.google.devtools.build.lib.remote.util.RxFutures$1.onError(RxFutures.java:221)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableAndThenCompletable$SourceObserver.onError(CompletableAndThenCompletable.java:62)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableFromSingle$CompletableFromSingleObserver.onError(CompletableFromSingle.java:41)
	at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)
	at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)
	at com.google.devtools.build.lib.remote.util.AsyncTaskCache$1.onError(AsyncTaskCache.java:339)
	at com.google.devtools.build.lib.remote.util.AsyncTaskCache$Execution.onError(AsyncTaskCache.java:205)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableUsing$UsingObserver.onError(CompletableUsing.java:165)
	at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek$CompletableObserverImplementation.onError(CompletablePeek.java:95)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.tryOnError(CompletableCreate.java:91)
	at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.onError(CompletableCreate.java:77)
	at com.google.devtools.build.lib.remote.util.RxFutures$OnceCompletableOnSubscribe$1.onFailure(RxFutures.java:102)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1117)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
	at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:560)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:218)
	at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:194)
	at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:146)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:516)
	at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:108)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:516)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptiblyFailure(TrustedListenableFutureTask.java:138)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:89)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No consistent repro

Which operating system are you running Bazel on?

Amazon Linux 2023.6.20250115

What is the output of bazel info release?

release 8.2.0rc3

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

#25232 (comment)
@westrik westrik mentioned this issue Apr 14, 2025
Open
@fmeum fmeum assigned fmeum and unassigned sgowroji, iancha1992 and satyanandak Apr 14, 2025
@fmeum fmeum added P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team and removed untriaged labels Apr 14, 2025
@fmeum
Copy link
Collaborator

fmeum commented Apr 14, 2025

@bazel-io fork 8.2.0

@iancha1992
Copy link
Member

iancha1992 commented Apr 14, 2025 •
edited
Loading

@Wyverald @meteorcloudy @fmeum Do we need 8.2.1?

For reference, see #25232 (comment)

cc: @bazelbuild/triage

@fmeum fmeum mentioned this issue Apr 14, 2025
Closed
@fmeum
Copy link
Collaborator

fmeum commented Apr 14, 2025

@iancha1992 This is a regression that results in a crash when certain files are evicted by the remote cache. I would lean towards fixing this in a patch release.

@iancha1992
Copy link
Member

@bazel-io fork 8.2.1

@bazel-io bazel-io mentioned this issue Apr 14, 2025
Closed
@meteorcloudy
Copy link
Member

Do we know which change caused this issue?

@coeuvre
Copy link
Member

coeuvre commented Apr 15, 2025

Probably #25545.

visz11 added a commit to coderabbit-test/bazel that referenced this issue Apr 15, 2025
@visz11
Update AbstractActionInputPrefetcher.java
48fa3df 
The downloads in AbstractActionInputPrefetcher#prefetchInputs reported lost inputs that are symlinks to other artifacts with the exec path of the target, which results in a crash when the target is not also an input to the action. This is fixed by always reporting the exec path of the symlink.

Fixes bazelbuild#25841
@visz11 visz11 mentioned this issue Apr 15, 2025
Closed
@visz11 visz11 mentioned this issue Apr 15, 2025
Open
fmeum added a commit to fmeum/bazel that referenced this issue Apr 15, 2025
@fmeum
Fix crash on lost inputs that are symlinks
5bcbfb3 
The downloads in `AbstractActionInputPrefetcher#prefetchInputs` reported lost inputs that are symlinks to other artifacts with the exec path of the target, which results in a crash when the target is not also an input to the action. This is fixed by always reporting the exec path of the symlink.

Fixes bazelbuild#25841

Closes bazelbuild#25847.

PiperOrigin-RevId: 747819887
Change-Id: Iff9125c95a2598851b8a31ed4ca18fd7218a97aa
(cherry picked from commit a7da0d8)
@fmeum fmeum mentioned this issue Apr 15, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Apr 15, 2025
@fmeum
[8.2.1] Fix crash on lost inputs that are symlinks (#25856)
74da5dd 
The downloads in `AbstractActionInputPrefetcher#prefetchInputs` reported
lost inputs that are symlinks to other artifacts with the exec path of
the target, which results in a crash when the target is not also an
input to the action. This is fixed by always reporting the exec path of
the symlink.

Fixes #25841

Closes #25847.

PiperOrigin-RevId: 747819887
Change-Id: Iff9125c95a2598851b8a31ed4ca18fd7218a97aa 
(cherry picked from commit a7da0d8)

Closes #25849
@iancha1992
Copy link
Member

@bazel-io fork 8.3.0

@bazel-io bazel-io mentioned this issue Apr 15, 2025
Closed
@iancha1992
Copy link
Member

A fix for this issue has been included in Bazel 8.2.1 RC1. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=8.2.1rc1. Thanks!

iancha1992 pushed a commit to iancha1992/bazel that referenced this issue Apr 16, 2025
@fmeum @iancha1992
Fix crash on lost inputs that are symlinks
5d5f9b0 
The downloads in `AbstractActionInputPrefetcher#prefetchInputs` reported lost inputs that are symlinks to other artifacts with the exec path of the target, which results in a crash when the target is not also an input to the action. This is fixed by always reporting the exec path of the symlink.

Fixes bazelbuild#25841

Closes bazelbuild#25847.

PiperOrigin-RevId: 747819887
Change-Id: Iff9125c95a2598851b8a31ed4ca18fd7218a97aa
iancha1992 pushed a commit to iancha1992/bazel that referenced this issue Apr 16, 2025
@fmeum @iancha1992
Fix crash on lost inputs that are symlinks
36fa2a5 
The downloads in `AbstractActionInputPrefetcher#prefetchInputs` reported lost inputs that are symlinks to other artifacts with the exec path of the target, which results in a crash when the target is not also an input to the action. This is fixed by always reporting the exec path of the symlink.

Fixes bazelbuild#25841

Closes bazelbuild#25847.

PiperOrigin-RevId: 747819887
Change-Id: Iff9125c95a2598851b8a31ed4ca18fd7218a97aa
iancha1992 pushed a commit to iancha1992/bazel that referenced this issue Apr 16, 2025
@fmeum @iancha1992
Fix crash on lost inputs that are symlinks
7f91468 
The downloads in `AbstractActionInputPrefetcher#prefetchInputs` reported lost inputs that are symlinks to other artifacts with the exec path of the target, which results in a crash when the target is not also an input to the action. This is fixed by always reporting the exec path of the symlink.

Fixes bazelbuild#25841

Closes bazelbuild#25847.

PiperOrigin-RevId: 747819887
Change-Id: Iff9125c95a2598851b8a31ed4ca18fd7218a97aa
@iancha1992 iancha1992 mentioned this issue Apr 16, 2025
Closed
iancha1992 pushed a commit to iancha1992/bazel that referenced this issue Apr 18, 2025
@fmeum @iancha1992
[8.2.1] Fix crash on lost inputs that are symlinks (bazelbuild#25856)
0f05316 
The downloads in `AbstractActionInputPrefetcher#prefetchInputs` reported
lost inputs that are symlinks to other artifacts with the exec path of
the target, which results in a crash when the target is not also an
input to the action. This is fixed by always reporting the exec path of
the symlink.

Fixes bazelbuild#25841

Closes bazelbuild#25847.

PiperOrigin-RevId: 747819887
Change-Id: Iff9125c95a2598851b8a31ed4ca18fd7218a97aa 
(cherry picked from commit a7da0d8)

Closes bazelbuild#25849
@iancha1992 iancha1992 mentioned this issue Apr 18, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Apr 22, 2025
@iancha1992 @fmeum
[8.3.0] Fix crash on lost inputs that are symlinks (#25856) (#25890)
f265f40 
The downloads in `AbstractActionInputPrefetcher#prefetchInputs` reported
lost inputs that are symlinks to other artifacts with the exec path of
the target, which results in a crash when the target is not also an
input to the action. This is fixed by always reporting the exec path of
the symlink.

Fixes #25841

Closes #25847.

PiperOrigin-RevId: 747819887
Change-Id: Iff9125c95a2598851b8a31ed4ca18fd7218a97aa (cherry picked from
commit a7da0d8)

Closes #25849

Commit
74da5dd

Co-authored-by: Fabian Meumertzheim <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fmeum fmeum

Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Milestone
No milestone
7 participants
@westrik @meteorcloudy @fmeum @coeuvre @sgowroji @iancha1992 @satyanandak