Add option to let activities heartbeat during worker shutdown

[baekgyu-kim]

What was changed

Added an experimental worker option, WorkerOptions.Builder#setAllowActivityHeartbeatDuringShutdown(boolean) (default false, preserving the existing behavior).

• When enabled, on a graceful worker shutdown the activity heartbeat executor is shut down only after all outstanding activity tasks have finished executing, so in-flight activities keep heartbeating through the existing mechanism — no separate heartbeat code path.
• When disabled (default), or whenever shutdownNow is used, the behavior is unchanged: the heartbeat executor is shut down first, so ActivityExecutionContext.heartbeat() throws ActivityWorkerShutdownException.
• Gated by allowActivityHeartbeatDuringShutdown && !interruptTasks, so non-graceful shutdown (WorkerFactory.shutdownNow) always behaves as if the option were disabled.

Touched: WorkerOptions, SingleWorkerOptions, Worker, SyncActivityWorker.

Why?

• Before: once a graceful shutdown was requested, heartbeat() threw ActivityWorkerShutdownException for the rest of the awaitTermination grace period. An activity that wanted to run to completion during that window could not refresh its server-side heartbeat deadline → the server timed it out and retried it → duplicate executions, even though the worker deliberately gave the activity time to finish.
• After (opt-in): with the option enabled, in-flight activities keep heartbeating during the grace period, so each call refreshes the heartbeat deadline and the server does not prematurely time out / retry the activity.
• Design: reuses the existing heartbeat path by only reordering executor shutdown — no new heartbeat logic. The default stays false, so existing behavior is preserved, and shutdownNow is intentionally excluded. Originates from Add the ability to keep heartbeating while the worker is shutting down #2075.

Note: with the option enabled, activities are no longer notified of shutdown via ActivityWorkerShutdownException, so they are expected to complete within the termination grace period on their own. This is documented on the setter.

Checklist

1. Closes Add the ability to keep heartbeating while the worker is shutting down #2075

2. How was this tested:

   New HeartbeatDuringWorkerShutdownTest (standalone activity + two-semaphore handshake; gated behind SDKTestWorkflowRule.useExternalService):

   • testHeartbeatingActivityCompletesDuringShutdown — option enabled + graceful shutdown(): the activity heartbeats and runs to completion.
   • testHeartbeatingActivityFailsDuringShutdownNow — shutdownNow(): the option is ignored, and the heartbeat fails the activity.

   WorkerOptionsTest — copy-builder round-trips the new option.

   # the integration tests require an external server
   USE_EXTERNAL_SERVICE=true TEMPORAL_SERVICE_ADDRESS=localhost:7233 \
     ./gradlew :temporal-sdk:test --tests "io.temporal.worker.shutdown.HeartbeatDuringWorkerShutdownTest"
   ./gradlew :temporal-sdk:test --tests "io.temporal.worker.WorkerOptionsTest"
   ./gradlew :temporal-sdk:spotlessJavaCheck
   

3. Any docs updates needed?

   • The behavior is documented in the WorkerOptions.Builder#setAllowActivityHeartbeatDuringShutdown Javadoc.

[maciejdudko]

Hi @baekgyu-kim, thank you for your contribution! It's great to see someone taking on these long standing issues. However, this is not the right implementation.

There should be a new worker option to enable heartbeating during shutdown, It should default to disabled, and when disabled, the behavior should be identical to existing behavior for backward compatibility purposes.

When the option is enabled, the heartbeat behavior should be identical to what happens during normal heartbeat when the worker is not shutting down. There should be no additional code path that calls sendHeartbeatRequest a different way, the existing mechanism should be used. The way to achieve that is to modify SyncActivityWorker.shutdown so that heartbeatExecutor.shutdown is only called after all outstanding activity tasks have finished executing.

If you need assistance with implementation, feel free to reach out on community Slack, either message me directly or post on #java-sdk channel.

[baekgyu-kim]

Hi @maciejdudko,
Thank you for the thoughtful review! I've reworked the PR as suggested.

It now adds an experimental WorkerOptions.Builder#setActivityHeartbeatDuringShutdown option. (default false, preserving the existing behavior)
When enabled, the heartbeat executor is shut down only after all outstanding activity tasks have finished executing, so heartbeats go through the existing mechanism without any separate code path.

Whenever you have a chance, I'd appreciate another look. Thanks again!

[maciejdudko]

shutdownNow behavior stays the same, see comment in SyncActivityWorker.

Suggested change

	* io.temporal.client.ActivityWorkerShutdownException}, unless {@link
	* WorkerOptions.Builder#setActivityHeartbeatDuringShutdown(boolean)} is enabled, in which case
	* heartbeats keep working until the activity tasks finish executing.<br>
	* io.temporal.client.ActivityWorkerShutdownException}.<br>

[baekgyu-kim]

Reverted. The WorkerFactory.shutdown Javadoc no longer references the option, so shutdownNow keeps its original documented behavior.

[baekgyu-kim]

Hi @maciejdudko,
Thanks again for your review.
Sorry for making you review this twice, and I really appreciate the detailed feedback.

I've addressed all of your comments:

• Naming: Renamed consistently to allowActivityHeartbeatDuringShutdown (field, getAllowActivityHeartbeatDuringShutdown() getter, setAllowActivityHeartbeatDuringShutdown(...) setter) across WorkerOptions, SingleWorkerOptions, and Worker.
• shutdownNow behavior: Changed the condition to allowActivityHeartbeatDuringShutdown && !interruptTasks, so shutdownNow behaves exactly as if the option were disabled (heartbeat executor shut down first).
• WorkerFactory.shutdown javadoc: Reverted. It no longer references the option, so both shutdown and shutdownNow keep their original documented behavior.
• WorkerOptions setter javadoc: Replaced with your suggested wording, dropping the implementation details.
• Test race condition: Reworked into a standalone activity with the two-semaphore handshake you outlined (wait for the activity to start → trigger shutdown → release the activity to heartbeat and return), eliminating the race where shutdown could land before the task is picked up.
• shutdownNow test case: Added testHeartbeatingActivityFailsDuringShutdownNow, which asserts the heartbeat fails the activity instead of letting it complete.

Thanks again for the careful review!

[maciejdudko]

Almost there! Just one small change to a test remaining. Also, I see you removed the doc change for WorkerFactory.shutdown. I liked that one; I believe we should mention that shutdown behavior is different when allowActivityHeartbeatDuringShutdown is true (unlike shutdownNow which is the same both ways).

[maciejdudko]

Also assert that cause is ActivityWorkerShutdownException.

[baekgyu-kim]

• Restored the WorkerFactory.shutdown javadoc note that heartbeats keep working until the activity tasks finish when allowActivityHeartbeatDuringShutdown is enabled.
• testHeartbeatingActivityFailsDuringShutdownNow now also asserts the cause by its type.

[baekgyu-kim]

Hi @maciejdudko,
Thank you for the thorough reviews, and sorry for the extra rounds.

Both points are addressed:

• WorkerFactory.shutdown javadoc: Restored the note about heartbeats continuing when allowActivityHeartbeatDuringShutdown is enabled.
• Test: testHeartbeatingActivityFailsDuringShutdownNow now asserts the cause, checking the failure type matches ActivityWorkerShutdownException.

Thanks again for the careful reviews. Have a great day.