[WIP - DO NOT REVIEW] Add DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED for Kafka consumers#11993
Draft
bm1549 wants to merge 2 commits into
Draft
[WIP - DO NOT REVIEW] Add DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED for Kafka consumers#11993bm1549 wants to merge 2 commits into
bm1549 wants to merge 2 commits into
Conversation
When a Kafka consumer buffers records inside the per-record iterator loop and does its real work (for example a JDBC write) after the loop, that work runs with no active consume scope, so its spans start a new root trace disconnected from the kafka.consume span. This adds an opt-in flag, DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED (default false), that keeps the last record's kafka.consume span active past the end of the poll loop so same-thread post-loop work nests under it. The deferred span is finished on the next poll(), close(), or unsubscribe(); if none of those arrive, the existing root-iteration-scope keep-alive finishes it as a backstop. Committing offsets is deliberately not a close trigger. The behavior is engaged only under the legacy context manager, whose ITERATION scopes have native cross-thread-safe cleanup; it is a no-op under the new context-swap manager. Cleanup is owner-aware: each consumer records a handle to the span it deferred on its KafkaConsumerInfo, so the exact span is finished even when records were iterated on a different thread than the one that triggers the close. Implemented for both kafka-clients-0.11 and kafka-clients-3.8, with flag-on forked test variants (legacy and context-swap modes) covering post-loop reparenting, the per-trigger close semantics, the keep-alive backstop, and cross-thread cleanup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bm1549
added
type: feature
This comment has been minimized.
This comment has been minimized.
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Kafka / producer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
Kafka / consumer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
bm1549
changed the title
DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED previously deferred the last record's consume span only under the legacy context manager and was a silent no-op under the new context-swap manager. Extend it to both managers: shouldDeferConsumerScope() no longer requires the legacy manager, and closeLingeringConsumeScope() restores the caller's context in a manager-aware way -- popping the ITERATION scope under the legacy manager, or doing Context.root().swap() + finish under the context-swap manager (the same swap-back the iterator already performs at end-of-iteration). The per-consumer owner-aware span handle finishes the exact deferred span across threads under both managers. Under the context-swap manager there is no iteration-keep-alive timer backstop; the deferred span is finished on the next poll(), close(), or unsubscribe(). Covered for both kafka-clients-0.11 and kafka-clients-3.8 by flipping the flag-on context-swap forked test variants to assert deferral, with the keep-alive-only leak-safety case gated to the legacy manager. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Adds an opt-in configuration flag,
DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED(system propertydd.trace.kafka.create-consumer-scope.enabled, defaultfalse), that keeps the last consumed record'skafka.consumespan active past the end of the Kafka poll loop. With the flag on, work a consumer does after its per-record iterator loop (a common "buffer records in the loop, write to the DB after" pattern) nests under the consume span instead of starting a disconnected root trace.The deferred span is finished by the next
poll(),close(), orunsubscribe(). Committing offsets is deliberately not a close trigger. Cleanup is owner-aware: each consumer records the exact span it deferred on itsKafkaConsumerInfo, so the span is finished correctly even when records are iterated on a different thread than the one that closes the consumer.Works under both context managers:
Contextand finished on the next poll/close/unsubscribe. There is no keep-alive timer backstop here, so a consumer that stops polling without closing can leave the span lingering, and later same-thread work would nest under it until the next trigger. This is a best-effort, opt-in tradeoff under the non-default manager.Implemented for both
kafka-clients-0.11andkafka-clients-3.8.Motivation
When a consumer buffers records inside the iterator loop and does its real work after the loop, that work runs with no active consume scope. Downstream spans (for example JDBC) are parented to nothing and recorded as separate root traces rather than as children of the
kafka.consumespan, breaking the connected consume to process to downstream trace.Additional Notes
Contributor Checklist
type:andinst:labels assignedclose/fix/linking keywords used🤖 Generated with Claude Code