Skip to content

Add continuous profiling and update telemetry for Illuminate - #4807

Open
johnoliver wants to merge 11 commits into
microsoft:mainfrom
johnoliver:continuous-profiling
Open

Add continuous profiling and update telemetry for Illuminate#4807
johnoliver wants to merge 11 commits into
microsoft:mainfrom
johnoliver:continuous-profiling

Conversation

@johnoliver

@johnoliver johnoliver commented Jul 31, 2026

Copy link
Copy Markdown
Member

…quirements
This pull request introduces several important changes related to the diagnostics and profiling system, primarily focused on removing the calibration logic and context switching measurement, and improving support for continuous diagnostics. The changes simplify the codebase by removing unused calibration components, refactoring machine stats to MachineInfo, and adding mechanisms for continuous diagnostics collection.

Diagnostics and Profiling Improvements:

  • Added startContinuousDiagnostics and stopContinuousDiagnostics methods to the DiagnosticEngine interface and implemented them in CodeOptimizerDiagnosticEngineJfr, enabling continuous diagnostics collection for use with continuous profiling. [1] [2]
  • Refactored CodeOptimizerDiagnosticEngineJfr to use AtomicInteger and AtomicBoolean for thread-safe management of process ID and continuous mode state. [1] [2]
  • Updated diagnostics emission logic to use MachineInfo instead of the removed calibration-based MachineStats; emission now uses emitMachineInfo() and omits context switching rate. [1] [2]

Removal of Calibration and Context Switching Logic:

  • Deleted the calibration and context switching measurement subsystem, including Calibration, Calibrator, CalibratorDefault, and ContextSwitchingRunner classes, as well as all related usages. [1] [2] [3] [4] [5] [6] [7]
  • Removed legacy calibration field population in MachineInfo; the field remains for backward compatibility but is always zero in new recordings. [1] [2] [3] [4]

Refactoring and Renaming:

  • Renamed MachineStats to MachineInfo throughout the codebase to better reflect its purpose and removed calibration-related logic. [1] [2] [3] [4] [5] [6] [7]

These changes streamline the diagnostics system by removing unused complexity and improving support for continuous profiling scenarios.

Fix # .

For significant contributions please make sure you have completed the following items:

  • Design discussion issue #
  • Changes in public surface reviewed
  • CHANGELOG.md updated

johnoliver and others added 9 commits July 31, 2026 19:40
@johnoliver
add MachineStats schema version
a1d4f01
@johnoliver
Add missing applicationinsights-continuous.json smoke test resource
8acde24 
The DiagnosticsTest ContinuousProfilingJava11Test references
applicationinsights-continuous.json via @useagent, but the resource was
never committed. On CI the file is absent, so the continuous profiling
config fails to load and all three DiagnosticsTest cases fail (the missing
config also leaves the shared test setup in a state that breaks the
subsequently-run Java11Test.getJfr).

Committing the resource fixes the Diagnostics:DiagnosticsTest build failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@johnoliver
Honor configured profile duration for continuous profiling dumps
44b0a92 
Previously a profile request against a continuous (circular-buffer)
recording always dumped the full local continuousProfilingMaxAge window,
silently ignoring the portal-/JMX-configured profile duration carried on
the alert breach.

captureContinuousRecording now resolves the capture window as
min(requestedDuration, maxAge): when the requested duration is shorter
than the buffer it streams out only the trailing requested window via
Recording.getStream(start, end); when it covers (or exceeds) the buffer it
keeps using the more robust dump() path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@johnoliver
Fix startup ordering race that tears down continuous diagnostic emitters
ff50bee 
A breach processed during startup, before startContinuousDiagnostics has
run, would schedule a diagnostic-cycle shutdown. Once continuous profiling
started and registered the continuous emitters, that stale scheduled
shutdown fired and permanently tore them down.

Guard the scheduled teardown under a lifecycle lock that checks whether
continuous diagnostics has since been enabled, and synchronize the
start/stop transitions so the check-and-stop cannot interleave with a
concurrent startContinuousDiagnostics. Refactor the JFR-touching helpers
into overridable instance methods so the race can be covered by a
deterministic unit test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@johnoliver
Restore null guard for NoOpProcessDumper in getThisProcess
a2493a8 
NoOpProcessDumper.thisProcess() returns null on operating systems that do
not support diagnostics. The continuous profiling refactor dropped the
guard around the process dumper result, so getThisProcess() would NPE on
thisProcess.getPid(). Restore a guard that returns null when no process is
available, matching the previous defensive behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@johnoliver
Fix continuous profiling timestamp, concurrency and config robustness
60e7e04 
Addresses several review findings on the continuous profiling dump path:

- Timestamp the uploaded profile (and encode the file name) at the start of
  the captured window (now - captureWindow) instead of at dump time, so the
  profile lands at the correct point on the portal timeline (B4).
- Keep the dumped file in a local variable and advance the global cooldown
  inside the recording lock, so concurrent triggers cannot publish or delete
  each other's files via the shared activeRecordingFile field (B7).
- Use a dedicated RecordingOptions.Builder for the continuous recording so
  the maxAge/disk options no longer permanently mutate the builder shared
  with on-demand recordings (S4).
- Clamp a non-positive continuousProfilingMaxAgeSeconds to the default
  rather than passing '0 ms'/'-1000 ms' straight to JFR (C4).
- Reuse the CPU recording configuration for the continuous recording instead
  of opening a second stream on the same .jfc resource (B2, partial).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@johnoliver
Address continuous profiling review follow-ups (logging, docs, test app)
634d365 
- Log the failure when emitting continuous diagnostic breach information,
  since the returned future is discarded by the caller and the error would
  otherwise be swallowed (S3).
- Drop the redundant inner init() guard in SystemStatsProvider that keyed
  off a singleton it populates itself; init() is already single-shot via the
  initialised flag (R5).
- Close the Files.walk stream via try-with-resources and select the most
  recently modified .jfr file in the Diagnostics smoke-test controller, so
  continuous profiling's extra dumps cannot cause a stale-file false green;
  extract the shared polling loop (S5).
- Document enableContinuousProfiling / continuousProfilingMaxAgeSeconds in
  docs/README.md, including their limitations, and add CHANGELOG entries for
  the feature and the MachineStats -> MachineInfo event rename (D1, D2).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@johnoliver
Fix continuous profiling dump and harden Diagnostics smoke test
43e24c5 
The continuous circular buffer is always in the RECORDING state, but the
attempt to honor a shorter requested profile duration streamed a sub-window
via Recording.getStream(start, end), which the JFR connection only permits on
a STOPPED recording. Every manual/portal/JMX trigger therefore threw
IllegalStateException ("Recording state RECORDING not in [STOPPED]"), so no
continuous profile was ever produced and the DiagnosticsTest smoke tests
failed. A live recording can only be dumped in its entirety, so always use
Recording.dump() and drop the infeasible sub-window streaming path. The
uploaded profile is still timestamped at the start of the captured window
(recordingEnd - maxAge). Update the README limitation note accordingly.

Also harden the Diagnostics smoke test controller: continuous profiling can
produce several dumps and a dump may be observed mid-write, so iterate every
candidate .jfr file (newest first) and skip any that cannot be decompressed
yet instead of letting an EOFException fail the whole request.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@johnoliver
Spotless
d06e8b2
@johnoliver
johnoliver requested a review from Copilot August 3, 2026 16:17
Copilot AI reviewed Aug 3, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request updates the Java agent’s profiling/diagnostics pipeline to support continuous profiling (always-on circular-buffer JFR recording) and adjusts diagnostics emission to ensure required diagnostic events are present in continuous snapshots. It also removes the legacy calibration/context-switching subsystem and renames the machine diagnostic event from MachineStats to MachineInfo.

Changes:

  • Add continuous profiling configuration and implementation (circular-buffer JFR recording dumped on demand), plus docs/changelog updates.
  • Add “continuous diagnostics” lifecycle hooks and update diagnostics emission to support continuous profiling snapshots.
  • Remove calibration/context-switching measurement and refactor MachineStats → MachineInfo across code and JFC templates.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
smoke-tests/apps/Diagnostics/src/smokeTest/resources/applicationinsights-continuous.json Adds smoke-test config enabling continuous profiling + diagnostics.
smoke-tests/apps/Diagnostics/src/smokeTest/java/com/microsoft/applicationinsights/smoketest/DiagnosticsTest.java Adds smoke test validating continuous recordings include required diagnostics.
smoke-tests/apps/Diagnostics/src/main/java/com/microsoft/applicationinsights/smoketestapp/TestController.java Adds endpoint + polling logic to validate diagnostic events exist in continuous dumps.
smoke-tests/apps/Diagnostics/JfrFileReader/src/main/java/com/microsoft/applicationinsights/jfrfile/JfrFileReader.java Adds helper to confirm at least one instance of an event type exists in a recording.
docs/README.md Documents continuous profiling settings and limitations.
CHANGELOG.md Notes continuous profiling enhancement and MachineStats → MachineInfo breaking change.
agent/agent-tooling/src/test/java/com/microsoft/applicationinsights/agent/internal/profiler/ProfilerContinuousProfilingTest.java Adds unit tests for continuous profiling dump/upload behavior.
agent/agent-tooling/src/main/resources/com/microsoft/applicationinsights/agent/internal/profiler/reduced-memory-profile.jfc Renames enabled diagnostics event to MachineInfo.
agent/agent-tooling/src/main/resources/com/microsoft/applicationinsights/agent/internal/profiler/reduced-cpu-profile.jfc Renames enabled diagnostics event to MachineInfo.
agent/agent-tooling/src/main/resources/com/microsoft/applicationinsights/agent/internal/profiler/diagnostic-memory-profile.jfc Renames enabled diagnostics event to MachineInfo.
agent/agent-tooling/src/main/resources/com/microsoft/applicationinsights/agent/internal/profiler/diagnostic-cpu-profile.jfc Renames enabled diagnostics event to MachineInfo.
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/profiler/triggers/AlertingSubsystemInit.java Adjusts when diagnostics are emitted relative to profiling start for continuous vs traditional mode.
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/profiler/Profiler.java Implements continuous circular-buffer recording creation + snapshot dump/upload path.
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/profiler/PerformanceMonitoringService.java Starts continuous diagnostics when continuous profiling is enabled.
agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/configuration/Configuration.java Adds continuous profiling configuration fields.
agent/agent-profiler/agent-diagnostics/src/test/java/com/microsoft/applicationinsights/diagnostics/appinsights/CodeOptimizerDiagnosticEngineJfrTest.java Adds lifecycle tests for continuous diagnostics vs per-breach cycles.
agent/agent-profiler/agent-diagnostics/src/main/java/com/microsoft/applicationinsights/diagnostics/jfr/SystemStatsProvider.java Removes calibration usage and refactors machine stats to MachineInfo.
agent/agent-profiler/agent-diagnostics/src/main/java/com/microsoft/applicationinsights/diagnostics/collection/calibration/ContextSwitchingRunner.java Removes legacy calibration/context-switching runner.
agent/agent-profiler/agent-diagnostics/src/main/java/com/microsoft/applicationinsights/diagnostics/collection/calibration/CalibratorDefault.java Removes legacy calibrator implementation.
agent/agent-profiler/agent-diagnostics/src/main/java/com/microsoft/applicationinsights/diagnostics/collection/calibration/Calibrator.java Removes legacy calibrator interface.
agent/agent-profiler/agent-diagnostics/src/main/java/com/microsoft/applicationinsights/diagnostics/collection/calibration/Calibration.java Removes legacy calibration model.
agent/agent-profiler/agent-diagnostics/src/main/java/com/microsoft/applicationinsights/diagnostics/appinsights/CodeOptimizerDiagnosticEngineJfr.java Adds continuous diagnostics lifecycle + thread-safety refinements; switches to MachineInfo.
agent/agent-profiler/agent-diagnostics/build.gradle.kts Adds test dependencies for new diagnostics tests.
agent/agent-profiler/agent-diagnostics-jfr/src/main/java/com/microsoft/applicationinsights/diagnostics/jfr/MachineInfo.java Renames event + introduces schema versioning for payload evolution.
agent/agent-profiler/agent-diagnostics-api/src/main/java/com/microsoft/applicationinsights/diagnostics/DiagnosticEngine.java Adds default startContinuousDiagnostics/stopContinuousDiagnostics hooks.

Comment thread ...ics/src/main/java/com/microsoft/applicationinsights/diagnostics/jfr/SystemStatsProvider.java
Comment thread ...ics/src/main/java/com/microsoft/applicationinsights/diagnostics/jfr/SystemStatsProvider.java
Comment thread ...Diagnostics/src/main/java/com/microsoft/applicationinsights/smoketestapp/TestController.java
@johnoliver
johnoliver requested a review from Copilot August 3, 2026 16:54
Copilot AI reviewed Aug 3, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

Suppressed comments (2)

agent/agent-profiler/agent-diagnostics-jfr/src/main/java/com/microsoft/applicationinsights/diagnostics/jfr/MachineInfo.java:47

  • MachineInfo Javadoc says recordings produced before schemaVersion existed should be treated as schemaVersion 1 (implicit legacy version), but schemaVersion currently defaults to 0 when the field is absent during deserialization. This makes the in-memory representation disagree with the documented schema semantics for older recordings.
    agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/profiler/Profiler.java:122
  • continuousProfilingMaxAgeSeconds is validated (and can log a warning) even when enableContinuousProfiling is false. Since continuousProfilingMaxAge is unused unless continuous profiling is enabled, this can create noisy startup logs for configurations that set continuousProfilingMaxAgeSeconds <= 0 but keep the feature disabled.
    continuousProfilingEnabled = config.enableContinuousProfiling;
    continuousProfilingMaxAge =
        resolveContinuousProfilingMaxAge(config.continuousProfilingMaxAgeSeconds);
    // Continuous profiling uses a single always-on recording, so it can only carry one JFC. Reuse
    // the CPU configuration rather than opening a second stream on the same resource.
    continuousRecordingConfiguration = cpuRecordingConfiguration;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@harsimar harsimar Awaiting requested review from harsimar harsimar is a code owner
@mattsains-msft mattsains-msft Awaiting requested review from mattsains-msft mattsains-msft is a code owner
@rajkumar-rangaraj rajkumar-rangaraj Awaiting requested review from rajkumar-rangaraj rajkumar-rangaraj is a code owner
@ramthi ramthi Awaiting requested review from ramthi ramthi is a code owner
@trask trask Awaiting requested review from trask trask is a code owner
@xiang17 xiang17 Awaiting requested review from xiang17 xiang17 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@johnoliver