diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 90cf564b3..45c0a75a0 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -175,6 +175,7 @@ to the symptom table below, so the next similar issue costs fewer reads. | Design properties are written free-form: a `ColorPicker`/`ToggleButtonGroup` value serializes as a plain option (wrong `$Type` for Studio Pro's Appearance tab), and a typo'd key/value (they're case-sensitive) passes `mxcli check`. Also `show design properties ` reports "No design properties found for widget type container" for a valid widget | Root bug: `resolveDesignPropsKey` upper-cased the MDL keyword but the lookup map is **lowercase-keyed**, so `container`→`DivContainer` never resolved — leaving `resolveDesignPropertyValueType` dead code and the theme registry unused on the write/validate paths | `mdl/executor/theme_reader.go` (`resolveDesignPropsKey` case fix) + `mdl/executor/cmd_pages_builder_v3.go` (`astDesignPropToValue` takes theme props) + `mdl/executor/validate_design_properties.go` (new, MDL-WIDGET11/12) wired from `cmd/mxcli/cmd_check.go` + `cmd/mxcli/lsp_diagnostics.go` (cached `themeRegistry`) | Fix `resolveDesignPropsKey` to lower-case the lookup. On write, resolve each flat value's type from the registry **by matching the value against the property's declared options** (see the CE6084 correction below — the control type alone does NOT decide it). On check (`-p` only, when themesource defines properties), walk page/snippet/alter-page widget trees and warn: **MDL-WIDGET11** unknown key (case-sensitivity hint / valid-key list), **MDL-WIDGET12** invalid value (lists allowed values). Warnings, not errors — a newer theme may add keys/values (forward-compat, per `page-styling-support.md:402`). Skip compound (registry doesn't model sub-props) and widgets with no type-specific metadata (pluggable). Bug-test `mdl-examples/bug-tests/typed-design-properties.mdl` | | Follow-up regression from the row above: after typed design properties merged, `mx check` fails **CE6084** "Expected design property _Flex container_ / _Column gap_ / _Align items Y_ … to be of type **Toggle button group**, but found **Custom**" on any page using a flat `ToggleButtonGroup` value (Atlas flex/spacing/typography, e.g. `'Column gap': 'Medium'`). Broke `TestMxCheck_DoctypeScripts` on `12-styling`, `15c-fragment-bindings`, `31-pluggable-datagrid-gallery-v010` (both engines) — green on unit tests, red only in `make test-integration` | `resolveDesignPropertyValueType` mapped `ToggleButtonGroup`→`custom` by control type. But a ToggleButtonGroup selection picks one of a **fixed option set**, so Studio Pro stores it as an **Option** — a `Custom` value type mismatches the declaration. Only a ColorPicker's **off-list** value (a free-form hex) is genuinely Custom. The value type is decided by the **value**, not the control | `mdl/executor/cmd_pages_builder_v3.go` (`resolveDesignPropertyValueType`, now takes the value and reuses `themeOptionAllowed`) | Make it value-aware: value ∈ declared options → `option` (Dropdown, ToggleButtonGroup, predefined ColorPicker swatch alike); off-list **and** `ColorPicker` → `custom`; else `option`; no metadata → `option`. Verified: the three doctype examples pass `mx check` = 0 errors on both engines. Test `TestAstDesignPropToValue_Typed` extended with the `Column gap: Medium` + ColorPicker swatch/hex cases. **Diagnosis pattern**: a value-type/BSON-`$Type` mapping keyed on a *declared control type* is a trap — verify it against `mx check`, never assert it from the type name alone (this is exactly how the original bug slipped in). **Process lesson**: this shipped red because `make test-integration` (mx-check doctype roundtrips) was not run before merge — run it, not just unit tests, for any page/widget-serialization change | | The **nightly** matrix (Mendix 10.24 / 11.6 / 11.12) fails only on **10.24**: `TestMxCheck_DoctypeScripts/15c-fragment-bindings-examples` → `Execution error: failed to build page: building block not found: Atlas_Web_Content.List_Cards` (both engines). 11.6/11.12 pass; unit tests + push-test (single-version) pass. Looks like a "design property on 10.24" issue but isn't | `15c` demonstrates `use building block Atlas_Web_Content.List_Cards`, an Atlas UI building block that ships in **11.x but is absent from the 10.x Atlas** (the example comment wrongly said "present in every standard Mendix app"). The example had **no `-- @version:` gate**, so on a 10.24 project the whole file ran and mxbuild couldn't resolve the block. (12-styling gates its design-property section at line 186; 31 gates the whole file at line 1 — both already skip on 10.24) | `mdl-examples/doctype-tests/15c-fragment-bindings-examples.mdl` | Add `-- @version: 11.0+` immediately before the `create page … P002_Rebound_Block` block (its last statement) so `filterByVersion` skips only the building-block demo on 10.x; the fragment-binding statements above stay ungated and keep 10.24 coverage. Verified: 15c passes on **10.24** (10 lines skipped, 0 errors) and still runs+passes the section on **11.6.3** (0 errors). **Diagnosis pattern**: a nightly-only, version-specific doctype failure = an example using a construct (building block, widget, syntax) that doesn't exist in the oldest matrix version and lacks a `-- @version:` gate; reproduce locally with `MX_BINARY=~/.mxcli/mxbuild//modeler/mx go test -tags integration -run TestMxCheck_DoctypeScripts/` | +| Same nightly pattern, **CE6083** this time: `TestMxCheck_DoctypeScripts/15b-fragment-slots-examples` fails only on **10.24** (both engines) — `[CE6083] "Design property Card style is not supported by your theme"` at every `cardWrap` container. `Card style` is an **Atlas v3 design property (11.x); the 10.x Atlas theme doesn't define it** | The shared `define fragment Card` used `designproperties: ['Card style': on]` and is instantiated by every page in the file, so a `-- @version:` gate would have to gate the whole file (killing 10.24 coverage of the slot feature the example is actually about). Unlike 15c's building block, the design property was **incidental** to the example | `mdl-examples/doctype-tests/15b-fragment-slots-examples.mdl` | **Drop the incidental v3 design property**, keep `class: 'card'` (Atlas card styling works on every version) — the example demonstrates content *slots*, not design properties (those live in 12-styling, gated 11.0+). Verified: 15b passes on 10.24 **and** 11.6.3 (0 errors, both engines). **Gate vs remove rule**: if the version-specific construct IS the point of a self-contained section → `-- @version:` gate it (15c); if it's incidental and in a shared/expanded definition → remove it and use a cross-version equivalent (15b). **Proactive sweep** after any such fix: `grep -lE 'designproperties|use building block' mdl-examples/doctype-tests/*.mdl` and confirm each usage is either gated or version-safe (note the doctype test skips `*.test.mdl`/`*.tests.mdl`) | | `mxcli run --local`: when a page action throws, the browser shows the generic Mendix error dialog and there is nothing to correlate it against — the runtime's own stdout/stderr (server stack trace, microflow `LOG` output) is swallowed, so a server-side bug can't be told apart from a client one | The runtime JVM was spawned with `cmd.Stdout=log; cmd.Stderr=log` where `log` is an in-memory `syncBuffer` surfaced only on a *startup* failure; during normal operation it goes nowhere on disk | `cmd/mxcli/docker/localboot.go` (`spawnAndConfigure`, `openRuntimeLog`, `LocalRuntime.logFile`, `LocalRuntimeOptions.RuntimeLogPath`) + `cmd/mxcli/docker/runlocal.go` (default `/.mxcli/runtime.log`) + `cmd/mxcli/cmd_run.go` (`--runtime-log`) | Tee the JVM's stdout+stderr to `/.mxcli/runtime.log` via `io.MultiWriter(log, file)` (the in-memory buffer still backs startup-error reporting). Append across restarts with a `=== runtime start … ===` marker; close the handle on Stop/reopen. Default on; `--runtime-log ` relocates, `-` disables. Print the path at boot. Test `TestOpenRuntimeLog`. Findings #25 | | Follow-up to the above (#25 re-test): `run --local` writes `runtime.log` but it stays **nearly empty** — the JVM tee captures startup/JVM output only; **application** logs (microflow `LOG`, server-side exception stack traces) never reach stdout, so a page-action error still can't be diagnosed | A standalone runtime (launched via `runtimelauncher.jar`) attaches **no log subscriber** by default — unlike a Studio Pro / m2ee run, which calls `create_log_subscriber` **after** start. Mendix application logs flow to log *subscribers*, not stdout, so with none attached they go nowhere | `cmd/mxcli/docker/runtime_controller.go` (`RuntimeController.LogSubscriberFile`/`Stdout`, `attachFileLogSubscriber`, called at the end of `Start`) + `cmd/mxcli/docker/localboot.go` (`StartLocalRuntime` sets `ctrl.LogSubscriberFile` to the abs runtime-log path) | After a successful `start` (and on every restart's `Start`, since each fresh JVM has no subscriber), call the `create_log_subscriber` admin action with `{type:"file", name:"mxcli-run-local", autosubscribe:"INFO", filename:, max_size:1GiB, max_rotate:0}`. **`max_rotate:0` is load-bearing**: the JVM stdout tee holds an fd on the same file, and a rotate-rename would detach it. Best-effort (a logging failure must not fail an up runtime — warn to Stdout instead). Pass an **absolute** path (the runtime's cwd is `/runtime`, not mxcli's). Tests `TestStart_AttachesLogSubscriber`, `TestStart_NoLogSubscriberWhenUnset`, `TestStart_LogSubscriberFailureNonFatal`. Findings #25 (round 2) | | Follow-up to round 2 (#25 re-test): the subscriber is registered but `runtime.log` **still** holds only the 4 JVM-banner lines — a probe microflow's `log info/warning/error` and a forced runtime exception produce **zero** lines. Isolation test: boot+subscriber → 0 probe lines; then call `start_logging` → 6 lines | A standalone runtime boots with logging **not started**, so a registered subscriber sits **inert** — nothing is delivered until the logging subsystem is activated. `create_log_subscriber` alone is necessary but not sufficient | `cmd/mxcli/docker/runtime_controller.go` (`configureRuntimeLogging` — renamed from `attachFileLogSubscriber`; now also calls `start_logging`) | After `create_log_subscriber`, call `CallM2EE(c.opts, "start_logging", nil)` in the same step (order: create subscriber → start_logging). Treat an "already started" response as success (`start` re-runs on the DB-update retry / restart paths on a still-running JVM). Still best-effort (warn to Stdout, never fail an up runtime). Test `TestStart_StartLoggingAlreadyStartedIsSuccess`; `TestStart_AttachesLogSubscriber` now asserts the `[start, create_log_subscriber, start_logging]` sequence. **Diagnosis pattern**: when a registered sink receives nothing, check whether the subsystem that feeds it is even *running* — registration ≠ activation. Findings #25 (round 3) | diff --git a/.claude/skills/mendix/debug-microflows.md b/.claude/skills/mendix/debug-microflows.md new file mode 100644 index 000000000..9f286fd13 --- /dev/null +++ b/.claude/skills/mendix/debug-microflows.md @@ -0,0 +1,133 @@ +# Debug Microflows — `mxcli debug` + +## Overview + +`mxcli debug` drives the Mendix runtime's **microflow debugger** from the command +line: set breakpoints **by name**, inspect a paused microflow's variables, and +step/continue — against an app started by `mxcli run --local`. It is the headless +counterpart to Studio Pro's debugger, so you can debug a server-side microflow +without leaving the warm loop. + +mxcli is uniquely able to offer breakpoints **by name** because it owns both +halves: the admin password + app URL (from `run --local`) and the activity model +GUIDs (from the `.mpr`). You never deal with raw GUIDs. + +## When to use this skill + +- A page action throws or misbehaves and you need to see *where* in a microflow it + goes wrong, with the in-scope variables. +- You want to confirm a microflow takes the branch/value you expect. + +For a server **stack trace / `LOG` output** (not stepping), you usually just want +the runtime log — see `run-local.md` (`--runtime-log`). Use the debugger when you +need to **pause and inspect** live execution. + +## Prerequisites + +- The app running under `mxcli run --local` (Mendix 11.x). +- Start it with **`--debug`** so the debugger is enabled and a session is ready: + + ```bash + mxcli run --local -p app.mpr --debug + ``` + + `--debug` alone does **not** change runtime behaviour — nothing pauses until you + set a breakpoint. It caches a debug session token under `/.mxcli/` + so the `mxcli debug` commands below (run from another terminal, **same `-p`**) + work immediately, with no separate `mxcli debug enable`. + +## The loop + +```bash +# terminal 1: app + debugger +mxcli run --local -p app.mpr --debug + +# terminal 2: find the activity, break on it by name +mxcli debug activities Sudoku.ACT_Hint -p app.mpr +mxcli debug break Sudoku.ACT_Hint --activity 'Retrieve' -p app.mpr + +# now trigger the microflow in the browser — the request pauses. Then: +mxcli debug paused -p app.mpr # which flow is paused + its variables +mxcli debug inspect Game -p app.mpr # one variable in detail +mxcli debug step over -p app.mpr # over | into | out +mxcli debug continue -p app.mpr # resume (the browser request completes) + +# when done — ALWAYS: +mxcli debug disable -p app.mpr +``` + +## Commands + +| Command | What it does | +|---------|--------------| +| `mxcli debug status` | Is the debugger on? How many microflows are paused? | +| `mxcli debug enable` / `disable` | Turn the debugger on/off (use `--debug` on `run --local` instead of `enable` for the warm loop) | +| `mxcli debug activities ` | List a microflow's activities with the object IDs you can break on | +| `mxcli debug break --activity <#n\|caption> [--if ]` | Set a breakpoint, resolved by name (`--if` = conditional) | +| `mxcli debug unbreak --activity <#n\|caption>` | Clear a breakpoint | +| `mxcli debug breaks` | List the breakpoints mxcli has set this session (name → object ID) | +| `mxcli debug paused` | Show paused microflows + full state (variables) | +| `mxcli debug inspect [--list] [--flow ]` | Inspect one variable of a paused flow (`--list` for a list variable → `get_list`) | +| `mxcli debug step [over\|into\|out] [--flow ]` | Advance one step (default `over`) | +| `mxcli debug continue [--all]` | Resume the paused flow (or all with `--all`) | + +Selecting an activity: `--activity '#2'` (the index from `activities`) or a +caption substring like `--activity 'Retrieve'` (must match exactly one, case- +insensitive). Selecting a paused flow: `--flow ` (from `paused`); with a +single paused flow it is auto-selected. + +## Nanoflows (client-side) + +`mxcli debug` works for **nanoflows** too — `break`/`activities`/`unbreak` auto-detect +whether `Module.Flow` is a microflow or a nanoflow and set the breakpoint the right +way (a nanoflow needs the `nanoflow_name` param; the wrong key NPEs the runtime — +mxcli handles this for you). Break by name exactly as for a microflow: + +```bash +mxcli debug break Sudoku.NF_ToggleNotes --activity 'Change' -p app.mpr +``` + +A paused **nanoflow** does not appear in `get_paused_microflows` — it surfaces only +in the runtime's `poll_events`. `mxcli debug paused` (and `step`/`inspect`/`continue`) +merge both sources, so a paused nanoflow shows up with its `debug_id` like any other; +its variables are in the "Client events (poll_events)" section of `paused`. + +Symptom of a paused nanoflow **without** mxcli: a frozen browser, the console logging +"Starting execution" but never "Finished", and `mxcli debug status` showing +`client_connected: true`. + +**Nanoflow `debug_id` is single-use.** Unlike a microflow (stable id), a nanoflow +gets a **new** `debug_id` after every step — the old one is invalidated. Because each +`mxcli debug` command re-reads the current state, just let `step`/`inspect`/`continue` +**auto-resolve** the flow (don't pass `--flow`): a bare `mxcli debug step over` picks up +the fresh id each time. Reusing a `--flow ` copied from an earlier `paused` +will fail on the second nanoflow step with "could not find … in debug with id". + +For **nanoflow log output**, see `write-nanoflows.md` — the runtime rewrites the log +node to `Client_Nanoflow`, so grep `runtime.log` for `Client_Nanoflow`, not your node +name. + +## Gotchas + +1. **A breakpoint pauses whoever hits it — the browser included.** The triggering + request hangs until `continue` (or `disable`). This is normal; just don't walk + away from a paused session. +2. **Always finish with `mxcli debug disable`.** `run --local --debug` disables it + for you on shutdown, but if you enabled it by hand, turn it off by hand. +3. **Use the same `-p` everywhere.** The session token and breakpoint record live + under `/.mxcli/`; a different `-p` (or none) looks in a different + place and won't see the session `run --local --debug` started. +4. **Conditions are Mendix expressions** (`--if '$Game/Solved = false'`), same + syntax as a Studio Pro conditional breakpoint. +5. **Overriding the target runtime:** `--app-url`, `--admin-port`, `--admin-pass`, + `--debug-pass` (or `MXCLI_APP_URL` / `MXCLI_ADMIN_PASS` / `MXCLI_DEBUG_PASS`) + default to a `run --local` runtime; set them to debug a differently-configured + or remote runtime. + +## Validation checklist + +- [ ] App started with `mxcli run --local --debug`. +- [ ] `mxcli debug status` shows `enabled`. +- [ ] `mxcli debug activities ` lists the activity you want. +- [ ] After triggering the flow, `mxcli debug paused` shows it with variables. +- [ ] Finished with `mxcli debug disable`. diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index d571b7022..474f119fc 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -117,6 +117,55 @@ Launch `run --local` as the **sole** command in its invocation (don't chain a tr | `--screenshot-path` / `--screenshot-url` | `.mxcli/run-local.png` / app root | Screenshot output / page (URL or `/path`) | | `--screenshot-user` / `--screenshot-password` | — | Log in once, reuse session (pages behind login) | | `--runtime-log` | `.mxcli/runtime.log` | Runtime log file: JVM stdout/stderr **and** the application log (microflow `LOG` output + server stack traces, via an attached file log subscriber). `-` disables. | +| `--debug` | off | Enable the microflow debugger at boot + start a session, so `mxcli debug break/paused/…` works from another terminal (see `debug-microflows.md`). No breakpoints = no behaviour change; disabled on shutdown. | +| `--debug-pass` | `mxdebug` | Debugger password when `--debug` is set | +| `--metrics` | off | Register a Prometheus meter registry at boot; the runtime serves metrics at `http://127.0.0.1:/prometheus` | +| `--trace` | off | Enable OpenTelemetry tracing (bundled agent, console exporter → the runtime log) with default span filters | +| `--trace-service` | `.mpr` name | `OTEL_SERVICE_NAME` under `--trace` | +| `--runtime-setting Key=Value` | — | Merge an extra runtime setting into the boot config (Value parsed as JSON when possible). Repeatable. | + +## Metrics and OpenTelemetry + +**Metrics (`--metrics`):** the Mendix runtime ships Micrometer registries but starts +with none. `--metrics` registers a **Prometheus** registry at boot, so +`http://127.0.0.1:8090/prometheus` (the admin port) serves ~70+ metric families +(`connectionbus_*`, `handler_requests_total`, `sessions_*`, `taskqueue_*`, …). For a +different registry use `--runtime-setting`, e.g. +`--runtime-setting 'Metrics.Registries=[{"type":"otlp","settings":{"step":"PT10S"}}]'` +(also `influx`, `statsd`, `jmx`). + +**Why this is a flag, not a post-boot API call:** the admin `update_configuration` +action **replaces** the whole config (there's no read-back), so a separate call to add +metrics would wipe the DB/BasePath settings. `--metrics`/`--runtime-setting` merge into +mxcli's single boot `update_configuration`, which is the only safe way. + +**Traces (`--trace`):** the runtime bundles the OpenTelemetry Java agent; +`--trace` attaches it to the runtime JVM and applies the default span filters: + +```bash +mxcli run --local -p app.mpr --trace # spans -> runtime.log +mxcli run --local -p app.mpr --trace --runtime-log - # spans to console only +``` + +Spans (tracer `com.mendix.runtime`, attrs `mx.microflow.name`/`mx.microflow.depth`) go +to the console exporter → `runtime.log` (so `tail -f .mxcli/runtime.log` shows them). +`--trace` sets `OTEL_SERVICE_NAME` (default the `.mpr` name, override with +`--trace-service`) and, unless you set them yourself, `OTEL_TRACES_EXPORTER=console` +with metrics/logs exporters off. + +**Why the default span filters matter:** unfiltered per-activity tracing is +**~10× slower**, so `--trace` ships `OpenTelemetry._RuntimeSpanFilters` = +`["CreateOrChangeVariable","Loop","Gateway","RetrieveFromCache"]` by default (keeping +the microflow-level spans). Override with +`--runtime-setting 'OpenTelemetry._RuntimeSpanFilters=[…]'`. + +**Export to a collector (OTLP) instead of the console:** set the OTEL env yourself +before running — `--trace` won't override an exporter you've already set: + +```bash +export OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 +mxcli run --local -p app.mpr --trace +``` | `--app-port` / `--admin-port` / `--serve-port` | 8080 / 8090 / 6543 | Ports | | `--db-host` / `--db-name` / `--db-user` / `--db-password` | 127.0.0.1:5432 / derived / mendix / mendix | Database | diff --git a/.claude/skills/mendix/write-nanoflows.md b/.claude/skills/mendix/write-nanoflows.md index 1d8de4268..6cb756bcd 100644 --- a/.claude/skills/mendix/write-nanoflows.md +++ b/.claude/skills/mendix/write-nanoflows.md @@ -108,6 +108,22 @@ DECLARE $IsValid Boolean = true; SET $IsValid = false; ``` +**Where nanoflow log output goes (and a filtering trap):** a nanoflow runs on the +client, so its `LOG` output takes two paths: + +- **Browser console** — with automatic timing, e.g. + `[Nanoflow] [flow_…] Starting execution of Sudoku.NF_ToggleNotes` … `Finished … 6.7 ms`. +- **Server runtime log** (`.mxcli/runtime.log` under `run --local`) — but the runtime + **rewrites the log node** to `Client_Nanoflow`. So `LOG INFO NODE 'Sudoku' '…'` from a + nanoflow appears as `Client_Nanoflow: …`, **not** `Sudoku: …`. A log filter built + around your microflow node names will silently drop every nanoflow line — grep for + `Client_Nanoflow` (or the message text) to see nanoflow logs. This is a Mendix + platform behaviour, not an mxcli one. +- **`LOG DEBUG` never reaches the server log.** Only `INFO`/`WARNING`/`ERROR` reach + `runtime.log`; a nanoflow `LOG DEBUG` line is sent but dropped server-side. All four + levels still show in the **browser console**, so use the console (not `runtime.log`) + when debugging at `DEBUG` level. + ### Control Flow ```mdl IF $Cart/ItemCount = 0 THEN diff --git a/CLAUDE.md b/CLAUDE.md index efd27ce3b..5d2b53550 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -525,7 +525,9 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) - Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--ensure-db` provisions the local Postgres + app database if missing; `--setup` does the non-blocking prerequisites (cache mxbuild+runtime, ensure DB) and exits — `mxcli init` wires it into a Claude Code SessionStart hook so a fresh/reaped web session self-bootstraps, and `docs-site/src/tools/bootstrap-prompt.md` is the empty-repo seed prompt. `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - External browser preview (`mxcli run --hub ` + `mxcli tunnel-hub`): the app stays local and reverse-tunnels out over a single 443 connection (embedded chisel) to a static relay, so it is reachable in a browser at a public URL — works from egress-only environments (Claude Code web), verified live through the session's MITM egress proxy. `run --hub` implies `--local`, boots the runtime with `ApplicationRootUrl` set to the assigned URL (so the SPA/`originURI` work under the public origin), resolves the control proxy honouring `NO_PROXY`, and retries forever. `mxcli tunnel-hub --domain ` is the **multi-tenant** relay: a registry keyed by prefix/project/solution/branch/worktree (stable URLs on reconnect) fronts many previews at per-subdomain hosts (`[prefix-]project[-branch].`; main collapses to the project) over one 443 with per-subdomain autocert, a registration API (`/api/register|status|deregister|backends`), and a sortable availability overview at `hub./`. Client identity flags: `--hub-prefix`/`--hub-project`/`--hub-solution`/`--hub-branch`/`--hub-worktree` (project + branch auto-detected). Package: `cmd/mxcli/tunnelhub/`. See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` (slices 3–4) +- Runtime metrics + settings passthrough (`mxcli run --local --metrics` / `--runtime-setting Key=Value`): `--metrics` registers a Prometheus Micrometer registry at boot (served at `http://127.0.0.1:/prometheus`); `--runtime-setting` merges arbitrary runtime config (e.g. `Metrics.Registries` for otlp/influx/statsd, or `OpenTelemetry._RuntimeSpanFilters`) into mxcli's **single** boot `update_configuration` call — the admin action replaces rather than merges, so folding settings into the one boot call is the only safe way. OTel traces via `--trace` attach the bundled `opentelemetry-javaagent` to the runtime JVM (console exporter → the tee'd runtime log) and ship default `OpenTelemetry._RuntimeSpanFilters` (unfiltered per-activity tracing is ~10× slower); `--trace-service` sets `OTEL_SERVICE_NAME`, and user-set `OTEL_*` env (e.g. an OTLP collector) is respected. See `.claude/skills/mendix/run-local.md` - OQL query execution against running runtime (`mxcli oql`) +- Microflow/nanoflow debugger (`mxcli debug`): set breakpoints **by name** (activity resolved from the model), inspect paused flows + variables, step over/into/out, continue — against a `run --local` runtime. Two M2EE planes wired behind one command (admin `enable/disable/status`, app `/debugger/` session); `run --local --debug` enables it at boot. **Nanoflows** are auto-detected (uses the `nanoflow_name` breakpoint param; paused nanoflows are merged from `poll_events`, which `get_paused_microflows` omits). Nanoflow `LOG` output is rewritten to the `Client_Nanoflow` node in the runtime log. See `.claude/skills/mendix/debug-microflows.md` and `docs/11-proposals/PROPOSAL_microflow_debugger.md` - Business event services (SHOW/DESCRIBE/CREATE/DROP) - Project settings (SHOW/DESCRIBE/ALTER) - External SQL query execution against PostgreSQL, Oracle, SQL Server (`mxcli sql`, MDL `sql connect/query`) diff --git a/cmd/mxcli/cmd_debug.go b/cmd/mxcli/cmd_debug.go new file mode 100644 index 000000000..e19736d11 --- /dev/null +++ b/cmd/mxcli/cmd_debug.go @@ -0,0 +1,525 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "net/url" + "os" + "path/filepath" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/spf13/cobra" +) + +// cmd_debug.go is slice 1 of the microflow debugger (see +// docs/11-proposals/PROPOSAL_microflow_debugger.md): connect to a runtime's +// debugger and toggle it (status/enable/disable), starting a session on enable. +// Breakpoints, paused-microflow inspection, and stepping are later slices. + +var debugCmd = &cobra.Command{ + Use: "debug", + Short: "Debug microflows against a running Mendix runtime", + Long: `Drive the Mendix runtime microflow debugger. + +The debugger spans two APIs: the M2EE admin plane toggles it on/off, and the +app's /debugger/ endpoint runs the debug session. Defaults match a runtime +started by 'mxcli run --local' (app http://127.0.0.1:8080, admin :8090, admin +password "mxcli-local-dev"), so in the common case no flags are needed. + +Enabling the debugger CHANGES runtime behaviour: once breakpoints exist (a later +slice), any execution that reaches one pauses until you continue — including a +browser request, which will hang. Always finish with 'mxcli debug disable'. + +Examples: + mxcli debug status + mxcli debug enable + mxcli debug disable`, +} + +var debugStatusCmd = &cobra.Command{ + Use: "status", + Short: "Show whether the debugger is enabled and how many microflows are paused", + RunE: func(cmd *cobra.Command, args []string) error { + c, err := resolveDebuggerClient(cmd) + if err != nil { + return err + } + st, err := c.Status() + if err != nil { + return err + } + fmt.Printf("Debugger: %s\n", enabledLabel(st.Enabled)) + fmt.Printf("Debug client connected: %v\n", st.ClientConnected) + fmt.Printf("Paused microflows: %d\n", st.NumberOfPausedMicroflows) + if st.NumberOfPausedMicroflows > 0 { + fmt.Println("\nNote: paused microflows hold their requests open until 'mxcli debug continue' (a later slice) or 'mxcli debug disable'.") + } + return nil + }, +} + +var debugEnableCmd = &cobra.Command{ + Use: "enable", + Short: "Enable the debugger and start a debug session", + RunE: func(cmd *cobra.Command, args []string) error { + c, err := resolveDebuggerClient(cmd) + if err != nil { + return err + } + if err := c.Enable(); err != nil { + return err + } + if _, err := c.StartSession(); err != nil { + return fmt.Errorf("debugger enabled but starting a session failed: %w", err) + } + fmt.Println("Debugger enabled; debug session started.") + fmt.Println("Breakpoints/stepping arrive in a later slice; for now the session is ready.") + fmt.Println("Remember to run 'mxcli debug disable' when done — a breakpoint pauses whoever hits it, the browser included.") + return nil + }, +} + +var debugDisableCmd = &cobra.Command{ + Use: "disable", + Short: "Disable the debugger and clear the cached session", + RunE: func(cmd *cobra.Command, args []string) error { + c, err := resolveDebuggerClient(cmd) + if err != nil { + return err + } + if err := c.Disable(); err != nil { + return err + } + // Clear the local breakpoint record too — the runtime dropped them with the + // session, so mxcli's view must not claim they're still set. + _ = os.Remove(breakpointsPath(cmd)) + fmt.Println("Debugger disabled.") + return nil + }, +} + +var debugActivitiesCmd = &cobra.Command{ + Use: "activities ", + Short: "List a microflow's activities with the object IDs you can break on", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + project, _ := cmd.Flags().GetString("project") + if project == "" { + return fmt.Errorf("--project (-p) is required to read the microflow") + } + acts, kind, err := resolveFlowActivities(project, args[0]) + if err != nil { + return err + } + if len(acts) == 0 { + fmt.Printf("No objects found in %s\n", args[0]) + return nil + } + fmt.Printf("Activities in %s (%s):\n\n", args[0], kind) + fmt.Printf(" %-4s %-22s %-38s %s\n", "#", "Type", "Object ID", "Caption") + for _, a := range acts { + fmt.Printf(" %-4d %-22s %-38s %s\n", a.Index, a.Type, a.ObjectID, a.Caption) + } + fmt.Println("\nBreak with: mxcli debug break " + args[0] + " --activity '#' (or a caption substring)") + return nil + }, +} + +var debugBreakCmd = &cobra.Command{ + Use: "break --activity <#n|caption>", + Short: "Set a breakpoint on a microflow activity, resolved by name", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + project, _ := cmd.Flags().GetString("project") + if project == "" { + return fmt.Errorf("--project (-p) is required to resolve the activity") + } + selector, _ := cmd.Flags().GetString("activity") + if selector == "" { + return fmt.Errorf("--activity is required (an '#' or caption substring); run 'mxcli debug activities %s' to see them", args[0]) + } + condition, _ := cmd.Flags().GetString("if") + + acts, kind, err := resolveFlowActivities(project, args[0]) + if err != nil { + return err + } + act, err := matchActivity(acts, selector) + if err != nil { + return err + } + + c, err := resolveDebuggerClient(cmd) + if err != nil { + return err + } + if err := c.AddBreakpoint(args[0], act.ObjectID, condition, kind == flowNanoflow); err != nil { + return err + } + + // Record it locally for 'breaks' (the runtime has no list call). + label := act.Caption + if label == "" { + label = fmt.Sprintf("%s#%d", act.Type, act.Index) + } + bpPath := breakpointsPath(cmd) + bps, _ := loadBreakpoints(bpPath) + bps = upsertBreakpoint(bps, localBreakpoint{Microflow: args[0], Activity: label, ObjectID: act.ObjectID, Condition: condition}) + if err := saveBreakpoints(bpPath, bps); err != nil { + fmt.Printf(" (warning: could not record breakpoint locally: %v)\n", err) + } + + fmt.Printf("Breakpoint set on %s → %s (%s)\n", args[0], label, act.ObjectID) + if condition != "" { + fmt.Printf(" condition: %s\n", condition) + } + fmt.Println("Any execution that reaches it — including a browser request — pauses until 'continue' (a later slice) or 'mxcli debug disable'.") + return nil + }, +} + +var debugUnbreakCmd = &cobra.Command{ + Use: "unbreak --activity <#n|caption>", + Short: "Clear a breakpoint on a microflow activity", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + project, _ := cmd.Flags().GetString("project") + if project == "" { + return fmt.Errorf("--project (-p) is required to resolve the activity") + } + selector, _ := cmd.Flags().GetString("activity") + if selector == "" { + return fmt.Errorf("--activity is required (an '#' or caption substring)") + } + acts, _, err := resolveFlowActivities(project, args[0]) + if err != nil { + return err + } + act, err := matchActivity(acts, selector) + if err != nil { + return err + } + c, err := resolveDebuggerClient(cmd) + if err != nil { + return err + } + if err := c.RemoveBreakpoint(act.ObjectID); err != nil { + return err + } + bpPath := breakpointsPath(cmd) + bps, _ := loadBreakpoints(bpPath) + bps = removeBreakpoint(bps, act.ObjectID) + _ = saveBreakpoints(bpPath, bps) + fmt.Printf("Breakpoint cleared on %s (%s)\n", args[0], act.ObjectID) + return nil + }, +} + +var debugBreaksCmd = &cobra.Command{ + Use: "breaks", + Short: "List the breakpoints mxcli has set this session (name → object ID)", + RunE: func(cmd *cobra.Command, args []string) error { + bps, err := loadBreakpoints(breakpointsPath(cmd)) + if err != nil { + return err + } + if len(bps) == 0 { + fmt.Println("No breakpoints recorded. Set one with 'mxcli debug break --activity …'.") + return nil + } + fmt.Println("Breakpoints set this session (mxcli's view):") + for _, b := range bps { + line := fmt.Sprintf(" %s → %s (%s)", b.Microflow, b.Activity, b.ObjectID) + if b.Condition != "" { + line += " if " + b.Condition + } + fmt.Println(line) + } + return nil + }, +} + +var debugPausedCmd = &cobra.Command{ + Use: "paused", + Short: "Show microflows currently paused at a breakpoint, with their variables", + RunE: func(cmd *cobra.Command, args []string) error { + c, err := resolveDebuggerClient(cmd) + if err != nil { + return err + } + flows, raw, events, err := allPausedFlows(c) + if err != nil { + return err + } + if len(flows) == 0 { + fmt.Println("No microflows or nanoflows are paused.") + return nil + } + fmt.Printf("%d paused flow(s):\n", len(flows)) + for _, f := range flows { + fmt.Printf(" %s (debug_id: %s)\n", f.Microflow, f.DebugID) + } + fmt.Println("\nMicroflow state (get_paused_microflows):") + printJSON(raw) + // A paused nanoflow's variables live in the poll_events payload, not in + // get_paused_microflows — print it too when it carries paused entries. + if len(extractPausedFromEvents(events)) > 0 { + fmt.Println("\nClient events (poll_events) — includes paused nanoflows:") + printJSON(events) + } + return nil + }, +} + +var debugInspectCmd = &cobra.Command{ + Use: "inspect [--list] [--flow ]", + Short: "Inspect a variable of a paused microflow (use --list for a list variable)", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + c, err := resolveDebuggerClient(cmd) + if err != nil { + return err + } + flow, _ := cmd.Flags().GetString("flow") + debugID, err := resolveDebugID(c, flow) + if err != nil { + return err + } + asList, _ := cmd.Flags().GetBool("list") + var raw []byte + if asList { + raw, err = c.GetList(debugID, args[0]) + } else { + raw, err = c.GetObject(debugID, args[0]) + } + if err != nil { + return err + } + printJSON(raw) + return nil + }, +} + +var debugStepCmd = &cobra.Command{ + Use: "step [over|into|out] [--flow ]", + Short: "Advance a paused microflow one step (default: over)", + Args: cobra.MaximumNArgs(1), + ValidArgs: []string{"over", "into", "out"}, + RunE: func(cmd *cobra.Command, args []string) error { + kind := "over" + if len(args) == 1 { + kind = args[0] + } + c, err := resolveDebuggerClient(cmd) + if err != nil { + return err + } + flow, _ := cmd.Flags().GetString("flow") + debugID, err := resolveDebugID(c, flow) + if err != nil { + return err + } + if err := c.Step(kind, debugID); err != nil { + return err + } + fmt.Printf("Stepped %s (debug_id: %s).\n", kind, debugID) + return nil + }, +} + +var debugContinueCmd = &cobra.Command{ + Use: "continue [--all]", + Short: "Resume a paused microflow (or all with --all)", + RunE: func(cmd *cobra.Command, args []string) error { + c, err := resolveDebuggerClient(cmd) + if err != nil { + return err + } + all, _ := cmd.Flags().GetBool("all") + if err := c.Continue(all); err != nil { + return err + } + if all { + fmt.Println("Continued all paused microflows.") + } else { + fmt.Println("Continued.") + } + return nil + }, +} + +// resolveDebugID returns the explicit --flow value, or auto-selects the single +// paused microflow. It errors (asking for --flow) when zero or several are paused +// so an action never targets the wrong flow. +func resolveDebugID(c *docker.DebuggerClient, flag string) (string, error) { + if flag != "" { + return flag, nil + } + flows, _, _, err := allPausedFlows(c) + if err != nil { + return "", err + } + switch { + case len(flows) == 1 && flows[0].DebugID != "": + return flows[0].DebugID, nil + case len(flows) == 0: + return "", fmt.Errorf("no paused microflows — nothing to act on") + default: + return "", fmt.Errorf("%d microflows are paused — pass --flow (see 'mxcli debug paused')", len(flows)) + } +} + +// allPausedFlows merges the two sources of paused flows: get_paused_microflows +// (microflows) and poll_events (nanoflows, which do NOT appear in the former). +// Returns the merged summary plus both raw payloads for full-state printing. +func allPausedFlows(c *docker.DebuggerClient) (flows []pausedFlowSummary, paused, events []byte, err error) { + paused, err = c.PausedMicroflows() + if err != nil { + return nil, nil, nil, err + } + flows = extractPausedFlows(paused) + // poll_events is best-effort: a runtime that lacks it shouldn't break `paused`. + if ev, evErr := c.PollEvents(); evErr == nil { + events = ev + for _, f := range extractPausedFromEvents(ev) { + flows = appendUniqueFlow(flows, f) + } + } + return flows, paused, events, nil +} + +// appendUniqueFlow appends f unless a flow with the same debug_id is already present. +func appendUniqueFlow(flows []pausedFlowSummary, f pausedFlowSummary) []pausedFlowSummary { + for _, e := range flows { + if e.DebugID == f.DebugID { + return flows + } + } + return append(flows, f) +} + +// printJSON pretty-prints a raw JSON message, falling back to the raw bytes if it +// isn't valid JSON. +func printJSON(raw []byte) { + var buf bytes.Buffer + if json.Indent(&buf, raw, "", " ") == nil { + fmt.Println(buf.String()) + return + } + fmt.Println(string(raw)) +} + +// upsertBreakpoint replaces an existing entry for the same object ID or appends. +func upsertBreakpoint(bps []localBreakpoint, bp localBreakpoint) []localBreakpoint { + for i := range bps { + if bps[i].ObjectID == bp.ObjectID { + bps[i] = bp + return bps + } + } + return append(bps, bp) +} + +// removeBreakpoint drops the entry with the given object ID. +func removeBreakpoint(bps []localBreakpoint, objectID string) []localBreakpoint { + out := bps[:0] + for _, b := range bps { + if b.ObjectID != objectID { + out = append(out, b) + } + } + return out +} + +// debugStateDir is the /.mxcli directory holding the session token +// and local breakpoint record (cwd/.mxcli when no project is given). +func debugStateDir(cmd *cobra.Command) string { + project, _ := cmd.Flags().GetString("project") + dir := "." + if project != "" { + dir = filepath.Dir(project) + } + return filepath.Join(dir, ".mxcli") +} + +func breakpointsPath(cmd *cobra.Command) string { + return filepath.Join(debugStateDir(cmd), "debug-breakpoints.json") +} + +// resolveDebuggerClient builds a DebuggerClient from the flags/env, defaulting to +// a `run --local` runtime. The admin host is derived from --app-url so a remote +// runtime works too; the token is cached under /.mxcli/. +func resolveDebuggerClient(cmd *cobra.Command) (*docker.DebuggerClient, error) { + appURL, _ := cmd.Flags().GetString("app-url") + adminPort, _ := cmd.Flags().GetInt("admin-port") + adminPass, _ := cmd.Flags().GetString("admin-pass") + debugPass, _ := cmd.Flags().GetString("debug-pass") + + u, err := url.Parse(appURL) + if err != nil || u.Hostname() == "" { + return nil, fmt.Errorf("invalid --app-url %q", appURL) + } + + tokenPath := filepath.Join(debugStateDir(cmd), "debug-session.token") + + c := docker.NewDebuggerClient(docker.DebuggerOptions{ + Admin: docker.M2EEOptions{ + Host: u.Hostname(), + Port: adminPort, + Token: adminPass, + Direct: true, + }, + AppURL: appURL, + DebugPass: debugPass, + TokenPath: tokenPath, + }) + // Best-effort: pick up a session token cached by a prior 'enable'. + _ = c.LoadToken() + return c, nil +} + +func enabledLabel(enabled bool) string { + if enabled { + return "enabled" + } + return "disabled" +} + +func envOr(key, def string) string { + if v := os.Getenv(key); v != "" { + return v + } + return def +} + +func init() { + debugCmd.PersistentFlags().StringP("project", "p", "", "Path to the .mpr (used for the session-token cache location)") + debugCmd.PersistentFlags().String("app-url", envOr("MXCLI_APP_URL", "http://127.0.0.1:8080"), "App base URL hosting the /debugger/ endpoint") + debugCmd.PersistentFlags().Int("admin-port", 8090, "M2EE admin API port") + debugCmd.PersistentFlags().String("admin-pass", envOr("MXCLI_ADMIN_PASS", "mxcli-local-dev"), "M2EE admin password") + debugCmd.PersistentFlags().String("debug-pass", envOr("MXCLI_DEBUG_PASS", "mxdebug"), "Debugger password (passed to enable_debugger and used as the debugger-endpoint credential)") + + debugBreakCmd.Flags().String("activity", "", "Which activity: an '#' (see 'debug activities') or a caption substring") + debugBreakCmd.Flags().String("if", "", "Only pause when this Mendix expression is true (conditional breakpoint)") + debugUnbreakCmd.Flags().String("activity", "", "Which activity: an '#' or a caption substring") + debugInspectCmd.Flags().String("flow", "", "Which paused microflow (debug_id); defaults to the only paused one") + debugInspectCmd.Flags().Bool("list", false, "Inspect a list variable (uses get_list instead of get_object)") + debugStepCmd.Flags().String("flow", "", "Which paused microflow (debug_id); defaults to the only paused one") + debugContinueCmd.Flags().Bool("all", false, "Continue all paused microflows, not just one") + + debugCmd.AddCommand(debugStatusCmd) + debugCmd.AddCommand(debugActivitiesCmd) + debugCmd.AddCommand(debugBreakCmd) + debugCmd.AddCommand(debugUnbreakCmd) + debugCmd.AddCommand(debugBreaksCmd) + debugCmd.AddCommand(debugPausedCmd) + debugCmd.AddCommand(debugInspectCmd) + debugCmd.AddCommand(debugStepCmd) + debugCmd.AddCommand(debugContinueCmd) + debugCmd.AddCommand(debugEnableCmd) + debugCmd.AddCommand(debugDisableCmd) + rootCmd.AddCommand(debugCmd) +} diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index 7eb75c5e9..2dc995351 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -46,9 +46,24 @@ subscriber after start, so the application log lands there too (a standalone runtime attaches no subscriber by default). The path is printed at boot; override with --runtime-log , or "-" to disable. +With --debug, the microflow debugger is enabled at boot and a session is started, +so 'mxcli debug break/paused/step/continue' works from another terminal (use the +same -p). No breakpoints exist until you set one, so --debug alone does not change +runtime behaviour; it is turned back off on shutdown. + +With --metrics, a Prometheus meter registry is registered at boot and the runtime +serves metrics at http://127.0.0.1:/prometheus. With --trace, the +bundled OpenTelemetry agent is attached (spans -> the runtime log via the console +exporter) with default span filters (unfiltered tracing is ~10x slower). Use +--runtime-setting Key=Value (repeatable) to merge any other runtime setting into +the boot config (the admin config action replaces rather than merges, so mxcli +folds these into its single boot call), e.g. a different Metrics.Registries type or +custom OpenTelemetry span filters. + Examples: mxcli run --local -p app.mpr mxcli run --local -p app.mpr --watch + mxcli run --local -p app.mpr --debug # then: mxcli debug break … -p app.mpr mxcli run --local -p app.mpr --app-port 8081 --db-name myapp mxcli run --hub https://hub.example.com -p app.mpr # browser preview mxcli run --hub https://hub.example.com --hub-secret u:pass -p app.mpr --watch @@ -99,6 +114,12 @@ Examples: screenshotUser, _ := cmd.Flags().GetString("screenshot-user") screenshotPassword, _ := cmd.Flags().GetString("screenshot-password") runtimeLog, _ := cmd.Flags().GetString("runtime-log") + debug, _ := cmd.Flags().GetBool("debug") + debugPass, _ := cmd.Flags().GetString("debug-pass") + metrics, _ := cmd.Flags().GetBool("metrics") + runtimeSettings, _ := cmd.Flags().GetStringArray("runtime-setting") + trace, _ := cmd.Flags().GetBool("trace") + traceService, _ := cmd.Flags().GetString("trace-service") opts := docker.LocalRunOptions{ ProjectPath: projectPath, @@ -121,6 +142,12 @@ Examples: ScreenshotUser: screenshotUser, ScreenshotPassword: screenshotPassword, RuntimeLogPath: runtimeLog, + Debug: debug, + DebugPass: debugPass, + Metrics: metrics, + RuntimeSettings: runtimeSettings, + Trace: trace, + TraceService: traceService, DB: docker.DBConfig{ Host: dbHost, Name: dbName, @@ -163,5 +190,11 @@ func init() { runCmd.Flags().String("screenshot-user", "", "Log in with this user before screenshotting (for pages behind login)") runCmd.Flags().String("screenshot-password", "", "Password for --screenshot-user") runCmd.Flags().String("runtime-log", "", "Write the Mendix runtime log (server stack traces + microflow LOG output) to this file for debugging (default /.mxcli/runtime.log; \"-\" to disable)") + runCmd.Flags().Bool("debug", false, "Enable the microflow debugger at boot and start a session, so 'mxcli debug break/paused/…' works from another terminal (no breakpoints = no behaviour change)") + runCmd.Flags().String("debug-pass", "", "Debugger password when --debug is set (default \"mxdebug\")") + runCmd.Flags().Bool("metrics", false, "Register a Prometheus meter registry at boot; the runtime serves metrics at http://127.0.0.1:/prometheus") + runCmd.Flags().StringArray("runtime-setting", nil, "Extra runtime setting Key=Value merged into the boot configuration (Value parsed as JSON when possible), e.g. --runtime-setting 'OpenTelemetry._RuntimeSpanFilters=[\"Loop\",\"Gateway\"]'. Repeatable.") + runCmd.Flags().Bool("trace", false, "Enable OpenTelemetry tracing: attach the bundled agent (console exporter → the runtime log) and apply default span filters (unfiltered tracing is ~10x slower)") + runCmd.Flags().String("trace-service", "", "OTEL_SERVICE_NAME under --trace (default: the .mpr name)") rootCmd.AddCommand(runCmd) } diff --git a/cmd/mxcli/debug_resolve.go b/cmd/mxcli/debug_resolve.go new file mode 100644 index 000000000..61f82b132 --- /dev/null +++ b/cmd/mxcli/debug_resolve.go @@ -0,0 +1,269 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "reflect" + "strconv" + "strings" + + "github.com/mendixlabs/mxcli/sdk/microflows" + "github.com/mendixlabs/mxcli/sdk/mpr" +) + +// debug_resolve.go turns a microflow name + activity selector into the model GUID +// the debugger uses as object_id, and keeps a small local record of the +// breakpoints mxcli has set (the runtime exposes no "list breakpoints" call). +// Slice 2 of the microflow debugger. + +// activityInfo is one microflow object, with the model GUID the debugger wants as +// its object_id (mxcli's activity GetID() already yields the Microsoft-GUID / +// bytes_le form the debugger expects — see types.BlobToUUID). +type activityInfo struct { + Index int // 1-based, in object-collection order + Type string // struct name, e.g. ActionActivity / ExclusiveSplit / StartEvent + Caption string // Mendix caption, e.g. "Create 'Game'" (empty for plain events) + ObjectID string // model GUID == debugger object_id +} + +// flowKind distinguishes a server microflow from a client nanoflow. It matters +// for the debugger: a nanoflow breakpoint uses the nanoflow_name param and a +// paused nanoflow surfaces only via poll_events (findings — nanoflow debugging). +type flowKind string + +const ( + flowMicroflow flowKind = "microflow" + flowNanoflow flowKind = "nanoflow" +) + +// resolveFlowActivities opens the project and lists a microflow's OR nanoflow's +// objects, auto-detecting which it is (microflow first, then nanoflow). The +// object-collection format is identical, so extractActivities handles both. +func resolveFlowActivities(projectPath, qualifiedName string) ([]activityInfo, flowKind, error) { + r, err := mpr.Open(projectPath) + if err != nil { + return nil, "", err + } + defer r.Close() + + if contents, err := r.GetRawMicroflowByName(qualifiedName); err == nil { + mf, err := mpr.ParseMicroflowBSON(contents, "", "") + if err != nil { + return nil, "", fmt.Errorf("parsing microflow %s: %w", qualifiedName, err) + } + return extractActivities(mf), flowMicroflow, nil + } + // Not a microflow — try a nanoflow (same ObjectCollection shape). + if u, err := r.GetRawUnitByName("nanoflow", qualifiedName); err == nil && u != nil { + nf, err := mpr.ParseMicroflowBSON(u.Contents, "", "") + if err != nil { + return nil, "", fmt.Errorf("parsing nanoflow %s: %w", qualifiedName, err) + } + return extractActivities(nf), flowNanoflow, nil + } + return nil, "", fmt.Errorf("no microflow or nanoflow named %s", qualifiedName) +} + +// extractActivities flattens a microflow's object collection into activityInfo. +func extractActivities(mf *microflows.Microflow) []activityInfo { + var out []activityInfo + if mf == nil || mf.ObjectCollection == nil { + return out + } + for i, o := range mf.ObjectCollection.Objects { + typeName, caption := objectTypeCaption(o) + out = append(out, activityInfo{ + Index: i + 1, + Type: typeName, + Caption: caption, + ObjectID: string(o.GetID()), + }) + } + return out +} + +// objectTypeCaption returns a microflow object's struct name and its Caption (if +// the concrete type carries one — action activities, splits, loops, annotations +// do; bare start/end events do not). +func objectTypeCaption(o microflows.MicroflowObject) (typeName, caption string) { + v := reflect.ValueOf(o) + for v.Kind() == reflect.Ptr { + v = v.Elem() + } + if v.Kind() != reflect.Struct { + return fmt.Sprintf("%T", o), "" + } + typeName = v.Type().Name() + if f := v.FieldByName("Caption"); f.IsValid() && f.Kind() == reflect.String { + caption = f.String() + } + return typeName, caption +} + +// matchActivity selects one activity by an "#" (1-based) or a +// case-insensitive caption substring that must match exactly one object. +func matchActivity(acts []activityInfo, selector string) (activityInfo, error) { + selector = strings.TrimSpace(selector) + if selector == "" { + return activityInfo{}, fmt.Errorf("empty --activity; use '#' or a caption substring (see 'mxcli debug activities')") + } + if strings.HasPrefix(selector, "#") { + n, err := strconv.Atoi(strings.TrimPrefix(selector, "#")) + if err != nil || n < 1 || n > len(acts) { + return activityInfo{}, fmt.Errorf("activity index %q out of range (1..%d)", selector, len(acts)) + } + return acts[n-1], nil + } + low := strings.ToLower(selector) + var matches []activityInfo + for _, a := range acts { + if a.Caption != "" && strings.Contains(strings.ToLower(a.Caption), low) { + matches = append(matches, a) + } + } + switch len(matches) { + case 1: + return matches[0], nil + case 0: + return activityInfo{}, fmt.Errorf("no activity caption matches %q — run 'mxcli debug activities' or use --activity '#'", selector) + default: + return activityInfo{}, fmt.Errorf("%q matches %d activities — be more specific or use --activity '#'", selector, len(matches)) + } +} + +// pausedFlowSummary is a best-effort extraction of one paused microflow from the +// get_paused_microflows result. Field names in the runtime response are not +// contract-stable across versions, so parsing is defensive and used only for the +// friendly summary + single-flow default — the full detail is always the raw JSON. +type pausedFlowSummary struct { + DebugID string + Microflow string +} + +// extractPausedFlows pulls {debug_id, microflow} pairs out of the paused-flows +// result, tolerating a top-level array or an array nested one level under an +// object key, and several field-name spellings. +func extractPausedFlows(raw []byte) []pausedFlowSummary { + var v any + if json.Unmarshal(raw, &v) != nil { + return nil + } + list := firstList(v) + var out []pausedFlowSummary + for _, it := range list { + m, ok := it.(map[string]any) + if !ok { + continue + } + id := firstString(m, "debug_id", "debugId", "id") + mf := firstString(m, "microflow_name", "microflowName", "microflow", "name") + if id != "" || mf != "" { + out = append(out, pausedFlowSummary{DebugID: id, Microflow: mf}) + } + } + return out +} + +// extractPausedFromEvents pulls paused flows out of a poll_events response. A +// paused NANOFLOW does not appear in get_paused_microflows — it surfaces only as +// a poll_events entry {"type":"paused_microflow","data":{debug_id, microflow_name, +// …}} (the data field is named microflow_name even for a nanoflow). Parsed +// defensively: any object anywhere with type=="paused_microflow" and a data map. +func extractPausedFromEvents(raw []byte) []pausedFlowSummary { + var v any + if json.Unmarshal(raw, &v) != nil { + return nil + } + var out []pausedFlowSummary + var walk func(any) + walk = func(n any) { + switch t := n.(type) { + case []any: + for _, e := range t { + walk(e) + } + case map[string]any: + if s, _ := t["type"].(string); s == "paused_microflow" { + if data, ok := t["data"].(map[string]any); ok { + id := firstString(data, "debug_id", "debugId", "id") + mf := firstString(data, "microflow_name", "nanoflow_name", "microflowName", "name") + if id != "" || mf != "" { + out = append(out, pausedFlowSummary{DebugID: id, Microflow: mf}) + } + } + } + for _, val := range t { + walk(val) + } + } + } + walk(v) + return out +} + +// firstList returns v if it is an array, else the first array value found among a +// map's values (one level deep). +func firstList(v any) []any { + if a, ok := v.([]any); ok { + return a + } + if m, ok := v.(map[string]any); ok { + for _, val := range m { + if a, ok := val.([]any); ok { + return a + } + } + } + return nil +} + +// firstString returns the first non-empty string value among the given keys. +func firstString(m map[string]any, keys ...string) string { + for _, k := range keys { + if s, ok := m[k].(string); ok && s != "" { + return s + } + } + return "" +} + +// localBreakpoint is one breakpoint mxcli has set, recorded so 'breaks' can show +// the name→GUID reverse map (the runtime has no read-back). +type localBreakpoint struct { + Microflow string `json:"microflow"` + Activity string `json:"activity"` // caption or type#index, for display + ObjectID string `json:"objectId"` + Condition string `json:"condition,omitempty"` +} + +// loadBreakpoints reads the local breakpoint record (missing file = empty). +func loadBreakpoints(path string) ([]localBreakpoint, error) { + b, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return nil, nil + } + return nil, err + } + var bps []localBreakpoint + if err := json.Unmarshal(b, &bps); err != nil { + return nil, err + } + return bps, nil +} + +// saveBreakpoints writes the local breakpoint record. +func saveBreakpoints(path string, bps []localBreakpoint) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + b, err := json.MarshalIndent(bps, "", " ") + if err != nil { + return err + } + return os.WriteFile(path, b, 0o600) +} diff --git a/cmd/mxcli/debug_resolve_test.go b/cmd/mxcli/debug_resolve_test.go new file mode 100644 index 000000000..64b13263d --- /dev/null +++ b/cmd/mxcli/debug_resolve_test.go @@ -0,0 +1,181 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +func TestExtractActivities(t *testing.T) { + mf := µflows.Microflow{ + ObjectCollection: µflows.MicroflowObjectCollection{ + Objects: []microflows.MicroflowObject{ + µflows.StartEvent{BaseMicroflowObject: microflows.BaseMicroflowObject{BaseElement: model.BaseElement{ID: "id-start"}}}, + µflows.Annotation{BaseMicroflowObject: microflows.BaseMicroflowObject{BaseElement: model.BaseElement{ID: "id-note"}}, Caption: "Give hint"}, + }, + }, + } + acts := extractActivities(mf) + if len(acts) != 2 { + t.Fatalf("got %d activities, want 2", len(acts)) + } + if acts[0].Index != 1 || acts[0].Type != "StartEvent" || acts[0].Caption != "" || acts[0].ObjectID != "id-start" { + t.Errorf("act[0] = %+v", acts[0]) + } + if acts[1].Index != 2 || acts[1].Type != "Annotation" || acts[1].Caption != "Give hint" || acts[1].ObjectID != "id-note" { + t.Errorf("act[1] = %+v", acts[1]) + } +} + +func TestExtractActivities_NilCollection(t *testing.T) { + if got := extractActivities(µflows.Microflow{}); got != nil { + t.Errorf("nil ObjectCollection should yield nil, got %v", got) + } +} + +func TestMatchActivity(t *testing.T) { + acts := []activityInfo{ + {Index: 1, Type: "StartEvent", Caption: "", ObjectID: "g1"}, + {Index: 2, Type: "ActionActivity", Caption: "Create 'Game'", ObjectID: "g2"}, + {Index: 3, Type: "ActionActivity", Caption: "Commit 'Game'", ObjectID: "g3"}, + } + cases := []struct { + selector string + wantID string + wantErr bool + }{ + {"#2", "g2", false}, + {"#1", "g1", false}, + {"create", "g2", false}, // caption substring, case-insensitive + {"Commit 'Game'", "g3", false}, // exact caption + {"#0", "", true}, // out of range + {"#9", "", true}, // out of range + {"nope", "", true}, // no caption match + {"game", "", true}, // ambiguous (matches g2 and g3) + {"", "", true}, // empty + } + for _, c := range cases { + got, err := matchActivity(acts, c.selector) + if c.wantErr { + if err == nil { + t.Errorf("matchActivity(%q): want error, got %+v", c.selector, got) + } + continue + } + if err != nil { + t.Errorf("matchActivity(%q): unexpected error %v", c.selector, err) + continue + } + if got.ObjectID != c.wantID { + t.Errorf("matchActivity(%q) = %q, want %q", c.selector, got.ObjectID, c.wantID) + } + } +} + +func TestExtractPausedFlows(t *testing.T) { + cases := []struct { + name string + json string + want []pausedFlowSummary + }{ + { + name: "top-level array", + json: `[{"debug_id":"d1","microflow_name":"Sudoku.ACT_Hint"}]`, + want: []pausedFlowSummary{{DebugID: "d1", Microflow: "Sudoku.ACT_Hint"}}, + }, + { + name: "nested under a key, alt field names", + json: `{"paused_microflows":[{"id":"d2","microflow":"M.F"},{"debugId":"d3","name":"M.G"}]}`, + want: []pausedFlowSummary{{DebugID: "d2", Microflow: "M.F"}, {DebugID: "d3", Microflow: "M.G"}}, + }, + { + name: "empty object", + json: `{}`, + want: nil, + }, + { + name: "invalid json", + json: `not json`, + want: nil, + }, + } + for _, c := range cases { + got := extractPausedFlows([]byte(c.json)) + if len(got) != len(c.want) { + t.Errorf("%s: got %d flows, want %d (%+v)", c.name, len(got), len(c.want), got) + continue + } + for i := range got { + if got[i] != c.want[i] { + t.Errorf("%s[%d] = %+v, want %+v", c.name, i, got[i], c.want[i]) + } + } + } +} + +func TestExtractPausedFromEvents(t *testing.T) { + // A paused nanoflow surfaces only in poll_events, as a paused_microflow event + // whose data uses the microflow_name field. + json := `{"events":[ + {"type":"log","data":{"message":"hi"}}, + {"type":"paused_microflow","data":{"debug_id":"d-nano","microflow_name":"Sudoku.NF_ToggleNotes","object_id":"o1"}} + ]}` + got := extractPausedFromEvents([]byte(json)) + if len(got) != 1 { + t.Fatalf("got %d, want 1 (%+v)", len(got), got) + } + if got[0].DebugID != "d-nano" || got[0].Microflow != "Sudoku.NF_ToggleNotes" { + t.Errorf("got %+v", got[0]) + } + // No paused entries → nil. + if g := extractPausedFromEvents([]byte(`{"events":[{"type":"log"}]}`)); g != nil { + t.Errorf("want nil for no paused entries, got %+v", g) + } + if g := extractPausedFromEvents([]byte(`not json`)); g != nil { + t.Errorf("want nil for invalid json, got %+v", g) + } +} + +func TestBreakpointRegistry_UpsertRemove(t *testing.T) { + var bps []localBreakpoint + bps = upsertBreakpoint(bps, localBreakpoint{Microflow: "M.F", Activity: "A", ObjectID: "g1"}) + bps = upsertBreakpoint(bps, localBreakpoint{Microflow: "M.F", Activity: "B", ObjectID: "g2"}) + if len(bps) != 2 { + t.Fatalf("want 2 breakpoints, got %d", len(bps)) + } + // Upsert with the same object ID replaces, not appends. + bps = upsertBreakpoint(bps, localBreakpoint{Microflow: "M.F", Activity: "A2", ObjectID: "g1", Condition: "x > 0"}) + if len(bps) != 2 { + t.Fatalf("upsert on same ID should replace; got %d", len(bps)) + } + if bps[0].Activity != "A2" || bps[0].Condition != "x > 0" { + t.Errorf("upsert did not replace: %+v", bps[0]) + } + // Remove. + bps = removeBreakpoint(bps, "g1") + if len(bps) != 1 || bps[0].ObjectID != "g2" { + t.Errorf("after remove: %+v", bps) + } +} + +func TestBreakpointRegistry_SaveLoad(t *testing.T) { + path := t.TempDir() + "/.mxcli/debug-breakpoints.json" + want := []localBreakpoint{{Microflow: "M.F", Activity: "Create", ObjectID: "g1", Condition: "y"}} + if err := saveBreakpoints(path, want); err != nil { + t.Fatalf("save: %v", err) + } + got, err := loadBreakpoints(path) + if err != nil { + t.Fatalf("load: %v", err) + } + if len(got) != 1 || got[0] != want[0] { + t.Errorf("round-trip = %+v, want %+v", got, want) + } + // A missing file loads as empty, not an error. + if bps, err := loadBreakpoints(t.TempDir() + "/nope.json"); err != nil || bps != nil { + t.Errorf("missing file: got %v err=%v, want nil,nil", bps, err) + } +} diff --git a/cmd/mxcli/docker/debugger.go b/cmd/mxcli/docker/debugger.go new file mode 100644 index 000000000..710e1893a --- /dev/null +++ b/cmd/mxcli/docker/debugger.go @@ -0,0 +1,320 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" + "os" + "path/filepath" + "strings" + "time" +) + +// debugger.go is slice 1 of the Mendix microflow debugger (see +// docs/11-proposals/PROPOSAL_microflow_debugger.md). It covers the two-plane +// wiring only: the M2EE admin plane toggles the debugger on/off and reports +// status, and the app's /debugger/ endpoint starts a debug session. Breakpoints, +// paused-microflow inspection, and stepping are later slices. +// +// Two APIs, two auth schemes: +// +// admin plane POST http://:/ X-M2EE-Authentication: base64(adminPass) +// actions: enable_debugger {password}, disable_debugger, get_debugger_status +// debugger plane POST /debugger/ X-Debugger-Authentication: base64(debugPass) +// body {action, session_token, params}; params is mandatory even when {} + +// DebuggerOptions configures a DebuggerClient. +type DebuggerOptions struct { + // Admin is the M2EE admin connection (Host/Port/Token/Direct) used for + // enable/disable/status. For a `run --local` runtime: 127.0.0.1:8090, token + // "mxcli-local-dev", Direct: true. + Admin M2EEOptions + // AppURL is the app base URL the /debugger/ endpoint lives under + // (e.g. http://127.0.0.1:8080). + AppURL string + // DebugPass is the debugger password. It is passed to enable_debugger and + // used as the X-Debugger-Authentication credential; the two must match. + DebugPass string + // TokenPath, when set, caches the session token across invocations (the CLI + // is one-shot per command). Empty keeps the token in memory only. + TokenPath string + // Timeout bounds a single debugger HTTP request (default 30s). Note: a + // breakpoint-driven call in a later slice can block far longer while paused — + // those will use their own timeout. + Timeout time.Duration +} + +// DebuggerClient drives both planes of the runtime debugger. +type DebuggerClient struct { + opts DebuggerOptions + http *http.Client + token string +} + +// DebuggerStatus mirrors the get_debugger_status feedback. +type DebuggerStatus struct { + Enabled bool `json:"enabled"` + ClientConnected bool `json:"client_connected"` + NumberOfPausedMicroflows int `json:"number_of_paused_microflows"` +} + +// NewDebuggerClient returns a client with defaults applied. +func NewDebuggerClient(opts DebuggerOptions) *DebuggerClient { + if opts.AppURL == "" { + opts.AppURL = "http://127.0.0.1:8080" + } + if opts.DebugPass == "" { + opts.DebugPass = "mxdebug" + } + if opts.Timeout == 0 { + opts.Timeout = 30 * time.Second + } + return &DebuggerClient{opts: opts, http: &http.Client{Timeout: opts.Timeout}} +} + +// Token returns the current in-memory session token (may be empty). +func (c *DebuggerClient) Token() string { return c.token } + +// Status returns the debugger state via the admin plane. +func (c *DebuggerClient) Status() (*DebuggerStatus, error) { + resp, err := CallM2EE(c.opts.Admin, "get_debugger_status", nil) + if err != nil { + return nil, err + } + if msg := resp.M2EEError(); msg != "" { + return nil, fmt.Errorf("get_debugger_status: %s", msg) + } + var st DebuggerStatus + if len(resp.RawFeedback) > 0 { + if err := json.Unmarshal(resp.RawFeedback, &st); err != nil { + return nil, fmt.Errorf("decoding debugger status: %w", err) + } + } + return &st, nil +} + +// Enable turns the debugger on (admin plane). The runtime requires the password +// here; the same value must be used as the debugger-endpoint credential. +func (c *DebuggerClient) Enable() error { + resp, err := CallM2EE(c.opts.Admin, "enable_debugger", map[string]any{"password": c.opts.DebugPass}) + if err != nil { + return err + } + if msg := resp.M2EEError(); msg != "" { + return fmt.Errorf("enable_debugger: %s", msg) + } + return nil +} + +// Disable turns the debugger off (admin plane) and clears the cached session +// token, so a stale token can't be reused against a fresh session. +func (c *DebuggerClient) Disable() error { + resp, err := CallM2EE(c.opts.Admin, "disable_debugger", nil) + if err != nil { + return err + } + if msg := resp.M2EEError(); msg != "" { + return fmt.Errorf("disable_debugger: %s", msg) + } + c.clearToken() + return nil +} + +// StartSession opens a debug session on the /debugger/ endpoint and caches the +// returned token. It is the only debugger-plane call that carries no token +// (it mints one). +func (c *DebuggerClient) StartSession() (string, error) { + result, err := c.post("start_session", false, map[string]any{"breakpoints": []any{}}) + if err != nil { + return "", err + } + var r struct { + SessionToken string `json:"session_token"` + } + if err := json.Unmarshal(result, &r); err != nil || r.SessionToken == "" { + return "", fmt.Errorf("start_session returned no session_token (response: %s)", string(result)) + } + c.token = r.SessionToken + if err := c.saveToken(r.SessionToken); err != nil { + return "", fmt.Errorf("caching session token: %w", err) + } + return r.SessionToken, nil +} + +// AddBreakpoint sets a breakpoint on an activity (object_id) of a microflow or +// nanoflow. A non-empty condition is a Mendix expression that gates the pause. +// Requires an active session (run 'mxcli debug enable' first). +// +// A nanoflow breakpoint uses the nanoflow_name param, not microflow_name — the +// runtime NPEs on the wrong key (findings — nanoflow debugging). +func (c *DebuggerClient) AddBreakpoint(flowName, objectID, condition string, nanoflow bool) error { + nameKey := "microflow_name" + if nanoflow { + nameKey = "nanoflow_name" + } + params := map[string]any{ + nameKey: flowName, + "object_id": objectID, + } + if condition != "" { + params["condition"] = condition + } + _, err := c.post("add_breakpoint", true, params) + return err +} + +// PollEvents returns the runtime's pending debugger events. A paused NANOFLOW +// appears here (as a paused_microflow event) but NOT in PausedMicroflows. +func (c *DebuggerClient) PollEvents() (json.RawMessage, error) { + return c.post("poll_events", true, nil) +} + +// RemoveBreakpoint clears the breakpoint on an activity (object_id). +func (c *DebuggerClient) RemoveBreakpoint(objectID string) error { + _, err := c.post("remove_breakpoint", true, map[string]any{"object_id": objectID}) + return err +} + +// PausedMicroflows returns the runtime's paused-microflow state (each paused +// flow, its current activity, and in-scope variables) as the raw result object. +func (c *DebuggerClient) PausedMicroflows() (json.RawMessage, error) { + return c.post("get_paused_microflows", true, nil) +} + +// GetObject inspects one variable of a paused microflow (by its debug_id). +func (c *DebuggerClient) GetObject(debugID, variableName string) (json.RawMessage, error) { + return c.post("get_object", true, map[string]any{ + "debug_id": debugID, + "variable_name": variableName, + }) +} + +// GetList inspects a LIST variable of a paused flow (by its debug_id). Use this +// instead of GetObject when the variable is a list of objects. +func (c *DebuggerClient) GetList(debugID, variableName string) (json.RawMessage, error) { + return c.post("get_list", true, map[string]any{ + "debug_id": debugID, + "variable_name": variableName, + }) +} + +// Step advances a paused microflow one step. kind is "over", "into", or "out". +func (c *DebuggerClient) Step(kind, debugID string) error { + var action string + switch kind { + case "over": + action = "step_over" + case "into": + action = "step_into" + case "out": + action = "step_out" + default: + return fmt.Errorf("unknown step %q (want over|into|out)", kind) + } + _, err := c.post(action, true, map[string]any{"debug_id": debugID}) + return err +} + +// Continue resumes execution: the current paused flow, or all paused flows when +// all is true. +func (c *DebuggerClient) Continue(all bool) error { + action := "continue" + if all { + action = "continue_all" + } + _, err := c.post(action, true, nil) + return err +} + +// LoadToken reads a previously cached session token into memory (best-effort: +// a missing file is not an error, since the session may not exist yet). +func (c *DebuggerClient) LoadToken() error { + if c.opts.TokenPath == "" { + return nil + } + b, err := os.ReadFile(c.opts.TokenPath) + if err != nil { + if os.IsNotExist(err) { + return nil + } + return err + } + c.token = strings.TrimSpace(string(b)) + return nil +} + +func (c *DebuggerClient) saveToken(token string) error { + if c.opts.TokenPath == "" { + return nil + } + if err := os.MkdirAll(filepath.Dir(c.opts.TokenPath), 0o755); err != nil { + return err + } + return os.WriteFile(c.opts.TokenPath, []byte(token), 0o600) +} + +func (c *DebuggerClient) clearToken() { + c.token = "" + if c.opts.TokenPath != "" { + _ = os.Remove(c.opts.TokenPath) + } +} + +// post drives the /debugger/ plane. The envelope is {action, session_token?, +// params}; params is always serialized (the runtime rejects a missing params +// with "Missing property"). Returns the raw "result" object from the response. +func (c *DebuggerClient) post(action string, withToken bool, params map[string]any) (json.RawMessage, error) { + if params == nil { + params = map[string]any{} + } + body := map[string]any{"action": action, "params": params} + if withToken { + if c.token == "" { + return nil, fmt.Errorf("no debug session — run 'mxcli debug enable' first") + } + body["session_token"] = c.token + } + + payload, err := json.Marshal(body) + if err != nil { + return nil, fmt.Errorf("marshaling debugger request: %w", err) + } + + url := strings.TrimRight(c.opts.AppURL, "/") + "/debugger/" + req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(payload)) + if err != nil { + return nil, fmt.Errorf("creating debugger request: %w", err) + } + req.Header.Set("Content-Type", "application/json") + // The debugger endpoint accepts ONLY this header form — raw password, Basic, + // and Bearer all 401 (verified against a running runtime). + req.Header.Set("X-Debugger-Authentication", m2eeAuthHeader(c.opts.DebugPass)) + + resp, err := c.http.Do(req) + if err != nil { + return nil, fmt.Errorf("cannot reach debugger endpoint at %s -- is the app running (mxcli run --local)?", url) + } + defer resp.Body.Close() + respBody, _ := io.ReadAll(resp.Body) + + switch resp.StatusCode { + case http.StatusOK: + // fall through + case http.StatusUnauthorized, http.StatusForbidden: + return nil, fmt.Errorf("debugger auth failed (HTTP %d) -- is the debugger enabled ('mxcli debug enable') and --debug-pass correct?", resp.StatusCode) + default: + return nil, fmt.Errorf("debugger endpoint returned HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(respBody))) + } + + var env struct { + Result json.RawMessage `json:"result"` + } + if err := json.Unmarshal(respBody, &env); err != nil { + return nil, fmt.Errorf("decoding debugger response: %w (body: %s)", err, strings.TrimSpace(string(respBody))) + } + return env.Result, nil +} diff --git a/cmd/mxcli/docker/debugger_test.go b/cmd/mxcli/docker/debugger_test.go new file mode 100644 index 000000000..bcacef36e --- /dev/null +++ b/cmd/mxcli/docker/debugger_test.go @@ -0,0 +1,365 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" +) + +// debuggerTestServer stubs BOTH planes on one httptest server, routed by path: +// "/" is the M2EE admin plane, "/debugger/" is the debugger endpoint. It records +// what it received so tests can assert the envelope and auth. +type debuggerTestServer struct { + *httptest.Server + adminActions []string + enablePass string // password seen on enable_debugger + dbgActions []string + dbgAuth string // X-Debugger-Authentication seen on the last /debugger/ call + dbgHadParams bool // whether the last /debugger/ body had a params key + dbgToken string // session_token seen on the last /debugger/ call + dbgParams map[string]any // params of the last /debugger/ call +} + +func newDebuggerTestServer(t *testing.T) (*debuggerTestServer, DebuggerOptions) { + t.Helper() + ts := &debuggerTestServer{} + ts.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/debugger/" { + ts.dbgAuth = r.Header.Get("X-Debugger-Authentication") + var body map[string]json.RawMessage + _ = json.NewDecoder(r.Body).Decode(&body) + var action string + _ = json.Unmarshal(body["action"], &action) + ts.dbgActions = append(ts.dbgActions, action) + _, ts.dbgHadParams = body["params"] + ts.dbgParams = nil + if p, ok := body["params"]; ok { + _ = json.Unmarshal(p, &ts.dbgParams) + } + if tok, ok := body["session_token"]; ok { + _ = json.Unmarshal(tok, &ts.dbgToken) + } + switch action { + case "start_session": + _, _ = w.Write([]byte(`{"result":{"session_token":"tok-123","runtime_version":"11.6"}}`)) + default: + _, _ = w.Write([]byte(`{"result":{}}`)) + } + return + } + // admin plane + var req struct { + Action string `json:"action"` + Params struct { + Password string `json:"password"` + } `json:"params"` + } + _ = json.NewDecoder(r.Body).Decode(&req) + ts.adminActions = append(ts.adminActions, req.Action) + switch req.Action { + case "enable_debugger": + ts.enablePass = req.Params.Password + _ = json.NewEncoder(w).Encode(M2EEResponse{}) + case "get_debugger_status": + _, _ = w.Write([]byte(`{"result":0,"feedback":{"enabled":true,"client_connected":true,"number_of_paused_microflows":2}}`)) + default: + _ = json.NewEncoder(w).Encode(M2EEResponse{}) + } + })) + t.Cleanup(ts.Close) + + host, port := parseTestServerAddr(t, ts.URL) + opts := DebuggerOptions{ + Admin: M2EEOptions{Host: host, Port: port, Token: "adminpass", Direct: true}, + AppURL: ts.URL, + DebugPass: "mxdebug", + TokenPath: filepath.Join(t.TempDir(), "debug-session.token"), + } + return ts, opts +} + +func TestDebugger_Status(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + st, err := NewDebuggerClient(opts).Status() + if err != nil { + t.Fatalf("Status: %v", err) + } + if !st.Enabled || !st.ClientConnected || st.NumberOfPausedMicroflows != 2 { + t.Errorf("status = %+v, want enabled+connected+2 paused", st) + } + if len(ts.adminActions) != 1 || ts.adminActions[0] != "get_debugger_status" { + t.Errorf("adminActions = %v", ts.adminActions) + } +} + +func TestDebugger_EnableSendsPassword(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + if err := NewDebuggerClient(opts).Enable(); err != nil { + t.Fatalf("Enable: %v", err) + } + if len(ts.adminActions) != 1 || ts.adminActions[0] != "enable_debugger" { + t.Fatalf("adminActions = %v, want [enable_debugger]", ts.adminActions) + } + if ts.enablePass != "mxdebug" { + t.Errorf("enable password = %q, want mxdebug", ts.enablePass) + } +} + +func TestDebugger_StartSessionCachesToken(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + tok, err := c.StartSession() + if err != nil { + t.Fatalf("StartSession: %v", err) + } + if tok != "tok-123" || c.Token() != "tok-123" { + t.Errorf("token = %q / %q, want tok-123", tok, c.Token()) + } + // start_session carries no session_token but MUST carry params. + if len(ts.dbgActions) != 1 || ts.dbgActions[0] != "start_session" { + t.Errorf("dbgActions = %v", ts.dbgActions) + } + if !ts.dbgHadParams { + t.Error("start_session body must include a params key") + } + if ts.dbgToken != "" { + t.Errorf("start_session should not send a session_token, got %q", ts.dbgToken) + } + // The debugger auth header is base64(debugPass). + if ts.dbgAuth != m2eeAuthHeader("mxdebug") { + t.Errorf("X-Debugger-Authentication = %q, want base64(mxdebug)", ts.dbgAuth) + } + // Token cached to disk and reloadable. + if b, err := os.ReadFile(opts.TokenPath); err != nil || string(b) != "tok-123" { + t.Errorf("cached token file = %q err=%v, want tok-123", string(b), err) + } + fresh := NewDebuggerClient(opts) + if err := fresh.LoadToken(); err != nil || fresh.Token() != "tok-123" { + t.Errorf("LoadToken -> %q err=%v, want tok-123", fresh.Token(), err) + } +} + +func TestDebugger_DisableClearsToken(t *testing.T) { + _, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + if _, err := os.Stat(opts.TokenPath); err != nil { + t.Fatalf("token file should exist before disable: %v", err) + } + if err := c.Disable(); err != nil { + t.Fatalf("Disable: %v", err) + } + if c.Token() != "" { + t.Errorf("in-memory token not cleared: %q", c.Token()) + } + if _, err := os.Stat(opts.TokenPath); !os.IsNotExist(err) { + t.Errorf("token file should be removed after disable, stat err=%v", err) + } +} + +func TestDebugger_AddBreakpoint(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + if err := c.AddBreakpoint("Sudoku.ACT_Hint", "guid-1", "$Game/Solved = false", false); err != nil { + t.Fatalf("AddBreakpoint: %v", err) + } + if got := ts.dbgActions[len(ts.dbgActions)-1]; got != "add_breakpoint" { + t.Fatalf("last action = %q, want add_breakpoint", got) + } + // Breakpoint calls MUST carry the session token from start_session. + if ts.dbgToken != "tok-123" { + t.Errorf("add_breakpoint session_token = %q, want tok-123", ts.dbgToken) + } + if ts.dbgParams["microflow_name"] != "Sudoku.ACT_Hint" || ts.dbgParams["object_id"] != "guid-1" { + t.Errorf("params = %v, want microflow_name+object_id", ts.dbgParams) + } + if ts.dbgParams["condition"] != "$Game/Solved = false" { + t.Errorf("condition = %v, want the expression", ts.dbgParams["condition"]) + } +} + +func TestDebugger_AddBreakpointOmitsEmptyCondition(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + if err := c.AddBreakpoint("M.F", "guid-2", "", false); err != nil { + t.Fatalf("AddBreakpoint: %v", err) + } + if _, ok := ts.dbgParams["condition"]; ok { + t.Errorf("empty condition must be omitted, got %v", ts.dbgParams["condition"]) + } +} + +func TestDebugger_RemoveBreakpoint(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + if err := c.RemoveBreakpoint("guid-1"); err != nil { + t.Fatalf("RemoveBreakpoint: %v", err) + } + if got := ts.dbgActions[len(ts.dbgActions)-1]; got != "remove_breakpoint" { + t.Fatalf("last action = %q, want remove_breakpoint", got) + } + if ts.dbgParams["object_id"] != "guid-1" { + t.Errorf("params = %v, want object_id guid-1", ts.dbgParams) + } +} + +func TestDebugger_BreakpointNeedsSession(t *testing.T) { + // Without a session token, a breakpoint call must fail with a clear message. + _, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) // no StartSession / LoadToken + err := c.AddBreakpoint("M.F", "guid-1", "", false) + if err == nil || !strings.Contains(err.Error(), "debug enable") { + t.Errorf("want a 'run debug enable first' error, got %v", err) + } +} + +func TestDebugger_StepActions(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + cases := []struct{ kind, want string }{ + {"over", "step_over"}, + {"into", "step_into"}, + {"out", "step_out"}, + } + for _, tc := range cases { + if err := c.Step(tc.kind, "dbg-9"); err != nil { + t.Fatalf("Step(%s): %v", tc.kind, err) + } + if got := ts.dbgActions[len(ts.dbgActions)-1]; got != tc.want { + t.Errorf("Step(%s) action = %q, want %q", tc.kind, got, tc.want) + } + if ts.dbgParams["debug_id"] != "dbg-9" { + t.Errorf("Step(%s) debug_id = %v, want dbg-9", tc.kind, ts.dbgParams["debug_id"]) + } + } + if err := c.Step("sideways", "dbg-9"); err == nil { + t.Error("Step with an unknown kind should error") + } +} + +func TestDebugger_Continue(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + if err := c.Continue(false); err != nil { + t.Fatalf("Continue: %v", err) + } + if got := ts.dbgActions[len(ts.dbgActions)-1]; got != "continue" { + t.Errorf("Continue(false) = %q, want continue", got) + } + if err := c.Continue(true); err != nil { + t.Fatalf("Continue(all): %v", err) + } + if got := ts.dbgActions[len(ts.dbgActions)-1]; got != "continue_all" { + t.Errorf("Continue(true) = %q, want continue_all", got) + } +} + +func TestDebugger_GetObject(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + if _, err := c.GetObject("dbg-9", "Game"); err != nil { + t.Fatalf("GetObject: %v", err) + } + if got := ts.dbgActions[len(ts.dbgActions)-1]; got != "get_object" { + t.Errorf("action = %q, want get_object", got) + } + if ts.dbgParams["debug_id"] != "dbg-9" || ts.dbgParams["variable_name"] != "Game" { + t.Errorf("params = %v, want debug_id+variable_name", ts.dbgParams) + } +} + +func TestDebugger_GetList(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + if _, err := c.GetList("dbg-9", "Games"); err != nil { + t.Fatalf("GetList: %v", err) + } + if got := ts.dbgActions[len(ts.dbgActions)-1]; got != "get_list" { + t.Errorf("action = %q, want get_list", got) + } + if ts.dbgParams["debug_id"] != "dbg-9" || ts.dbgParams["variable_name"] != "Games" { + t.Errorf("params = %v, want debug_id+variable_name", ts.dbgParams) + } +} + +func TestDebugger_AddBreakpointNanoflow(t *testing.T) { + // A nanoflow breakpoint must use nanoflow_name, not microflow_name (the + // runtime NPEs otherwise). + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + if err := c.AddBreakpoint("Sudoku.NF_ToggleNotes", "guid-n", "", true); err != nil { + t.Fatalf("AddBreakpoint(nanoflow): %v", err) + } + if ts.dbgParams["nanoflow_name"] != "Sudoku.NF_ToggleNotes" { + t.Errorf("want nanoflow_name, got params %v", ts.dbgParams) + } + if _, ok := ts.dbgParams["microflow_name"]; ok { + t.Errorf("nanoflow breakpoint must not send microflow_name, got %v", ts.dbgParams) + } +} + +func TestDebugger_PollEvents(t *testing.T) { + ts, opts := newDebuggerTestServer(t) + c := NewDebuggerClient(opts) + if _, err := c.StartSession(); err != nil { + t.Fatalf("StartSession: %v", err) + } + if _, err := c.PollEvents(); err != nil { + t.Fatalf("PollEvents: %v", err) + } + if got := ts.dbgActions[len(ts.dbgActions)-1]; got != "poll_events" { + t.Errorf("action = %q, want poll_events", got) + } + if ts.dbgToken != "tok-123" { + t.Errorf("poll_events must carry the session token, got %q", ts.dbgToken) + } +} + +func TestDebugger_AuthFailure(t *testing.T) { + // A 401 from the debugger endpoint must produce an actionable error. + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusUnauthorized) + _, _ = w.Write([]byte(`{}`)) + })) + defer ts.Close() + c := NewDebuggerClient(DebuggerOptions{AppURL: ts.URL, DebugPass: "wrong"}) + _, err := c.StartSession() + if err == nil { + t.Fatal("expected an auth error on HTTP 401") + } + if got := err.Error(); !strings.Contains(got, "auth failed") || !strings.Contains(got, "debug enable") { + t.Errorf("error = %q, want it to mention auth + how to enable", got) + } +} diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index d5e064c40..573a91d07 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -10,6 +10,7 @@ import ( "os" "os/exec" "path/filepath" + "strings" "syscall" "time" ) @@ -66,6 +67,17 @@ type LocalRuntimeOptions struct { // when the app is served through an external tunnel/reverse proxy rather than // localhost. Empty for a plain local run. ApplicationRootUrl string + // RuntimeSettings are extra update_configuration keys merged into the boot + // payload (e.g. "Metrics.Registries", "OpenTelemetry._RuntimeSpanFilters"). + // Merged here because the admin action replaces rather than merges. + RuntimeSettings map[string]any + // Trace attaches the bundled OpenTelemetry Java agent to the runtime JVM + // (traces via the console exporter → the tee'd runtime log). The caller should + // also set OpenTelemetry._RuntimeSpanFilters via RuntimeSettings — unfiltered + // per-activity tracing is ~10x slower. + Trace bool + // TraceServiceName is OTEL_SERVICE_NAME when Trace is set (default: the app). + TraceServiceName string // DB is the database the runtime connects to. DB DBConfig // ReadyTimeout bounds how long StartLocalRuntime waits for the admin API @@ -136,6 +148,66 @@ func localRuntimeEnv(o LocalRuntimeOptions) []string { ) } +// otelAgentJar locates the OpenTelemetry Java agent bundled with the runtime +// (/agents/opentelemetry-javaagent*.jar). The version suffix varies, +// so it is globbed. +func (o *LocalRuntimeOptions) otelAgentJar() (string, error) { + pattern := filepath.Join(o.runtimeDir(), "agents", "opentelemetry-javaagent*.jar") + matches, _ := filepath.Glob(pattern) + if len(matches) == 0 { + return "", fmt.Errorf("OpenTelemetry agent not found at %s (this runtime may not bundle it)", pattern) + } + return matches[0], nil +} + +// withTraceEnv layers the OpenTelemetry Java-agent + OTEL_* env onto base. The +// agent is always attached (via JAVA_TOOL_OPTIONS, appended to any existing +// value); the OTEL_* exporters default to console traces / no metrics+logs but +// are NOT overridden if the caller already set them (so OTLP-to-a-collector via +// the user's own env still works). Traces on the console exporter land in the +// tee'd runtime log. +func withTraceEnv(base []string, agentJar, serviceName string) []string { + has := func(key string) bool { + for _, e := range base { + if strings.HasPrefix(e, key+"=") { + return true + } + } + return false + } + get := func(key string) string { + for _, e := range base { + if strings.HasPrefix(e, key+"=") { + return e[len(key)+1:] + } + } + return "" + } + jto := strings.TrimSpace(get("JAVA_TOOL_OPTIONS") + " -javaagent:" + agentJar) + + out := make([]string, 0, len(base)+5) + for _, e := range base { + if strings.HasPrefix(e, "JAVA_TOOL_OPTIONS=") { + continue // replaced below with the agent appended + } + out = append(out, e) + } + out = append(out, "JAVA_TOOL_OPTIONS="+jto) + if !has("OTEL_SERVICE_NAME") && serviceName != "" { + out = append(out, "OTEL_SERVICE_NAME="+serviceName) + } + if !has("OTEL_TRACES_EXPORTER") { + out = append(out, "OTEL_TRACES_EXPORTER=console") + } + if !has("OTEL_METRICS_EXPORTER") { + out = append(out, "OTEL_METRICS_EXPORTER=none") + } + if !has("OTEL_LOGS_EXPORTER") { + out = append(out, "OTEL_LOGS_EXPORTER=none") + } + return out +} + // appContainerParams is the update_appcontainer_configuration payload (which port // and address the app itself listens on). func appContainerParams(o LocalRuntimeOptions) map[string]any { @@ -168,6 +240,14 @@ func runtimeConfigParams(o LocalRuntimeOptions, constants map[string]string) map if o.ApplicationRootUrl != "" { params["ApplicationRootUrl"] = o.ApplicationRootUrl } + // Overlay extra runtime settings (e.g. Metrics.Registries, + // OpenTelemetry._RuntimeSpanFilters) into this SAME payload. The admin + // update_configuration action REPLACES rather than merges and has no + // read-back, so merging here — into mxcli's single boot call — is the only + // safe way to add settings without clobbering the DB/BasePath config. + for k, v := range o.RuntimeSettings { + params[k] = v + } return params } @@ -282,6 +362,13 @@ func (rt *LocalRuntime) spawnAndConfigure() error { cmd := exec.Command(javaExe, "-jar", rt.opts.launcherJar(), rt.opts.DeployDir) cmd.Dir = rt.opts.runtimeDir() cmd.Env = localRuntimeEnv(rt.opts) + if rt.opts.Trace { + jar, err := rt.opts.otelAgentJar() + if err != nil { + return fmt.Errorf("--trace: %w", err) + } + cmd.Env = withTraceEnv(cmd.Env, jar, rt.opts.TraceServiceName) + } PrepareMxCommand(cmd) // FreeType LD_PRELOAD workaround, layered on cmd.Env setProcessGroup(cmd) // reap any JVM child on Stop so the port is freed // The in-memory buffer always captures output for startup-failure reporting. diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index f59dfd6b1..0ee417fc6 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -3,6 +3,7 @@ package docker import ( + "encoding/json" "fmt" "io" "io/fs" @@ -92,8 +93,28 @@ type LocalRunOptions struct { // warm loop is debuggable (server stack traces, microflow LOG output). // Default /.mxcli/runtime.log; set to "-" to disable. Findings #25. RuntimeLogPath string - Stdout io.Writer - Stderr io.Writer + // Debug enables the runtime microflow debugger at boot and starts a session, + // caching its token under /.mxcli so `mxcli debug break/paused/…` + // (in another terminal, same -p) works immediately. Enabling with no + // breakpoints does not change runtime behaviour — only a set breakpoint pauses. + Debug bool + // DebugPass is the debugger password used when Debug is set (default "mxdebug"). + DebugPass string + // Metrics registers a Prometheus meter registry at boot; the runtime then + // serves /prometheus on the admin port. Sugar for a Metrics.Registries setting. + Metrics bool + // Trace attaches the bundled OpenTelemetry Java agent (traces → the runtime + // log via the console exporter) and applies default span filters. + Trace bool + // TraceService is OTEL_SERVICE_NAME under --trace (default: the .mpr name). + TraceService string + // RuntimeSettings are raw "Key=Value" runtime settings merged into the boot + // update_configuration payload (Value is parsed as JSON, else a string), e.g. + // 'Metrics.Registries=[{"type":"otlp"}]' or + // 'OpenTelemetry._RuntimeSpanFilters=["Loop","Gateway"]'. + RuntimeSettings []string + Stdout io.Writer + Stderr io.Writer } // defaultLocalAdminPass is the admin password for a local dev runtime. The admin @@ -140,6 +161,9 @@ func (o *LocalRunOptions) applyDefaults() { if o.RuntimeLogPath == "" { o.RuntimeLogPath = filepath.Join(filepath.Dir(o.ProjectPath), ".mxcli", "runtime.log") } + if o.Debug && o.DebugPass == "" { + o.DebugPass = "mxdebug" + } if o.Stdout == nil { o.Stdout = os.Stdout } @@ -148,6 +172,57 @@ func (o *LocalRunOptions) applyDefaults() { } } +// defaultOtelSpanFilters are the internal runtime spans suppressed under --trace. +// Unfiltered per-activity tracing is ~10x slower; these bring it near baseline +// while keeping the microflow-level spans (findings — OpenTelemetry). +var defaultOtelSpanFilters = []any{"CreateOrChangeVariable", "Loop", "Gateway", "RetrieveFromCache"} + +// buildRuntimeSettings turns --metrics/--trace + repeatable --runtime-setting +// Key=Value into the map merged into the boot update_configuration payload. A +// Value that parses as JSON is used as-is; otherwise it is a plain string. +// --metrics adds a Prometheus registry and --trace adds default span filters, +// each unless the corresponding key was already set explicitly. +func buildRuntimeSettings(metrics, trace bool, raw []string) (map[string]any, error) { + out := map[string]any{} + for _, s := range raw { + k, v, err := parseRuntimeSetting(s) + if err != nil { + return nil, err + } + out[k] = v + } + if metrics { + if _, ok := out["Metrics.Registries"]; !ok { + out["Metrics.Registries"] = []any{map[string]any{"type": "prometheus"}} + } + } + if trace { + if _, ok := out["OpenTelemetry._RuntimeSpanFilters"]; !ok { + out["OpenTelemetry._RuntimeSpanFilters"] = defaultOtelSpanFilters + } + } + if len(out) == 0 { + return nil, nil + } + return out, nil +} + +// parseRuntimeSetting splits "Key=Value"; Value is parsed as JSON when possible +// (so arrays/objects/numbers/bools pass through typed), else kept as a string. +func parseRuntimeSetting(s string) (string, any, error) { + i := strings.IndexByte(s, '=') + if i <= 0 { + return "", nil, fmt.Errorf("invalid --runtime-setting %q (want Key=Value)", s) + } + key := strings.TrimSpace(s[:i]) + rawVal := s[i+1:] + var v any + if json.Unmarshal([]byte(rawVal), &v) == nil { + return key, v, nil + } + return key, rawVal, nil +} + // deriveDBName turns a project file name into a safe Postgres database name: // lowercased, non-alphanumerics collapsed to underscores, leading digit prefixed. func deriveDBName(projectPath string) string { @@ -472,6 +547,14 @@ func RunLocal(opts LocalRunOptions) error { if runtimeLog == "-" { runtimeLog = "" } + runtimeSettings, err := buildRuntimeSettings(opts.Metrics, opts.Trace, opts.RuntimeSettings) + if err != nil { + return err + } + traceService := opts.TraceService + if opts.Trace && traceService == "" { + traceService = strings.TrimSuffix(filepath.Base(opts.ProjectPath), filepath.Ext(opts.ProjectPath)) + } rt, err := StartLocalRuntime(LocalRuntimeOptions{ DeployDir: opts.DeployDir, InstallPath: installPath, @@ -482,6 +565,9 @@ func RunLocal(opts LocalRunOptions) error { ApplicationRootUrl: appRootURL, DB: opts.DB, RuntimeLogPath: runtimeLog, + RuntimeSettings: runtimeSettings, + Trace: opts.Trace, + TraceServiceName: traceService, Stdout: w, Stderr: stderr, }) @@ -494,6 +580,42 @@ func RunLocal(opts LocalRunOptions) error { if runtimeLog != "" { fmt.Fprintf(w, "Runtime log: %s\n", runtimeLog) } + if opts.Metrics { + // The Prometheus registry is served from the admin port (loopback). + fmt.Fprintf(w, "Metrics (Prometheus): http://127.0.0.1:%d/prometheus\n", opts.AdminPort) + } + if opts.Trace { + if runtimeLog != "" { + fmt.Fprintf(w, "Tracing enabled (OpenTelemetry, service %q); spans -> %s\n", traceService, runtimeLog) + } else { + fmt.Fprintf(w, "Tracing enabled (OpenTelemetry, service %q); spans go to the console only (pass --runtime-log to capture them)\n", traceService) + } + } + + // 6·debug: --debug enables the microflow debugger and starts a session now, so + // breakpoints can be set from another terminal without a separate + // `mxcli debug enable`. No breakpoints exist yet, so nothing pauses; the token + // is cached under /.mxcli for the `mxcli debug …` commands. + if opts.Debug { + dbg := NewDebuggerClient(DebuggerOptions{ + Admin: rt.m2ee, + AppURL: rt.AppURL(), + DebugPass: opts.DebugPass, + TokenPath: filepath.Join(filepath.Dir(opts.ProjectPath), ".mxcli", "debug-session.token"), + }) + if err := dbg.Enable(); err != nil { + fmt.Fprintf(stderr, " (debugger not enabled: %v)\n", err) + } else if _, err := dbg.StartSession(); err != nil { + fmt.Fprintf(stderr, " (debugger enabled but starting a session failed: %v)\n", err) + } else { + fmt.Fprintf(w, "Debugger enabled. Set a breakpoint from another terminal:\n") + fmt.Fprintf(w, " mxcli debug break --activity <#n|caption> -p %s\n", opts.ProjectPath) + // Best-effort: turn the debugger back off on shutdown so a breakpoint + // can't be left pausing requests. (The runtime dies with the process + // anyway; this also clears the cached session token.) + defer func() { _ = dbg.Disable() }() + } + } // 6a. With --hub, open a reverse tunnel so the app is reachable in a browser at // its public URL, and heartbeat so it shows as available in the hub overview. diff --git a/cmd/mxcli/docker/runlocal_test.go b/cmd/mxcli/docker/runlocal_test.go index 2f65ab599..4d1345138 100644 --- a/cmd/mxcli/docker/runlocal_test.go +++ b/cmd/mxcli/docker/runlocal_test.go @@ -52,6 +52,151 @@ func TestLocalRunOptions_Defaults(t *testing.T) { } } +func TestLocalRunOptions_DebugPassDefault(t *testing.T) { + // --debug with no password defaults to "mxdebug". + o := LocalRunOptions{ProjectPath: "/proj/App.mpr", Debug: true} + o.applyDefaults() + if o.DebugPass != "mxdebug" { + t.Errorf("DebugPass = %q, want mxdebug", o.DebugPass) + } + // An explicit password is preserved. + o2 := LocalRunOptions{ProjectPath: "/proj/App.mpr", Debug: true, DebugPass: "secret"} + o2.applyDefaults() + if o2.DebugPass != "secret" { + t.Errorf("DebugPass override lost: %q", o2.DebugPass) + } + // Without --debug, no password is set. + o3 := LocalRunOptions{ProjectPath: "/proj/App.mpr"} + o3.applyDefaults() + if o3.DebugPass != "" { + t.Errorf("DebugPass should stay empty without --debug, got %q", o3.DebugPass) + } +} + +func TestBuildRuntimeSettings(t *testing.T) { + // --metrics alone → a Prometheus registry. + s, err := buildRuntimeSettings(true, false, nil) + if err != nil { + t.Fatalf("buildRuntimeSettings: %v", err) + } + regs, ok := s["Metrics.Registries"].([]any) + if !ok || len(regs) != 1 { + t.Fatalf("Metrics.Registries = %v, want one registry", s["Metrics.Registries"]) + } + if m, _ := regs[0].(map[string]any); m["type"] != "prometheus" { + t.Errorf("registry = %v, want type prometheus", regs[0]) + } + + // --runtime-setting with a JSON array value passes through typed, and an + // explicit Metrics.Registries is not overridden by --metrics. + s, err = buildRuntimeSettings(true, false, []string{ + `OpenTelemetry._RuntimeSpanFilters=["Loop","Gateway"]`, + `Metrics.Registries=[{"type":"otlp"}]`, + }) + if err != nil { + t.Fatalf("buildRuntimeSettings: %v", err) + } + filters, ok := s["OpenTelemetry._RuntimeSpanFilters"].([]any) + if !ok || len(filters) != 2 || filters[0] != "Loop" { + t.Errorf("span filters = %v", s["OpenTelemetry._RuntimeSpanFilters"]) + } + if regs, _ := s["Metrics.Registries"].([]any); len(regs) != 1 || regs[0].(map[string]any)["type"] != "otlp" { + t.Errorf("explicit Metrics.Registries should win over --metrics, got %v", s["Metrics.Registries"]) + } + + // --trace adds the default span filters… + s, _ = buildRuntimeSettings(false, true, nil) + if f, _ := s["OpenTelemetry._RuntimeSpanFilters"].([]any); len(f) != len(defaultOtelSpanFilters) { + t.Errorf("--trace should add %d default span filters, got %v", len(defaultOtelSpanFilters), s["OpenTelemetry._RuntimeSpanFilters"]) + } + // …but an explicit filter set wins over --trace defaults. + s, _ = buildRuntimeSettings(false, true, []string{`OpenTelemetry._RuntimeSpanFilters=["Only"]`}) + if f, _ := s["OpenTelemetry._RuntimeSpanFilters"].([]any); len(f) != 1 || f[0] != "Only" { + t.Errorf("explicit span filters should win over --trace, got %v", s["OpenTelemetry._RuntimeSpanFilters"]) + } + + // A non-JSON value stays a plain string. + s, _ = buildRuntimeSettings(false, false, []string{"DTAPMode=A"}) + if s["DTAPMode"] != "A" { + t.Errorf("DTAPMode = %v, want string A", s["DTAPMode"]) + } + + // Nothing requested → nil (no overlay). + if s, _ := buildRuntimeSettings(false, false, nil); s != nil { + t.Errorf("want nil for no settings, got %v", s) + } + + // Malformed setting errors. + if _, err := buildRuntimeSettings(false, false, []string{"noequals"}); err == nil { + t.Error("want error for a setting without '='") + } +} + +func TestWithTraceEnv(t *testing.T) { + base := []string{"PATH=/bin", "JAVA_TOOL_OPTIONS=-Xmx512m"} + env := withTraceEnv(base, "/agents/otel.jar", "sudoku") + + get := func(key string) (string, bool) { + for _, e := range env { + if strings.HasPrefix(e, key+"=") { + return e[len(key)+1:], true + } + } + return "", false + } + // Agent appended to the existing JAVA_TOOL_OPTIONS (not duplicated). + jto, _ := get("JAVA_TOOL_OPTIONS") + if !strings.Contains(jto, "-Xmx512m") || !strings.Contains(jto, "-javaagent:/agents/otel.jar") { + t.Errorf("JAVA_TOOL_OPTIONS = %q, want existing + agent", jto) + } + n := 0 + for _, e := range env { + if strings.HasPrefix(e, "JAVA_TOOL_OPTIONS=") { + n++ + } + } + if n != 1 { + t.Errorf("JAVA_TOOL_OPTIONS appears %d times, want 1", n) + } + if v, _ := get("OTEL_SERVICE_NAME"); v != "sudoku" { + t.Errorf("OTEL_SERVICE_NAME = %q, want sudoku", v) + } + if v, _ := get("OTEL_TRACES_EXPORTER"); v != "console" { + t.Errorf("OTEL_TRACES_EXPORTER = %q, want console", v) + } + + // A user-provided OTEL_* is respected (not overridden). + env = withTraceEnv([]string{"OTEL_TRACES_EXPORTER=otlp"}, "/agents/otel.jar", "svc") + if v, _ := get2(env, "OTEL_TRACES_EXPORTER"); v != "otlp" { + t.Errorf("user OTEL_TRACES_EXPORTER should win, got %q", v) + } +} + +func get2(env []string, key string) (string, bool) { + for _, e := range env { + if strings.HasPrefix(e, key+"=") { + return e[len(key)+1:], true + } + } + return "", false +} + +func TestRuntimeConfigParams_OverlaysSettings(t *testing.T) { + o := LocalRuntimeOptions{ + DeployDir: "/d", DB: DBConfig{Type: "PostgreSQL", Name: "app"}, + RuntimeSettings: map[string]any{"Metrics.Registries": []any{map[string]any{"type": "prometheus"}}}, + } + p := runtimeConfigParams(o, nil) + // Base keys still present… + if p["DatabaseName"] != "app" { + t.Errorf("DatabaseName lost: %v", p["DatabaseName"]) + } + // …and the overlay merged in (not a separate replace call). + if _, ok := p["Metrics.Registries"]; !ok { + t.Errorf("Metrics.Registries not merged into config params: %v", p) + } +} + func TestLocalRunOptions_DefaultsRespectOverrides(t *testing.T) { o := LocalRunOptions{ ProjectPath: "/proj/App.mpr", diff --git a/docs-site/src/SUMMARY.md b/docs-site/src/SUMMARY.md index a744de98d..f55ef7904 100644 --- a/docs-site/src/SUMMARY.md +++ b/docs-site/src/SUMMARY.md @@ -153,6 +153,7 @@ - [Database Connector Generation](tools/connector-generation.md) - [Local Dev Loop](tools/run-local.md) - [Bootstrap Prompt](tools/bootstrap-prompt.md) + - [Debug Microflows](tools/debug-microflows.md) - [Docker Integration](tools/docker.md) - [mxcli docker build](tools/docker-build.md) - [mxcli docker check](tools/docker-check.md) diff --git a/docs-site/src/tools/debug-microflows.md b/docs-site/src/tools/debug-microflows.md new file mode 100644 index 000000000..b04db8eca --- /dev/null +++ b/docs-site/src/tools/debug-microflows.md @@ -0,0 +1,110 @@ +# Debug Microflows — `mxcli debug` + +`mxcli debug` drives the Mendix runtime's **microflow debugger** from the command +line: set breakpoints **by name**, inspect a paused microflow's variables, and +step/continue — against an app started by [`mxcli run --local`](run-local.md). It +is the headless counterpart to Studio Pro's debugger. + +Because mxcli owns both halves — the admin password + app URL (from `run --local`) +and the activity model GUIDs (from the `.mpr`) — it can offer breakpoints **by +name**, so you never handle raw GUIDs. + +## Quick start + +```bash +# terminal 1 — app with the debugger enabled and a session ready +mxcli run --local -p app.mpr --debug + +# terminal 2 — break by name, then inspect/step/continue (same -p) +mxcli debug activities Sudoku.ACT_Hint -p app.mpr +mxcli debug break Sudoku.ACT_Hint --activity 'Retrieve' -p app.mpr +# ...trigger the microflow in the browser (the request pauses)... +mxcli debug paused -p app.mpr +mxcli debug inspect Game -p app.mpr +mxcli debug step over -p app.mpr +mxcli debug continue -p app.mpr +mxcli debug disable -p app.mpr # always finish here +``` + +`--debug` on `run --local` enables the debugger and starts a session (cached under +`/.mxcli/`), so you skip a separate `mxcli debug enable`. With **no +breakpoints set, nothing pauses** — `--debug` alone is behaviour-neutral. + +## Two APIs, one command + +Under the hood the debugger spans two runtime APIs, which `mxcli debug` hides: + +- the **M2EE admin** API toggles the debugger (`enable`/`disable`/`status`); +- the app's **`/debugger/`** endpoint runs the session (breakpoints, paused state, + stepping). + +## Commands + +| Command | What it does | +|---------|--------------| +| `mxcli debug status` | Debugger on? How many microflows paused? | +| `mxcli debug enable` / `disable` | Turn on/off (prefer `run --local --debug` for the warm loop) | +| `mxcli debug activities ` | List activities + the object IDs you can break on | +| `mxcli debug break --activity <#n\|caption> [--if ]` | Set a breakpoint by name; `--if` is a conditional (Mendix expression) | +| `mxcli debug unbreak --activity <#n\|caption>` | Clear a breakpoint | +| `mxcli debug breaks` | List the breakpoints set this session (name → object ID) | +| `mxcli debug paused` | Paused microflows + full state (variables) | +| `mxcli debug inspect [--list] [--flow ]` | Inspect one variable of a paused flow; `--list` inspects a list variable (`get_list`) | +| `mxcli debug step [over\|into\|out] [--flow ]` | Advance one step (default `over`) | +| `mxcli debug continue [--all]` | Resume the paused flow (or all with `--all`) | + +**Selecting an activity:** `--activity '#2'` (the index from `activities`) or a +caption substring like `--activity 'Retrieve'` (case-insensitive; must match one). + +**Selecting a paused flow:** `--flow ` (from `paused`); a single paused +flow is auto-selected. + +## Connection flags + +Defaults target a `run --local` runtime; override to debug a differently-configured +or remote runtime. + +| Flag | Env | Default | +|------|-----|---------| +| `--app-url` | `MXCLI_APP_URL` | `http://127.0.0.1:8080` | +| `--admin-port` | — | `8090` | +| `--admin-pass` | `MXCLI_ADMIN_PASS` | `mxcli-local-dev` | +| `--debug-pass` | `MXCLI_DEBUG_PASS` | `mxdebug` | +| `-p, --project` | — | (for the `.mxcli/` session + breakpoint files) | + +## Nanoflows (client-side) + +`mxcli debug` works for **nanoflows** as well as microflows. `break` / `activities` / +`unbreak` auto-detect the document type and set the breakpoint correctly — a nanoflow +uses the runtime's `nanoflow_name` parameter (the wrong key NPEs the runtime; mxcli +handles it). Break by name the same way: + +```bash +mxcli debug break Sudoku.NF_ToggleNotes --activity 'Change' -p app.mpr +``` + +A paused **nanoflow** does not appear in `get_paused_microflows`; it surfaces only in +the runtime's `poll_events`. `mxcli debug paused` (and `step`/`inspect`/`continue`) +merge both sources, so a paused nanoflow appears with its `debug_id`; its variables +are shown under the *Client events (poll_events)* section of `paused`. + +A nanoflow's `debug_id` is **single-use** — it changes after every step (a microflow's +is stable). Let `step`/`inspect`/`continue` **auto-resolve** the flow (omit `--flow`); +each command re-reads the current state and picks up the fresh id. A `--flow` value +copied from an earlier `paused` goes stale after the first nanoflow step. + +**Nanoflow logging:** a nanoflow logs to the browser console and to the server +runtime log, but the runtime **rewrites the log node to `Client_Nanoflow`** — so in +`.mxcli/runtime.log` a nanoflow's `LOG NODE 'Sudoku' …` appears as `Client_Nanoflow:`, +not `Sudoku:`. Grep for `Client_Nanoflow` (or the message text); a node-name filter +built for microflows drops nanoflow lines. Note `LOG DEBUG` from a nanoflow is dropped +server-side (browser console only) — only `INFO`/`WARNING`/`ERROR` reach `runtime.log`. + +## Important behaviour + +- **A breakpoint pauses whoever hits it — the browser included.** The triggering + request hangs until `continue` or `disable`. Don't leave a paused session idle. +- **Always `mxcli debug disable` when done.** `run --local --debug` does this on + shutdown; a manual `enable` is your responsibility. +- **Use the same `-p` for every command.** The session token and breakpoint record + live under `/.mxcli/`. diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md index 5e311a50e..49a106cd7 100644 --- a/docs-site/src/tools/run-local.md +++ b/docs-site/src/tools/run-local.md @@ -78,6 +78,48 @@ so structural changes need a restart; behavioural changes do not. | `--screenshot-url` | app root | Page to shoot: full URL, or a path relative to the app root (e.g. `/p/customers`). Repeat for a multi-page set. | | `--screenshot-user` / `--screenshot-password` | — | Log in once (Mendix form auth) and reuse the session, so pages behind login render authenticated | | `--runtime-log` | `/.mxcli/runtime.log` | Runtime log file — JVM stdout/stderr **and** the application log (server stack traces + microflow `LOG` output); `-` disables | +| `--debug` | off | Enable the microflow debugger at boot; then use [`mxcli debug`](debug-microflows.md) from another terminal. Behaviour-neutral until a breakpoint is set | +| `--debug-pass` | `mxdebug` | Debugger password when `--debug` is set | +| `--metrics` | off | Register a Prometheus meter registry; metrics served at `http://127.0.0.1:/prometheus` | +| `--trace` | off | Enable OpenTelemetry tracing (bundled agent → runtime log) with default span filters | +| `--trace-service` | `.mpr` name | `OTEL_SERVICE_NAME` under `--trace` | +| `--runtime-setting Key=Value` | — | Merge an extra runtime setting into the boot config (Value parsed as JSON when possible); repeatable | + +## Metrics and OpenTelemetry + +`--metrics` registers a **Prometheus** meter registry at boot, so +`http://127.0.0.1:8090/prometheus` (the admin port) serves the runtime's Micrometer +metrics (`connectionbus_*`, `handler_requests_total`, `sessions_*`, `taskqueue_*`, …). +For another registry, use `--runtime-setting`: + +```bash +mxcli run --local -p app.mpr --metrics +mxcli run --local -p app.mpr --runtime-setting 'Metrics.Registries=[{"type":"otlp"}]' +``` + +These are flags rather than a post-boot call because the admin `update_configuration` +action **replaces** the whole config (no read-back), so a separate call would wipe the +DB/BasePath settings — `--metrics`/`--runtime-setting` merge into mxcli's single boot +`update_configuration`. + +**Traces (`--trace`):** attaches the bundled OpenTelemetry Java agent to the runtime +JVM (console exporter → `runtime.log`) and applies default span filters — unfiltered +per-activity tracing is ~10× slower, so `--trace` ships +`OpenTelemetry._RuntimeSpanFilters=["CreateOrChangeVariable","Loop","Gateway","RetrieveFromCache"]` +(override via `--runtime-setting`). + +```bash +mxcli run --local -p app.mpr --trace +tail -f .mxcli/runtime.log # microflow spans: mx.microflow.name / mx.microflow.depth +``` + +To export to a collector instead of the console, set the OTEL env yourself before +running (`--trace` won't override an exporter you've set): + +```bash +export OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 +mxcli run --local -p app.mpr --trace +``` ## Debugging a server-side error diff --git a/docs/11-proposals/PROPOSAL_microflow_debugger.md b/docs/11-proposals/PROPOSAL_microflow_debugger.md new file mode 100644 index 000000000..1f2acf3a7 --- /dev/null +++ b/docs/11-proposals/PROPOSAL_microflow_debugger.md @@ -0,0 +1,189 @@ +--- +title: mxcli microflow debugger — breakpoints by name against a running runtime +status: proposed +date: 2026-07-26 +--- + +# Proposal: `mxcli debug` — the Mendix microflow debugger, breakpoints by name + +**Status:** Proposed +**Date:** 2026-07-26 +**Author:** Generated with Claude Code + +Add first-class microflow-debugger support to mxcli: set breakpoints, inspect paused +microflows and their variables, and step/continue — all **by microflow (and activity) +name**, against a runtime started by `mxcli run --local` (or any reachable runtime). + +Origin: the `ako/mxcli-sudoku` findings + README documented the full debugger protocol +and a hand-rolled `scripts/mfdebug.sh` wrapper, and concluded that **mxcli is the only +tool positioned to offer breakpoints by name** — it already owns both halves: the admin +password + app URL (from `run --local`) and the activity model GUIDs (from the `.mpr`). +This proposal turns that wrapper into a supported command. + +## Problem + +Debugging a server-side microflow today means either opening Studio Pro (defeats the +headless `run --local` loop) or driving the runtime's debugger APIs by hand. The latter +is awkward because: + +- It spans **two** APIs with **two** auth schemes (see *Background*). +- A breakpoint's `object_id` is the **model GUID of an activity**, stored in the `.mpr` + as a little-endian .NET GUID — "nothing in the runtime will tell you what it is." The + sudoku wrapper shells out to `mxcli bson dump` + a Python GUID conversion to find it. +- `params` is mandatory on every debugger call even when empty; the session token must be + threaded through; and a live breakpoint **pauses whoever hits it (the browser too)**, + so a forgotten `disable` hangs the app. + +mxcli already parses microflows (`sdk/mpr/parser_microflow.go` → each activity carries a +`$ID` via `extractBsonID`) and already owns the admin connection (`docker.CallM2EE`), so it +can hide all of this behind names. + +## Background: the runtime debugger protocol (as verified by sudoku) + +**Plane 1 — M2EE admin (`:8090`, `X-M2EE-Authentication: base64(adminPass)`)** toggles +debugger state only: + +| Action | Params | Purpose | +|---|---|---| +| `enable_debugger` | `{"password": ""}` | turn on; **password is required** | +| `disable_debugger` | — | turn off | +| `get_debugger_status` | — | `{enabled, client_connected, number_of_paused_microflows}` | + +**Plane 2 — app debugger endpoint (`/debugger/`, +`X-Debugger-Authentication: base64(debugPass)`)** drives breakpoints. Body is always +`{action, session_token, params}` and **`params` is mandatory even when `{}`**: + +| Action | Params | Notes | +|---|---|---| +| `start_session` | `{breakpoints: []}` | **no token**; response has `result.session_token` | +| `add_breakpoint` | `{microflow_name, object_id, condition?}` | `object_id` = activity model GUID | +| `remove_breakpoint` | `{object_id}` | | +| `get_paused_microflows` | `{}` | flows + all in-scope variables | +| `get_object` | `{debug_id, variable_name}` | inspect one variable | +| `step_over` / `step_into` / `step_out` | `{debug_id}` | | +| `continue` / `continue_all` | `{}` | resume | + +Auth quirk (documented): the debugger endpoint accepts only +`X-Debugger-Authentication: base64(pass)` — raw password, `Basic`, and `Bearer` all 401, +and the 401 body is `{}` with no `WWW-Authenticate`. + +## Goals + +- Set/clear breakpoints, list paused microflows with variables, inspect a variable, step, + and continue — **by microflow name** (and activity name/index), never a raw GUID. +- Reuse `run --local`'s admin password and app URL automatically (zero flags in the + common case). +- Resolve activity GUIDs from mxcli's own model — no `bson dump`/Python detour. +- Fail safe: make it hard to leave the debugger enabled (auto-disable, clear warnings). + +## Non-goals (this cut) + +- A DAP (Debug Adapter Protocol) bridge for VS Code / the vscode-mdl extension (attractive + follow-up — see *Future work*). +- Nanoflow (client-side) debugging — this is the **microflow** (server) debugger. +- Conditional-breakpoint expression validation beyond passing `condition` through. +- Replacing Studio Pro's debugger UI; this is the headless/CLI counterpart. + +## Design + +### Command surface + +``` +mxcli debug status # get_debugger_status (enabled? paused? client connected?) +mxcli debug enable [--debug-pass] # enable_debugger + start_session, cache session token +mxcli debug break Module.Flow [--activity ] [--if ''] +mxcli debug breaks # list active breakpoints (name → activity → GUID) +mxcli debug unbreak Module.Flow [--activity …] # remove_breakpoint +mxcli debug paused # get_paused_microflows: flow, current activity, vars +mxcli debug inspect [--flow …] # get_object for a paused flow +mxcli debug step [over|into|out] # default: over +mxcli debug continue [--all] +mxcli debug disable # disable_debugger (also run on Ctrl-C / defer) +``` + +- **Name → GUID resolution.** `break Module.Flow --activity ACT_Name` looks up the + microflow in the model, finds the activity by caption/name (or `#index` in flow order), + and uses its `$ID` as `object_id`. Bare `break Module.Flow` (no `--activity`) breaks on + the flow's **start** activity. `mxcli debug breaks` prints the reverse map so a user sees + what a GUID corresponds to. +- **Session/token handling** is internal: `enable` (or the first `break`) does + `start_session`, caches the token under the project (e.g. `.mxcli/debug-session.token`), + and every subsequent call threads it. `disable` clears it. +- **Human + JSON output.** `paused`/`inspect` render a readable tree by default and + `--format json` for scripting/agents. + +### Reusing `run --local` + +`run --local` already binds the admin API at `mxcli-local-dev` on `:8090` and serves the +app on `:8080`. `mxcli debug` resolves the same defaults (admin pass, app URL, project) so +the common case is flagless. Overridable with `--admin-url/--app-url/--admin-pass/--debug-pass/-p` +and env vars mirroring the sudoku wrapper (`MXCLI_ADMIN_PASS`, `MXCLI_DEBUG_PASS`, …). + +Optional convenience: `mxcli run --local --debug[=pass]` enables the debugger at boot so a +fresh session is immediately debuggable. + +### Code shape + +- **Admin plane**: reuse `docker.CallM2EE` for `enable_debugger`/`disable_debugger`/ + `get_debugger_status` (it already speaks `{action, params}` + `X-M2EE-Authentication`). +- **Debugger plane**: a small new client `docker.DebuggerClient` (or `mdl/.../debug`) — + POST `/debugger/` with `X-Debugger-Authentication`, the `{action, session_token, + params}` envelope, and typed responses. `params` always serialized (never omitted). +- **GUID resolver**: a helper over the existing microflow reader that maps + `Module.Flow[.activity]` → `object_id` and back. `extractBsonID` already yields the + activity `$ID`; confirm its string form matches the debugger's `object_id` + (little-endian .NET GUID → canonical UUID) and normalize if needed. +- **CLI**: `cmd/mxcli/cmd_debug.go` (Cobra), subcommands as above. + +## Safety + +- **Enabling the debugger changes runtime behaviour**: a breakpoint pauses *any* execution + that reaches it — including a browser request, which then hangs until `continue`. Every + command that can leave it enabled prints this once, and `mxcli debug` registers a + best-effort `disable` on interrupt. +- `status` surfaces `number_of_paused_microflows` so a hung app is diagnosable at a glance. +- The debugger password is a local dev secret (default `mxdebug`, overridable); document + not enabling the debugger on a shared/hosted runtime. + +## Implementation slices + +1. **Debugger client + admin toggles**: `DebuggerClient`, `enable/disable/status`, + `start_session` + token cache. Tests with an httptest stub of both planes. +2. **GUID resolver + `break`/`unbreak`/`breaks`**: name→activity→`object_id`; reverse map. + Tests over a fixture microflow (assert the resolved GUID equals the model `$ID`). +3. **Inspect/step/continue + `paused`**: response rendering (tree + `--format json`). +4. **`run --local` integration + safety**: default resolution from the warm loop, + `--debug` boot flag, interrupt-disable, warnings. +5. **Docs + skill**: a `debug-microflows` skill, `mxcli debug` help, docs-site page, and a + worked example mirroring the sudoku session (enable → break by name → paused → step → + continue → disable). + +## Testing + +- Unit: envelope/auth (both planes) against httptest stubs; token threading; `params` + always present; GUID resolver name↔GUID over a fixture. +- Integration (`-tags integration`): against a `run --local` runtime — enable, break on a + known activity by name, drive a request that hits it, assert `get_paused_microflows` + shows the flow + variables, step, continue, disable; assert `status` returns to + `enabled:false`. + +## Open questions + +1. **Activity naming** — break by activity **caption** (user-facing, may be blank/dup) vs + a stable **`#index`** in flow order vs both? Proposal offers both, caption first. +2. **GUID form** — verify `extractBsonID`'s output equals the debugger's expected + `object_id` string across versions; if the runtime wants a different casing/format, + normalize in the resolver. +3. **Interactive REPL** — ship one-shot subcommands first (scriptable, agent-friendly), or + also an interactive `mxcli debug repl` stepping loop? Proposal: subcommands first. +4. **Default-on with `run --local`?** — leave the debugger opt-in (`--debug`), or always + enable it locally? Proposal: opt-in (it changes runtime behaviour). + +## Future work + +- **DAP bridge** so the vscode-mdl extension (and any DAP client) can set breakpoints in + `.mdl`/microflow views and step with a real debugger UI — mxcli becomes the debug + adapter over this same protocol. +- Nanoflow (client-side) debugging. +- `mxcli debug watch ` and conditional-breakpoint expression checking via the + existing `exprcheck` package. diff --git a/mdl-examples/doctype-tests/15b-fragment-slots-examples.mdl b/mdl-examples/doctype-tests/15b-fragment-slots-examples.mdl index d80f054a6..c0a90ea6f 100644 --- a/mdl-examples/doctype-tests/15b-fragment-slots-examples.mdl +++ b/mdl-examples/doctype-tests/15b-fragment-slots-examples.mdl @@ -41,7 +41,11 @@ create persistent entity SlotTest.Order ( * widgets are spliced in. */ define fragment Card as { - container cardWrap (class: 'card', designproperties: ['Card style': on]) { + -- Card styling via the Atlas `card` class (works on every Mendix version). The + -- v3 `'Card style'` design property is intentionally NOT used here: it doesn't + -- exist in the 10.x Atlas (CE6083) and this example is about content slots, not + -- design properties — see 12-styling-examples.mdl (gated 11.0+) for those. + container cardWrap (class: 'card') { container cardBody (class: 'card-body') { slot content }