From d39820ada7b7e1079caee06c244d3babf3783b22 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 23 Jul 2026 20:17:23 +0000 Subject: [PATCH 01/39] =?UTF-8?q?docs(warm-loop):=20add=20slice=205=20?= =?UTF-8?q?=E2=80=94=20multi-project=20solutions=20in=20one=20container?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the multi-project-solution orchestration as slice 5: several apps of one solution running side by side in a single container, each its own runtime + database with auto-allocated ports, all registered under one --hub-solution and grouped in the hub overview, with sibling URLs wired from the hub-assigned subdomains via a mxcli.solution.yaml manifest driven by `mxcli run --solution`. It's an orchestration layer over shipped primitives (per-project DB/deploy isolation, port-triple flags, the slice-4 hub's solution grouping) — no new runtime or hub mechanics. Also marks slices 3 & 4 verified-live and merged (PR #11). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_mxcli_dev_warm_loop.md | 48 +++++++++++++++++-- 1 file changed, 45 insertions(+), 3 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md index edd18c286..f9912c088 100644 --- a/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md +++ b/docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md @@ -599,19 +599,61 @@ builds on the previous. |---|-------|----------|------------|-------------| | 1 | **Warm local loop** — shipped as `mxcli run --local [--watch]` (serve daemon + M2EE admin client + `restartRequired` branching; + client bundling & Playwright screenshots) | Docker-free ~1 s edit→test loop, locally | nothing new | ✅ **shipped** | | 2 | **Provisioning** — `run --local --ensure-db` (DB) + `run --local --setup` (non-blocking bring-up) + `mxcli init` SessionStart hook + bootstrap prompt template | a fresh Claude Code Web session comes up testable; iPad-native start | slice 1 | ✅ **shipped** | -| 3 | **Single-app external preview** — `mxcli run --hub ` (embedded chisel client, proxy honouring `NO_PROXY`, `ApplicationRootUrl` boot wiring) + `mxcli tunnel-hub` (embedded chisel server, autocert, single-443 `--backend`) | a shareable live preview URL (the iPad two-container flow) | slice 1 | ✅ **shipped** (2026-07-23) — code + in-process tunnel test + local end-to-end boot; Mendix renders through the hub's Host-rewriting backend, no shim needed. External E2E against the Scaleway hub is the remaining confirmation | -| 4 | **Tunnel hub** — `mxcli tunnel-hub` (multi-tenant) + `mxcli run --hub` registration + admin overview | many dev containers behind one ingress; per-preview subdomains across projects/solutions/branches/worktrees; sortable overview with availability | slice 3 | ✅ **built** (2026-07-23) — registry + registration API + single-443 host-routing front with per-subdomain autocert + sortable admin page + client registration; unit + in-process tunnel integration tests. External E2E against the Scaleway hub (wildcard DNS) is the remaining confirmation; deeper multi-tenant auth (per-container tokens, admin auth) is a follow-on | +| 3 | **Single-app external preview** — `mxcli run --hub ` (embedded chisel client, proxy honouring `NO_PROXY`, `ApplicationRootUrl` boot wiring) + `mxcli tunnel-hub` (embedded chisel server, autocert, single-443 `--backend`) | a shareable live preview URL (the iPad two-container flow) | slice 1 | ✅ **shipped** (2026-07-23) — code + in-process tunnel test + local end-to-end boot; Mendix renders through the hub's Host-rewriting backend, no shim needed. Verified live against a self-run VPS hub and merged (PR #11) | +| 4 | **Tunnel hub** — `mxcli tunnel-hub` (multi-tenant) + `mxcli run --hub` registration + admin overview | many dev containers behind one ingress; per-preview subdomains across projects/solutions/branches/worktrees; sortable overview with availability | slice 3 | ✅ **shipped** (2026-07-23) — registry + registration API + single-443 host-routing front with per-subdomain autocert + sortable admin page + client registration; unit + in-process tunnel integration tests; verified live against a self-run VPS hub (wildcard DNS, per-subdomain autocert) and merged (PR #11). Deeper multi-tenant auth (per-container tokens, admin auth) is a follow-on | +| 5 | **Multi-project solutions** — `mxcli run --solution` + a `mxcli.solution.yaml` manifest | boot several apps of one solution in a single container — each its own runtime + database, ports auto-allocated, all registered under one `--hub-solution` with sibling URLs wired from the hub | slice 4 | proposed — an orchestration layer over shipped primitives; see § Slice 5 below | Recommended sequencing: **1 → 2** delivers the complete solo dev experience (Scenario A plus provisioning) with no external moving parts, and can ship first. **3** adds external preview for a single app. **4** is the scale-out and deserves its own design/security review — build it only once 1–3 are proven. Slices 1–2 stand alone if the tunnel/hub work -is deferred indefinitely. +is deferred indefinitely. **5** is a pure orchestration layer on top of 4, for running the +several apps of one solution together in a single container. Cross-cut: the instant `mxcli check` gate (`PROPOSAL_check_mxbuild_gap_heuristics.md`) should front slice 1's build on every iteration, so most errors never reach even the ~0.8 s warm build. +## Slice 5: multi-project solutions in one container + +A multi-app **solution** — several Mendix apps that integrate (e.g. a web front end, an +API app, an admin app) — can already run **side by side in one container** with today's +primitives. Each `mxcli run --local` / `run --hub`: + +- takes its own `-p ` (projects live in subdirs of one repo, e.g. `apps/web/Web.mpr`); +- **derives its own database name** from the project file (`Web.mpr` → `web`), so N apps + share one Postgres with distinct databases (`--ensure-db` creates each); +- writes its own `deployment/` next to the `.mpr` (no collision); +- accepts `--app-port` / `--admin-port` / `--serve-port`, so distinct port triples let N + runtimes coexist (the stale-port guard is per-triple); +- registers under one `--hub-solution`, which the multi-tenant hub (slice 4) already + **groups** in the overview. + +So the capability exists; what's missing is **orchestration ergonomics** — today you +hand-launch N processes with hand-picked ports. Slice 5 makes it one command: + +- **`mxcli.solution.yaml`** — a manifest at the repo root listing the projects (each a + path to its `.mpr` and an optional app name), the shared solution name + hub, and any + inter-app constant wiring (which constant on app A points at app B). +- **`mxcli run --solution`** — reads the manifest, **auto-allocates** the port triples + (`8080/8090/6543`, `8081/8091/6544`, …), ensures each database, boots each app (in + parallel to hide cold-start), and — with `--hub` — registers each under the one + `--hub-solution`. +- **Sibling URL wiring** — the hub hands each app a stable subdomain, so a constant like + `Web.ApiBaseUrl` resolves to `https://api-.` (for the browser / other + origins) or `http://127.0.0.1:` (intra-container); the manifest declares the + linkage and `run --solution` fills in the assigned ports/subdomains at boot. + +**Constraints** (see the coexistence analysis): the binding limit is **memory** — each app +is a JVM runtime plus an in-memory `mxbuild --serve`, so a web container realistically +holds a handful of small apps; larger solutions want a bigger container. And **inter-app +integration** must be wired as above. Neither is a blocker — sizing + config, not new +mechanics. + +This adds **no new runtime or hub machinery** — it is an orchestration layer over shipped +primitives (`run --local`/`--hub`, per-project DB/deploy isolation, the hub's solution +grouping), which is why it slots cleanly after slice 4. + ## Version Compatibility - `mxbuild --serve` and `reload_model`: **≥ 11.6.3** (verified on 11.6.3 and From 19ecd3241d996fdb2d83ea18c5036a16763d561a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 12:20:38 +0000 Subject: [PATCH 02/39] fix(run): reap child process groups so ports free on Ctrl-C MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The warm loop's three long-lived children — mxbuild --serve, the runtime JVM, and the rollup web-client bundler — were each tracked by a single PID, and every Stop() signalled only that direct child. Because mxbuild is a shell-script wrapper that spawns a Temurin JVM grandchild, SIGTERM to the wrapper left the JVM orphaned, still holding its port (6543/8080/8090). The next `mxcli run --local` then refused to start with "port already in use". Start each child in its own process group (Setpgid) and have Stop() signal the whole group (negative-PID SIGTERM, then SIGKILL after the grace period), so wrapper grandchildren are reaped too. Cross-platform via a Windows no-op stub (no POSIX process groups there; the warm loop is a Linux-devcontainer feature). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/docker/localboot.go | 5 +- cmd/mxcli/docker/mxserve.go | 5 +- cmd/mxcli/docker/procgroup_unix.go | 48 ++++++++++++++++ cmd/mxcli/docker/procgroup_unix_test.go | 76 +++++++++++++++++++++++++ cmd/mxcli/docker/procgroup_windows.go | 35 ++++++++++++ cmd/mxcli/docker/webclient_watch.go | 5 +- 6 files changed, 168 insertions(+), 6 deletions(-) create mode 100644 cmd/mxcli/docker/procgroup_unix.go create mode 100644 cmd/mxcli/docker/procgroup_unix_test.go create mode 100644 cmd/mxcli/docker/procgroup_windows.go diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index 0df5974cd..cabedc567 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -264,6 +264,7 @@ func (rt *LocalRuntime) spawnAndConfigure() error { cmd.Dir = rt.opts.runtimeDir() cmd.Env = localRuntimeEnv(rt.opts) PrepareMxCommand(cmd) // FreeType LD_PRELOAD workaround, layered on cmd.Env + setProcessGroup(cmd) // reap any JVM child on Stop so the port is freed log := &syncBuffer{} cmd.Stdout = log cmd.Stderr = log @@ -353,13 +354,13 @@ func (rt *LocalRuntime) stopProcess() error { if rt.cmd == nil || rt.cmd.Process == nil { return nil } - _ = rt.cmd.Process.Signal(syscall.SIGTERM) + _ = signalProcessGroup(rt.cmd.Process, syscall.SIGTERM) done := make(chan error, 1) go func() { done <- rt.cmd.Wait() }() select { case <-done: case <-time.After(8 * time.Second): - _ = rt.cmd.Process.Kill() + _ = killProcessGroup(rt.cmd.Process) <-done } rt.cmd = nil diff --git a/cmd/mxcli/docker/mxserve.go b/cmd/mxcli/docker/mxserve.go index 23f235e83..b19557eda 100644 --- a/cmd/mxcli/docker/mxserve.go +++ b/cmd/mxcli/docker/mxserve.go @@ -139,6 +139,7 @@ func StartServe(opts ServeOptions) (*ServeServer, error) { fmt.Sprintf("--java-exe-path=%s", javaExe), ) PrepareMxCommand(cmd) // FreeType LD_PRELOAD workaround + setProcessGroup(cmd) // reap the JVM grandchild on Stop (mxbuild is a wrapper) log := &syncBuffer{} cmd.Stdout = log cmd.Stderr = log @@ -227,13 +228,13 @@ func (s *ServeServer) Stop() error { if s.cmd == nil || s.cmd.Process == nil { return nil } - _ = s.cmd.Process.Signal(syscall.SIGTERM) + _ = signalProcessGroup(s.cmd.Process, syscall.SIGTERM) done := make(chan error, 1) go func() { done <- s.cmd.Wait() }() select { case <-done: case <-time.After(5 * time.Second): - _ = s.cmd.Process.Kill() + _ = killProcessGroup(s.cmd.Process) <-done } return nil diff --git a/cmd/mxcli/docker/procgroup_unix.go b/cmd/mxcli/docker/procgroup_unix.go new file mode 100644 index 000000000..f15bc72c1 --- /dev/null +++ b/cmd/mxcli/docker/procgroup_unix.go @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build !windows + +package docker + +import ( + "os" + "os/exec" + "syscall" +) + +// procgroup_unix.go isolates each long-lived child of the warm loop (mxbuild +// --serve, the runtime JVM, the rollup bundler) into its own process group, so +// teardown can signal the WHOLE group rather than just the direct child. This +// matters because mxbuild is a shell-script wrapper that spawns a Temurin JVM +// grandchild: signalling only the wrapper PID leaves the JVM orphaned, still +// holding its port (6543/8080/8090), so the next `mxcli run --local` refuses to +// start ("port already in use"). Killing the group reaps the grandchild too. + +// setProcessGroup makes cmd the leader of a new process group (so its PGID equals +// its PID once started). Call before cmd.Start(). It preserves any SysProcAttr a +// caller already set (e.g. none of ours do today, but PrepareMxCommand might). +func setProcessGroup(cmd *exec.Cmd) { + if cmd.SysProcAttr == nil { + cmd.SysProcAttr = &syscall.SysProcAttr{} + } + cmd.SysProcAttr.Setpgid = true +} + +// signalProcessGroup sends sig to the whole process group led by p (started via +// setProcessGroup). A negative PID targets the group. If the group can't be +// resolved (p was not made a leader, or it already exited), it falls back to +// signalling p alone so behaviour is never worse than before. +func signalProcessGroup(p *os.Process, sig syscall.Signal) error { + if p == nil { + return nil + } + if err := syscall.Kill(-p.Pid, sig); err != nil { + return p.Signal(sig) + } + return nil +} + +// killProcessGroup force-terminates (SIGKILL) the whole group led by p. +func killProcessGroup(p *os.Process) error { + return signalProcessGroup(p, syscall.SIGKILL) +} diff --git a/cmd/mxcli/docker/procgroup_unix_test.go b/cmd/mxcli/docker/procgroup_unix_test.go new file mode 100644 index 000000000..5a0c9a6d8 --- /dev/null +++ b/cmd/mxcli/docker/procgroup_unix_test.go @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build !windows + +package docker + +import ( + "bufio" + "os/exec" + "strconv" + "strings" + "syscall" + "testing" + "time" +) + +// TestProcessGroup_ReapsGrandchild proves setProcessGroup + killProcessGroup +// terminate a wrapper's grandchild — the real-world case where mxbuild is a +// shell-script that spawns a Temurin JVM. A single-PID signal (the old Stop +// behaviour) would leave that grandchild orphaned, still holding its port, so the +// next `mxcli run --local` refuses to start. +func TestProcessGroup_ReapsGrandchild(t *testing.T) { + // sh (the "wrapper") backgrounds a grandchild that sleeps, prints its PID, then + // waits so the whole group stays alive until we kill it. + cmd := exec.Command("sh", "-c", "sleep 60 & echo $!; wait") + setProcessGroup(cmd) + stdout, err := cmd.StdoutPipe() + if err != nil { + t.Fatalf("stdout pipe: %v", err) + } + if err := cmd.Start(); err != nil { + t.Fatalf("start: %v", err) + } + + sc := bufio.NewScanner(stdout) + if !sc.Scan() { + t.Fatalf("did not read grandchild pid: %v", sc.Err()) + } + gpid, err := strconv.Atoi(strings.TrimSpace(sc.Text())) + if err != nil { + t.Fatalf("parse grandchild pid %q: %v", sc.Text(), err) + } + if err := syscall.Kill(gpid, 0); err != nil { + t.Fatalf("grandchild %d should be alive before the kill: %v", gpid, err) + } + + // Kill the whole group, then reap the leader. + if err := killProcessGroup(cmd.Process); err != nil { + t.Fatalf("killProcessGroup: %v", err) + } + _ = cmd.Wait() + + // The grandchild must be gone. Poll briefly: after SIGKILL it is a zombie until + // its reparented init reaps it, during which Kill(pid,0) still returns nil. + deadline := time.Now().Add(3 * time.Second) + for { + if err := syscall.Kill(gpid, 0); err != nil { + return // gone — success + } + if time.Now().After(deadline) { + _ = syscall.Kill(gpid, syscall.SIGKILL) // don't leak on failure + t.Fatalf("grandchild %d survived the group kill (orphaned)", gpid) + } + time.Sleep(50 * time.Millisecond) + } +} + +// TestSetProcessGroup_SetsFlag is a cheap guard that setProcessGroup requests a +// new process group (so the negative-PID kill above has a group to target). +func TestSetProcessGroup_SetsFlag(t *testing.T) { + cmd := exec.Command("true") + setProcessGroup(cmd) + if cmd.SysProcAttr == nil || !cmd.SysProcAttr.Setpgid { + t.Fatal("setProcessGroup should set SysProcAttr.Setpgid") + } +} diff --git a/cmd/mxcli/docker/procgroup_windows.go b/cmd/mxcli/docker/procgroup_windows.go new file mode 100644 index 000000000..73014576f --- /dev/null +++ b/cmd/mxcli/docker/procgroup_windows.go @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build windows + +package docker + +import ( + "os" + "os/exec" + "syscall" +) + +// procgroup_windows.go provides no-op / single-process fallbacks for the +// process-group helpers. Windows has no POSIX process groups; the warm loop is a +// Linux-devcontainer feature, so this only needs to keep the package compiling +// and preserve the prior single-PID signalling behaviour. + +// setProcessGroup is a no-op on Windows. +func setProcessGroup(cmd *exec.Cmd) {} + +// signalProcessGroup signals the process itself (no group semantics on Windows). +func signalProcessGroup(p *os.Process, sig syscall.Signal) error { + if p == nil { + return nil + } + return p.Signal(sig) +} + +// killProcessGroup force-terminates the process. +func killProcessGroup(p *os.Process) error { + if p == nil { + return nil + } + return p.Kill() +} diff --git a/cmd/mxcli/docker/webclient_watch.go b/cmd/mxcli/docker/webclient_watch.go index 9476f8903..87aa08999 100644 --- a/cmd/mxcli/docker/webclient_watch.go +++ b/cmd/mxcli/docker/webclient_watch.go @@ -148,6 +148,7 @@ func StartWebClientWatch(opts WebClientOptions) (*WebClientWatcher, error) { "CHOKIDAR_INTERVAL=300", "MX_WEB_CLIENT_BUILD_LOG="+filepath.Join(opts.DeployDir, "log", "web-client-build.log"), ) + setProcessGroup(cmd) // reap any node child on Stop so a rebuild can't be orphaned log := &syncBuffer{} stdout, err := cmd.StdoutPipe() if err != nil { @@ -276,13 +277,13 @@ func (wc *WebClientWatcher) Stop() error { if wc.cmd == nil || wc.cmd.Process == nil { return nil } - _ = wc.cmd.Process.Signal(syscall.SIGTERM) + _ = signalProcessGroup(wc.cmd.Process, syscall.SIGTERM) done := make(chan error, 1) go func() { done <- wc.cmd.Wait() }() select { case <-done: case <-time.After(5 * time.Second): - _ = wc.cmd.Process.Kill() + _ = killProcessGroup(wc.cmd.Process) <-done } return nil From 2d352e84f3d2369b76b22c64e104ef07a2a02e3e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 12:20:48 +0000 Subject: [PATCH 03/39] fix(run): re-serve the browser bundle after a structural --watch change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Under `run --local --watch`, a structural model change (new/removed page, nav or home change) went through a runtime restart, but the app then 404'd on /dist/index.js and booted only the