Add spark-gluten-clickhouse entry (Spark + Gluten with the CH backend)#861
Open
alexey-milovidov wants to merge 8 commits into
Open
Add spark-gluten-clickhouse entry (Spark + Gluten with the CH backend)#861alexey-milovidov wants to merge 8 commits into
alexey-milovidov wants to merge 8 commits into
Conversation
Adds a spark-gluten-clickhouse/ entry that runs the ClickBench query
suite against Apache Spark with Apache Gluten configured to use the
ClickHouse backend ('ch'), in which Gluten loads libch.so (a fork of
ClickHouse v23.1) into the Spark executor JVM and runs the columnar
plan natively through it.
Compared with spark-gluten/ (which uses the Velox backend), this
exercises a meaningfully different execution path: Catalyst -> Substrait
-> ClickHouse engine, rather than Catalyst -> Substrait -> Velox.
No pre-built bundle is published for the CH backend (the Apache Gluten
release tarball ships only the Velox bundle), so benchmark.sh builds
both libch.so and the Gluten Spark plugin from source. The build is
memory-hungry; a 64 GB host (c6a.8xlarge or larger) is recommended.
Queries use ClickHouse-style regex backreferences (\1) since the regex
evaluation runs inside libch.so, as anticipated in the spark-gluten/
README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pinned Kyligence/ClickHouse fork now rejects Clang < 19 in cmake/tools.cmake, so installing Clang 18 fails the configure step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 22, 2026 00:31 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
The run of Logs:
|
The entry was written in the old self-contained benchmark.sh style, which predates lib/benchmark-common.sh. The c6a.4xlarge run (PR #861) failed immediately at `../download-hits-parquet-single: No such file or directory` — the download helpers moved into lib/ and are now invoked by the harness, not by each entry. Refactor to match the sibling spark-gluten/ (Velox) and spark-velox/ entries: - benchmark.sh: thin wrapper that sets BENCH_* env and exec's ../lib/benchmark-common.sh (RESTARTABLE=no, concurrent QPS skipped per #946). - install: the from-source build (libch.so + Gluten CH plugin), moved out of benchmark.sh. Fixes verified against Gluten v1.4.0's own build_clickhouse.sh: * clone apache/incubator-gluten (canonical repo, matches docs) at v1.4.0; * read CH_ORG as well as CH_BRANCH from cpp-ch/clickhouse.version; * the build lands in cpp-ch/build_ch/ (not build/ as the old path assumed and the prose docs still say) — glob for libch.so instead of hardcoding; * set MAVEN_OPTS per the Gluten docs so the Maven build doesn't OOM; * idempotent (skips clone/build when artifacts already present). - load/query/start/stop/check/data-size: standard harness scripts. query runs Spark under JDK 17 (Gluten is built under JDK 8). - query.py: single run per invocation, timing as the last stderr line, exit 1 on failure — the harness contract, replacing the old 3-try/"Time:" loop. - Drop run.sh (the harness drives the query loop). - .gitignore: __pycache__/ and the from-source build artifacts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
alexey-milovidov
had a problem deploying
to
benchmark-approval
July 22, 2026 21:21 — with
GitHub Actions
Error
GitHub Actions
Error
alexey-milovidov
added
the
machine:c6a.metal
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 22, 2026 21:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
The run of Logs:
|
…JNI/AWT
The c6a.metal re-run got past the download fix and into the ClickHouse
compile, then failed at the extern-local-engine cmake configure:
-- Could NOT find JNI (missing: AWT)
CMake Error: ... JAVA_AWT_INCLUDE_PATH (ADVANCED) ... set to NOTFOUND
ClickHouse's extern-local-engine links against JNI including AWT, but the
install used openjdk-17-jdk-headless, which ships neither jawt.h nor
libjawt.so (verified against the Ubuntu noble package file lists). Switch to
the full openjdk-17-jdk (ships jawt.h; depends on the non-headless JRE that
ships libjawt.so) and export JAVA_HOME to it so cmake's FindJNI probes the
right, AWT-complete JDK deterministically. JDK 8 stays headless — it is only
the Maven build, which doesn't need AWT.
Also corrects the README: the outer wrapper cmakes into cpp-ch/build_ch, but
the inner ClickHouse cmake builds libch.so under cpp-ch/build/ (install globs
for it under cpp-ch/ either way).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 22, 2026 22:32 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
The run of Logs:
|
The c6a.metal run got all the way through the libch.so compile (all 12976
ninja steps linked — the JDK/AWT fix worked) and built every Gluten Maven
module except `backends-clickhouse`, which failed its `enforce-delta-profile`
enforcer rule:
"-P delta" must be set when building Gluten with ClickHouse backend.
Profile "delta" is not activated.
Add `-Pdelta` to the `mvn` invocation. The `spark-3.5` profile already pins
the matching `delta.version` (3.2.0, package `delta-spark`), so no extra
version flag is needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 23, 2026 02:28 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
The run of Logs:
|
Run 4 (with -Pdelta) got all the way through the libch.so compile, the Maven
backends-clickhouse build, and data loading (~14.8 GB), then failed at query
time when the driver JVM lazily loaded the native engine:
java.lang.UnsatisfiedLinkError: .../libch.so:
cannot allocate memory in static TLS block
at org.apache.gluten.backendsapi.clickhouse.CHListenerApi.initialize
... CHBackend.onDriverStart ... GlutenDriverPlugin.init
libch.so carries initial-exec-model TLS (from its statically linked deps), and
the static TLS block is sized at process startup, so a lazy `dlopen` from the
already-running JVM has no room left. Gluten's own docs work around this with
`spark.executorEnv.LD_PRELOAD=<libch.so>`, but in local[*] mode the driver JVM
*is* the executor and is launched before any Spark config is read, so
executorEnv never applies.
Preload it via the JVM's environment instead: set `LD_PRELOAD` in query.py
before `getOrCreate()`. This does not affect the already-started Python
process, but pyspark copies os.environ into the JVM it spawns, so the JVM
preloads libch.so at startup while the static TLS block still has room;
Gluten's later `System.load()` then reuses the already-loaded library. Also set
`spark.executorEnv.LD_PRELOAD` for correctness under a future cluster-mode run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
alexey-milovidov
added
machine:c6a.metal
alexey-milovidov
requested a deployment
to
benchmark-approval
July 23, 2026 07:53 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
spark-gluten-clickhouse/entry that runs Apache Spark with Apache Gluten configured to use the ClickHouse backend (spark.gluten.sql.columnar.backend.lib=ch). Gluten loadslibch.so(a fork of ClickHouse v23.1) into the Spark executor JVM and runs the columnar physical plan natively through it.spark-gluten/(Velox backend) and the proposedspark-velox/(Add spark-velox entry (Spark + Velox via Apache Gluten) #858) — this entry exercises a meaningfully different execution path: Catalyst → Substrait → ClickHouse engine, rather than Catalyst → Substrait → Velox.Build
No pre-built bundle is published for the CH backend (Apache Gluten v1.4.0 ships only the Velox bundle, and Maven Central has nothing).
benchmark.shtherefore builds two things from source:libch.so— built from Kyligence/ClickHouse at the branch pinned ingluten/cpp-ch/clickhouse.version(currentlyrebase_ch/20250326). Uses Clang 18 / cmake / ninja.-Pbackends-clickhouse,spark-3.5,scala-2.12under JDK 8.Limitations
libch.socompile is essentially a ClickHouse build and is RAM-hungry; Gluten's docs recommend ≥64 GB. On c6a.4xlarge (32 GB) it may OOM —c6a.8xlargeor larger is recommended for a clean run, hence the default machine label inbenchmark.sh.Notes
\1) rather than Spark's$1, because regex evaluation runs insidelibch.so. This was anticipated in the existingspark-gluten/README.mdand Gluten issue #7545.Test plan
benchmark.shclones gluten + Kyligence/ClickHouse, buildslibch.soand the Spark plugin, runs all 43 queries, and writesresults/<machine>.json.🤖 Generated with Claude Code