unified: Add Swift parser based on swift-syntax
#22195
+1,920
−42
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| /target | ||
| /swift/.build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 6.2.4 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_test") | ||
| load("@rules_swift//swift:swift.bzl", "swift_library") | ||
|
|
||
| package(default_visibility = ["//visibility:public"]) | ||
|
|
||
| # The pinned Swift version, shared with the local `cargo` build and | ||
| # `swift/Package.swift`. Referenced by the `swift.toolchain` extension in | ||
| # //:MODULE.bazel via `swift_version_file`. | ||
| exports_files([".swift-version"]) | ||
|
|
||
| # The Swift FFI shim: wraps swift-syntax and exposes a small C ABI | ||
| # (`ssr_parse_json` / `ssr_string_free`). Built with the hermetic Swift | ||
| # toolchain registered in //:MODULE.bazel; provides a `CcInfo` that the Rust | ||
| # targets below link against. | ||
| swift_library( | ||
| name = "swift_syntax_ffi", | ||
| srcs = ["swift/Sources/SwiftSyntaxFFI/SwiftSyntaxFFI.swift"], | ||
| module_name = "SwiftSyntaxFFI", | ||
| deps = [ | ||
| "@swift-syntax//:SwiftParser", | ||
| "@swift-syntax//:SwiftSyntax", | ||
| ], | ||
| ) | ||
|
|
||
| # Safe Rust bindings on top of the C ABI. `build.rs` is intentionally *not* | ||
| # wired up here (no `cargo_build_script`): under Bazel the Swift side is | ||
| # provided by `:swift_syntax_ffi`, whereas `build.rs` only exists to build the | ||
| # Swift shim in the local `cargo` workflow. | ||
| rust_library( | ||
| name = "swift_syntax_rs", | ||
| srcs = ["src/lib.rs"], | ||
| edition = "2024", | ||
| deps = [":swift_syntax_ffi"], | ||
| ) | ||
|
|
||
| rust_binary( | ||
| name = "swift-syntax-parse", | ||
| srcs = ["src/main.rs"], | ||
| # The Swift toolchain propagates a runfiles-relative RPATH to its runtime | ||
| # `.so`s via `swift_syntax_ffi`'s `CcInfo`, but (unlike `swift_binary`) | ||
| # `rust_binary` does not copy those libraries into runfiles. Add the | ||
| # toolchain files as `data` so the runtime is found at execution time. | ||
| # (rules_swift does not yet support statically linking the runtime.) | ||
| data = ["@swift_toolchain_ubuntu24.04//:files"], | ||
| edition = "2024", | ||
| deps = [":swift_syntax_rs"], | ||
| ) | ||
|
|
||
| rust_test( | ||
| name = "swift_syntax_rs_test", | ||
| size = "small", | ||
| crate = ":swift_syntax_rs", | ||
| data = ["@swift_toolchain_ubuntu24.04//:files"], | ||
| edition = "2024", | ||
| ) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| [package] | ||
| name = "swift-syntax-rs" | ||
| description = "Rust wrapper around the swift-syntax package for parsing Swift source" | ||
| version = "0.1.0" | ||
| authors = ["GitHub"] | ||
| edition = "2024" | ||
|
|
||
| [lib] | ||
| name = "swift_syntax_rs" | ||
| path = "src/lib.rs" | ||
|
|
||
| [[bin]] | ||
| name = "swift-syntax-parse" | ||
| path = "src/main.rs" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,180 @@ | ||
| # swift-syntax-rs | ||
|
|
||
| A Rust wrapper around the [swift-syntax](https://github.com/swiftlang/swift-syntax) | ||
| package, allowing Swift source code to be parsed from Rust. | ||
|
|
||
| Parsing is delegated to a small Swift shim (in [`swift/`](swift/)) that links | ||
| against `SwiftSyntax`/`SwiftParser` and exposes a tiny C ABI. The Rust crate | ||
| builds that shim (via `build.rs`) and provides safe bindings on top of it. | ||
|
|
||
| ## Output format | ||
|
|
||
| The emitted JSON tree preserves the AST's named structure. Every node has a | ||
| `kind` and a `range` with `start`/`end` positions (UTF-8 `offset` plus 1-based | ||
| `line`/`column`). Beyond that: | ||
|
|
||
| - **Tokens** carry `text`, `tokenKind`, and `leadingTrivia`/`trailingTrivia` | ||
| arrays of `{ kind, text }` pieces. | ||
| - **Layout nodes** (e.g. `functionDecl`) embed their children directly as | ||
| members keyed by the child's name in the parent (`name`, `signature`, | ||
| `body`, …), alongside `kind`/`range`. Absent optional children are omitted. | ||
| - **Collection nodes** (e.g. `codeBlockItemList`) are elided: a list-valued | ||
| field is simply a JSON array of its elements (e.g. `parameters`, `statements`). | ||
| This drops the collection node's own `kind`/`range`. | ||
|
|
||
| Only meaningful trivia is kept — the four comment kinds (`lineComment`, | ||
| `blockComment`, `docLineComment`, `docBlockComment`) and `unexpectedText` | ||
| (source the parser skipped). Whitespace trivia is dropped, since node ranges | ||
| already encode positions. | ||
|
|
||
| ### Example | ||
|
|
||
| Parsing `let x = 1 // c` produces the following (each `range` object is | ||
| abbreviated here as `…`): | ||
|
|
||
| ```jsonc | ||
| { | ||
| "kind": "sourceFile", | ||
| "range": …, | ||
| "statements": [ // collection node elided to an array | ||
| { | ||
| "kind": "codeBlockItem", | ||
| "range": …, | ||
| "item": { | ||
| "kind": "variableDecl", | ||
| "range": …, | ||
| "attributes": [], // empty collection → empty array | ||
| "modifiers": [], | ||
| "bindingSpecifier": { // a token | ||
| "kind": "token", | ||
| "text": "let", | ||
| "tokenKind": "keyword(SwiftSyntax.Keyword.let)", | ||
| "range": …, | ||
| "leadingTrivia": [], | ||
| "trailingTrivia": [] | ||
| }, | ||
| "bindings": [ | ||
| { | ||
| "kind": "patternBinding", | ||
| "range": …, | ||
| "pattern": { | ||
| "kind": "identifierPattern", | ||
| "range": …, | ||
| "identifier": { "kind": "token", "text": "x", "tokenKind": "identifier(\"x\")", "range": …, "leadingTrivia": [], "trailingTrivia": [] } | ||
| }, | ||
| "initializer": { | ||
| "kind": "initializerClause", | ||
| "range": …, | ||
| "equal": { "kind": "token", "text": "=", "tokenKind": "equal", "range": …, "leadingTrivia": [], "trailingTrivia": [] }, | ||
| "value": { | ||
| "kind": "integerLiteralExpr", | ||
| "range": …, | ||
| "literal": { | ||
| "kind": "token", | ||
| "text": "1", | ||
| "tokenKind": "integerLiteral(\"1\")", | ||
| "range": …, | ||
| "leadingTrivia": [], | ||
| "trailingTrivia": [ { "kind": "lineComment", "text": "// c" } ] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ], | ||
| "endOfFileToken": { "kind": "token", "text": "", "tokenKind": "endOfFile", "range": …, "leadingTrivia": [], "trailingTrivia": [] } | ||
| } | ||
| ``` | ||
|
|
||
| Note how `statements`, `bindings`, `attributes`, and `modifiers` are plain | ||
| arrays (their collection nodes are elided), layout children such as | ||
| `bindingSpecifier` and `initializer` are embedded by name, and the `// c` | ||
| comment rides along as `trailingTrivia` on the token it follows. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| The build does not depend on any particular version manager. You need: | ||
|
|
||
| - **Rust** — pinned to `1.88` by the repo-root [`rust-toolchain.toml`](../../rust-toolchain.toml), | ||
| which `rustup` picks up automatically. | ||
| - **Swift** — pinned to the version in [`.swift-version`](.swift-version) | ||
| (currently `6.2.4`), used to build `swift-syntax` `602.0.0`. Install it any way | ||
| you like — [swift.org](https://www.swift.org/install/) or | ||
| [swiftly](https://www.swift.org/swiftly/) (which reads `.swift-version`), or a | ||
| system package. Just make sure `swift` is on your `PATH` (or point `build.rs` | ||
| at it with the `SWIFT` environment variable). | ||
|
|
||
| On Debian/Ubuntu the Swift runtime also needs `libncurses6` (and related libs) | ||
| available on the system. | ||
|
|
||
| ## Building & testing | ||
|
|
||
| With `cargo` and `swift` on `PATH`: | ||
|
|
||
| ```sh | ||
| cargo build | ||
| cargo test | ||
| ``` | ||
|
|
||
| If your `swift`/`swiftc` are not on `PATH`, point the build at them explicitly: | ||
|
|
||
| ```sh | ||
| SWIFT=/path/to/swift SWIFTC=/path/to/swiftc cargo build | ||
| ``` | ||
|
Comment on lines
+119
to
+123
Contributor
There was a problem hiding this comment. Is it common to have something like this in rust builds that also compile code in other languages? Note that I'm not suggesting to change this.
Contributor
Author
There was a problem hiding this comment. I don't think we do it anywhere else, so it's a valid concern. Originally I had |
||
|
|
||
| The first build compiles `swift-syntax` and can take several minutes. | ||
|
|
||
| ## Building with Bazel (CI) | ||
|
|
||
| CI builds this crate hermetically with Bazel. A Swift toolchain is downloaded | ||
| from swift.org by the official `rules_swift` standalone toolchain extension | ||
| (wired up in the repo-root `MODULE.bazel`), `swift-syntax` is pulled from the | ||
| Bazel Central Registry, and the FFI shim is compiled as a `swift_library` that | ||
| the Rust targets link against. `build.rs` is not used under Bazel; it only | ||
| builds the Swift shim for the local `cargo` workflow. | ||
|
|
||
| ```sh | ||
| bazel build //unified/swift-syntax-rs:swift-syntax-parse | ||
| bazel test //unified/swift-syntax-rs:swift_syntax_rs_test | ||
| bazel run //unified/swift-syntax-rs:swift-syntax-parse < some.swift | ||
| ``` | ||
|
|
||
| Requirements: | ||
|
|
||
| - **`clang`** must be installed on the runner. `rules_swift` requires the Bazel | ||
| CC toolchain to use clang; the repo's `.bazelrc` already sets | ||
| `--repo_env=CC=clang`, so no extra flags are needed. | ||
| - The registered Swift toolchain currently targets **ubuntu24.04 / x86_64** | ||
| only (Bazel cannot auto-select between Linux distributions). Add more | ||
| platforms in `MODULE.bazel` (`swift.toolchain` + `register_toolchains`) if CI | ||
| grows to cover them. | ||
|
|
||
| The Swift compiler version is read from [`.swift-version`](.swift-version) by | ||
| both the Bazel toolchain (`swift.toolchain(swift_version_file = …)`) and the | ||
| local build, and is kept in sync with the `swift-syntax` release pinned in | ||
| `swift/Package.swift`, so local and CI builds behave identically. | ||
|
tausbn marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## Usage | ||
|
|
||
| Library: | ||
|
|
||
| ```rust | ||
| let json = swift_syntax_rs::parse_to_json("let x = 1")?; | ||
| println!("{json}"); | ||
| ``` | ||
|
|
||
| CLI (reads a file argument or stdin, prints the syntax tree as JSON): | ||
|
|
||
| ```sh | ||
| echo 'let x = 1' | cargo run --bin swift-syntax-parse | ||
| ``` | ||
|
|
||
| ## Layout | ||
|
|
||
| - `swift/` — Swift package exposing the `ssr_parse_json` / `ssr_string_free` C ABI. | ||
| - `build.rs` — builds the Swift package and emits link/rpath flags (local `cargo` only). | ||
| - `BUILD.bazel` — Bazel targets for the hermetic CI build (swift_library + rust targets). | ||
| - `src/lib.rs` — safe Rust bindings (`parse_to_json`). | ||
| - `src/main.rs` — demo CLI. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.