Skip to content

Add Protocol Buffers and gRPC compiler - #21

Draft
OmarAlJarrah wants to merge 4 commits into
mainfrom
feat/protobuf-compiler
Draft

Add Protocol Buffers and gRPC compiler#21
OmarAlJarrah wants to merge 4 commits into
mainfrom
feat/protobuf-compiler

Conversation

@OmarAlJarrah

Copy link
Copy Markdown
Member

Summary

Adds compilers/protobuf, a compiler that lowers Protocol Buffers .proto
schemas into Morphic's spec-agnostic IR — the protobuf/gRPC counterpart to the
existing OpenAPI compiler. Parsing uses github.com/bufbuild/protocompile, a
pure-Go .proto compiler that produces fully linked, feature-resolved
descriptors and bundles the well-known-type imports. Like every compiler, it
does no file I/O: it compiles a single root .proto supplied as bytes, resolves
google/protobuf/*.proto imports from the parser's bundle, and reports any other
unresolved import as a diagnostic rather than crashing.

It supports proto2, proto3, and the 2023 edition through one lowering, since the
parser resolves edition features into the same descriptor surface the
proto2/proto3 paths expose. The mapping onto the IR:

  • Types. Messages, nested messages, and enums hoist into the flat, ID-keyed
    type registry, with IDs derived from their fully-qualified proto names (the
    stable structural identity, never a display name). Field numbers become
    Property.WireID; the package becomes Namespace.
  • Presence. proto2 required/optional, proto3 implicit vs optional, and
    editions field_presence lower to Property.Presence
    (implicit/explicit/required). Protobuf has no null, so TypeRef.Nullable stays
    off and presence carries the three-state discipline on its own axis.
  • oneof. A real oneof becomes a WireTagged, exclusive Union reached
    through one synthetic Flatten wrapper property, with each member's field
    number preserved as the variant wire ID. Synthetic proto3-optional oneofs
    are presence markers and stay ordinary fields, never unions.
  • Containers & encodings. repeated hoists a List (packed vs expanded on
    List.Encoding); map<K,V> hoists a MapT without exposing the synthetic
    map-entry message; sint/fixed/sfixed lower to an Encoding over the base
    primitive; groups/editions DELIMITED carry the delimited encoding.
  • Enums. Open vs closed follows the resolved semantics (proto2 closed, proto3
    open, per-enum editions feature); allow_alias duplicate values survive as
    distinct members.
  • Well-known types. Timestamp/Duration map to date-time primitives;
    Any/Struct/Value to the schemaless Any node; the wrapper types to a
    nullable primitive by default (or an External under an injectable policy
    option); FieldMask/Empty to External. Well-known internals are never
    hoisted.
  • Extensions & reserved. extend fields attach to the extended Model with
    Property.ExtensionOf; extensions ranges become Model.ExtensionRanges;
    reserved numbers/names and custom options (protovalidate and the like) are
    preserved verbatim in Extensions, guarded by the validate pass. Custom
    options render deterministically.
  • Services. Each proto service becomes an OperationGroup; each rpc an
    Operation with a gRPC RPCBinding, streaming direction (unary, client,
    server, bidi), and idempotency level. An Empty request or response lowers to
    an absent payload.

No IR structs were changed: everything maps onto the existing IR, which already
accounts for wire IDs, oneof, packed repeated, extension ranges, presence, and
editions. One minor, documented degradation: a non-finite proto2 default
(inf/nan) is dropped with an info diagnostic rather than stored as an
invalid decimal, and a delimited group encoding in a nested (repeated/oneof)
position is not re-expressed on the wrapped element.

The compiler is not yet wired into the engine's default registry: engine.Sniff
is YAML/JSON-based and would need a content-detection path for .proto, which is
a separate engine-layer change. The compiler is fully usable through its public
API and registers cleanly into any compilers.Registry.

Test plan

  • Conformance corpus — one minimal .proto per protobuf/gRPC capability from
    the cross-spec matrix under testdata/conformance/protobuf/, each with a
    focused capability assertion plus a byte-exact golden IR snapshot: messages,
    nested types, open/closed enums, oneof, proto3 optional, maps, packed/expanded
    repeated, scalar wire encodings, proto2 presence and defaults, extensions,
    reserved, packages, well-known types, all four streaming modes with
    idempotency, deprecation, doc comments, custom options, and a 2023-edition
    file.
  • Golden snapshot — a full gRPC library service (testdata/golden/protobuf/).
  • Round-trip property — compile, marshal, unmarshal, cmp.Diff-equal, and
    re-marshal for determinism.
  • Registry / options / errors — format registration, the wrapper-external
    policy, unresolved-import diagnostics, wrong-options and multi-source rejection.
  • Architecture test — extended so compilers/protobuf may import ir,
    compilers, protocompile, and the protobuf runtime only.

go build ./..., go vet ./..., gofmt -l . (clean), golangci-lint run
(clean), and go test ./... all pass.

Add a compiler that lowers .proto schemas into the spec-agnostic IR, the
protobuf/gRPC counterpart to the OpenAPI compiler. It parses a single root
.proto with bufbuild/protocompile, resolves well-known-type imports from the
parser's bundle, and treats any other unresolved import as a diagnostic
(compilers do no file I/O).

Coverage spans proto2, proto3, and the 2023 edition:

- messages, nested messages, and enums hoist into the flat type registry with
  IDs derived from their fully-qualified names; field numbers become wire IDs.
- presence lowers to Property.Presence (implicit / explicit / required) with
  proto2 required, proto3 implicit vs optional, and editions features resolved
  by the parser; protobuf has no null, so nullability stays off.
- oneof becomes a WireTagged, exclusive Union reached through one synthetic
  Flatten wrapper property; synthetic proto3-optional oneofs stay presence
  markers, not unions.
- repeated and map fields hoist List and MapT container nodes; packed vs
  expanded is recorded on List.Encoding; sint/fixed/zigzag scalar variants
  lower to Encoding over the base primitive.
- enums carry open/closed semantics and allow_alias duplicate values; reserved
  ranges and names, and custom options, are preserved verbatim in Extensions.
- well-known types map faithfully: Timestamp/Duration to date-time primitives,
  Any/Struct/Value to the schemaless Any node, wrappers to a nullable primitive
  (or an External under policy), FieldMask/Empty to External.
- extension fields attach to the message they extend with Property.ExtensionOf;
  extension ranges become Model.ExtensionRanges.
- services become operation groups; each rpc is an Operation with a gRPC
  RPCBinding, streaming direction, and idempotency level; an Empty request or
  response lowers to an absent payload.
…trip

Add the test suite for the protobuf compiler, mirroring the OpenAPI compiler:

- a conformance corpus of one minimal .proto per protobuf/gRPC capability from
  the cross-spec matrix (messages, nested types, open/closed enums, oneof,
  proto3 optional, maps, packed/expanded repeated, scalar wire encodings,
  proto2 presence and defaults, extensions, reserved, packages, well-known
  types, all four streaming modes with idempotency, deprecation, doc comments,
  custom options, and a 2023-edition file), each with a focused assertion and a
  byte-exact golden IR snapshot;
- a full gRPC library service golden snapshot;
- a JSON round-trip property test (compile, marshal, unmarshal, deep-equal, and
  re-marshal for determinism);
- registry, option-policy, unresolved-import, and multi-source-rejection tests.

Extend the architecture import-graph test so compilers/protobuf may import ir,
compilers, protocompile, and the protobuf runtime only.
Simplify the compiler by removing guards that can never fire, so the code
matches its actual reachable behavior:

- protocompile recovers parser panics internally and returns them as errors, so
  the compiler's own recover and errParse sentinel were dead; compileRoot now
  returns the linked descriptor directly and a PanicError flows through as a
  compile-error diagnostic like any other.
- a descriptor's Options() always returns a materialized options message (empty
  when nothing is set), so the nil-options guards in optionBool, optionsMessage,
  customOptions, and methodIdempotency were unreachable and are removed.
- json.Marshal of the fixed-shape reserved, file-option, and array payloads
  cannot fail, so those error branches are dropped.
- leafType folds MessageKind into its default and syntaxDigit folds proto3 into
  its default, since those are the only remaining reachable cases.
Extend the corpus and add a white-box helper test so every reachable branch is
exercised, matching the repository's 100%-coverage gate:

- enrich the corpus with an unmapped well-known type, unsigned/bytes/non-finite
  proto2 defaults, a repeated encoded scalar, a deprecated oneof member with an
  explicit json_name, an Empty-input rpc, and an rpc whose only option is
  deprecated;
- add rich-options (message/repeated/map/enum/bytes/unsigned custom options,
  NaN option values, a group field, and enum-value/service/method options),
  file-options, and no-package corpus files with assertions and goldens;
- add compile tests for a syntax error and an unused-import warning;
- add a white-box test covering the pure helpers and the option-rendering
  recursion-depth guards.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant