Replace the Ant build with Gradle, output verified byte-identical

[pacmano1]

Refs #52.

This replaces the Ant build with a standard Gradle build in one verified step. The product the build produces did not change, and that claim is checkable: the output of both builds was compared archive by archive, file by file. This description is the full write-up (it used to live in BUILD-MIGRATION.md in the repo; per review it now lives here, since it describes this change rather than the ongoing build).

What changed since the first review

Addressing the review feedback (thanks @mgaffigan):

• The verification tooling moved out of this repo to pacmano1/oie-build-parity (compare_builds.py, the jar-provenance audit, snapshot_distribution.py). It is migration scaffolding, not part of the build, so it no longer adds to this PR's review surface. If the maintainers would rather it live under the org, say the word and I'll move it to OpenIntegrationEngine/oie-build-parity. It's offered, not imposed.
• BUILD-MIGRATION.md is removed; this write-up is now the PR description (it would only go stale sitting in the tree).
• Build flags are now Gradle project properties (-P), and -D support is gone. A clean breaking change, as suggested, rather than carrying a compatibility shim. -PdisableSigning=true, -Pcoverage=true, -Pversion=x.y.z, and -Pkey.storepass=<PIN> for signing. The Dockerfile and CI were updated to match.
• The hand-written empty package-info.class generation is gone. Ant's javac emitted empty, inert marker classes for the three annotation-free package-info.java sources; the Gradle build no longer reproduces them. This is now a documented, zero-runtime-effect difference (the comparison tool classifies it), instead of being faked with a hand-built class file.
• A few incidentals: the in-build snapshotDistribution task moved to the external tooling; the writeArtifactInventory task description was clarified; some unrelated personal entries were removed from .gitignore.

The substance of the migration below is unchanged.

The short version

• The build system changed from Ant to Gradle. The product it builds did not change, and that claim is checkable.
• 374 of the 419 library jars that used to be stored in the repository are now downloaded from Maven Central, each accepted only after proving it is the exact same file, bit for bit (same SHA-1). The other 45 stay in the repository, each with a documented reason (appendix).
• Every check on the build's output is reproducible, so any reviewer can re-run the verification rather than trust this description. The recipe is at the bottom; the tooling is in oie-build-parity.
• Three commits, reviewable separately: 1,806 pure file moves to the standard layout (git show --stat -M100% confirms zero content change), the build swap, and the dependency swap (plus a Windows-docs note from @jonbartels on top). Only the head commit builds; squash-on-merge is fine.
• Substantial parts of this work were produced with AI assistance as a process experiment. The controls that made that safe, and the errors those controls caught, are described below.

Why one change instead of piecemeal

None of the direction here is new. @NicoPiel proposed the move to a dependency-managed build (#52) and drove three successive Gradle attempts (#54, #100, #214); @jessesaga built a fourth (#243). In those reviews @mgaffigan named the bar a one-step replacement would have to clear, byte-identical output, and wrote the verification toolkit this branch is cross-checked with. The three-commit structure follows the reviewable-commit standard Tony Germano held the project's last large dependency change (#146) to. This branch is built on that groundwork.

The incremental path (wrap the old build first, convert one piece at a time) is the default for a reason: replacing a build system in one step is reckless when you can't prove the output is the same.

This branch takes the other path because the risk became measurable. The bar is met, and here is exactly what "identical" means, including everything that is not identical:

• Every archive in the distribution has identical contents to an Ant build of this branch's parent commit (8c1111ba3), apart from build-tool fingerprints rather than product content: (1) 121 archives carry an Ant-version label Gradle doesn't write; (2) the build-written file timestamps inside those same archives differ, because both tools stamp a fixed fake date and chose different ones (third-party jars pass through byte-identical); (3) one small file (version.properties) embeds the build date, so it differs between any two builds; (4) two jars contain one extra folder entry (not a file); and (5) the empty package-info.class markers Ant's javac emitted for the three annotation-free package-info.java sources are not reproduced (inert, no runtime effect; see "What changed since the first review"). Every class file, resource, script, and configuration byte is the same. The comparison tool classifies each of these categories itself; nothing was waved through by hand.
• All 654 tests pass, and the list of test classes that ran is identical to the Ant build's. (The CI test counter drops to roughly half its historical number because the old build accidentally wrote a duplicate results file that counted every test twice; the true count was always 654.)
• The published v4.6.0-rc1 release was cross-checked against a local Ant build of the same source: 473 of 490 archives bit-for-bit identical in content. The 17 that differ are all the release pipeline, not compilation (the signing step repackages modular jars, rewriting one bookkeeping file; signature files and installer files can never match between builds). None differ in compiled code.
• The signing path works end to end: all 216 jars that ship signed are signed by the new build and pass jarsigner -verify (strict mode adds the expected self-signed-dev-keystore warning; the old build warns identically).
• The built product boots and runs (a human boot test of the Gradle-built server/setup; a sanity check on top of the archive comparison).

A verified cutover is one reviewable event with the proof attached, still split into three commits so each concern can be reviewed on its own.

What changed

1. File moves (commit 1): 1,806 source files moved to the standard layout (src/main/java, src/test/java, …). git show --stat -M100% confirms every one is a pure move.
2. The build swap (commit 2): the new Gradle build via a committed, checksum-pinned wrapper. Everything lands where it did: server/setup, server/dist. The Ant entry points become stubs that point to ./gradlew; the subproject Ant files are deleted. CI, the Dockerfile, and the contributor docs use the new build (with -P flags). Eclipse-era conveniences are now build commands (devRun, devLauncher, devClient, createDerbyDb).
3. The dependency swap (commit 3): the 374 verified jars come from Maven Central; the 45 unverifiable files (32 distinct libraries) stay vendored with per-jar reasons (appendix). Three safeguards: every download is checksum-verified on every build; the build fails loudly if a jar would silently go missing; and transitive resolution is kept off so the shipped set cannot drift (see Limitations).

How mistakes are caught before they ship

The design rule everywhere: a mismatch must stop the build with a clear message, never surface later as a broken product.

• Update a library version but forget to refresh the fingerprints? The build fails before compiling, naming the library.
• A downloaded jar with no assigned place in the shipped product? The build fails and names it.
• The startup files that list jar names are generated from the resolved versions, so they can't go stale.
• Changing a library version is two steps, and forgetting either is a build failure with instructions (the exact command, and why it needs a cold cache, is in CONTRIBUTING).

For changes to the build logic itself, CONTRIBUTING documents a habit: snapshot the built product before and after, diff the two, confirm only what you intended changed. The tooling for that is snapshot_distribution.py / compare_builds.py in oie-build-parity.

How this was built

Substantial parts of this migration were produced with AI assistance: the Gradle build scripts, the comparison and provenance tooling, the dependency audit, and the drafts of this write-up. The experiment: can AI-driven work of this size be held to machine-checked claims rather than taken on trust? The controls:

1. The pass/fail bar was set before the work began: the new build's output must match the old one, verified by machine. Every aggressive step happened behind that gate, and it caught real mistakes (ledger below).
2. Machine claims were treated as unproven until re-executed.
3. Every internal review ran its findings through a second, adversarial pass whose job was to disprove them, a fresh AI session with none of the first's context, re-running the commands. (The ledger shows one at work: the disproven "zero deprecation warnings" claim.)
4. Independent tooling. The primary comparison tool was itself written as part of this AI-assisted work, so passing it is partly self-grading. The output was additionally verified with oie-release-verifier, the release-checking toolkit @mgaffigan wrote for a different purpose.
5. A human approved every irreversible step. Approving commands is not code review, and neither is the output-equivalence proof; the build logic still needs human eyes, and the three-commit structure exists for that.
6. Discretionary improvements were kept out, even provably safe ones. This change rests on one claim, nothing changed, and every optional improvement folded in makes it harder to verify. (Example: thirteen vendored jars are content-identical to their public copies and could safely become downloads; left untouched here, queued as follow-ups.)

The error ledger

The mistakes made during this work, and what caught each:

• An early wrong AI guess blamed a compiler path setting for output differences; the real cause was a compiler feature silently disabled by Gradle's defaults. The output comparison caught it.
• The AI wrongly classified three libraries as "not on Maven Central." Re-execution caught it: Central's search index is missing fingerprint records for some files, so "no match" must be confirmed against the actual download server.
• An AI refactor dropped eleven historically renamed jars from the shipped product. The output comparison flagged all eleven.
• An internal AI review claimed "zero deprecation warnings." A fresh cold review re-ran the command and disproved it; the build was restructured. One further-out item remains (Limitations).
• The AI's first boot-test ran beside a live server instance, so its result couldn't be attributed to the new build. A human caught it; runtime testing moved to a human-run boot, which succeeded.
• The dependency checksum inventory was first generated on a warm cache without CI's coverage flag, missing parent-POM and JaCoCo-agent entries. The fail-closed gate stopped CI within seconds; it was regenerated cold with CI's flags, and CONTRIBUTING documents that.
• Project lore held the HAPI 2.3 jars were locally patched. A human question prompted an entry-by-entry comparison: content-identical to the public copies, only repackaged.

Known limitations

• Adding a new library takes more manual work (transitive resolution is off, so the shipped set provably can't drift). The way out is gradual: when a library is upgraded anyway, switch transitivity on for just that one and check with the comparison tool.
• The build logic is centralized and plain, not idiomatic. Large Groovy files transcribing the old Ant build one-to-one, guarded by the output comparison rather than unit tests. Restructuring into convention plugins is left for its own change.
• The configuration cache isn't supported yet. One call is slated for removal in Gradle 10 (years out); the build pins Gradle 8.14.1 and is unaffected. Tracked.
• Signed release builds are ~5 minutes slower (signing one jar at a time vs four). Only signed release builds notice.
• Of the three commits, only the last builds. Split this way so each part can be read in isolation.
• A few …Tests-named classes have never run, before or after (the pattern matches …Test). Preserved unchanged; a separate follow-up.
• The equivalence proof covers the cutover only. What protects every day after: per-build checksum verification, the fail-closed placement gate, the 654 tests, and the documented snapshot habit.
• Output claims are verifiable; process claims are not. Every claim about the build's output can be re-run. The claims about how the work was done can't be checked, and don't need to be: a reader who ignores the process story entirely can verify the build the same way.

Appendix: the 45 jars that stay in the repository

Generated from the committed audit record (jar-provenance.json in oie-build-parity). 32 distinct jars, 45 files across modules.

• Same contents as the public copy, different wrapper (13): unpacked and compared file-by-file against Maven Central; every file inside identical, but the zip container fingerprints differently (packing order/timestamps/compression). Strict file-level identity was the adoption rule, so these stayed. Safe to adopt later. autocomplete-2.5.4, the ten hapi-*-2.3 jars, jtds-1.3.1, rsyntaxtextarea-2.5.6, sqlite-jdbc-3.43.2.1.
• Genuinely changed from the public copy (3): compiled code differs (114 classes in javaparser-1.0.8, 60 in zip4j_1.3.3, 2 in not-going-to-be-commons-ssl-0.3.18). Local forks; replacing them would change the product.
• Nothing on Maven Central to replace them with (10): PDFRenderer, backport-util-concurrent-Java60-3.1, istack-commons-runtime-3.0.6, jai_imageio, language_support, looks-2.3.1, openjfx, webdavclient4j-core-0.92, wizard, wsdl4j-1.6.2-fixed.
• Known origins outside Maven Central (6): five dcm4che-*-2.0.29 jars (from dcm4che.org) and mirth-vocab.jar (built in-house by the generator module).

How to re-verify

Needs JDK 17, Apache Ant (acceptance used 1.10.14), Python 3, and network access for the first Gradle run. The tooling lives in oie-build-parity.

Run these from your checkout of this PR's branch (the first line captures it; the rest use absolute paths, so cwd doesn't matter after that):

ENGINE="$(pwd)"

# The comparison tooling
git clone https://github.com/pacmano1/oie-build-parity /tmp/oie-build-parity

# Ant baseline at the migration's parent commit, in a throwaway worktree
git worktree add /tmp/oie-ant 8c1111ba3
(cd /tmp/oie-ant/server && ant -f mirth-build.xml -DdisableSigning=true -DdisableTests=true build)

# Gradle build of this branch (same JDK build, same calendar day:
# version.properties embeds the build date). Gradle flags are -P;
# the Ant baseline above uses Ant's own -D syntax.
./gradlew clean build dist -PdisableSigning=true

# Compare every archive, entry by entry (absolute paths; cwd-independent)
python3 /tmp/oie-build-parity/compare_builds.py /tmp/oie-ant/server/setup "$ENGINE/server/setup"

git worktree remove --force /tmp/oie-ant

Expected: REAL differences (0), with the known tool-fingerprint deltas (Ant-version labels, timestamps, version.properties, the two directory entries, and the empty package-info.class markers) each classified and counted. The Ant baseline builds in seconds (normal); the Gradle build takes a few minutes and runs the full suite.

A successful Gradle build ends with Gradle's stock "incompatible with Gradle 9.0" banner; --warning-mode all shows it refers to the one disclosed Gradle-10 item under Limitations.

The dependency audit is re-checkable with python3 /tmp/oie-build-parity/sweep_provenance.py --verify-exact (expect 0 mismatches). The signed-build check and the rc1 cross-check each have their own recipe in the oie-build-parity README.

The Ant baseline hash file (oie-release-verifier format, uncompressed SHA-256 e54cb2ef8c30ea887fb6752fc83a4dbf9fd818de3267177d29d6d2a1eaf3f6e4) is published here. A second maintainer re-running the dual-build recipe before merge is invited.

[github-actions]

Test Results

107 files   -   4  107 suites   - 107   3m 33s ⏱️ - 2m 13s
651 tests ±  0  651 ✅ ±  0  0 💤 ±0  0 ❌ ±0 
651 runs   - 651  651 ✅  - 651  0 💤 ±0  0 ❌ ±0 

Results for commit 64e1a98. ± Comparison against base commit 88c8ee5.

♻️ This comment has been updated with latest results.

[mgaffigan]

Looking good! Tests pass, CI passes, released binaries are byte-identical with the main ant build.

[jonbartels]

Can the file moves be refactored to a separate PR. It is good to move the project to the standard gradle layout. However it would make the first commit smaller. https://docs.gradle.org/current/dsl/org.gradle.api.tasks.SourceSet.html and main.java.srcDirs

So that would be one PR with a non-standard layout. then a second PR would move to the standard layout. It could also be one PR with multiple commits showing the addition of gradle, removal of JARs, then move of project files.

I think this PR has the same too-large problem one of the earlier Nico PRs had

[mgaffigan]

Generally looks really good. (Obviously runs fine, since it is byte identical)

Changes requested:

• remove documentation of the change that will quickly grow stale (consider moving to github wiki or another destination)
• remove verifier to separate repo to avoid having to review
• remove/clarify purpose of apparently unused steps
• remove hardcoded "empty" class file generation. Check in the .class file or remove and consider this an intended difference.

Other incidentals:

• Consider having the dependency resolution per-project file (unless it makes this easier to review, we can do that in a separate PR)
• Consider omitting the -D legacy support. I don't think this will work well long term, and it is easier to make a breaking change now.

[mgaffigan]

What's the benefit of using this over the built-in wrapper? Wrapper validation?

[mgaffigan]

These seem like they are unrelated. I don't think it's a problem, but they don't seem strictly related to the gradle conversion.

[mgaffigan]

Seems like this should be part of the PR itself - not in-repo. It's going to be stale quickly. Remove.

[mgaffigan]

Should we maintain this? Or take this as an opportunity for a breaking change. -D to -P is not so big a lift for callers - easier to do so now than later.

[mgaffigan]

Can we avoid this being a central json that is going to have merge conflicts? Centralizing makes this unclear. Each project (client/server) should specify the deps it requires. Depending on details, might be better as a follow-up PR.

[mgaffigan]

I don't understand what this is doing for the build. Is this effectively a debug log? Presumably should be removed.

[mgaffigan]

Seems like this is duplicative of the tools/verifier and separate verifiers. The build cannot check itself. Remove/move to tools.

[mgaffigan]

This seems excessive for the getting-started. Remove to wiki.

[mgaffigan]

I don't think this directory landing in main is critical to this PR landing, I suggest the build-parity directory be removed to a separate PR if it is to be in main.

[mgaffigan]

Why is this changing?

[pacmano1]

Thanks, this was a useful pass. All of it is addressed; rundown:

Docs that go stale: BUILD-MIGRATION.md is removed; the write-up now lives in the PR description.

Verifier to a separate repo: done. compare_builds.py, the provenance audit, and the snapshot tool now live in https://github.com/pacmano1/oie-build-parity, out of this PR's review surface. If you'd rather it sit under the org, I'll move it to OpenIntegrationEngine/oie-build-parity; it's offered, not imposed.

Unused/debug steps: snapshotDistribution is removed (the snapshot protocol moved to the external tooling). writeArtifactInventory isn't a debug log; it records each module's resolved jars so :server:stageSetup can place foreign-module jars without resolving their configurations cross-project (the Gradle-9-safe path). I reworded its description so that's clear.

Empty package-info.class generation: gone. The build no longer hand-writes those three inert markers; they're now a documented, zero-runtime-effect difference, and the comparison tool classifies an ant-only empty package-info.class as expected. No raw bytestream in the build anymore.

Per-project dependency resolution: agree it's the better shape, and I'd like to do it as the follow-up you suggested, one step at a time with the parity check, rather than reshape staging inside the migration. The deps are already declared per-project (the catalog bundles); what's central is the placement map, which is version-independent and has had zero churn since it was created. Happy to take it on next.

Drop -D legacy support: done, clean break, no shim. Flags are -P now (-PdisableSigning, -Pcoverage, -Pversion, and -Pkey.storepass for signing). CONTRIBUTING and CI updated.

Wrapper action (build.yaml): you guessed it: validate-wrappers: true checks the committed gradle-wrapper.jar against Gradle's published known-good checksums (the supply-chain answer to "should we commit the jar"), plus the action wires up Gradle's dependency caching across runs.

install4j / .gitignore incidentals: the install4j change is reverted out of the PR (it was an unrelated local edit). The unrelated .gitignore entries are removed.

One thing worth flagging in good faith: dropping -D did break a caller I'd missed at first, the Dockerfile (ARG GRADLE_BUILD_ARGS="-DdisableSigning=true"), since a stale -D is silently ignored rather than rejected. It's fixed to -P in the same commit as the flag change. Called out because it's exactly the "does this break a caller" question your comment raised, and the answer was yes, now handled. I fixed the flag but didn't run docker build to exercise the container itself (it isn't CI-gated); the change just mirrors the unsigned Gradle build, which is verified green.

[pacmano1]

Fair concern, and the diff is genuinely large. The reason it's one change rather than a sequence is that the evidence is one piece: every archive in the distribution is verified byte-identical against an Ant build of the parent commit. An intermediate PR with a non-standard layout (or the build swap without the dependency swap) would be a state that was never verified and was never meant to ship, so splitting there would mean asking reviewers to sign off on something the proof doesn't cover.

The intended way to review this isn't to read all the files; it's to run the comparator and watch it report zero real differences. The recipe is in the PR description, and the tooling is in a separate repo now so it isn't part of what you have to read here. That's the review surface, not the line count.

On the srcDirs idea specifically: pointing Gradle at the old layout would keep a non-standard layout that a second PR then has to undo, so it adds a step rather than removing one, and the move is the cheap part to review anyway since git show --stat -M100% proves it's pure renames with zero content change.

The branch is already three commits split along the lines you describe: the 1,806 pure file moves (that same -M100% rename check), then the build swap, then the dependency swap, so each concern can be read on its own, and squash-on-merge is fine. Happy to walk through any one of them.

Also, thanks for the Windows SDKMan note, I rebased to address the review and kept your commit on top.

[jonbartels]

TY for splitting the commits up.