One interface.
Two Spark runtimes.
A runtime-aware compatibility architecture for DataComPy that allowed comparison logic to select the correct Spark Classic or Spark Connect implementation without depending on an active local SparkContext.
42
focused comparator and end-to-end tests reported passing
4
comparator families covered
2
Spark runtimes addressed
#548
public pull request
01 · The compatibility problem
Same Spark API.
Different expression engines.
DataComPy comparison logic imported classic PySpark functions and windows directly. Spark Connect uses different DataFrame, Column, functions, and Window implementations, so classic expression construction could fail even when the comparison logic itself was valid.
The contribution proposed dispatching from the actual object in use. Classic objects receive classic helpers; Connect objects receive Connect helpers. That decision keeps comparator logic shared while respecting the runtime boundary.
02 · Engineering scope
Compatibility applied across the comparison system
- 01
Runtime-aware dispatch between classic PySpark and Spark Connect functions.
- 02
Matching window implementations for classic and Connect DataFrame or Column objects.
- 03
Compatibility applied across array, boolean, numeric, and string comparators.
- 04
SparkSQLCompare paths covering merge behavior, duplicate handling, mismatch reporting, masking, caching, and window expressions.
- 05
Regression coverage proving Connect expressions can be built without an active local SparkContext.
03 · Public outcome
Closed PR.
Carried-forward design.
Pull request #548 was closed without merge. DataComPy's maintainer chose a separate implementation branch while several related pull requests were under review.
The maintainer publicly confirmed: “I've incorporated some of your design into this new standalone branch.”
This is documented technical influence, not a client engagement, partnership, endorsement, or claim that Capital One adopted the submitted patch wholesale.
Public evidence
Architecture can create value even when maintainers carry the idea forward through a different implementation.
Original pull request
https://github.com/capitalone/datacompy/pull/548Maintainer implementation branch
https://github.com/capitalone/datacompy/compare/main...spark-connect-sessions