Root cause identified.
Architecture adopted.

A Spark Connect compatibility defect in Capital One’s open-source DataComPy exposed a core architectural issue in how Spark expression APIs were selected.

When SparkSQLCompare received an already-created Connect session, expression selection relied on process-global Spark state rather than the actual DataFrame or Column being compared. As a result, classic Spark expressions could be constructed against Spark Connect DataFrames, causing runtime failures across distributed environments.

Root Cause: The legacy implementation assumed SPARK_CONNECT_MODE_ENABLED accurately represented the runtime environment. That assumption breaks down whenever a Connect session originates outside the library (e.g. in client code, microservices, or enterprise notebook kernels).

Architectural Correction: PR #548 authored by Patrick Ribbsaeter (@patrickswedish) introduced runtime dispatch based on the Spark object being operated on. Instead of trusting global process state, the system dynamically resolves the correct Spark functions and Window implementation from the actual DataFrame or Column object.

Capital One maintainers integrated the object-level dispatch approach into upstream Pull Request #552, explicitly crediting Patrick Ribbsaeter (@patrickswedish / PR #548) for the dispatch approach and for identifying the process-global flag as the root cause.

Capital One PR #552 description:

“Approach proposed by @patrickswedish / #548 ... This architecture resolves the root cause in Spark Connect.”

Capital One then extended the PR with comprehensive regression coverage, Connect validation, CI integration, and compatibility fixes. PR #552 is open, mergeable, and passing all continuous integration pipelines.

This pattern is critical to banking, fintech, and large-scale data engineering, where mission-critical reliability depends on predictable behavior across distributed systems, cloud runtimes, data reconciliation pipelines, and remote cluster execution.