3.x: Implement LWT Replica-Only Routing with Local DC Prioritization

[nikagra]

Implement LWT Replica-Only Routing with Local DC Prioritization

Overview

This PR adds specialized load balancing for Lightweight Transaction (LWT) queries in com.datastax.driver.core.policies.TokenAwarePolicy, optimizing performance by routing exclusively to replicas and prioritizing local datacenter replicas. This reduces coordinator forwarding overhead and minimizes contention during Paxos consensus phases.

Motivation

LWT queries require multi-round Paxos coordination and are sensitive to latency and coordinator variance. Prior behavior could:

• Route to non-replica nodes, adding an extra forwarding hop
• Not prioritize local DC replicas consistently
• Introduce unnecessary cross-DC traffic for LOCAL_SERIAL transactions

This change ensures LWT queries target replicas directly, with local replicas first, resulting in lower latency and reduced contention.

Changes

1. TokenAwarePolicy: LWT-Aware Routing Path

• Implemented LWT-specific routing within TokenAwarePolicy#newQueryPlan(...).
• Routing method selection:
  • REGULAR for non-LWT, or when QueryOptions.setLoadBalancingLwtRequestRoutingMethod(REGULAR) is configured.
  • PRESERVE_REPLICA_ORDER for LWT (default): uses a specialized iterator that returns replicas only, local-first, preserving primary replica order.
• The logic uses the statement’s routing key, keyspace, and optionally table metadata to resolve replicas via Metadata#getReplicasList(...).

2. LWTHostIterator: Replica-Only, Local-First

• New TokenAwarePolicy.LWTHostIterator that:
  • Returns only replicas (never non-replica hosts in the preferred path).
  • Prioritizes local DC replicas (childPolicy.distance(host) == LOCAL).
  • Then returns remote replicas if available and not ignored.
  • Filters out DOWN hosts and IGNORED hosts.
  • Preserves primary replica ordering from metadata; no randomization for LWT.
• Pragmatic fallback: If no suitable replicas are available (e.g., all are DOWN or IGNORED, or replicas list is empty due to missing routing info), it falls back to the child policy’s plan to preserve availability.

3. Key Design Decisions

✅ Rack Awareness Intentionally Omitted

• Do not apply rack-based prioritization within the local DC for LWT queries.
• Rationale: Rack-based ordering can create hotspots and contention during Paxos phases. Treating all local DC replicas equally reduces lockstep contention on hot partitions.

✅ Replica-Only with Pragmatic Fallback

• Default behavior: When routing info is available, the query plan contains only replicas.
• Fallback: When routing information is missing, metadata is unavailable, or all replicas are DOWN/IGNORED, falls back to child policy’s plan (may include non-replicas).
• Rationale: Hard failure would be overly strict; fallback preserves availability with a small performance penalty.

✅ Local DC First, Remote DC Second

• Local replicas (distance == LOCAL) are returned first; remote replicas (distance == REMOTE) follow.
• Rationale: For LOCAL_SERIAL, this keeps Paxos messaging within the local DC for best performance, while still allowing remote replicas when needed.

✅ Filter DOWN and IGNORED Hosts

• LWTHostIterator skips hosts that are !isUp() or have distance == IGNORED.
• Rationale: Attempting DOWN hosts wastes time; respecting IGNORED honors user-configured allowlists and filters.

Documentation

• Class-level Javadoc: Expanded section in com.datastax.driver.core.policies.TokenAwarePolicy explaining LWT routing behavior, requirements, fallback behavior, and rack avoidance rationale.
• Inline comments: LWTHostIterator documents filtering and ordering logic.

Requirements for LWT Replica-Only Routing

For the optimization to apply, the following must be true:

1. Statement routing key available: use prepared statements or manually set a routing key.
2. Effective keyspace known: set on statement or session (also uses table metadata when available for replicas resolution).
3. Child policy reports DC locality: use DCAwareRoundRobinPolicy or similar.
4. Cluster metadata available: driver must have resolved replica topology.

Configuration

LWT routing method is controlled via:

QueryOptions.setLoadBalancingLwtRequestRoutingMethod(QueryOptions.RequestRoutingMethod)

• Default: PRESERVE_REPLICA_ORDER (replicas-only, local-first)
• Alternative: REGULAR (uses standard token-aware routing with non-replica fallback)

Performance Impact

Expected Improvements (with routing info available)

• Reduced latency: Eliminates coordinator forwarding hops.
• Lower p99 variance: Consistent replica targeting reduces coordinator variance.
• Reduced cross-DC traffic: Local DC replicas prioritized for LOCAL_SERIAL.
• Lower contention: Rack avoidance helps distribute LWT load across local DC.

Fallback Scenario (routing info missing or replicas unavailable)

• Adds ~0.5–2ms median latency due to coordinator forwarding.
• Mitigated by using prepared statements (routing key always available) and ensuring metadata is up to date.

Compatibility

• ✅ No breaking changes: Public API unchanged.
• ✅ Backward compatible: Existing query behavior preserved for non-LWT and when REGULAR is configured.
• ✅ Java 8 compatible: No new APIs used.
• ✅ Clirr check: 0 errors, 0 warnings.

Testing

Unit/Integration Tests

• LWT load balancing integration tests verify single-coordinator behavior for LWT with known routing key and keyspace, and non-LWT behavior remains distributed.
• Existing token-aware tests continue to pass; replicas are prioritized when routing info is present.
• Added/updated tests around filtering and fallback:
  • Filter DOWN replicas for LWT.
  • Filter IGNORED replicas for LWT.
  • Combined filtering of DOWN and IGNORED hosts.
  • Empty plan when all replicas filtered → child policy fallback engaged.

Validation Performed

• Code formatting: mvn fmt:format (repo standard)
• Compilation: make compile-all or mvn compile -pl driver-core -am
• Unit tests: targeted tests in driver-core and integration-tests modules
• Full verification: make check or mvn verify -DskipTests
• API compatibility: Clirr checks pass

Migration Notes

No action required for existing users. The optimization is automatic when:

• Using TokenAwarePolicy (default in most configurations)
• Using prepared statements (best practice)
• LWT queries detected via Statement#isLWT() and server-provided metadata

Users not meeting these conditions continue to work with existing behavior.

Related Documentation

• Scylla LWT Documentation: https://docs.scylladb.com/using-scylla/lwt/
• Driver Load Balancing Manual: manual/load_balancing/
• Class Javadoc: com.datastax.driver.core.policies.TokenAwarePolicy

Closes

#752

[Copilot]

Pull request overview

Implements replica-only routing for LWT queries within TokenAwarePolicy, prioritizing local-DC replicas and allowing fallback to regular routing based on a new QueryOptions setting.

Changes:

• Removed RequestHandler’s LWT special-casing so LWTs flow through the configured load balancing policy chain.
• Added LWT-aware query planning to TokenAwarePolicy (replicas-only, local-first, filters DOWN/IGNORED), configurable via QueryOptions.RequestRoutingMethod.
• Added unit tests covering LWT replica filtering and updated README with the new optimization.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
driver-core/src/main/java/com/datastax/driver/core/policies/TokenAwarePolicy.java	Adds LWT-specific replica-only query plan behavior and related Javadoc.
driver-core/src/main/java/com/datastax/driver/core/RequestHandler.java	Removes LWT replica-routing bypass so LWT uses the LBP chain.
driver-core/src/main/java/com/datastax/driver/core/QueryOptions.java	Adds configurable LWT routing method (RequestRoutingMethod) and default.
driver-core/src/test/java/com/datastax/driver/core/policies/TokenAwarePolicyTest.java	Adds unit tests for LWT local-first ordering and filtering behavior.
README.md	Documents the LWT optimization feature at a high level.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.