Performance overhead of ReactiveCassandraTemplate

[samueldlightfoot]

As brought to light from someone on the Gitter channel, it appears there is a significant performance overhead for query flows that use the ReactiveCassandraTemplate. Notably this includes the reactive @'Repository classes.

The throughput test ran compared writing using ReactiveCassandraRepository::insert vs ReactiveCqlTemplate::execute (cql, args). Local testing gave the following results (writes per second):

• ReactiveCassandraRepository::insert 4000 writes/s
• ReactiveCqlTemplate::execute 7500 writes/s

As you can see there is a significant difference.

I have tried the test with prepared statements both enabled and disabled and it makes little difference. CPU profiling shows no hotspots for the repository inserts and no discernible difference in the overall profiles. Could it be the mapping layer that builds the statements? I will continue to dig into possibilities.

The test showing the throughput difference can be found here (credits to original author piddubnyi): https://github.com/samueldlightfoot/spring-data-cassandra-performnace

Here are the JProfiler snapshots for runs of both for anyone interested (I may be missing something in my analysis):
Spring Data Performance.zip

Entity:

@Table("snapshot")
@Value
@Builder
@EqualsAndHashCode(callSuper = false)
@RequiredArgsConstructor
public class SnapshotRecord {

    @PrimaryKeyColumn(ordinal = 0, type = PrimaryKeyType.PARTITIONED)
    long id;
    @PrimaryKeyColumn(ordinal = 1, type = PrimaryKeyType.PARTITIONED)
    short market;
    @PrimaryKeyColumn(ordinal = 3, type = PrimaryKeyType.CLUSTERED)
    Instant slot;

    double value;
}

Repository:

public interface SnapshotRepository extends ReactiveCrudRepository<SnapshotRecord, Long> {

    default Mono<Boolean> saveViaCql(ReactiveCqlOperations cqlOps, SnapshotRecord record) {
        return cqlOps.execute(
                "INSERT INTO snapshot (id, market,slot,value) VALUES (?,?,?,?) USING TIMESTAMP ?;",
                ps -> {
                    return ps.bind(
                            record.getId(),
                            record.getMarket(),
                            record.getSlot(),
                            record.getValue(),
                            record.getSlot().toEpochMilli() * 1000
                    );
                }
   );
    }
}

Runner:

Flux<SnapshotRecord> data = Flux.generate(Object::new, (state, sink) -> {
            ThreadLocalRandom random = ThreadLocalRandom.current();
            sink.next(
                new SnapshotRecord(
                    random.nextLong(),
                    (short) random.nextInt(),
                    Clock.systemUTC().instant(),
                    random.nextDouble()
                )
            );
            return state;
        });
        subscription = data
//.flatMap((SnapshotRecord record) -> repository.saveViaCql(cqlOps, record), 512, 2048)
.flatMap(repository::save, 512, 2048) //doing this runs almost 2x slower than previous line
            .doOnNext(d -> success.incrementAndGet())
            .onErrorContinue((throwable, object) -> fail.incrementAndGet())
            .subscribe();

[mp911de]

After a first investigation, the main difference comes from the fact of using the CassandraTemplate that works internally a lot with mapping while the CQL template is a tiny layer above the driver. Even without actual Cassandra interaction, I can yield about 20000 inserts/sec (with Cassandra its 10000 writes/sec) so likely the hotspot is somewhere in CassandraTemplate.

[mp911de]

Upon further investigation, it seems that Cassandra is the fastest write-store that we currently support. Disabling the database interaction helped to reveal a few things that didn't bubble up because the actual database was so slow in other Spring Data modules so we never noticed these.

We've identified a few things that we could optimize:

1. Caching of type information through annotation lookups (used typically to determine the Cassandra target type)
2. Caching of the CassandraColumnType
3. Avoid calling Spring's ConversionService (the assignability checks on our side that prevent calling the conversion service didn't consider primitives)
4. Reactive return type information caching (to decorate return types with Repository invocation listeners)

Applying these changes I can yield now about 60000 inserts/sec (without Cassandra, with Cassandra about 16000 which is close to 18726 using plain CQL).

The overhead in performance drag becomes way smaller and if you consider what CassandraTemplate gives you (entity callbacks, lifecycle events, statement creation, value conversion) then the overhead now of about 12% becomes much better than 50% overhead.

[mp911de]

During the analysis of object allocations a method became visible that constructs an INSERT from a collection of values. Due to the immutable nature, the construction seems rather expensive because the objects are created individually and not as batch. Maybe an optimization for the query builders, @adutra?