Kafka: Fixes needlessly low interpretation of maxRowsInMemory. (#5034)

AppenderatorImpl already applies maxRowsInMemory across all sinks. So dividing by
the number of Kafka partitions is pointless and effectively makes the interpretation
of maxRowsInMemory lower than expected.

This undoes one of the two changes from #3284, which fixed the original bug twice.
In this, that's worse than fixing it once.
This commit is contained in:
Gian Merlino 2017-11-02 13:45:04 -06:00 committed by GitHub
parent 21e1bf68f6
commit 5da0241ac8
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 1 additions and 20 deletions

View File

@ -957,12 +957,9 @@ public class KafkaIndexTask extends AbstractTask implements ChatHandler
private Appenderator newAppenderator(FireDepartmentMetrics metrics, TaskToolbox toolbox)
{
final int maxRowsInMemoryPerPartition = (tuningConfig.getMaxRowsInMemory() /
ioConfig.getStartPartitions().getPartitionOffsetMap().size());
return Appenderators.createRealtime(
dataSchema,
tuningConfig.withBasePersistDirectory(toolbox.getPersistDir())
.withMaxRowsInMemory(maxRowsInMemoryPerPartition),
tuningConfig.withBasePersistDirectory(toolbox.getPersistDir()),
metrics,
toolbox.getSegmentPusher(),
toolbox.getObjectMapper(),

View File

@ -185,22 +185,6 @@ public class KafkaTuningConfig implements TuningConfig, AppenderatorConfig
);
}
public KafkaTuningConfig withMaxRowsInMemory(int rows)
{
return new KafkaTuningConfig(
rows,
maxRowsPerSegment,
intermediatePersistPeriod,
basePersistDirectory,
maxPendingPersists,
indexSpec,
true,
reportParseExceptions,
handoffConditionTimeout,
resetOffsetAutomatically
);
}
@Override
public boolean equals(Object o)
{