druid/extensions-core
Agustin Gonzalez 8e5048e643
Avoid memory mapping hydrants after they are persisted & after they are merged for native batch ingestion (#11123)
* Avoid mapping hydrants in create segments phase for native ingestion

* Drop queriable indices after a given sink is fully merged

* Do not drop memory mappings for realtime ingestion

* Style fixes

* Renamed to match use case better

* Rollback memoization code and use the real time flag instead

* Null ptr fix in FireHydrant toString plus adjustments to memory pressure tracking calculations

* Style

* Log some count stats

* Make sure sinks size is obtained at the right time

* BatchAppenderator unit test

* Fix comment typos

* Renamed methods to make them more readable

* Move persisted metadata from FireHydrant class to AppenderatorImpl. Removed superfluous differences and fix comment typo. Removed custom comparator

* Missing dependency

* Make persisted hydrant metadata map concurrent and better reflect the fact that keys are Java references. Maintain persisted metadata when dropping/closing segments.

* Replaced concurrent variables with normal ones

* Added   batchMemoryMappedIndex "fallback" flag with default "false". Set this to "true" make code fallback to previous code path.

* Style fix.

* Added note to new setting in doc, using Iterables.size (and removing a dependency), and fixing a typo in a comment.

* Forgot to commit this edited documentation message
2021-05-11 14:34:26 -07:00
..
avro-extensions add avro stream input format (#11040) 2021-04-12 21:53:41 -07:00
azure-extensions Bump dev version to 0.22.0-SNAPSHOT (#10759) 2021-01-15 13:16:23 -08:00
datasketches fix count and average SQL aggregators on constant virtual columns (#11208) 2021-05-10 13:41:48 -07:00
druid-aws-rds-extensions Bump dev version to 0.22.0-SNAPSHOT (#10759) 2021-01-15 13:16:23 -08:00
druid-basic-security basic security extension ignore permissions that use unknown ResourceType or Action (#10896) 2021-02-23 14:49:09 -08:00
druid-bloom-filter fix count and average SQL aggregators on constant virtual columns (#11208) 2021-05-10 13:41:48 -07:00
druid-kerberos Bump dev version to 0.22.0-SNAPSHOT (#10759) 2021-01-15 13:16:23 -08:00
druid-pac4j Bump dev version to 0.22.0-SNAPSHOT (#10759) 2021-01-15 13:16:23 -08:00
druid-ranger-security Bump dev version to 0.22.0-SNAPSHOT (#10759) 2021-01-15 13:16:23 -08:00
ec2-extensions Bump dev version to 0.22.0-SNAPSHOT (#10759) 2021-01-15 13:16:23 -08:00
google-extensions GCS lookup support (#11026) 2021-03-30 01:40:41 +05:30
hdfs-storage DruidInputSource: Fix issues in column projection, timestamp handling. (#10267) 2021-03-25 10:32:21 -07:00
histogram fix count and average SQL aggregators on constant virtual columns (#11208) 2021-05-10 13:41:48 -07:00
kafka-extraction-namespace upgrade Apache Kafka to 2.8.0 (#11139) 2021-04-24 08:27:07 -07:00
kafka-indexing-service Avoid memory mapping hydrants after they are persisted & after they are merged for native batch ingestion (#11123) 2021-05-11 14:34:26 -07:00
kinesis-indexing-service Avoid memory mapping hydrants after they are persisted & after they are merged for native batch ingestion (#11123) 2021-05-11 14:34:26 -07:00
kubernetes-extensions k8s discovery module: fix issue for druid.host being more than 63chars not permitted as k8s resource label value (#10961) 2021-04-07 17:45:28 -07:00
lookups-cached-global Allow list for JDBC connection properties to address CVE-2021-26919 (#11047) 2021-04-01 17:30:47 -07:00
lookups-cached-single Enable rewriting certain inner joins as filters. (#11068) 2021-04-14 10:49:27 -07:00
mysql-metadata-storage Enforce allow list for JDBC properties by default (#11063) 2021-04-06 19:46:19 -07:00
orc-extensions DruidInputSource: Fix issues in column projection, timestamp handling. (#10267) 2021-03-25 10:32:21 -07:00
parquet-extensions DruidInputSource: Fix issues in column projection, timestamp handling. (#10267) 2021-03-25 10:32:21 -07:00
postgresql-metadata-storage Enforce allow list for JDBC properties by default (#11063) 2021-04-06 19:46:19 -07:00
protobuf-extensions add protobuf inputformat (#11018) 2021-04-12 22:03:13 -07:00
s3-extensions DruidInputSource: Fix issues in column projection, timestamp handling. (#10267) 2021-03-25 10:32:21 -07:00
simple-client-sslcontext Bump dev version to 0.22.0-SNAPSHOT (#10759) 2021-01-15 13:16:23 -08:00
stats fix count and average SQL aggregators on constant virtual columns (#11208) 2021-05-10 13:41:48 -07:00