Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,372 Commits 113 Branches 505 Tags 726 MiB
Commit Graph

20 Commits

Author SHA1 Message Date
Jihoon Son 0db696b7c9 Fix CompactionTask doc (#5351) 
* Fix CompactionTask doc

* Update coordinator doc
 2018-02-05 22:38:34 -08:00
Roman Leventov a7a6a0487e Replace IOPeon with SegmentWriteOutMedium; Improve buffer compression (#4762) 
* Replace IOPeon with OutputMedium; Improve compression

* Fix test

* Cleanup CompressionStrategy

* Javadocs

* Add OutputBytesTest

* Address comments

* Random access in OutputBytes and GenericIndexedWriter

* Fix bugs

* Fixes

* Test OutputBytes.readFully()

* Address comments

* Rename OutputMedium to SegmentWriteOutMedium and OutputBytes to WriteOutBytes

* Add comments to ByteBufferInputStream

* Remove unused declarations
 2017-12-04 18:04:27 -08:00
Jihoon Son 5f3c863d5e Add compaction task (#4985) 
* Add compaction task

* added doc

* use combining aggregators

* address comments

* add support for dimensionsSpec

* fix getUniqueDims and getUniqueMetics

* find unique dimensionsSpec

* fix compilation

* add unit test

* fix test

* fix test

* test for different dimension orderings and types, and doc for type and ordering

* add control for custom ordering and type

* update doc

* fix compile

* fix compile

* add segments param

* fix serde error

* fix build
 2017-11-03 21:55:27 -06:00
Jihoon Son dfa9cdc982 Prioritized locking (#4550) 
* Implementation of prioritized locking

* Fix build failure

* Fix tc fail

* Fix typos

* Fix IndexTaskTest

* Addressed comments

* Fix test

* Fix spacing

* Fix build error

* Fix build error

* Add lock status

* Cleanup suspicious method

* Add nullables

*  add doInCriticalSection to TaskLockBox and revert return type of task actions

* fix build

* refactor CriticalAction

* make replaceLock transactional

* fix formatting

* fix javadoc

* fix build
 2017-10-11 23:16:31 -07:00
Jihoon Son d5606bc558 Passing lockTimeout as a parameter for TaskLockbox.lock() (#4549) 
* Passing lockTimeout as a parameter for TaskLockbox.lock()

* Remove TIME_UNIT

* Fix tc fail

* Add taskLockTimeout to TaskContext

* Add caution
 2017-08-08 18:21:07 -07:00
Jihoon Son cc20260078 Early publishing segments in the middle of data ingestion (#4238) 
* Early publishing segments in the middle of data ingestion

* Remove unnecessary logs

* Address comments

* Refactoring the patch according to #4292 and address comments

* Set the total shard number of NumberedShardSpec to 0

* refactoring

* Address comments

* Fix tests

* Address comments

* Fix sync problem of committer and retry push only

* Fix doc

* Fix build failure

* Address comments

* Fix compilation failure

* Fix transient test failure
 2017-07-10 22:35:36 -07:00
Roman Leventov 05d58689ad Remove the ability to create segments in v8 format (#4420) 
* Remove ability to create segments in v8 format

* Fix IndexGeneratorJobTest

* Fix parameterized test name in IndexMergerTest

* Remove extra legacy merging stuff

* Remove legacy serializer builders

* Remove ConciseBitmapIndexMergerTest and RoaringBitmapIndexMergerTest
 2017-06-26 13:21:39 -07:00
Jihoon Son 1150bf7a2c Refactoring Appenderator Driver (#4292) 
* Refactoring Appenderator

1) Added publishExecutor and handoffExecutor for background publishing and handing segments off
2) Change add() to not move segments out in it

* Address comments

1) Remove publishTimeout for KafkaIndexTask
2) Simplifying registerHandoff()
3) Add increamental handoff test

* Remove unused variable

* Add persist() to Appenderator and more tests for AppenderatorDriver

* Remove unused imports

* Fix strict build

* Address comments
 2017-06-02 07:09:11 +09:00
Jihoon Son 733dfc9b30 Add PrefetchableTextFilesFirehoseFactory for cloud storage types (#4193) 
* Add PrefetcheableTextFilesFirehoseFactory

* fix comment

* exception handling

* Fix wrong json property

* Remove ReplayableFirehoseFactory and fix misspelling

* Defer object initialization

* Add a temporaryDirectory parameter to FirehoseFactory.connect()

* fix when cache and fetch are disabled

* Address comments

* Add more test

* Increase timeout for test

* Add wrapObjectStream

* Move methods to Firehose from PrefetchableFirehoseFactory

* Cleanup comment

* add directory listing to s3 firehose

* Rename a variable

* Addressing comments

* Update document

* Support disabling prefetch

* Fix race condition

* Add fetchLock

* Remove ReplayableFirehoseFactoryTest

* Fix compilation error

* Fix test failure

* Address comments

* Add default implementation for new method
 2017-05-18 15:37:18 +09:00
kaijianding 19ac1c7c2c Add SameIntervalMergeTask for easier usage of MergeTask (#3981) 
* Add SameIntervalMergeTask for easier usage of MergeTask

* fix a bug and add ut

* remove same_interval_merge_sub from Task.java and remove other no needed code
 2017-03-06 11:21:32 -06:00
kaijianding ef6a19c81b buildV9Directly in MergeTask and AppendTask (#3976) 
* buildV9Directly in MergeTask and AppendTask

* add doc
 2017-02-28 10:04:32 -08:00
David Lim ff52581bd3 IndexTask improvements (#3611) 
* index task improvements

* code review changes

* add null check
 2017-01-18 14:24:37 -08:00
Gian Merlino bcd20441be Make buildV9Directly the default. (#3688) 2016-11-14 09:29:32 -08:00
Gian Merlino 27bd5cb13a Add forceExtendableShardSpecs option to Hadoop indexing, IndexTask. (#3473) 
Fixes #3241.
 2016-09-21 13:40:04 -06:00
Dave Li c4e8440c22 Adds long compression methods (#3148) 
* add read

* update deprecated guava calls

* add write and vsizeserde

* add benchmark

* separate encoding and compression

* add header and reformat

* update doc

* address PR comment

* fix buffer order

* generate benchmark files

* separate encoding strategy and format

* fix benchmark

* modify supplier write to channel

* add float NONE handling

* address PR comment

* address PR comment 2
 2016-08-30 16:17:46 -07:00
kaijianding 50d52a24fc ability to not rollup at index time, make pre aggregation an option (#3020) 
* ability to not rollup at index time, make pre aggregation an option

* rename getRowIndexForRollup to getPriorIndex

* fix doc misspelling

* test query using no-rollup indexes

* fix benchmark fail due to jmh bug
 2016-08-02 11:13:05 -07:00
Gian Merlino ea03906fcf Configurable compressRunOnSerialization for Roaring bitmaps. (#3228) 
Defaults to true, which is a change in behavior (this used to be false and unconfigurable).
 2016-07-08 10:24:19 +05:30
Björn Zettergren 2462c82c0e New defaults for maxRowsInMemory rowFlushBoundary 
To bring consistency to docs and source this commit changes the default
values for maxRowsInMemory and rowFlushBoundary to 75000 after
discussion in PR https://github.com/druid-io/druid/pull/2457.

The previous default was 500000 and it's lower now on the grounds that
it's better for a default to be somewhat less efficient, and work,
than to reach for the stars and possibly result in
"OutOfMemoryError: java heap space" errors.
 2016-03-01 13:50:28 +01:00
Gian Merlino 3a996216bd Multivalued dimensions can be compressed since 0.8.0. 2016-02-17 08:33:21 -08:00
fjy 1aa363cea7 new quickstart 2016-02-04 09:37:38 -08:00