9dff0e5dbd
Add hadoop classloader isolation tests for HadoopTask
2016-02-19 11:15:53 -08:00
ddf913d626
Merge pull request #2508 from gianm/ftr-shutdown-logging
...
ForkingTaskRunner: Better logging during orderly shutdown.
2016-02-19 10:02:24 -08:00
c0c6cf77fa
ForkingTaskRunner: Better logging during orderly shutdown.
2016-02-19 09:17:16 -08:00
243ac5399b
Harmonize realtime indexing loop across the task and standalone nodes.
...
- Both now catch ParseExceptions on plumber.add (see https://groups.google.com/d/topic/druid-user/wmiRDvx2RvM/discussion )
- Standalone now treats IndexSizeExceededException as fatal (previously only the task did)
2016-02-19 07:34:15 -08:00
87752be740
Make HadoopTasks's classloader a single one
2016-02-18 20:58:09 -08:00
07d714b1b5
Fixed equal distribution strategy when exist disable middleManager with same currCapacityUsed.
2016-02-17 08:40:42 +01:00
5779b32742
Merge pull request #2439 from metamx/fix2435
...
Make QuotableWhiteSpaceSplitter able to take JSON
2016-02-11 13:14:43 -06:00
40ade32a1f
Fix dependencies.
...
* Don't put druid****selfcontained.jar at the end of the hadoop isolated classpath
* Add `<scope>provided</scope>` to prevent repeated dependency inclusion in the extension directories
2016-02-11 07:30:14 -08:00
3a6452c6d4
Make QuotableWhiteSpaceSplitter able to take json
...
* Fixes #2435
2016-02-10 16:42:14 -08:00
91f23583f5
Merge pull request #2436 from gianm/mm-less-suppressey
...
Harmonize znode writing code in RTR and Worker.
2016-02-10 16:11:30 -08:00
fa92b77f5a
Harmonize znode writing code in RTR and Worker.
...
- Throw most exceptions rather than suppressing them, which should help
detect problems. Continue suppressing exceptions that make sense to
suppress.
- Handle payload length checks consistently, and improve error message.
- Remove unused WorkerCuratorCoordinator.announceTaskAnnouncement method.
- Max znode length should be int, not long.
- Add tests.
2016-02-10 14:52:00 -08:00
2bde8b1d68
Make hadoop classpath isolation more explicit
...
* Fixes #2428
2016-02-10 12:09:17 -08:00
a0728fa854
Allow ScalingStats to be null
...
* Fixes #2378
2016-02-02 18:01:01 -08:00
7853a9cc41
clean up TaskLifecycleTest
2016-01-31 11:19:20 -06:00
5fd4b79373
RealtimeIndexTask: Fix NPE caused by calling stopGracefully before a firehose had been connected.
2016-01-29 11:20:23 -08:00
c4fde52160
Fix 'graceful shutdown aborted' log message in ThreadPoolTaskRunner.
2016-01-29 11:07:17 -08:00
dcb7830330
Merge pull request #984 from drcrallen/thread-priority-rebase
...
Use thread priorities. (aka set `nice` values for background-like tasks)
2016-01-21 15:02:34 +05:30
66e74b1a63
Minor field name change in RemoteTaskRunnerFactory to be more descriptive
...
* Addresses https://github.com/druid-io/druid/pull/2309#discussion_r50335081
2016-01-20 17:43:20 -08:00
3152d08844
Fix overlord scheduled executor injection
...
* Fixes https://github.com/druid-io/druid/issues/2308
2016-01-20 14:16:14 -08:00
2e1d6aaf3d
Use thread priorities. (aka set `nice` values for background-like tasks)
...
* Defaults the thread priority to java.util.Thread.NORM_PRIORITY in io.druid.indexing.common.task.AbstractTask
* Each exec service has its own Task Factory which is assigned a priority for spawned task. Therefore each priority class has a unique exec service
* Added priority to tasks as taskPriority in the task context. <0 means low, 0 means take default, >0 means high. It is up to any particular implementation to determine how to handle these numbers
* Add options to ForkingTaskRunner
* Add "-XX:+UseThreadPriorities" default option
* Add "-XX:ThreadPriorityPolicy=42" default option
* AbstractTask - Removed unneded @JsonIgnore on priority
* Added priority to RealtimePlumber executors. All sub-executors (non query runners) get Thread.MIN_PRIORITY
* Add persistThreadPriority and mergeThreadPriority to realtime tuning config
2016-01-20 14:00:31 -08:00
ac6c90e657
Merge pull request #1953 from metamx/taskRunnerResourceManagement
...
Move resource managemnt to be the responsibility of the TaskRunner
2016-01-20 22:27:47 +05:30
df2906a91c
Merge pull request #2290 from gianm/index-merger-v9-stuff
...
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime.
2016-01-19 13:04:00 -08:00
0c31f007fc
Merge pull request #1728 from himanshug/aggregators_in_segment_metadata
...
Store AggregatorFactory[] in segment metadata
2016-01-19 12:55:49 -08:00
fe841fd961
Merge pull request #2118 from guobingkun/fix_segment_loading
...
Fix loading segment for historical
2016-01-19 14:25:48 -06:00
a99aef29a1
adding aggregators to segment metadata
2016-01-19 14:23:39 -06:00
1dcf22edb7
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime nodes.
...
Also parameterize some tests to run with/without buildV9Directly:
- IndexGeneratorJobTest
- RealtimeIndexTaskTest
- RealtimePlumberSchoolTest
2016-01-19 12:15:06 -08:00
c4ad50f92c
Fix loading segment for historical
...
Historical will drop a segment that shouldn't be dropped in the following scenario:
Historical node tried to load segmentA, but failed with SegmentLoadingException,
then ZkCoordinator called removeSegment(segmentA, blah) to schedule a runnable that would drop segmentA by deleting its files. Now, before that runnable executed, another LOAD request was sent to this historical, this time historical actually succeeded on loading segmentA and announced it. But later on, the scheduled drop-of-segment runnable started executing and removed the segment files, while historical is still announcing segmentA.
2016-01-19 10:29:49 -06:00
164b0aad7a
removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class
...
instead of a Map to store segment metadata
2016-01-18 22:03:46 -06:00
82ff98c2bf
add config for build v9 directly and update docs
2016-01-16 11:26:34 +08:00
976d4c965b
Move resource managemnt to be the responsibility of the TaskRunner
2016-01-13 10:38:22 -08:00
82bdfbbbf1
Merge pull request #2155 from metamx/taskConfigTmpdir
...
Make TaskConfig pull from java.io.tmpdir
2016-01-05 13:58:39 -06:00
45f402f22f
increase timeout
...
tune timeouts
2016-01-05 19:06:04 +05:30
e18301d99c
Make TaskConfig pull from java.io.tmpdir
...
* Also makes paths built off of java.nio.file.Paths instead of String.format
2016-01-04 10:17:08 -08:00
b5c094d951
Fixes #2180
2016-01-01 16:56:41 -08:00
b68265399c
Merge pull request #2168 from druid-io/remove-indexmaker
...
Remove IndexMaker
2015-12-30 12:24:29 +05:30
df893dbaf8
Merge pull request #2141 from gianm/fix-restoring-realtime
...
Fix some problems with restoring
2015-12-30 10:44:45 +05:30
7ffa706655
Merge pull request #2152 from metamx/add-taskId
...
Add taskId to realtimeMetrics
2015-12-29 10:33:40 -08:00
38b0f1fbc2
fix transient failures in unit tests
2015-12-28 20:03:30 -08:00
faf421726b
remove IndexMaker
2015-12-28 14:19:02 -08:00
8cb52bddd8
Merge pull request #2140 from navis/fix-sporadic-testfail4
...
Fix sporadic fail of RemoteTaskRunnerTest#testWorkerRemoved
2015-12-27 14:55:49 -08:00
9aa62e4631
Merge pull request #2154 from navis/fix-testfail-WorkerTaskMonitorTest
...
Fix sporadic fail of WorkerTaskMonitorTest#testRunTask
2015-12-23 20:52:33 -08:00
a8f6c6110d
Fix sporadic fail of WorkerTaskMonitorTest#testRunTask
2015-12-24 02:31:30 +09:00
2c3c4a3f8f
Another try to fix xxServerViewTests
2015-12-24 02:13:40 +09:00
978a3fd8ae
Add taskId to realtimeMetrics
...
Add task Id to Realtime Metrics
2015-12-23 18:05:25 +05:30
32edd1538d
RealtimeIndexTask: Fix a couple of problems with restoring.
...
- Shedding locks at startup is bad, we actually want to keep them. Stop doing that.
- stopGracefully now interrupts the run thread if had started running finishJob. This avoids
waiting for handoff unnecessarily.
2015-12-22 16:04:47 -08:00
f4ce2b9bc5
TaskLockbox: Consider active tasks active even if they have no locks.
2015-12-22 16:04:16 -08:00
bad270b6c4
druid.indexer.task.restoreTasksOnRestart configuration.
2015-12-22 10:59:15 -08:00
8a179fc273
Fix sporadic fail of RemoteTaskRunnerTest#testWorkerRemoved
2015-12-22 14:33:37 +09:00
5e178499e8
trying to fix transient errors in testRealtimeIndexTask() by increasing overall timeout and unlimited wait for segment publish
2015-12-21 00:11:20 -06:00
14229ba0f2
Merge pull request #1922 from metamx/jsonIgnoresFinalFields
...
Change DefaultObjectMapper to NOT overwrite final fields unless explicitly asked to
2015-12-18 15:38:32 -08:00
1e5aa2f3ac
fix getType() and Json serialization in ClientMergeQuery and add serde tests
2015-12-15 12:08:43 -06:00
a32906c7fd
Remove FilteredServerView
2015-12-09 01:54:12 +05:30
9491e8de3b
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoffs
...
- fixes #1970
- extracted out segment handoff callbacks in SegmentHandoffNotifier
which is responsible for tracking segment handoffs and doing callbacks
when handoff is complete.
- Coordinator now maintains a view of segments in the cluster, this
will affect the jam heap requirements for the overlord for large
clusters.
realtime index task and nodes now use HTTP end points exposed by the
coordinator to get serverView
review comment
fix realtime node guide injection
review comments
make test not rely on scheduled exec
fix compilation
fix import
review comment
introduce immutableSegmentLoadInfo
fix son reading
remove unnecessary logging
2015-12-09 01:54:09 +05:30
62ba9ade37
unifying license header in all java files
2015-12-05 22:16:23 -06:00
20544d409b
Merge pull request #1988 from himanshug/multi-interval-batch-delta
...
support multiple intervals in dataSource inputSpec
2015-12-04 09:07:52 -08:00
020a5e7081
Merge pull request #2024 from metamx/fairBigTaskQueueLock
...
Make the TaskQueue big lock fair
2015-12-03 19:32:53 -08:00
61aaa09012
support multiple intervals in dataSource input spec
2015-12-03 21:28:04 -06:00
86f0a36e83
support multiple intervals in SegmentListUsedAction
2015-12-03 21:28:04 -06:00
221fb95d07
add support for getting used segments for multiple interval in IndexerMetadataStorageCoordinator
2015-12-03 21:28:04 -06:00
dbaaa6af92
Make the TaskQueue big lock fair
2015-12-01 19:13:07 -08:00
1eb8211346
Add datasource and taskId to metrics emitted by peons
...
This PR adds the datasource and taskId to the jvm and sys metrics
emitted by the peons.
fix spelling
review comment
review comment
2015-12-01 23:20:59 +05:30
8e83d800d6
Merge pull request #1881 from gianm/restartable-tasks
...
Restorable indexing tasks
2015-11-23 21:14:37 -08:00
501dcb43fa
Some changes that make it possible to restart tasks on the same hardware.
...
This is done by killing and respawning the jvms rather than reconnecting to existing
jvms, for a couple reasons. One is that it lets you restore tasks after server reboots
too, and another is that it lets you upgrade all the software on a box at once by just
restarting everything.
The main changes are,
1) Add "canRestore" and "stopGracefully" methods to Tasks that say if a task can
stop gracefully, and actually do a graceful stop. RealtimeIndexTask is the only
one that currently implements this.
2) Add "stop" method to TaskRunners that attempts to do an orderly shutdown.
ThreadPoolTaskRunner- call stopGracefully on restorable tasks, wait for exit
ForkingTaskRunner- close output stream to restorable tasks, wait for exit
RemoteTaskRunner- do nothing special, we actually don't want to shutdown
3) Add "restore" method to TaskRunners that attempts to bootstrap tasks from last run.
Only ForkingTaskRunner does anything here. It maintains a "restore.json" file with
a list of restorable tasks.
4) Have the CliPeon's ExecutorLifecycle lock the task base directory to avoid a restored
task and a zombie old task from stomping on each other.
2015-11-23 11:22:08 -08:00
666d785787
Switch TaskActions from Optionals to nullable.
...
Deserialization of Optionals does not work quite right- they come back as actual
nulls, rather than absent Optionals. So these probably only ever worked for the local
task action client.
2015-11-20 09:14:07 -08:00
21c84b5ff7
Merge pull request #1896 from gianm/allocate-segment
...
SegmentAllocateAction (fixes #1515 )
2015-11-18 21:05:46 -08:00
e52c156066
Merge pull request #1880 from gianm/rtr-adjust
...
RTR: Ensure that there is only one cleanup task scheduled for a worker at once.
2015-11-18 15:12:55 -08:00
8fcf2403e3
Merge pull request #1943 from metamx/realtime-caching
...
Enable caching on intermediate realtime persists
2015-11-17 15:06:43 -08:00
dbe201aeed
Merge pull request #1929 from pjain1/jetty_threads
...
separate ingestion and query thread pool
2015-11-17 12:14:25 -08:00
6c498b7d4a
separate ingestion and query thread pool
2015-11-17 13:42:41 -06:00
d7eb2f717e
enable query caching on intermediate realtime persists
2015-11-17 10:58:00 -08:00
46527a9610
Merge pull request #1954 from metamx/fix-stupid-aws-limit
...
EC2 autoscaler: avoid hitting aws filter limits
2015-11-13 10:52:35 -08:00
4f46d457f1
Merge pull request #1947 from noddi/feature/count-parameter-history-endpoints
...
Add count parameter to history endpoints
2015-11-12 10:23:44 -08:00
749ac12f88
EC2 autoscaler: avoid hitting aws filter limits
2015-11-11 20:28:06 -08:00
465cbcf9a7
Merge pull request #1956 from metamx/remove-unused-imports
...
Cleanup + remove unused imports
2015-11-11 17:36:47 -08:00
e4e5f0375b
SegmentAllocateAction ( fixes #1515 )
...
This is a feature meant to allow realtime tasks to work without being told upfront
what shardSpec they should use (so we can potentially publish a variable number
of segments per interval).
The idea is that there is a "pendingSegments" table in the metadata store that
tracks allocated segments. Each one has a segment id (the same segment id we know
and love) and is also part of a sequence.
The sequences are an idea from @cheddar that offers a way of doing replication.
If there are N tasks reading exactly the same data with exactly the same logic
(think Kafka tasks reading a fixed range of offsets) then you can place them
in the same sequence, and they will generate the same sequence of segments.
2015-11-11 16:54:35 -08:00
6e5d2c6745
Add count parameter to history endpoints.
2015-11-11 23:03:57 +01:00
fa6142e217
cleanup and remove unused imports
2015-11-11 12:25:21 -08:00
c197a4cf32
fix #1918 , add unit tests for RemoteTaskActionClient
2015-11-12 03:15:17 +08:00
abae47850a
Add backwards compatability for PR #1922
2015-11-11 10:27:00 -08:00
1df4baf489
Move Jackson Guice adapters into io.druid
...
* Removes access to protected methods in com.fasterxml
* Eliminates druid-common's use of foreign package com.fasterxml
2015-11-09 10:50:45 -08:00
fc55314d1c
ForkingTaskRunner: Log without buffering.
...
In #933 the ForkingTaskRunner's logging was changed to buffered from
unbuffered. This means that the last few KB of the logs are generally
not visible while a task is running, which makes debugging running
tasks difficult.
2015-11-07 15:16:53 -08:00
929b981710
Change DefaultObjectMapper to NOT overwrite final fields unless explicitly asked to
2015-11-05 18:10:13 -08:00
cb409ee928
RemoteTaskActionClient: Fix statusCode check.
2015-11-05 10:03:49 -08:00
8f231fd3e3
cleanup druid codebase
2015-11-04 13:59:53 -08:00
84f7d8d264
making static final variables in HadoopDruidIndexerConfig upper case
2015-11-02 23:24:26 -06:00
8b67417ac8
make methods in Index[Merger,Maker,IO] non-static so that they can have
...
appropriate ObjectMapper injected instead of creating one statically
2015-11-02 23:24:26 -06:00
16ae8866b8
Log and continue on failure to schedule cleanup for missing workers at startup.
2015-10-28 08:10:54 -07:00
513bc76252
RTR: Ensure that there is only one cleanup task scheduled for a worker at once.
...
This is accomplished by making sure that scheduleTasksCleanupForWorker is only called
from the PathChildrenCache event thread, having it cancel existing cleanup tasks when
it adds a new one, and having tasks check on finish that the thing they are removing
from the task list is actually themselves.
2015-10-27 21:16:58 -07:00
ea2267e08c
Merge pull request #1868 from gianm/fix-announcements
...
Historical and MiddleManager server announcements should not remove parents.
2015-10-27 14:50:05 -07:00
7df7370935
Merge pull request #1862 from metamx/indexingServiceMMGone
...
Add timeout to shutdown request to middle manager for indexing service
2015-10-27 14:38:01 -07:00
44a2b204df
Add timeout to shutdown request to middle manager for indexing service
2015-10-27 13:56:03 -07:00
4b92752deb
Historical and MiddleManager server announcements should not remove parents.
...
Removing parent paths causes watchers of the "announcements" path to get stuck
and stop seeing new updates.
2015-10-27 08:06:11 -07:00
4914925d65
New extension loading mechanism
...
1) Remove maven client from downloading extensions at runtime.
2) Provide a way to load Druid extensions and hadoop dependencies through file system.
3) Refactor pull-deps so that it can download extensions into extension directories.
4) Add documents on how to use this new extension loading mechanism.
5) Change the way how Druid tarball is generated. Now all the extensions + hadoop-client 2.3.0
are packaged within the Druid tarball.
2015-10-21 14:22:36 -05:00
b7c68ec449
Merge pull request #1842 from metamx/DRUID-1841
...
Do not pass `druid.indexer.runner.javaOpts` to Peon as a property
2015-10-21 13:15:36 -05:00
e4ac78e43d
bump next snapshot to 0.9.0
2015-10-20 13:46:13 -07:00
532e1c9fd5
Do not pass `druid.indexer.runner.javaOpts` to Peon as a property
...
* Still places `druid.indexer.runner.javaOpts` on the command line, but the Peon no longer tries to have the property `druid.indexer.runner.javaOpts` set
* Fixes https://github.com/druid-io/druid/issues/1841
2015-10-20 09:24:01 -07:00
4c2c7a2c37
update version to 0.8.3
2015-10-14 21:40:55 -07:00
bf11723a52
Update usages of io.druid.client.selector.Server to build URL or URI directly instead of using String.format
2015-10-12 12:30:56 -07:00
2d847ad654
Merge pull request #1730 from metamx/union-queries-fix
...
fix #1727 - Union bySegment queries fix
2015-09-29 12:23:25 -07:00
573aa96bd6
fix #1727 - Union bySegment queries fix
...
Fixes #1727 .
revert to doing merging for results for union queries on broker.
revert unrelated changes
Add test for union query runner
Add test
remove unused imports
fix imports
fix renamed file
fix test
update docs.
2015-09-29 23:32:36 +05:30
d2e400f063
Merge pull request #1740 from metamx/validate-locks
...
fix #1715
2015-09-29 09:38:42 -07:00
25bbc0b923
Merge pull request #1778 from gianm/redirect-fixes
...
Redirect fixes
2015-09-25 09:54:48 -07:00
348172203f
OverlordRedirectInfo: Fix ability to detect that there is no leader.
2015-09-25 09:30:09 -07:00
b630720164
fail task if finishjob throws any exception
...
add realtime task failure test
2015-09-25 10:55:45 -05:00
aa9d90355e
Merge pull request #1772 from gianm/fix-overlord-startup
...
RemoteTaskRunner: Fix for starting an overlord before any workers ever existed.
2015-09-24 21:55:03 -07:00
63bf021077
RemoteTaskRunner: Fix for starting an overlord before any workers ever existed.
2015-09-24 21:15:36 -07:00
6e550d5346
update doc about aggregation field in merge task and a null check
2015-09-24 22:25:07 -05:00
b638400acb
fix #1715
...
fixes #1715
- TaskLockBox has a set of active tasks
- lock requests throws exception for if they are from a task not in
active task set.
- TaskQueue is responsible for updating the active task set on
tasklockbox
fix #1715
fixes #1715
- TaskLockBox has a set of active tasks
- lock requests throws exception for if they are from a task not in
active task set.
- TaskQueue is responsible for updating the active task set on
tasklockbox
review comment
remove duplicate line
use ISE instead
organise imports
2015-09-24 10:06:50 +05:30
61b0743943
Merge pull request #1748 from metamx/forkingJavaOptionsWithQuotes
...
Allow ForkingTaskRunner javaOpts to have quoted arguments which contain spaces
2015-09-21 21:03:00 -05:00
465035e531
Allow ForkingTaskRunner javaOpts to have quoted arguments which contain spaces
2015-09-21 17:32:27 -07:00
e48f6dd660
Merge pull request #1736 from gianm/additional-ingest-segment-timeline-test
...
IngestSegmentFirehostFactoryTimelineTest for overshadowing of the middle of a segment.
2015-09-17 14:42:29 -07:00
64e33b2bcb
IngestSegmentFirehostFactoryTimelineTest for overshadowing of the middle of a segment.
2015-09-16 10:17:43 -07:00
74f4572bd4
Lazily deserialize "parser" to InputRowParser in DataSchema
...
so that user hadoop related InputRowParsers are created only when needed
this allows overlord to accept a HadoopIndexTask with a hadoopy InputRowParser
and not fail because hadoopy InputRowParser might need hadoop libraries
2015-09-16 10:58:13 -05:00
f5ed6e885c
Merge pull request #1702 from himanshug/double_datasource_in_storage_dir
...
do not have dataSource twice in path to segment storage on hdfs
2015-09-15 14:00:35 -07:00
4681ff22ed
add task duration in response for completed tasks
2015-09-10 13:51:50 +05:30
fe0233adf2
removing unused imports from HadoopIndexTask
2015-09-09 11:12:01 -05:00
47aac991ec
add null check for task context.
...
make variable final
2015-09-04 22:19:01 +05:30
75a582974b
Merge pull request #1639 from gianm/new-plumber
...
New plumber
2015-09-03 18:52:57 -07:00
062a47fba4
Modify Plumbers in these ways,
...
1) Persist using Committer instead of Runnable. (Although the metadata object
is ignored in this patch)
2) Remove the getSink method.
3) Plumbers are now responsible for time-based and hydrant-full-based periodic
committing. (FireChief, RealtimeIndexTask, and IndexTask used to do this)
2015-09-03 11:13:06 -07:00
726326abc3
Add Task Context and ability to override task specific properties
...
override javaOpts
fix compilation
review comments
Add Test for typecast
review comments - remove unused method.
2015-09-03 23:36:32 +05:30
940e1aa3eb
Replace funky imports with standard ones.
...
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
2015-08-28 18:02:05 -07:00
414a6fb477
Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat.
...
Fixes #1678 . IngestSegmentFirehose (and its users) need to remember which
windows of which segments should actually be read, based on a timeline.
2015-08-28 07:32:41 -07:00
2e0dd1d792
adding UTs and addressing review comments to
...
firehoseV2 addition to Realtime[Manager|Plumber],
essential segment metadata persist support,
kafka-simple-consumer-firehose extension patch
2015-08-27 20:50:46 -05:00
2237a8cf0f
kafka 8 simple consumer firehose
2015-08-27 20:50:46 -05:00
b306739e9c
fix convert segment task
...
1) fix serde
2) fix wrong parameter being passed when creating subtask
remove sysout
2015-08-27 11:34:41 +05:30
e38cf54bc8
Migrate TestDerbyConnector to a JUnit @Rule
2015-08-26 21:47:40 -07:00
fdb6a6651b
Merge pull request #1669 from metamx/upgrade-dependencies
...
Upgrade dependencies
2015-08-25 21:30:22 -07:00
5c19ffa98c
Merge pull request #1663 from gianm/segment-insert-constraints
...
TaskActionToolbox: Remove allowOlderVersions, lift interval constraint
2015-08-25 18:11:46 -07:00
51f6a9a2c9
update jackson to 2.6.1
2015-08-25 16:07:01 -07:00
33681525e3
TaskActionToolbox: Remove allowOlderVersions switch, lift interval constraint.
...
allowOlderVersions has been stuck true for a while due to a bug (introduced in
566a3a61
2015-08-25 14:17:38 -07:00
2301b60365
Add ability to provide taskResource for IndexTask.
2015-08-24 17:38:31 -07:00
3b2e41e42a
update for next release
2015-08-18 17:16:46 -07:00
15fa43dd43
changing DatasourcePathSpec, to get segment list, so that hadoop indexer uses overlord action to get list of segments and passes when running as an overlord task. and, uses metadata store directly when running as standalone hadoop indexer
...
also, serialized list of segments is passed to DatasourcePathSpec so that hadoop classloader issues do not creep up
2015-08-16 14:07:35 -05:00
4d4aa8bfc6
refactor IngestSegmentFirehoseFactory so that IngestSegmentFirehose becomes reusable
...
Conflicts:
indexing-service/src/main/java/io/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
2015-08-14 14:44:22 -05:00
bc0c7dd65d
Avoid the Hadoop objectMapper in the local IndexTask. Fixes #1545 .
2015-08-11 10:40:53 -07:00
1ddaa3fb33
Merge pull request #1592 from metamx/clean-test-files
...
clean temporary files
2015-08-03 11:47:20 -07:00
2679efee7a
clean temporary files
2015-08-03 23:32:58 +05:30
6f65e6d3ef
Merge pull request #1547 from pjain1/improve_overlord_test
...
add test to OverlordResourceTest
2015-07-28 07:35:48 -10:00
2e1b617346
add more tests
2015-07-24 15:12:08 -05:00
97242356b4
Merge pull request #1480 from guobingkun/kill_task_test
...
Unit tests for KillTask and MetadataTaskStorage
2015-07-20 16:31:45 -07:00
4cfb00bc8a
inrement version
2015-07-15 13:09:05 -07:00
3f7ba58227
Merge pull request #1504 from metamx/fix-1447
...
fix for #1447
2015-07-14 08:50:08 -07:00
e2ddfb7a1a
Merge pull request #1511 from pjain1/remove_test
...
remove flaky overlord test
2015-07-13 18:38:34 -05:00
59dec89f6a
remove flaky overlord test
2015-07-13 15:32:12 -05:00
725086cc89
Merge pull request #1506 from gianm/realtime-plumber-nulls
...
Consider null inputRows and parse errors as unparseable during realtime ingestion.
2015-07-13 10:12:12 -05:00
9068bcd062
Consider null inputRows and parse errors as unparseable during realtime ingestion.
...
Also, harmonize exception handling between the RealtimeIndexTask and the RealtimeManager.
Conditions other than null inputRows and parse errors bubble up in both.
2015-07-11 20:40:03 -07:00
cac722968e
Merge pull request #1503 from metamx/fix-leaking-zk-nodes
...
Fix leaking Status Path nodes in ZK
2015-07-10 17:40:18 -05:00
9f19e96658
Merge pull request #1477 from pjain1/overlord_test
...
overlord and task master test
2015-07-10 14:27:14 -07:00
55c4fe64f3
overlord and task master test
2015-07-10 16:17:45 -05:00
5fe27fe4ad
fix for #1447
...
fixes #1447
2015-07-09 19:05:48 +05:30
8d7a566bae
Fix leaking Status Path nodes in ZK
...
- remove ZK status path nodes for workers after they are removed
2015-07-09 17:20:09 +05:30
c0b60c0d2f
I'm not your mom, indexing-service/test... cleanup after yourself
2015-07-01 15:00:09 -07:00
282a0f9760
Unit tests for KillTask and MetadataTaskStorage
2015-06-29 17:55:41 -05:00
b5b9ca1446
Merge pull request #1470 from pjain1/rtindex_test
...
Realtime Index Task test
2015-06-29 16:51:35 -05:00
284b80b09e
Realtime Index Task test
2015-06-29 09:52:41 -05:00
4a83a22f8c
Merge pull request #1445 from metamx/JSWorkerSelectStrategy
...
JavaScript Worker Select Strategy
2015-06-22 17:19:13 -05:00
fb4052d577
JavaScript Worker Select Strategy
...
this PR adds a JavaScriptWorkerSelectStrategy which allows defining
arbitrary logic for selecting workers to run task using a JavaScript
function.
This gives users full control to implement complex worker selection
strategies based on task attributes.
more tests and a complex javascript config
fix for java8 modify for nashorn compatibility
2015-06-20 02:01:34 +05:30
0a5bb909a2
[maven-release-plugin] prepare for next development iteration
2015-06-18 17:35:19 -07:00
59c6b2b279
[maven-release-plugin] prepare release druid-0.8.0-rc1
2015-06-18 17:35:14 -07:00
acc0a3fbf7
Add jitter to the retries for RemoteTaskActionClient
2015-06-12 17:43:25 -07:00
e9afec4a2b
fix task status issues on zk outages
...
docs
review comments
fix test
review comments
Review comments
fix compilation
fix typo
2015-06-11 00:49:52 +05:30
78d468700b
Merge pull request #1388 from metamx/fix-1360
...
fix race described in 1360
2015-06-10 11:59:36 -07:00
f6b336ac3e
Merge pull request #1432 from metamx/config-fix
...
fix passing of config from IndexTuningConfig to RealtimeTuningConfig
2015-06-10 11:42:58 -07:00
963682d696
Add check for valid rowFlushBoundary configuration and fix tests
2015-06-10 21:38:34 +05:30
191b302f6a
fix passing of config from IndexTuningConfig to RealtimeTuningConfig
...
- pass rowFlushboundary correctly instead of using default.
- fixes indexTask failing with
io.druid.segment.incremental.IndexSizeExceededException when
rowFlushboundary is set higher than
RealtimeTuningConfig.defaultMaxRowsInMemory
rename test method
2015-06-10 21:07:25 +05:30
af9ea08041
fix race described in 1360
...
review comments
review comments
review comments
no need to remove
fix test
review comments
2015-06-10 12:19:12 +05:30
056cab93ed
Add Hadoop Converter Job and task
...
* Fixes https://github.com/druid-io/druid/issues/1363
* Add extra utils in JobHelper based on PR feedback
2015-06-09 14:47:38 -07:00
ef9b67cce3
Merge pull request #1422 from metamx/fix-ec2-public-ip
...
fix public IP not working in EC2 autoscaling
2015-06-03 16:30:51 -07:00
4ebdfea76f
fix public IP not working in EC2 autoscaling
2015-06-03 16:05:59 -07:00
8289914f76
Make AbstractTask.makeId use AbstractTask.joinId
...
* Also remove TaskUtil
2015-06-03 13:24:20 -07:00
ac9057c00e
Merge pull request #1401 from metamx/ec2-public-ip
...
flag to enable public IP in EC2 autoscaling
2015-05-28 20:21:32 -07:00
d834a974ba
flag to enable public IP in EC2-VPC autoscaling
2015-05-28 18:14:12 -07:00
bb1145ef56
Make the index task use indexmerger and not indexmaker
2015-05-28 13:34:57 -07:00
5ad5d7d18b
Merge pull request #1379 from flowbehappy/fix-hadoop-ha
...
bug fix: hdfs task log and indexing task not work properly with Hadoop HA
2015-05-22 09:14:50 -04:00
07659f30ab
bug fix: hdfs task log and indexing task not work properly with Hadoop HA
2015-05-21 20:49:42 +08:00
29ba05c04f
Abstractify HadoopTask
...
* Add `invokeForeignLoader` to commonize the way tasks are attempted to be launched in a foreign class loader
* Add `buildClassLoader` to accomplish the common tasks for hadoop jobs when building a ClassLoader
2015-05-14 17:04:43 -07:00
7a6acf5c1b
update pom to 0.8
2015-05-11 19:41:58 -06:00
e69d82a2b4
Realtime: Delay firehose connection until job is started.
...
Some firehoses (like the Kafka firehose) acquire input resources when they
connect, so it helps to delay this until after plumber.startJob() runs.
2015-05-04 10:54:07 -07:00
721505c017
Merge pull request #1208 from druid-io/rework-metrics
...
Schemaless metrics + additional metrics for things we care about
2015-04-27 15:04:54 -07:00
963e5765bf
Schemaless metrics + additional metrics for things we care about
2015-04-27 13:39:40 -07:00
633fdb029e
Add option to ConvertSegmentTask to skip validation
...
* Validation is enabled by default
2015-04-27 08:37:55 -07:00
29341f9837
Fix random unit test failure from NoopTask ID collision
2015-04-24 13:07:48 -07:00
f73f14ab91
Merge pull request #1297 from metamx/versionConverterTaskUpdates
...
Update VersionConverterTask for IndexSpec and allowing Forced updates
2015-04-20 16:44:35 -07:00
7479ac9012
Update VersionConverterTask for IndexSepc and allowing Forced updates
2015-04-20 16:17:06 -07:00
d260515a43
update druid-api version
2015-04-17 14:58:35 -07:00
ea5572d001
Merge pull request #1271 from metamx/strictErrorChecking
...
Add stricter checking for potential coding errors
2015-04-15 15:21:41 -07:00
abdeaa0746
Add stricter checking for potential coding errors
...
Can use via `mvn clean compile test-compile -P strict'
2015-04-15 14:52:25 -07:00
3a3046ccf3
add support for dimension compression
...
- compression for single-value dimensions using CompressedVSizeIntsIndexedSupplier
- makes dimension compression configurable via IndexSpec
- IndexSpec also enables configuring bitmap and metric compression
2015-04-14 10:44:18 -07:00
195a3b8bb8
ignore rows with invalid interval
2015-04-06 16:08:40 -07:00
208e307915
Merge pull request #1251 from metamx/uriSegmentLoaders
...
Revert "Revert "Overhaul of SegmentPullers to add consistency and retries""
2015-03-30 17:43:51 -07:00
aea7f9d192
[maven-release-plugin] prepare for next development iteration
2015-03-30 16:35:24 -07:00
060d7aef03
[maven-release-plugin] prepare release druid-0.7.1
2015-03-30 16:35:20 -07:00
1c6cbea89c
Revert "Revert "Overhaul of SegmentPullers to add consistency and retries""
...
This reverts commit f904bc7858
2015-03-30 13:40:04 -07:00
f904bc7858
Revert "Overhaul of SegmentPullers to add consistency and retries"
2015-03-30 13:15:50 -07:00
6d407e8677
Add URI handling to SegmentPullers
...
* Requires https://github.com/druid-io/druid-api/pull/37
* Requires https://github.com/metamx/java-util/pull/22
* Moves the puller logic to use a more standard workflow going through java-util helpers instead of re-writing the handlers for each impl
* General workflow goes like this: 1) LoadSpec makes sure the correct Puller is called with the correct parameters. 2) The Puller sets up general information like how to make an InputStream, how to find a file name (for .gz files for example), and when to retry. 3) CompressionUtils does most of the heavy lifting when it can
2015-03-30 12:33:23 -07:00
942c17a2aa
Remove timeline chunk count assumptions.
...
* Replace with generic iterables
2015-03-24 22:40:49 +01:00
b389cfe404
[maven-release-plugin] prepare for next development iteration
2015-03-19 12:38:17 -07:00
60e7d543cc
[maven-release-plugin] prepare release druid-0.7.1-rc1
2015-03-19 12:38:13 -07:00
9d6b728054
Merge pull request #1215 from metamx/log-audit-IP-Address
...
Add remote ip address in audit log.
2015-03-17 13:59:31 -07:00
bfe10bd156
This fixes arbitrary gran spec breaking
2015-03-17 12:19:43 -07:00
f9821d242f
also log author ip address in audit log
2015-03-17 23:15:15 +05:30
ddfafa0711
randomize task ID to fix spurious test failure
2015-03-12 18:08:48 -07:00
a508c0955f
Merge pull request #1195 from himanshug/task_storage_config_fix
...
correctly parse recentlyFinishedThreshold from config
2015-03-12 16:50:49 -07:00
3ec4a30ab5
initial commit
...
review comments
more refactoring and cleaning of redundant code
add UT + docs + more refactoring
fixes + review comments
more cleanup
end points to fetch history
review comments
remove unnecessary changes
review comments rename header name
review comments + add test for MetadataRulesManager
review comments docs
2015-03-12 22:50:29 +05:30
23545fc01c
correctly parse recentlyFinishedThreshold from config
2015-03-12 09:46:57 -05:00
d3f5bddc5c
Add ability to apply extraction functions to the time dimension
...
- Moves DimExtractionFn under a more generic ExtractionFn interface to
support extracting dimension values other than strings
- pushes down extractionFn to the storage adapter from query engine
- 'dimExtractionFn' parameter has been deprecated in favor of 'extractionFn'
- adds a TimeFormatExtractionFn, allowing to project the '__time' dimension
- JavascriptDimExtractionFn renamed to JavascriptExtractionFn, adding
support for any dimension value types that map directly to Javascript
- update documentation for time column extraction and related changes
2015-03-11 16:45:42 -07:00
b00c243786
Need a null check for iamProfile.
2015-03-10 17:52:15 -07:00
b810cdfe58
EC2AutoScaler: Allow setting "iamProfile".
2015-03-10 17:41:35 -07:00
d102a89760
Fix license on EC2AutoScalerSerdeTest.
2015-03-10 17:31:30 -07:00
9235b45063
EC2AutoScaler: Support for setting subnetId.
2015-03-10 11:29:56 -07:00
113d204b10
break up archive task actions, which was missed in #566a3a6112
2015-03-04 13:19:52 -08:00
bd5cecdd44
UTs update for indexing service
2015-02-25 15:45:58 -08:00
b167dcf82c
[maven-release-plugin] prepare for next development iteration
2015-02-23 14:28:06 -08:00
e81ac2ba43
[maven-release-plugin] prepare release druid-0.7.0
2015-02-23 14:27:58 -08:00
25db9abb7f
Merge pull request #1138 from metamx/better-default-hostname
...
Better default hostname
2015-02-18 17:37:34 -08:00
53d2b961c5
default to canonical hostname instead of localhost
2015-02-18 16:44:48 -08:00
78df7f6165
Move Druid release artifacts to Sonatype
...
- Switch to using Druid parent POM
- Add required fields for Sonatype
- Common plugin versions and settings have been moved to the parent pom
- Cleanup artifacts and POMs for consistent formatting
- Remove org.hyperic.sigar dependency and update docs to reflect necessary jars to add at runtime when sigar is needed
2015-02-13 14:26:31 -08:00
d29740ed9f
[maven-release-plugin] prepare for next development iteration
2015-02-12 16:16:00 -08:00
211fd15b7e
[maven-release-plugin] prepare release druid-0.7.0-rc3
2015-02-12 16:15:56 -08:00
708759e1e0
Update http-client to 1.0.0
2015-02-10 13:36:47 -08:00
79a3e8f59f
Fix overriding base of IndexerZkConfig to be absolute instead of relative
...
* Updated docs to clarify ZK config behavior
* Added unit tests for this case
2015-02-04 13:04:06 -08:00
1f12c5b2f1
[maven-release-plugin] prepare for next development iteration
2015-02-03 12:06:49 -08:00
e82d431be7
[maven-release-plugin] prepare release druid-0.7.0-rc2
2015-02-03 12:06:41 -08:00
92e616de11
Merge pull request #1077 from metamx/remove-unused-imports
...
remove unused imports
2015-02-02 10:45:27 -08:00
ba932bb1f2
remove unused imports
2015-02-02 21:53:39 +05:30
d05032b98a
towards a community led druid
2015-01-31 20:57:36 -08:00
a01a22dba1
Merge pull request #1074 from druid-io/overlord-leader
...
Add an endpoint to return the overlord leader
2015-01-30 13:44:49 -08:00
bd49528805
Merge pull request #1073 from druid-io/fix-statusPath
...
Fix worker status path announcement with indexer zk config
2015-01-30 12:51:21 -08:00
649f285feb
Add an endpoint to return the overlord leader
2015-01-30 12:37:48 -08:00
bc1405bee0
fix worker status path announcement with indexer zk config
2015-01-30 12:26:08 -08:00
2c2771b90e
Make dynamic worker selection actually work
2015-01-27 14:17:42 -08:00
0f3eac4705
fix dimension exclusion
2015-01-23 00:31:23 +05:30
1f94de22c6
[maven-release-plugin] prepare for next development iteration
2015-01-20 14:23:55 -08:00
17476edc31
[maven-release-plugin] prepare release druid-0.7.0-rc1
2015-01-20 14:23:51 -08:00
2d516fa591
Add a new equal distribution strategy for assigning tasks
2015-01-20 13:12:22 -08:00
cd9635ff5e
Merge pull request #1034 from druid-io/minor-rename
...
minor rename of things in hadoop ingestion config to match 0.6.x
2015-01-15 15:46:13 -08:00
ccddbf8747
minor rename of things in hadoop ingestion config to match 0.6.x
2015-01-15 14:04:55 -08:00
5bfcc43377
Merge pull request #1008 from metamx/stringConversionJavaUtilUpdate
...
Update all String conversions to and from byte[] to use the java-util StringUtils functions
2015-01-15 13:50:27 -08:00
67757b6aea
Change IndexerZkConfig to use @JacksonInject instead of just straight @Inject
...
* Updated IndexerZkConfig to use no setters, and take all arguments from constructor instead
* Also added more unit tests
2015-01-08 11:11:17 -08:00
f6fbb733b8
Added a few places where tests were using Object instead of Module
2015-01-05 13:47:25 -08:00
b1b5c9099e
Update all String conversions to and from byte[] to use the java-util StringUtils functions
...
* Speedup of GroupBy with javaScript filters by ~10%
* Requires https://github.com/metamx/java-util/pull/15
2015-01-05 11:22:32 -08:00
65286a24e0
Change zk configs to use Jackson injection instead of Skife
...
* Also added generic config testing class JsonConfigTesterBase
2014-12-29 10:36:12 -08:00
af1185b58c
Merge pull request #969 from metamx/fixRemoteLogViewing
...
Remove try-with-resources for log stream in WokerResource
2014-12-15 16:26:02 -07:00
54068e8b1d
Remove try-with-resources for log stream in WokerResource
2014-12-15 15:24:59 -08:00
ac407fb6ba
clean up defaults
2014-12-15 15:05:02 -08:00
e872952390
fix working path default bug
2014-12-15 14:51:58 -08:00
b3fe91bb50
Merge pull request #830 from metamx/union-merge-on-historical
...
Union merge on historical
2014-12-15 13:36:47 -07:00
bed3e7e1d2
Merge pull request #966 from metamx/fix-tasklog-streaming
...
fix task log streaming
2014-12-14 09:31:41 -08:00
bd91a40491
fix task log streaming
2014-12-13 15:22:55 -08:00
092dfe0309
fix IndexTaskTest tmp dir
...
- Create local firehose files in a clean temp directory to avoid
firehose reading other random temp files that start with 'druid'
2014-12-12 17:05:45 -08:00
Charles Allen
Fangjin Yang
Gian Merlino
Andrés Gomez
Himanshu
Xavier Léauté
Parag Jain
Nishant
Charles Allen
Jonathan Wei
Bingkun Guo
Kurt Young
navis.ryu
Xavier Léauté
Bartosz Ługowski
zhxiaog
lvjq
Paul Otto
Parag Jain
flow
msprunck
Himanshu Gupta