Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-25 04:16:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
7,244 Commits 116 Branches 508 Tags
Commit Graph

2718 Commits

Author SHA1 Message Date
Bingkun Guo
970b9cfcaa fix stale segment in ServerSelector 2015-12-07 14:04:34 -06:00
Fangjin Yang
00416e26ff Merge pull request #2012 from himanshug/add_comment 
adding more comments to why drops do not happen
 2015-12-07 11:50:06 -08:00
Himanshu Gupta
62ba9ade37 unifying license header in all java files 2015-12-05 22:16:23 -06:00
刘麒贇
3781aa380f Added an injection test for ZkCoordinator. 2015-12-05 11:56:22 +08:00
Himanshu Gupta
221fb95d07 add support for getting used segments for multiple interval in IndexerMetadataStorageCoordinator 2015-12-03 21:28:04 -06:00
Nishant
1eb8211346 Add datasource and taskId to metrics emitted by peons 
This PR adds the datasource and taskId to the jvm and sys metrics
emitted by the peons.

fix spelling

review comment

review comment
 2015-12-01 23:20:59 +05:30
Himanshu Gupta
7c91219130 adding more comments to why drops do not happen when there are no segments in db 2015-11-25 16:26:51 -06:00
Fangjin Yang
8e83d800d6 Merge pull request #1881 from gianm/restartable-tasks 
Restorable indexing tasks
 2015-11-23 21:14:37 -08:00
Gian Merlino
13af260102 Merge pull request #1957 from metamx/skip-incremental-segment 
Ability to skip Incremental Index during query using query context
 2015-11-23 17:32:57 -08:00
Gian Merlino
501dcb43fa Some changes that make it possible to restart tasks on the same hardware. 
This is done by killing and respawning the jvms rather than reconnecting to existing
jvms, for a couple reasons. One is that it lets you restore tasks after server reboots
too, and another is that it lets you upgrade all the software on a box at once by just
restarting everything.

The main changes are,

1) Add "canRestore" and "stopGracefully" methods to Tasks that say if a task can
   stop gracefully, and actually do a graceful stop. RealtimeIndexTask is the only
   one that currently implements this.

2) Add "stop" method to TaskRunners that attempts to do an orderly shutdown.
   ThreadPoolTaskRunner- call stopGracefully on restorable tasks, wait for exit
   ForkingTaskRunner- close output stream to restorable tasks, wait for exit
   RemoteTaskRunner- do nothing special, we actually don't want to shutdown

3) Add "restore" method to TaskRunners that attempts to bootstrap tasks from last run.
   Only ForkingTaskRunner does anything here. It maintains a "restore.json" file with
   a list of restorable tasks.

4) Have the CliPeon's ExecutorLifecycle lock the task base directory to avoid a restored
   task and a zombie old task from stomping on each other.
 2015-11-23 11:22:08 -08:00
Gian Merlino
36569094ce EventReceiverFirehose: Drain buffer when closed, until empty. 2015-11-20 19:12:59 -08:00
Fangjin Yang
21c84b5ff7 Merge pull request #1896 from gianm/allocate-segment 
SegmentAllocateAction (fixes #1515)
 2015-11-18 21:05:46 -08:00
Nishant
60f649dab1 Ability to skip Incremental Index during query using query context 
This PR adds adds the ability to skip incremental index when querying
results from realtime nodes. default behaviour is to include
incrementalIndex in queries.

review comment
 2015-11-19 00:29:41 +05:30
Charles Allen
8fcf2403e3 Merge pull request #1943 from metamx/realtime-caching 
Enable caching on intermediate realtime persists
 2015-11-17 15:06:43 -08:00
Charles Allen
8fa34ee671 Merge pull request #1973 from metamx/cachemonitor-optional-injection 
CacheMonitor - make cache injection optional
 2015-11-17 13:02:06 -08:00
Himanshu
9118b010c0 Merge pull request #1976 from noddi/feature/count-parameter-rules-history 
Add count parameter to rules history endpoint.
 2015-11-17 14:41:50 -06:00
Charles Allen
dbe201aeed Merge pull request #1929 from pjain1/jetty_threads 
separate ingestion and query thread pool
 2015-11-17 12:14:25 -08:00
Parag Jain
6c498b7d4a separate ingestion and query thread pool 2015-11-17 13:42:41 -06:00
Bartosz Ługowski
b0a444eafc Add count parameter to rules history endpoint. 2015-11-17 20:14:12 +01:00
Xavier Léauté
d7eb2f717e enable query caching on intermediate realtime persists 2015-11-17 10:58:00 -08:00
Xavier Léauté
71376efd57 make cache injection optional 
allows enabling the CacheMonitor for all index tasks, even if some don't
bind any cache instance.
 2015-11-16 22:40:01 -08:00
Fangjin Yang
4f46d457f1 Merge pull request #1947 from noddi/feature/count-parameter-history-endpoints 
Add count parameter to history endpoints
 2015-11-12 10:23:44 -08:00
Fangjin Yang
465cbcf9a7 Merge pull request #1956 from metamx/remove-unused-imports 
Cleanup + remove unused imports
 2015-11-11 17:36:47 -08:00
Gian Merlino
e4e5f0375b SegmentAllocateAction (fixes #1515) 
This is a feature meant to allow realtime tasks to work without being told upfront
what shardSpec they should use (so we can potentially publish a variable number
of segments per interval).

The idea is that there is a "pendingSegments" table in the metadata store that
tracks allocated segments. Each one has a segment id (the same segment id we know
and love) and is also part of a sequence.

The sequences are an idea from @cheddar that offers a way of doing replication.
If there are N tasks reading exactly the same data with exactly the same logic
(think Kafka tasks reading a fixed range of offsets) then you can place them
in the same sequence, and they will generate the same sequence of segments.
 2015-11-11 16:54:35 -08:00
Charles Allen
dc4ae59aaf Merge pull request #1865 from noddi/bugfix/issue-560 
Fix #560 - datasource segments summary shouldn't include replication.
 2015-11-11 15:29:08 -08:00
Bartosz Ługowski
6e5d2c6745 Add count parameter to history endpoints. 2015-11-11 23:03:57 +01:00
Gian Merlino
8e743b70c6 SegmentIdentifier, like a pre-DataSegment. 2015-11-11 12:25:43 -08:00
Xavier Léauté
e26f7fca36 cleanup cache interface 2015-11-11 12:25:22 -08:00
Xavier Léauté
fa6142e217 cleanup and remove unused imports 2015-11-11 12:25:21 -08:00
Gian Merlino
f2c271e500 Better error message when LocalDataSegmentPusher cannot create its directory. 2015-11-10 19:49:01 -08:00
Xavier Léauté
cf779946ef Merge pull request #1791 from guobingkun/event_receiver_firehose_monitor 
EventReceiverFirehoseMonitor
 2015-11-10 11:09:42 -08:00
Xavier Léauté
a57cbfd2c3 Merge pull request #1387 from metamx/enableShutdownLogging 
Add special handler to allow logger messages during shutdown
 2015-11-09 17:20:09 -08:00
Charles Allen
1df4baf489 Move Jackson Guice adapters into io.druid 
* Removes access to protected methods in com.fasterxml
* Eliminates druid-common's use of foreign package com.fasterxml
 2015-11-09 10:50:45 -08:00
Fangjin Yang
fe6efb6367 Merge pull request #1932 from gianm/data-schema-exclude-metric-names 
DataSchema: Exclude metric names from dimension list.
 2015-11-07 15:22:33 -08:00
Bartosz Ługowski
8b5de492cc Fix #560 - datasource segments summary shouldn't include replication. 2015-11-07 21:43:47 +01:00
Gian Merlino
6b908a5061 DataSchema: Exclude metric names from dimension list. 
Otherwise we could end up with a metric and a dimension with the same column name.
 2015-11-07 12:25:36 -08:00
Xavier Léauté
e9533db987 Merge pull request #1850 from metamx/friendlyBardCache 
Allow setting upper limit on the number of cache segments a broker will try to fetch.
 2015-11-06 10:25:49 -08:00
Xavier Léauté
c896818241 Update curator to 2.9.1 
Lots of bugfixes since 2.8.0
- https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12333324
- https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12332392
 2015-11-05 15:53:01 -08:00
fjy
4efc90dda3 fix zkcoordinator inject 2015-11-04 17:30:17 -08:00
fjy
8f231fd3e3 cleanup druid codebase 2015-11-04 13:59:53 -08:00
Xavier Léauté
3b3d88bd03 Merge pull request #1857 from noddi/feature/intervals-route 
Queryable intervals
 2015-11-03 13:19:04 -08:00
Himanshu Gupta
8b67417ac8 make methods in Index[Merger,Maker,IO] non-static so that they can have 
appropriate ObjectMapper injected instead of creating one statically
 2015-11-02 23:24:26 -06:00
Fangjin Yang
f90ddfdb89 Merge pull request #1745 from himanshug/numbered_to_elastic 
making NumberedShardSpec elastic
 2015-10-30 16:01:03 -07:00
Bingkun Guo
c3b6fcce9d Add EventReceiverFirehoseMonitor 
add an EventReceiverFirehoseMonitor so that we can monitor how many
events have been queued in the EventReceiverFirehose and get a sense
about whether the firehose is under too much pressure.
 2015-10-30 11:40:02 -05:00
Bartosz Ługowski
6de51e022d Coordinator - add intervals route. 2015-10-30 11:51:42 +01:00
Xavier Léauté
149333d77d forward cancellation request to all brokers, fixes #1802 2015-10-29 16:58:38 -07:00
Charles Allen
dfce14ed17 Allow setting upper limit on the number of cache segments a broker will try to fetch. 2015-10-29 11:50:00 -07:00
Xavier Léauté
59872bd0cd Merge pull request #1809 from metamx/fifoPriorityExecutorService 
Make PrioritizedExecutorService optionally FIFO
 2015-10-27 15:19:32 -07:00
Fangjin Yang
ea2267e08c Merge pull request #1868 from gianm/fix-announcements 
Historical and MiddleManager server announcements should not remove parents.
 2015-10-27 14:50:05 -07:00
Gian Merlino
7df7370935 Merge pull request #1862 from metamx/indexingServiceMMGone 
Add timeout to shutdown request to middle manager for indexing service
 2015-10-27 14:38:01 -07:00