deb6ecf919
handle review comments for PR 2784
...
https://github.com/druid-io/druid/pull/2784#discussion_r59062021
2016-04-12 21:52:00 +05:30
edd74f2b67
Allow Lite DataSegment Announcements
...
separate config for each skipping dimensions, metrics and loadSpec
Add test
fix test comment
Add docs
2016-04-07 18:24:12 +05:30
1e02eeab13
Merge pull request #2683 from metamx/default_retry
...
Better defaults for Retry policy for task actions
2016-03-29 08:02:59 -07:00
9cb197adec
Merge pull request #2722 from himanshug/fix_hadoop_jar_upload
...
config to explicitly specify classpath for hadoop container during hadoop ingestion
2016-03-28 14:49:03 -07:00
e78a469fb7
UTs for ExtensionsConfig
2016-03-25 10:51:28 -05:00
004b00bb96
config to explicitly specify classpath for hadoop container during hadoop ingestion
2016-03-25 10:51:28 -05:00
0fa04305a6
refine description for mergeBytesLimit
2016-03-24 13:17:24 -05:00
448e0127b9
dynamic config endpoint is at coordinator
2016-03-23 17:22:19 -05:00
451c0bc6d8
Merge pull request #2702 from pjain1/improve_docs
...
how to query in the querying section, correct default for select strategy, formatting
2016-03-22 16:40:35 -07:00
39ecb9929d
how to query, correct default for select strategy, formatting
2016-03-22 17:06:15 -05:00
ed8f39fcfe
Better defaults for Retry policy for task actions
...
This PR changes the retry of task actions to be a bit more aggressive
by reducing the maxWait. Current defaults were 1 min to 10 mins, which
lead to a very delayed recovery in case there are any transient network
issues between the overlord and the peons.
doc changes.
2016-03-18 11:59:55 -07:00
5da9a280b6
Query Time Lookup - Dynamic Configuration
2016-03-18 09:45:05 -07:00
5ec5ac92c6
Merge pull request #2382 from himanshug/broker_segment_tier_selection
...
at broker, if configured, only add segments from specific tiers to the timeline
2016-03-14 16:53:06 -07:00
96c981cd0a
fix broken link for Tasks
2016-03-11 11:36:34 -06:00
ca5de3f583
only allow lowering maxResults and maxIntermediateRows from groupBy query context
2016-03-08 15:03:59 -06:00
099acb4966
allow groupBy max[Intermediate]Rows limit be overridable by context
2016-03-07 15:22:41 -06:00
177b575d41
fix default number of connections on broker config documentation
2016-03-03 13:50:48 -08:00
2462c82c0e
New defaults for maxRowsInMemory rowFlushBoundary
...
To bring consistency to docs and source this commit changes the default
values for maxRowsInMemory and rowFlushBoundary to 75000 after
discussion in PR https://github.com/druid-io/druid/pull/2457 .
The previous default was 500000 and it's lower now on the grounds that
it's better for a default to be somewhat less efficient, and work,
than to reach for the stars and possibly result in
"OutOfMemoryError: java heap space" errors.
2016-03-01 13:50:28 +01:00
c6803c4364
Allow specifying peon javaOpts as an array
2016-02-26 13:24:35 -08:00
eb13d7afe3
Merge pull request #2521 from himanshug/fix_2497
...
RTR has multiple threads for assignment of pending tasks now
2016-02-26 08:14:15 -08:00
9f8faabddb
Merge pull request #2469 from pdeva/patch-10
...
correct service names
2016-02-26 21:15:58 +05:30
bc156effe7
RTR has multiple threads for assignment of pending tasks now.
2016-02-26 09:27:03 -06:00
23c993c9e7
Add druid.indexer.server.maxChatRequests for QoS; deprecate separate ports.
...
- Add druid.indexer.server.maxChatRequests, which sets up a QoSFilter on the main Jetty server.
- Deprecate druid.indexer.runner.separateIngestionEndpoint
- Deprecate druid.indexer.server.chathandler.*
2016-02-19 13:36:09 -08:00
dd81b5ebe4
correct service names
...
use a `/` instead of `:` cause thats how the service names are declared in the respective config files of coordinator and overlord
2016-02-13 15:26:19 -08:00
e0c049c0b0
Make startup properties logging optional.
...
Off by default, but enabled in the example config files. See also #2452 .
2016-02-12 14:12:16 -08:00
3a6452c6d4
Make QuotableWhiteSpaceSplitter able to take json
...
* Fixes #2435
2016-02-10 16:42:14 -08:00
d1cb17d3f7
at broker - only add segments from specific tiers to the timeline
2016-02-09 22:33:22 -06:00
1d57e3267d
some minor doc changes
2016-02-09 08:20:53 -06:00
b40c342cd1
make Global stupid pool cache size configurable
2016-02-05 14:18:06 -06:00
9e2295aa61
whitespace fixes
2016-02-04 16:25:51 -08:00
003f54e268
add doc rendering
2016-02-04 14:21:59 -08:00
1aa363cea7
new quickstart
2016-02-04 09:37:38 -08:00
c9393e5289
Add more docs around timezone handling
...
* Fixes #2356
2016-02-01 08:51:07 -08:00
b07db4089c
fix doc: Setting druid.coordinator.merge.on will trigger an Append Task instead of Merge Task.
2016-01-26 10:20:32 -06:00
e0d90f875c
Graphite emitter
2016-01-21 13:43:37 -06:00
c9368702fa
do some editing of the instructions for using mysql for metadata
2016-01-21 10:37:30 -06:00
2e54553a8f
Merge pull request #1990 from himanshug/schedule_kill_task
...
support periodic hard delete of segments
2016-01-15 15:22:33 -06:00
1908d63162
acl for zookeeper is added
2016-01-13 14:56:05 -08:00
eb2d251ac8
support periodic hard delete of segments
2016-01-12 16:55:05 -06:00
82bdfbbbf1
Merge pull request #2155 from metamx/taskConfigTmpdir
...
Make TaskConfig pull from java.io.tmpdir
2016-01-05 13:58:39 -06:00
e18301d99c
Make TaskConfig pull from java.io.tmpdir
...
* Also makes paths built off of java.nio.file.Paths instead of String.format
2016-01-04 10:17:08 -08:00
77863285e9
fix typo
2015-12-27 14:28:23 -08:00
b308a13483
correct docs
2015-12-27 14:27:20 -08:00
bad270b6c4
druid.indexer.task.restoreTasksOnRestart configuration.
2015-12-22 10:59:15 -08:00
951a4e9b35
Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer
2015-12-21 00:05:53 -06:00
2b5a010332
Change sample worker config spec with host:port instead of ip:port.
...
Also extend description of the 'affinity' property of the worker strategy
fillCapacityWithAffinity and fix a couple typos of middle manager (to
be more consistent throughout the page).
Add additional verbiage about appropriate middle manager host value.
2015-12-14 14:59:23 -08:00
b0ab363022
Merge pull request #2052 from gianm/service-names
...
Change service names in docs, examples to match defaults in the code.
2015-12-08 15:40:35 -08:00
9491e8de3b
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoffs
...
- fixes #1970
- extracted out segment handoff callbacks in SegmentHandoffNotifier
which is responsible for tracking segment handoffs and doing callbacks
when handoff is complete.
- Coordinator now maintains a view of segments in the cluster, this
will affect the jam heap requirements for the overlord for large
clusters.
realtime index task and nodes now use HTTP end points exposed by the
coordinator to get serverView
review comment
fix realtime node guide injection
review comments
make test not rely on scheduled exec
fix compilation
fix import
review comment
introduce immutableSegmentLoadInfo
fix son reading
remove unnecessary logging
2015-12-09 01:54:09 +05:30
8e594a2e72
Change service names in docs, examples to match defaults in the code.
2015-12-06 10:04:21 -08:00
501dcb43fa
Some changes that make it possible to restart tasks on the same hardware.
...
This is done by killing and respawning the jvms rather than reconnecting to existing
jvms, for a couple reasons. One is that it lets you restore tasks after server reboots
too, and another is that it lets you upgrade all the software on a box at once by just
restarting everything.
The main changes are,
1) Add "canRestore" and "stopGracefully" methods to Tasks that say if a task can
stop gracefully, and actually do a graceful stop. RealtimeIndexTask is the only
one that currently implements this.
2) Add "stop" method to TaskRunners that attempts to do an orderly shutdown.
ThreadPoolTaskRunner- call stopGracefully on restorable tasks, wait for exit
ForkingTaskRunner- close output stream to restorable tasks, wait for exit
RemoteTaskRunner- do nothing special, we actually don't want to shutdown
3) Add "restore" method to TaskRunners that attempts to bootstrap tasks from last run.
Only ForkingTaskRunner does anything here. It maintains a "restore.json" file with
a list of restorable tasks.
4) Have the CliPeon's ExecutorLifecycle lock the task base directory to avoid a restored
task and a zombie old task from stomping on each other.
2015-11-23 11:22:08 -08:00
8fcf2403e3
Merge pull request #1943 from metamx/realtime-caching
...
Enable caching on intermediate realtime persists
2015-11-17 15:06:43 -08:00
dbe201aeed
Merge pull request #1929 from pjain1/jetty_threads
...
separate ingestion and query thread pool
2015-11-17 12:14:25 -08:00
6c498b7d4a
separate ingestion and query thread pool
2015-11-17 13:42:41 -06:00
d7eb2f717e
enable query caching on intermediate realtime persists
2015-11-17 10:58:00 -08:00
6e5d2c6745
Add count parameter to history endpoints.
2015-11-11 23:03:57 +01:00
cf779946ef
Merge pull request #1791 from guobingkun/event_receiver_firehose_monitor
...
EventReceiverFirehoseMonitor
2015-11-10 11:09:42 -08:00
e9533db987
Merge pull request #1850 from metamx/friendlyBardCache
...
Allow setting upper limit on the number of cache segments a broker will try to fetch.
2015-11-06 10:25:49 -08:00
3ee28c35ce
fix curator compress doc
2015-11-03 16:48:59 -06:00
962f65cc76
fix metadata typo and rename default extension directory
2015-11-03 14:50:42 -06:00
95a5ae0373
Docs improved by adding more details about local cache and memory for segments on historicals.
2015-11-01 21:56:28 +02:00
c3b6fcce9d
Add EventReceiverFirehoseMonitor
...
add an EventReceiverFirehoseMonitor so that we can monitor how many
events have been queued in the EventReceiverFirehose and get a sense
about whether the firehose is under too much pressure.
2015-10-30 11:40:02 -05:00
dfce14ed17
Allow setting upper limit on the number of cache segments a broker will try to fetch.
2015-10-29 11:50:00 -07:00
59872bd0cd
Merge pull request #1809 from metamx/fifoPriorityExecutorService
...
Make PrioritizedExecutorService optionally FIFO
2015-10-27 15:19:32 -07:00
7df7370935
Merge pull request #1862 from metamx/indexingServiceMMGone
...
Add timeout to shutdown request to middle manager for indexing service
2015-10-27 14:38:01 -07:00
ecdafa87c5
Make PrioritizedExecutorService optionally FIFO
2015-10-27 14:16:22 -07:00
44a2b204df
Add timeout to shutdown request to middle manager for indexing service
2015-10-27 13:56:03 -07:00
5f23703216
Merge pull request #1638 from guobingkun/remove_maven_client_code
...
Remove Maven client at runtime + Provide a way to load Druid extensions through local file system
2015-10-26 09:30:05 -07:00
d068ec3b9d
"druid.manager.segment" should be "druid.manager.segments"
2015-10-21 15:50:44 -07:00
4914925d65
New extension loading mechanism
...
1) Remove maven client from downloading extensions at runtime.
2) Provide a way to load Druid extensions and hadoop dependencies through file system.
3) Refactor pull-deps so that it can download extensions into extension directories.
4) Add documents on how to use this new extension loading mechanism.
5) Change the way how Druid tarball is generated. Now all the extensions + hadoop-client 2.3.0
are packaged within the Druid tarball.
2015-10-21 14:22:36 -05:00
020a706ac4
Merge pull request #1800 from metamx/hybrid-cache
...
Hybrid L1/L2 cache
2015-10-06 15:18:07 -07:00
b11c0859e8
hybrid l1/l2 cache to combine local and remote cache
2015-10-06 14:32:08 -07:00
9006e255ef
fixing the link to chunkPeriod doc
2015-10-01 13:03:46 -05:00
df4c8a3aa5
balance memcached cache across multiple connections
2015-09-18 09:52:16 -07:00
9705c5139b
Merge pull request #1732 from jon-wei/segmentmeta
...
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
2015-09-16 12:36:25 -07:00
193fb4fdfc
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
2015-09-14 19:41:42 -07:00
39d23ffc45
in configuration/index.md s/instantialize/initialize
2015-09-09 21:18:26 -05:00
ebdb612933
composing emitter module to use multiple emitters together
2015-09-09 16:45:50 -05:00
10946610f4
Merge pull request #1656 from druid-io/all-the-docs
...
more docs for common questions
2015-08-25 17:49:47 -07:00
4055f9ca48
more docs for common questions
2015-08-25 17:49:04 -07:00
5a48376788
add postgresql setup docs
2015-08-21 13:35:58 -07:00
43978eba99
Fixes and more docs across many areas
2015-08-13 10:35:53 -07:00
3078d905fa
Merge pull request #1586 from onlychoice/fix_javaOpts
...
Fix configuration for javaOpts, remove quotation marks
2015-08-03 21:27:45 -05:00
27997f9682
added required config option
2015-08-03 19:08:05 -07:00
d3e838ff34
Fix configuration for javaOpts, remove quotation marks
2015-08-01 11:26:37 +08:00
c17bc3fd74
update doc with useCache and populateCache flags
2015-07-28 15:46:22 -05:00
ba59f8afc4
Merge pull request #1550 from himanshug/optionally_log_all_requests
...
print *all* HTTP requests to log if configured
2015-07-28 11:19:36 -07:00
90b475975e
Log all the requests to debug level in logger io.druid.jetty.RequestLog
2015-07-28 12:56:04 -05:00
4801de62a2
make "announce" the chathandler default in realtime node,
...
remove doc references to chathandler type "announce" since it is the default now,
2015-07-27 12:14:28 -05:00
e051e93d19
Merge pull request #1518 from RealROI/more-azure-features
...
Azure Blob Store support for Firehose and Indexing Service Logs
2015-07-17 16:10:22 -07:00
0bda7af52c
Add more support for Azure Blob Store
...
Azure Blob Store support for Task Logs and a firehose for data ingestion
2015-07-17 15:38:21 -07:00
e21195f987
Merge pull request #1469 from guobingkun/table_config
...
Inconsistent property names for "druid.metadata.storage.tables.xxx"
2015-07-17 07:43:19 -07:00
fc438e5a72
add alert on errors polling for rules
...
reset retrystarttime after alerting
2015-07-13 09:03:51 -07:00
4a0ae7d8d5
Fix inconsistent druid property names for "druid.metadata.storage.tables.xxx" between document and code
2015-06-29 10:12:30 -05:00
726ed432a1
Merge pull request #1451 from rasahner/doc_minorFixes
...
minor documentation fixes in Tasks.md, index.md, indexing-service.md
2015-06-23 10:15:47 -07:00
4ba34fe43d
minor documentation fixes in Tasks.md, index.md, indexing-service.md
2015-06-19 17:09:53 -05:00
fb4052d577
JavaScript Worker Select Strategy
...
this PR adds a JavaScriptWorkerSelectStrategy which allows defining
arbitrary logic for selecting workers to run task using a JavaScript
function.
This gives users full control to implement complex worker selection
strategies based on task attributes.
more tests and a complex javascript config
fix for java8 modify for nashorn compatibility
2015-06-20 02:01:34 +05:30
6763e3780a
Merge pull request #1417 from metamx/rtr-fixes
...
fix behaviour of middlemanagers around ZK disconnects , fixes #709
2015-06-10 14:42:49 -07:00
bdf4d541e3
Fix issue #1016 that if user specifies extension coordinates, the local extensions will be loaded twice,
...
which could cause Guice duplicate binding errors.
Add unit test to replicate duplicate extension issue. Update documents. Add an package accessible getter for loadersMap for testing only.
Notice that extensions explicitly specified in druid.extensions.coordinates have a higher priority than ones included in the classpath.
Extension modules that don't have a canonical class name will be ignored.
2015-06-10 15:48:03 -05:00
e9afec4a2b
fix task status issues on zk outages
...
docs
review comments
fix test
review comments
Review comments
fix compilation
fix typo
2015-06-11 00:49:52 +05:30
afdd9304cf
rename servermonitor to historicalmetricsmonitor in docs
2015-06-05 17:32:04 -07:00
d2346b6834
shorten links and file names
...
* remove redundant parts in file names
* delete unsupported "Druid-Personal-Demo-Cluster"
2015-05-29 20:55:42 -05:00
8edc2aaca3
renaming all *.md filenames to only have lowercase and dashes
...
so that they are editable on case-insensitive os as well
2015-05-29 20:55:42 -05:00
Nishant
Fangjin Yang
Himanshu Gupta
Bingkun Guo
Robin
Gian Merlino
Parag Jain
Charles Allen
Jonathan Wei
jisookim
Björn Zettergren
pdeva
Slim Bouguerra
Nikita Geer
Steve M
Xavier Léauté
Bartosz Ługowski
Xavier Léauté
Oleg Zaezdny
jon-wei
Zhihui Jiao
Zak Kristjanson
samjhecht