67d0b0ee42
Add taskType dimension to task metrics ( #5664 )
2018-05-07 09:42:26 -07:00
c12c16385e
support throw duplcate row during realtime ingestion in RealtimePlumber ( #5693 )
2018-05-04 10:12:25 -07:00
e2431ae161
Update defaultHadoopCoordinates in documentation. ( #5720 )
...
* Update defaultHadoopCoordinates in documentation.
To match changes applied in #5382 .
* Remove a parameter with defaults from example configuration file.
If it has reasonable defaults, then why would it be in an example config file?
Also, it is yet another place that has been forgotten to be updated and will be forgotten in the future.
Also, if someone is running different hadoop version, then there's much more work to be done than just changing this property, so why give users false hopes?
* Fix typo in documentation.
2018-04-30 20:49:14 -07:00
8ec2d2fe18
Use unique segment paths for Kafka indexing ( #5692 )
...
* support unique segment file paths
* forbiddenapis
* code review changes
* code review changes
* code review changes
* checkstyle fix
2018-04-29 21:59:48 -07:00
762f8829e4
Add task action metrics, add taskId metric dimension. ( #5714 )
...
* Add task action metrics, add taskId metric dimension.
Adds two new metrics: task/action/log/time and task/action/run/time. Also
adds taskId as a dimension, to give us the ability to drill down into metrics
for an individual task. Also standardizes metrics-attachment using two helper
methods in IndexTaskUtils.
* Fix typo
2018-04-29 21:24:06 -07:00
b084075279
Add http/https proxy options to PullDependencies.java ( #5450 )
2018-03-07 15:05:43 -08:00
80419752b5
Add metamx emitter, http clients, and metrics packages to druid java-util ( #5289 )
...
* Add metamx java-util emitter, http clients, and metrics packages to druid java-util
* Remove metamx java-util from pom.xml files
* Checkstyle fixes
* Import fix
* TeamCity inspection fixes
* Use slf4j, move some version defs to master pom.xml
* Use parent jvm-attach-api and maven-surefire-plugin versions
* Add ] to log msg, suppress inspection
2018-01-24 22:10:36 +01:00
241efafbb2
Automatic compaction by coordinators ( #5102 )
...
* Automatic compaction by coordinator
* add links
* skip compaction for very recent segments if they are small
* fix finding search interval
* fix finding search interval
* fix TimelineHolder iteration
* add test for newestSegmentFirstPolicy
* add CompactionSegmentIterator
* add numTargetCompactionSegments
* add missing config
* fix skipping huge shards
* fix handling large number of segments per shard
* fix test failure
* change recursive call to loop
* fix logging
* fix build
* fix test failure
* address comments
* change dataSources type
* check running pendingTasks at each run
* fix test
* address comments
* fix build
* fix test
* address comments
* address comments
* add doc for segment size optimization
* address comment
2018-01-13 13:52:37 +09:00
2ecebb3173
Fix coordinator/overlord redirects when TLS is enabled ( #5037 )
...
* Fix coordinator/overlord redirects when TLS is enabled
* address review comment
* fix UTs
* workaround to not ignore URL instance to fix the teamcity build
* update tls doc
2017-11-09 13:10:28 -08:00
52d7f74226
Add streaming aggregation as the last step of ConcurrentGrouper if data are spilled ( #4704 )
...
* Add steaming grouper
* Fix doc
* Use a single dictionary while combining
* Revert GroupByBenchmark
* Removed unused code
* More cleanup
* Remove unused config
* Fix some typos and bugs
* Refactor Groupers.mergeIterators()
* Add comments for combining tree
* Refactor buildCombineTree
* Refactor iterator
* Add ParallelCombiner
* Add ParallelCombinerTest
* Handle InterruptedException
* use AbstractPrioritizedCallable
* Address comments
* [maven-release-plugin] prepare release druid-0.11.0-sg
* [maven-release-plugin] prepare for next development iteration
* Address comments
* Revert "[maven-release-plugin] prepare for next development iteration"
This reverts commit 5c6b31e4880f5c3a8b82
2017-10-17 23:24:08 -07:00
7cc18226cd
add more tls configs to enable/disable specific cipher suites and protocols ( #4902 )
...
* add more tls configs to enable/disable specific cipher suites and protocols
* fix doc, allow empty list
2017-10-09 13:53:12 -07:00
f69c9280c4
remove ServerConfig from DruidNode as all information needs to be present in DruidNode serialized form ( #4858 )
...
* remove ServerConfig from DruidNode as all information needs to be present in DruidNode serialized form
* sanitize output of /druid/coordinator/v1/cluster endpoint
2017-09-28 10:40:59 -05:00
a36adc63e4
[documentation] add more jvm and os guidelines ( #4793 )
...
* add more jvm and os guidelines
* address review comments
* add not so general guidelines too
* duplicate statement removal
2017-09-20 13:12:57 -07:00
b5e839b3db
injectable sslcontextfactory for jetty server and key manager factory algorithm ( #4769 )
...
* injectable sslcontextfactory for jetty server
key manager factory algorithm
* explicitly set trustAll certificates to false in sslcontextfactory
2017-09-12 11:45:03 -07:00
1bddfc089c
Additional docs/log for direct memory usage ( #4631 )
...
* Additional docs/log for direct memory usage
* Tweak docs
* Doc rewording
2017-08-10 23:33:20 -07:00
ae6780f62a
rolling upgrade order change to bring coordinator and overlord together ( #4281 )
...
* rolling upgrade order change to bring coordinator and overlord together
* mentioned merged Coordinator-Overlord in upgrade order doc
* revert autoscaling doc change
* auto scaling doc fix
2017-07-25 12:54:12 -05:00
6e2f78f552
TLS support ( #4270 )
2017-07-06 17:40:12 -07:00
4502c207af
fix injection bug and documentation ( #4243 )
2017-05-03 15:07:43 -05:00
7411b18df9
Add BroadcastDistributionRule ( #4077 )
...
* Add BroadcastDistributionRule
* Add missing null check
* Rename variable 'colocateDataSource' to 'colocatedDatasource'
* Address comments
* Document for broadcast rules
* Drop segments which are not co-located anymore
* Remove duplicated segment loading and dropping
* Add caveat
* address comments
2017-05-01 09:55:17 -07:00
94da5e80f9
Namespace optimization for hdfs data segments. ( #3877 )
...
* NN optimization for hdfs data segments.
* HdfsDataSegmentKiller, HdfsDataSegment finder changes to use new storage
format.Docs update.
* Common utility function in DataSegmentPusherUtil.
* new static method `makeSegmentOutputPathUptoVersionForHdfs` in JobHelper
* reuse getHdfsStorageDirUptoVersion in
DataSegmentPusherUtil.getHdfsStorageDir()
* Addressed comments.
* Review comments.
* HdfsDataSegmentKiller requested changes.
* extra newline
* Add maprfs.
2017-03-01 09:51:20 -08:00
35160e5595
Add metrics for Query Count statistics ( #3470 )
...
* Add metrics for Query Count statistics
This PR adds a new metrics monitor “QueryCountStatsMonitor” which emits
three new metrics -
1) query/success/count - number of successful queries
2) query/failed/count - number of failed queries
3) query/interrupted/count - number of interrupted/timedout queries
fix bindings
* make fields final
* fix imports
* AsyncQueryForwardingServlet implement QueryStatsProvider
* remove unused import
2016-12-19 09:47:58 -08:00
4be3eb0ce7
report message gap, source gap and sink count in RealtimePlumber ( #3744 )
...
* report message gap, source gap and sink count in RealtimePlumber
* report message gap, sink count in Appenderator
* add ingest/events/sourceGap in metrics.md
* remove source gap
2016-12-13 11:23:02 -06:00
7c63bee7f5
Add mapreduce.job.classloader.system.classes property to 'Other Hadoop Versions' docs ( #3706 )
2016-11-18 16:16:50 -08:00
b76b3f8d85
reset-cluster command to clean up druid state stored on metadata and deep storage ( #3670 )
2016-11-09 11:07:01 -06:00
6f21778364
Support finding segments in AWS S3. ( #3399 )
...
* support finding segments from a AWS S3 storage.
* add more Uts
* address comments and add a document for the feature.
* update docs indentation
* update docs indentation
* address comments.
1. add a Ut for json ser/deser for the config object.
2. more informant error message in a Ut.
* address comments.
1. use @Min to validate the configuration object
2. change updateDescriptor to a string as it does not take an argument otherwise
* fix a Ut failure - delete a Ut for testing default max length.
2016-10-10 17:27:09 -07:00
6b40bf8b32
doc: added note to README, about necessary hdfs config after insert-segment-to-db ( #3402 )
2016-08-28 16:39:33 -07:00
d624037698
Pull-deps: correct the library directory in the document ( #3361 )
...
* Pull-deps: correct the library directory in the document
* Pull-deps: correct the library directory in the document in the last example command
2016-08-16 17:18:15 -07:00
6beb8ac342
fix some docs and add new content ( #3369 )
2016-08-16 15:00:18 -07:00
ed5b92d612
document how to check MM enabled/disabled ( #3331 )
2016-08-06 05:56:51 +08:00
e5397ed316
Link up Hadoop class loading docs better. ( #3302 )
2016-07-29 10:19:54 -07:00
546e4f79b0
Add size of pending deletes to historical metrics ( #3295 )
...
* Add size of pending deletes to historical metrics
2016-07-27 11:30:47 -07:00
b1e3fe77f5
More logging around how the coordinator balancer is happening ( #3279 )
...
* More logging around how the coordinator balancer is happening
* Address comments
* Address code review comments and add actual logging
2016-07-27 13:24:32 +05:30
dd4ec751d0
Update docs for working with Hadoop dependencies. ( #3252 )
...
- Attempt to make things clearer in general
- Point out that HDFS deep storage and MR jobs don't use the same loading mechanism
- Recommend using mapreduce.job.classloader = true when possible
2016-07-18 07:47:58 -05:00
6a03a0cfec
Fix ingest/persist/backPressure docs. ( #3243 )
2016-07-13 21:56:28 -07:00
b8a4f4ea7b
DumpSegment: Add --dump bitmaps option. ( #3221 )
...
Also make --dump metadata respect --column.
2016-07-06 12:42:50 -07:00
99844dfeb5
remove need for tmp extensions dir ( #3211 )
...
correct lib path relative to base distribution dir
2016-07-01 12:55:57 -07:00
66d8ad36d7
adds new coordinator metrics 'segment/unavailable/count' and ( #3176 )
...
'segment/underReplicated/count' (#3173 )
2016-06-23 14:53:15 -07:00
da660bb592
DumpSegment tool. ( #3182 )
...
Fixes #2723 .
2016-06-23 14:37:50 -07:00
3b3e772748
Add --no-default-remote-repositories flag to pull-deps. ( #3120 )
2016-06-13 17:01:18 +05:30
4ab675e863
Fix command name in example ( #3088 )
2016-06-07 10:44:27 -07:00
cd5c5419bb
Make docs deploying better. ( #3040 )
...
- Make redirects for old links based on _redirects.json
- Replace #{DRUIDVERSION} tokens in docs with current version
- Allow origins named something other than "origin"
- Can use either s3cmd or awscli, depending on availability
2016-05-31 15:34:58 -07:00
00de26c76a
fix extensions docs ( #2995 )
...
* fix extensions docs
* fix mistakes
2016-05-19 14:01:06 -07:00
6c5bf91f9a
publish metrics numJettyConns to see how number of active jetty connections change over time ( #2839 )
...
this can be compared with numer of active queries to see if requests are waiting in jetty queue
2016-05-02 14:08:25 -07:00
639d0630b8
jackson conflict workaround in hadooop ingestio & parquet extension coordinate update ( #2817 )
2016-04-13 14:20:33 -07:00
37d2ab623e
Merge pull request #2815 from slaunay/documentation/hadoop-classpath-issue-fix-with-configuration
...
Doc for mapreduce.job.user.classpath.first=true
2016-04-12 10:51:51 -07:00
e3e932a4d4
refactor extensions into core and contrib
2016-03-08 17:12:09 -08:00
18f9e05f0f
improve doc on including druid and hadoop extensions
2016-02-26 13:53:08 -06:00
083f019a48
Merge pull request #2465 from druid-io/more-doc-fix
...
more doc fixes
2016-02-17 11:00:38 -08:00
7da6594bfe
more doc fixes
2016-02-17 09:43:47 -08:00
e9f1c94822
Update metrics.md
2016-02-17 09:27:15 -06:00
Kirill Kozlov
kaijianding
Jakub Kukul
David Lim
Gian Merlino
Hongze Zhang
Jonathan Wei
Jihoon Son
Himanshu
Parag Jain
Parag Jain
Akash Dwivedi
Nishant
jaehong choi
Ashish
Chanh Le
Fangjin Yang
Charles Allen
michaelschiff
du00cs
Sébastien Launay
Bingkun Guo
Slim