Commit Graph

1078 Commits

Author SHA1 Message Date
Roman Leventov 2fa4b10145 More fine-grained DI for management node types. Don't allocate processing resources on Router (#4429)
* Remove DruidProcessingModule, QueryableModule and QueryRunnerFactoryModule from DI for coordinator, overlord, middle-manager. Add RouterDruidProcessing not to allocate processing resources on router

* Fix examples

* Fixes

* Revert Peon configs and add comments

* Remove qualifier
2017-06-27 22:58:01 -07:00
Slim a2584d214a Delagate creation of segmentPath/LoadSpec to DataSegmentPushers and add S3a support (#4116)
* Adding s3a schema and s3a implem to hdfs storage module.

* use 2.7.3

* use segment pusher to make loadspec

* move getStorageDir and makeLoad spec under DataSegmentPusher

* fix uts

* fix comment part1

* move to hadoop 2.8

* inject deep storage properties

* set version to 2.7.3

* fix build issue about static class

* fix comments

* fix default hadoop default coordinate

* fix create filesytem

* downgrade aws sdk

* bump the version
2017-06-04 00:55:09 -06:00
Kenji Noguchi 3400f601db Protobuf extension (#4039)
* move ProtoBufInputRowParser from processing module to protobuf extensions

* Ported PR #3509

* add DynamicMessage

* fix local test stuff that slipped in

* add license header

* removed redundant type name

* removed commented code

* fix code style

* rename ProtoBuf -> Protobuf

* pom.xml: shade protobuf classes, handle .desc resource file as binary file

* clean up error messages

* pick first message type from descriptor if not specified

* fix protoMessageType null check. add test case

* move protobuf-extension from contrib to core

* document: add new configuration keys, and descriptions

* update document. add examples

* move protobuf-extension from contrib to core (2nd try)

* touch

* include protobuf extensions in the distribution

* fix whitespace

* include protobuf example in the distribution

* example: create new pb obj everytime

* document: use properly quoted json

* fix whitespace

* bump parent version to 0.10.1-SNAPSHOT

* ignore Override check

* touch
2017-05-30 13:11:58 -07:00
Jihoon Son 733dfc9b30 Add PrefetchableTextFilesFirehoseFactory for cloud storage types (#4193)
* Add PrefetcheableTextFilesFirehoseFactory

* fix comment

* exception handling

* Fix wrong json property

* Remove ReplayableFirehoseFactory and fix misspelling

* Defer object initialization

* Add a temporaryDirectory parameter to FirehoseFactory.connect()

* fix when cache and fetch are disabled

* Address comments

* Add more test

* Increase timeout for test

* Add wrapObjectStream

* Move methods to Firehose from PrefetchableFirehoseFactory

* Cleanup comment

* add directory listing to s3 firehose

* Rename a variable

* Addressing comments

* Update document

* Support disabling prefetch

* Fix race condition

* Add fetchLock

* Remove ReplayableFirehoseFactoryTest

* Fix compilation error

* Fix test failure

* Address comments

* Add default implementation for new method
2017-05-18 15:37:18 +09:00
Roman Leventov b7a52286e8 Make @Override annotation obligatory (#4274)
* Make MissingOverride an error

* Make travis stript to fail fast

* Add missing Override annotations

* Comment
2017-05-16 13:30:30 -05:00
Gian Merlino 2ca7b00346 Update versions to 0.10.1-SNAPSHOT. (#4191) 2017-04-20 18:12:28 -07:00
Nishant Bangarwa 801ea5efa4 Fix: Broker fails throws OOME with conf-quickstart (#4127)
when running the the packaged conf-quickstart druid broker fails to
start and throws OOME. increasing the direct memory to get around this.
2017-03-29 11:43:58 -07:00
Gian Merlino 12317fd001 Bump version to 0.10.0-SNAPSHOT. (#3913) 2017-02-06 17:54:35 -08:00
Nishant 351d570684 Improve startup script - create PID and LOG dir if they do not exist (#3808) 2017-01-02 09:20:22 -08:00
Nishant 93c34d3c3f Ability to add hadoop config directory via environment variable (#3781) 2016-12-16 11:19:15 -08:00
Gian Merlino 657e4512d2 Checkstyle checks for AvoidStaticImport, UnusedImports. (#3660)
Excludes tests from AvoidStaticImport, since those are used often there and
I didn't want to make this changeset too large. Production code use was minimal
and I switched those to non-static imports.
2016-11-05 11:34:36 -07:00
kaijianding f1dee037d6 fix 'No Such File' error when execute script out of druid installation directory (#3517) 2016-11-01 09:57:09 -07:00
Aveplatter 317d62e18c teeny tiny wording change (#3623) 2016-10-31 09:46:54 -07:00
Gian Merlino 40f2fe7893 Bump versions to 0.9.3-SNAPSHOT (#3524) 2016-09-29 13:53:32 -07:00
Maciej Bryński d0ea84149f Changing num threads to 9 (#3492) 2016-09-28 10:44:51 -06:00
Gian Merlino 50db86cb17 Quickstart: Use hadoopyString for batch indexing instead of string. (#3263) 2016-07-19 10:18:10 -07:00
Nishant 94b3c74cdc Druid launch script improvements (#3175)
* Add status command to launch scripts

* make druid init script to pick up config directories from environment variables

make druid init script to pick up config directories from environment
variables
2016-06-24 15:02:34 -07:00
Gian Merlino ebf890fe79 Update master version to 0.9.2-SNAPSHOT. (#3133) 2016-06-13 13:10:38 -07:00
Charles Allen 8024b915e2 [QTL] Implement LookupExtractorFactory of namespaced lookup (#2926)
* support LookupReferencesManager registration of namespaced lookup and eliminate static configurations for lookup from namespecd lookup extensions

- druid-namespace-lookup and druid-kafka-extraction-namespace are modified
- However, druid-namespace-lookup still has configuration about ON/OFF
  HEAP cache manager selection, which is not namespace wide
  configuration but node wide configuration as multiple namespace shares
  the same cache manager

* update KafkaExtractionNamespaceTest to reflect argument signature changes

* Add more synchronization functionality to NamespaceLookupExtractorFactory

* Remove old way of using extraction namespaces

* resolve compile error by supporting LookupIntrospectHandler

* Remove kafka lookups

* Remove unused stuff

* Fix start and stop behavior to be consistent with new javadocs

* Remove unused strings

* Add timeout option

* Address comments on configurations and improve docs

* Add more options and update hash key and replaces

* Move monitoring to the overriding classes

* Add better start/stop logging

* Remove old docs about namespace names

* Fix bad comma

* Add `@JsonIgnore` to lookup factory

* Address code review comments

* Remove ExtractionNamespace from module json registration

* Fix problems with naming and initialization. Add tests

* Optimize imports / reformat

* Fix future not being properly cancelled on failed initial scheduling

* Fix delete returns

* Add more docs about whole introspection

* Add `/version` introspection point for lookups

* Add more tests and address comments

* Add StaticMap extraction namespace for testing. Also add a bunch of tests

* Move cache system property to `druid.lookup.namespace.cache.type`

* Make VERSION lower case

* Change poll period to 0ms  for StaticMap

* Move cache key to bytebuffer

* Change hashCode and equals on static map extraction fn

* Add more comments on StaticMap

* Address comments

* Make scheduleAndWait use a latch

* Sanity renames and fix imports

* Remove extra info in docs

* Fix review comments

* Strengthen failure on start from warn to error

* Address comments

* Rename namespace-lookup to lookups-cached-global

* Fix injective mis-naming
* Also add serde test
2016-05-24 10:56:40 -07:00
Charles Allen 2863adef60 Add jconsole.sh example script for connecting (#2947)
* Can change logging at run-time
2016-05-16 15:37:15 -07:00
Walton Seymour 13a2b265a5 Adds init scripts (#2939) 2016-05-10 09:00:13 -07:00
Gian Merlino 738dcd8cd9 Update version to 0.9.1-SNAPSHOT.
Fixes #2462
2016-03-17 10:34:20 -07:00
Gian Merlino d63473e0d5 examples: Switch druid-server, druid-common to "provided". 2016-03-10 18:43:29 -08:00
Gian Merlino e0c049c0b0 Make startup properties logging optional.
Off by default, but enabled in the example config files. See also #2452.
2016-02-12 14:12:16 -08:00
fjy 1aa363cea7 new quickstart 2016-02-04 09:37:38 -08:00
Gian Merlino 8e594a2e72 Change service names in docs, examples to match defaults in the code. 2015-12-06 10:04:21 -08:00
Himanshu Gupta 62ba9ade37 unifying license header in all java files 2015-12-05 22:16:23 -06:00
Bingkun Guo 4914925d65 New extension loading mechanism
1) Remove maven client from downloading extensions at runtime.
2) Provide a way to load Druid extensions and hadoop dependencies through file system.
3) Refactor pull-deps so that it can download extensions into extension directories.
4) Add documents on how to use this new extension loading mechanism.
5) Change the way how Druid tarball is generated. Now all the extensions + hadoop-client 2.3.0
are packaged within the Druid tarball.
2015-10-21 14:22:36 -05:00
Xavier Léauté e4ac78e43d bump next snapshot to 0.9.0 2015-10-20 13:46:13 -07:00
Xavier Léauté 4c2c7a2c37 update version to 0.8.3 2015-10-14 21:40:55 -07:00
David Lim f42f6247ee Modified the Twitter firehose to process more properties
Add dimensions such as screen name, retweet and verified booleans,
source, location, and originator information to support additional
analytics.
2015-09-25 00:21:15 -06:00
David Lim 13e808368f Disable metadata publishing attempt in example script
Example config no-ops DataSegmentPusher and DataSegmentAnnouncer but
still binds SegmentPublisher to MetadataSegmentPublisher which was
throwing exceptions when it tried to publish and no DB was configured.
Setting this binds NoopSegmentPublisher instead.
2015-09-16 15:57:04 -07:00
Gian Merlino 5ca440ff15 run_druid_server.sh 2015-08-18 18:32:33 -07:00
Gian Merlino 601cd82987 Use local repository instead of maven central:
- Use local repo in run_example_server.
- Add extensions-repo to common.runtime.properties for the tarball.
2015-08-18 18:32:33 -07:00
Xavier Léauté 3b2e41e42a update for next release 2015-08-18 17:16:46 -07:00
Zhihui Jiao d3e838ff34 Fix configuration for javaOpts, remove quotation marks 2015-08-01 11:26:37 +08:00
Himanshu Gupta 034e0a4f34 adding example config for starting middle manager 2015-07-24 16:57:12 -05:00
Himanshu Gupta 299367b0da adding router config to examples/config 2015-07-23 22:47:47 -05:00
fjy 0074138bb9 fix versions in tutorials 2015-07-15 16:18:16 -07:00
Xavier Léauté 4cfb00bc8a inrement version 2015-07-15 13:09:05 -07:00
Fangjin Yang 798c3320d0 Merge pull request #1475 from himanshug/derby_in_examples
for example/tutorial, use derby instead of mysql
2015-07-14 10:18:21 -07:00
fjy 08d00cc80f rework the realtime examples a bit; add more faq 2015-07-07 14:07:14 -07:00
Himanshu Gupta 473bc101d7 for example/tutorial, use derby instead of mysql 2015-06-30 20:47:32 -05:00
Xavier Léauté 0a5bb909a2 [maven-release-plugin] prepare for next development iteration 2015-06-18 17:35:19 -07:00
Xavier Léauté 59c6b2b279 [maven-release-plugin] prepare release druid-0.8.0-rc1 2015-06-18 17:35:14 -07:00
fjy 7a6acf5c1b update pom to 0.8 2015-05-11 19:41:58 -06:00
Charles Allen 27e25fe513 fix IRCFirehoseFactory json name/nick mismatch
* Now it is nick as per wikipedia example
2015-05-01 12:35:44 -07:00
fjy 4575c5378c Rework the druid docs 2015-04-21 10:11:11 -07:00
Himanshu Gupta 457edb5d25 correcting storageDirectory property typo 2015-04-01 00:54:38 -05:00
fjy aea7f9d192 [maven-release-plugin] prepare for next development iteration 2015-03-30 16:35:24 -07:00