Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,322 Commits 111 Branches 503 Tags 721 MiB
Commit Graph

1006 Commits

Author SHA1 Message Date
Bingkun Guo 89b477970f DataSegmentFinder tool 
`insert-segment-to-db` is a tool that can insert segments into Druid metadata storage. It is intended to be used
to update the segment table in metadata storage after people manually migrate segments from one place to another.
It can also be used to insert missing segment into Druid, or even recover metadata storage by telling it where the
segments are stored.

Note: This tool expects users to have Druid cluster running in a "safe" mode, where there are no active tasks to interfere
the segments being inserted. Users can optionally bring down the cluster to make 100% sure nothing is interfering.
 2015-12-21 00:02:04 -06:00
Fangjin Yang d95adb5749 Merge pull request #1949 from metamx/overlord-console-redirect 
redirect to overlord console by default
 2015-12-10 15:11:00 -08:00
Xavier Léauté dcd15735cd Merge pull request #2015 from metamx/handoff-notifier-coordinator 
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoff information
 2015-12-08 14:06:39 -08:00
Nishant 9491e8de3b Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoffs 
- fixes #1970
- extracted out segment handoff callbacks in SegmentHandoffNotifier
which is responsible for tracking segment handoffs and doing callbacks
when handoff is complete.
- Coordinator now maintains a view of segments in the cluster, this
will affect the jam heap requirements for the overlord for large
clusters.
realtime index task and nodes now use HTTP end points exposed by the
coordinator to get serverView

review comment

fix realtime node guide injection

review comments

make test not rely on scheduled exec

fix compilation

fix import

review comment

introduce immutableSegmentLoadInfo

fix son reading

remove unnecessary logging
 2015-12-09 01:54:09 +05:30
Charles Allen cacd8f7a27 Merge pull request #2028 from navis/parse-spec-validator 
fix for #1939, Druid ParseSpec validator
 2015-12-07 11:53:01 -08:00
Himanshu Gupta 62ba9ade37 unifying license header in all java files 2015-12-05 22:16:23 -06:00
navis.ryu 4a70d8e775 ParseSpecValidator 2015-12-04 11:08:02 +09:00
Fangjin Yang 21c84b5ff7 Merge pull request #1896 from gianm/allocate-segment 
SegmentAllocateAction (fixes #1515)
 2015-11-18 21:05:46 -08:00
Charles Allen 8fcf2403e3 Merge pull request #1943 from metamx/realtime-caching 
Enable caching on intermediate realtime persists
 2015-11-17 15:06:43 -08:00
Charles Allen dbe201aeed Merge pull request #1929 from pjain1/jetty_threads 
separate ingestion and query thread pool
 2015-11-17 12:14:25 -08:00
Parag Jain 6c498b7d4a separate ingestion and query thread pool 2015-11-17 13:42:41 -06:00
Xavier Léauté d7eb2f717e enable query caching on intermediate realtime persists 2015-11-17 10:58:00 -08:00
Gian Merlino e4e5f0375b SegmentAllocateAction (fixes #1515) 
This is a feature meant to allow realtime tasks to work without being told upfront
what shardSpec they should use (so we can potentially publish a variable number
of segments per interval).

The idea is that there is a "pendingSegments" table in the metadata store that
tracks allocated segments. Each one has a segment id (the same segment id we know
and love) and is also part of a sequence.

The sequences are an idea from @cheddar that offers a way of doing replication.
If there are N tasks reading exactly the same data with exactly the same logic
(think Kafka tasks reading a fixed range of offsets) then you can place them
in the same sequence, and they will generate the same sequence of segments.
 2015-11-11 16:54:35 -08:00
Xavier Léauté fa6142e217 cleanup and remove unused imports 2015-11-11 12:25:21 -08:00
Xavier Léauté b519fcf99b redirect to overlord console by default 2015-11-11 09:55:18 -08:00
Xavier Léauté a57cbfd2c3 Merge pull request #1387 from metamx/enableShutdownLogging 
Add special handler to allow logger messages during shutdown
 2015-11-09 17:20:09 -08:00
fjy 8f231fd3e3 cleanup druid codebase 2015-11-04 13:59:53 -08:00
Gian Merlino 8defe29270 Merge pull request #1901 from guobingkun/fix_typo_and_rename 
Fix metadata typo and rename default extension directory
 2015-11-03 14:02:11 -08:00
Xavier Léauté 3b3d88bd03 Merge pull request #1857 from noddi/feature/intervals-route 
Queryable intervals
 2015-11-03 13:19:04 -08:00
Bingkun Guo 962f65cc76 fix metadata typo and rename default extension directory 2015-11-03 14:50:42 -06:00
Himanshu Gupta 84f7d8d264 making static final variables in HadoopDruidIndexerConfig upper case 2015-11-02 23:24:26 -06:00
Bingkun Guo 657a5ac346 fix pull-deps remoteRepository option 2015-10-30 11:32:56 -05:00
Bartosz Ługowski 6de51e022d Coordinator - add intervals route. 2015-10-30 11:51:42 +01:00
Charles Allen 194f632b00 Check for non-daemon threads when CliPeon exits 2015-10-29 11:56:38 -07:00
Charles Allen 7a2ceef690 Add special handler to allow logger messages during shutdown 
* Adds a special PropertyChecker interface which is ONLY for setting string properties at the very start of psvm
 2015-10-27 14:33:36 -07:00
Bingkun Guo 4914925d65 New extension loading mechanism 
1) Remove maven client from downloading extensions at runtime.
2) Provide a way to load Druid extensions and hadoop dependencies through file system.
3) Refactor pull-deps so that it can download extensions into extension directories.
4) Add documents on how to use this new extension loading mechanism.
5) Change the way how Druid tarball is generated. Now all the extensions + hadoop-client 2.3.0
are packaged within the Druid tarball.
 2015-10-21 14:22:36 -05:00
Xavier Léauté e4ac78e43d bump next snapshot to 0.9.0 2015-10-20 13:46:13 -07:00
Xavier Léauté 4c2c7a2c37 update version to 0.8.3 2015-10-14 21:40:55 -07:00
Xavier Léauté 5db4c5a089 move cache bindings into a dedicated module 2015-10-02 00:29:05 -04:00
Gian Merlino 940e1aa3eb Replace funky imports with standard ones. 
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
 2015-08-28 18:02:05 -07:00
Xavier Léauté 9a0c15c52c update airline to 0.7 2015-08-25 16:07:00 -07:00
Xavier Léauté 2093187c91 rework tarball distribution: 
- move assembly out of druid-services into a 'distribution' module
- create separate 'extensions-distribution' module and assembly to
  package extensions and their dependencies into a local maven
  repository
- include this extensions maven repository in the binaries tarball
 2015-08-18 18:32:33 -07:00
Xavier Léauté 3b2e41e42a update for next release 2015-08-18 17:16:46 -07:00
Himanshu Gupta 15fa43dd43 changing DatasourcePathSpec, to get segment list, so that hadoop indexer uses overlord action to get list of segments and passes when running as an overlord task. and, uses metadata store directly when running as standalone hadoop indexer 
also, serialized list of segments is passed to DatasourcePathSpec so that hadoop classloader issues do not creep up
 2015-08-16 14:07:35 -05:00
Charles Allen db19d2d547 Revert "Update to guice 4.0" 2015-08-14 09:26:07 -07:00
Nishant 08f50fcc37 explicitly call lifecycle stop 2015-08-14 17:35:17 +05:30
Charles Allen 7e61216287 Update to guice 4.0 
- Mark a lot of `@Provides` methods as final since guice 4.0 disallows overriding them
 2015-08-10 13:57:18 -07:00
Charles Allen 9dd18de1a5 Merge pull request #1596 from metamx/console-artifact 
make console a maven dependency instead of filedump
 2015-08-04 15:28:45 -07:00
Xavier Léauté 07ee1acca2 make console a maven dependency instead of filedump 2015-08-04 09:49:34 -07:00
Charles Allen ba59f8afc4 Merge pull request #1550 from himanshug/optionally_log_all_requests 
print *all* HTTP requests to log if configured
 2015-07-28 11:19:36 -07:00
Himanshu Gupta 90b475975e Log all the requests to debug level in logger io.druid.jetty.RequestLog 2015-07-28 12:56:04 -05:00
sahner 4801de62a2 make "announce" the chathandler default in realtime node, 
remove doc references to chathandler type "announce" since it is the default now,
 2015-07-27 12:14:28 -05:00
Fangjin Yang e21195f987 Merge pull request #1469 from guobingkun/table_config 
Inconsistent property names for "druid.metadata.storage.tables.xxx"
 2015-07-17 07:43:19 -07:00
Charles Allen 456ad9ffba Merge pull request #1529 from metamx/update-versions 
inrement version
 2015-07-15 13:25:31 -07:00
Xavier Léauté 4cfb00bc8a inrement version 2015-07-15 13:09:05 -07:00
Charles Allen 2b87ad307d Make CliPeon run shutdown hooks properly 
* Also make jetty threads daemon
 2015-07-14 15:15:26 -07:00
Bingkun Guo 4a0ae7d8d5 Fix inconsistent druid property names for "druid.metadata.storage.tables.xxx" between document and code 2015-06-29 10:12:30 -05:00
Davide Anastasia 4a3a7dd1ad read hadoop-indexer configuration file from HDFS 2015-06-24 14:08:53 -07:00
Xavier Léauté 0a5bb909a2 [maven-release-plugin] prepare for next development iteration 2015-06-18 17:35:19 -07:00
Xavier Léauté 59c6b2b279 [maven-release-plugin] prepare release druid-0.8.0-rc1 2015-06-18 17:35:14 -07:00
Charles Allen 056cab93ed Add Hadoop Converter Job and task 
* Fixes https://github.com/druid-io/druid/issues/1363
* Add extra utils in JobHelper based on PR feedback
 2015-06-09 14:47:38 -07:00
fjy 7a6acf5c1b update pom to 0.8 2015-05-11 19:41:58 -06:00
Himanshu Gupta aa10626b61 adding druid.db.tables.xxx renames and druid.db.connectory.*validation* from ConvertProperties 2015-05-03 08:41:15 -05:00
Himanshu Gupta 8722f4cc18 UTs for ConvertProperties 2015-05-03 08:40:17 -05:00
Fangjin Yang 40121bc800 Merge pull request #1324 from metamx/fixMasterPropertiesTest 
Partial revert of 467fcfae03
 2015-04-29 22:15:21 -07:00
Charles Allen de16eb34af Partial revert of 467fcfae03 2015-04-29 17:20:51 -07:00
Xavier Léauté bfdb5dcb5d Merge pull request #1317 from metamx/overlord-endpoints 
proxy /druid/indexer requests on the coordinator to the indexing overlord
 2015-04-29 16:07:42 -07:00
Charles Allen e055a7e869 Add options to the properties converter to update from 0.6.x to 0.7.x 
* Removed 0.5 --> 0.6 converter
 2015-04-29 15:46:16 -07:00
nishant 6e84caf3d5 Proxy Requests from coordinator to overlord 
Add unit test and support https scheme

review comments
 2015-04-29 20:58:26 +05:30
fjy aea7f9d192 [maven-release-plugin] prepare for next development iteration 2015-03-30 16:35:24 -07:00
fjy 060d7aef03 [maven-release-plugin] prepare release druid-0.7.1 2015-03-30 16:35:20 -07:00
Himanshu Gupta b131a4fd3b adding router numMaxThread configuration for ProxyServlet HttpClient executor 2015-03-23 21:58:01 -05:00
fjy b389cfe404 [maven-release-plugin] prepare for next development iteration 2015-03-19 12:38:17 -07:00
fjy 60e7d543cc [maven-release-plugin] prepare release druid-0.7.1-rc1 2015-03-19 12:38:13 -07:00
Xavier Léauté c847b3b861 Merge pull request #1218 from himanshug/task_storage_config_fix 
add correct guice init of TaskStorageConfig to CliPeon
 2015-03-17 13:08:43 -07:00
Himanshu Gupta 4a8f5dfa0f add correct guice init of TaskStorageConfig to CliPeon 2015-03-17 14:03:54 -05:00
nishantmonu51 3ec4a30ab5 initial commit 
review comments

more refactoring and cleaning of redundant code

add UT + docs + more refactoring

fixes + review comments

more cleanup

end points to fetch history

review comments

remove unnecessary changes

review comments rename header name

review comments + add test for MetadataRulesManager

review comments docs
 2015-03-12 22:50:29 +05:30
Himanshu Gupta 956361dbbd Refactoring: Moved BaseJettyServerInitializer.java to JettyServerInitUtils.java and changed it from abstract class to a utils class containing static methods 
And, moved jetty stuff to its own package io.druid.server.initialization.jetty
 2015-03-05 19:33:27 -06:00
Himanshu Gupta a8648d8f3d feature to add servlet filters in a druid node via extension modules 2015-03-03 22:35:02 -06:00
Xavier Léauté 17743b94f3 Cleanup router servlet + tests for gzip proxying 
- Uses method overrides instead of modified Jetty code, now that
  ProxyServlet provides enough method hooks for proper overrides.
  This means we may also benefit from any Jetty ProxyServlet fixes
- Adds test for async proxy servlet to make sure gzip encoding is
  properly proxied.
- Router now proxies POST requests for requests that are not Druid
  queries, by only treating /druid/v2/* endpoints as queries.
 2015-03-02 11:36:15 -08:00
Himanshu Gupta 467fcfae03 UTs for services 2015-02-25 15:45:59 -08:00
Xavier Léauté b167dcf82c [maven-release-plugin] prepare for next development iteration 2015-02-23 14:28:06 -08:00
Xavier Léauté e81ac2ba43 [maven-release-plugin] prepare release druid-0.7.0 2015-02-23 14:27:58 -08:00
Xavier Léauté 8e33e58422 rename Druid binary assembly 2015-02-18 17:42:55 -08:00
Xavier Léauté 78df7f6165 Move Druid release artifacts to Sonatype 
- Switch to using Druid parent POM
- Add required fields for Sonatype
- Common plugin versions and settings have been moved to the parent pom
- Cleanup artifacts and POMs for consistent formatting
- Remove org.hyperic.sigar dependency and update docs to reflect necessary jars to add at runtime when sigar is needed
 2015-02-13 14:26:31 -08:00
fjy d29740ed9f [maven-release-plugin] prepare for next development iteration 2015-02-12 16:16:00 -08:00
fjy 211fd15b7e [maven-release-plugin] prepare release druid-0.7.0-rc3 2015-02-12 16:15:56 -08:00
fjy 1f12c5b2f1 [maven-release-plugin] prepare for next development iteration 2015-02-03 12:06:49 -08:00
fjy e82d431be7 [maven-release-plugin] prepare release druid-0.7.0-rc2 2015-02-03 12:06:41 -08:00
fjy 3e5d338c8e Remove non friendly dependencies from Druid 2015-02-03 11:36:08 -08:00
Eric Tschetter 42eba986ce Towards consistent null handling 
This commit also includes
1) the addition of a context parameter on timeseries queries that allows it to ignore empty buckets instead of generating results for them
2) A cleanup of an unused method on an interface
 2015-02-02 12:53:07 -08:00
fjy d05032b98a towards a community led druid 2015-01-31 20:57:36 -08:00
Xavier Léauté 2c2771b90e Make dynamic worker selection actually work 2015-01-27 14:17:42 -08:00
fjy 1f94de22c6 [maven-release-plugin] prepare for next development iteration 2015-01-20 14:23:55 -08:00
fjy 17476edc31 [maven-release-plugin] prepare release druid-0.7.0-rc1 2015-01-20 14:23:51 -08:00
Fangjin Yang 852e863425 Merge pull request #981 from druid-io/strictModuleTyping 
Use Module instead of generic Object in Guice related items
 2015-01-05 12:43:20 -08:00
Charles Allen 65286a24e0 Change zk configs to use Jackson injection instead of Skife 
* Also added generic config testing class JsonConfigTesterBase
 2014-12-29 10:36:12 -08:00
Charles Allen 7c8d4a7433 Use Module instead of generic Object in Guice related items 2014-12-19 10:54:06 -08:00
fjy c0af7b2a5d fix broken ut 2014-12-15 18:29:15 -08:00
Charles Allen d1fcd4e92e Add log4j in services 2014-12-15 17:39:19 -08:00
fjy 4f0c0c5ae2 i hate guice 2014-12-15 16:04:19 -08:00
fjy ac407fb6ba clean up defaults 2014-12-15 15:05:02 -08:00
Charles Allen a520835972 Merge pull request #957 from metamx/fix-gzip-encoding 
fix gzip compression
 2014-12-15 13:20:36 -08:00
fjy 56dfe5d883 Merge branch 'master' of github.com:metamx/druid 2014-12-15 09:42:08 -08:00
fjy 88858ea1c3 make default handler service announcing 2014-12-15 09:42:03 -08:00
Xavier Léauté 4b5282d224 add loadstatus endpoint for serverView status 2014-12-12 13:18:31 -08:00
Xavier Léauté 0de56efe5c fix gzip compression 
- fixes compression not working for POST requests
- router now forwards raw bytes instead of decompressing
- cleanded up router servlet initialization
- add test for gzip on get and post methods
- use Jersey annotation when possible in QueryResource
 2014-12-10 17:30:17 -08:00
Xavier Léauté 5f3f4e0a60 prevent /status from redirect, fixes #953 2014-12-10 11:32:22 -08:00
Fangjin Yang 332e4a4d1c Merge pull request #902 from metamx/cache-config 
Commonalize the cache config and change default to disable cache
 2014-12-04 11:30:24 -07:00
Gian Merlino d388a8fe89 Replace google-http-client imports with real guava imports. 2014-12-03 10:52:57 -08:00