Commit Graph

9955 Commits

Author SHA1 Message Date
Vadim Ogievetsky ddec5ea82e Web console: Improve data loader time column detection (#8199)
* better timestamp suggestion

* misc fixes

* update dependancies and licenses

* add indexer, update snapshots

* add explicit render types

* pass through auto refresh in lookup view
2019-07-30 17:54:39 -07:00
Jihoon Son 385f492a55
Use PartitionsSpec for all task types (#8141)
* Use partitionsSpec for all task types

* fix doc

* fix typos and revert to use isPushRequired

* address comments

* move partitionsSpec to core

* remove hadoopPartitionsSpec
2019-07-30 17:24:39 -07:00
Aaron Bossert aba65bb675 removed hard-coded Kafka key and value deserializer (#8112)
* removed hard-coded Kafka key and value deserializer, leaving default deserializer as org.apache.kafka.common.serialization.ByteArrayDeserializer.  Also added checks to ensure that any provided deserializer class extends org.apache.kafka.serialization.Deserializer and outputs a byte array.

* Addressed all comments from original pull request and also added a
unit test.

* Added additional test that uses "poll" to ensure that custom deserializer
works properly.
2019-07-30 16:25:32 -07:00
Clint Wylie 653b558134 sql firehose and firehose doc adjustments (#8067)
* firehose doc adjustments

* fix typo

* additional information on parser types in ingestion docs

* clarify ingest segment firehose docs, add sql firehose examples to sql extension pages

* fixit

* make sql firehose more forgiving my always constructing a MapInputRowParser from the parseSpec of whatever actual InputRowParser impl is provided, remove doc references to map based parsers

* transforms

* fix tests
2019-07-30 15:28:10 -07:00
mcbrewster 4e60afc86e Web-console: add Supervisor statistics table and show json query manager (#8176)
* add statistics table

* change error message

* add query managers

* update snapshots

* fixes

* add stricter types

* fix error type

* prettier fic

* fix undefined

* add typing

* fixe query manager
2019-07-30 12:53:46 -07:00
Fokko Driesprong 0f8c902e94 Spotbugs fix DM_NUMBER_CTOR (#8072) 2019-07-30 20:08:27 +03:00
Fokko Driesprong e016995d1f Enable Spotbugs: WMI_WRONG_MAP_ITERATOR (#8005)
* WMI_WRONG_MAP_ITERATOR

* Fixed missing loop
2019-07-30 19:51:53 +03:00
Jihoon Son fb653ceef9 Add benchmark for VersionedIntervalTimeline (#8161)
* Add benchmark for VersionedIntervalTimeline

* rename
2019-07-30 08:10:00 -07:00
mcbrewster f7abc300a8 Web-console: add loader to show-json, add status-dialog (#8172)
* add loader to show json, add status-dialog

* fix ordered import

* fix changes

* order inputs

* update snapshots

* fix bad merge conflicts
2019-07-30 08:08:36 -07:00
mcbrewster a0270512a2 Web-console: add history menu to supervisors modal (#8164)
* Add history menu to supervisors dialog

* add scrolling

* change how tabs render

* remove copy button

* user query manager
2019-07-30 08:07:58 -07:00
Eyal Yurman 80a0fbf254 fix circular reference in moving average query (#8192) 2019-07-29 22:31:44 -07:00
Vadim Ogievetsky 771c04841e remove husky hooks (#8193) 2019-07-29 19:32:16 -07:00
Jonathan Wei 640b7afc1c Add CliIndexer process type and initial task runner implementation (#8107)
* Add CliIndexer process type and initial task runner implementation

* Fix HttpRemoteTaskRunnerTest

* Remove batch sanity check on PeonAppenderatorsManager

* Fix paralle index tests

* PR comments

* Adjust Jersey resource logging

* Additional cleanup

* Fix SystemSchemaTest

* Add comment to LocalDataSegmentPusherTest absolute path test

* More PR comments

* Use Server annotated with RemoteChatHandler

* More PR comments

* Checkstyle

* PR comments

* Add task shutdown to stopGracefully

* Small cleanup

* Compile fix

* Address PR comments

* Adjust TaskReportFileWriter and fix nits

* Remove unnecessary closer

* More PR comments

* Minor adjustments

* PR comments

* ThreadingTaskRunner: cancel  task run future not shutdownFuture and remove thread from workitem
2019-07-29 17:06:33 -07:00
Vadim Ogievetsky cc4450db12 Web console: add reindex (ingestSegment firehose) to the data loader (#8181)
* tidy up nulls

* standardize more on undefined

* updated licenses

* do not do heavy handed rendering

* reindex from druid

* tidy up

* add inline firehose

* add husky

* sass lint

* better suggestion

* fix script typo

* adjust time formats

* add missing time formats

* use term 'reindex'

* fix lodash.compact
2019-07-29 14:41:27 -07:00
Chi Cao Minh ab71a2e1e4 Revert "Fix dependency analyze warnings (#8128)" (#8189)
This reverts commit 5dd0d8e873.
2019-07-29 11:42:16 -07:00
Vadim Ogievetsky 0b3ede193e Web console: adding Slack channel integration (#8182)
* adding slack channel integration

* update snapshots
2019-07-28 16:17:54 -07:00
Vadim Ogievetsky 8bd0f8c2ac Web console: code quality improvements (null tidy up) (#8162)
* tidy up nulls

* standardize more on undefined

* updated licenses

* update snapshot

* do not do heavy handed rendering

* add placeholder to SQL view

* remove pointelss fragment
2019-07-27 01:46:37 -07:00
Jihoon Son adf7bafb9f Fix race between canHandle() and addSegment() in StorageLocation (#8114)
* Fix race between canHandle() and addSegment() in StorageLocation

* add comment

* add comments

* fix test

* address comments

* remove <p> tag from javadoc

* address comments

* comparingLong
2019-07-27 11:11:06 +03:00
Chi Cao Minh 5dd0d8e873 Fix dependency analyze warnings (#8128)
* Fix dependency analyze warnings

Update the maven dependency plugin to the latest version and fix all
warnings for unused declared and used undeclared dependencies in the
compile scope. Added new travis job to add the check to CI. Also fixed
some source code files to use the correct packages for their imports.

* Fix licenses and dependencies

* Fix licenses and dependencies again

* Fix integration test dependency

* Address review comments

* Fix unit test dependencies

* Fix integration test dependency

* Fix integration test dependency again

* Fix integration test dependency third time

* Fix integration test dependency fourth time

* Fix compile error

* Fix assert package
2019-07-26 10:49:03 -07:00
Alexandre YANG 7b9bd5359d [statsd-emitter] Fix ingest persists metrics typo (#8160) 2019-07-26 07:20:41 -07:00
Parag Jain 31a29d8883 add noop type name to prevent jackson exception when setting type to noop (#8133) 2019-07-25 16:07:08 -07:00
mcbrewster ef9230f916 Web-console: Add continue to spec view (#8144)
* Add continue to spec view

* update spec when type selected

* remove redundancy

* fix mistakes
2019-07-25 14:48:26 -07:00
mcbrewster 8ba1f06632 Web-console: add lookups tile to home view (#8124)
* add lookups-tile

* add uninitailized state

* remove error:
2019-07-25 13:08:58 -07:00
Jihoon Son 61f4abece4
Add more warning to the doc for resetOffsetAutomatically (#8153)
* Add more warnings to the doc for resetOffsetAutomatically

* fix kinesis doc

* fix typos

* revise the description

* capital

* capitalize
2019-07-24 17:37:32 -07:00
Jihoon Son db14946207
Add support minor compaction with segment locking (#7547)
* Segment locking

* Allow both timeChunk and segment lock in the same gruop

* fix it test

* Fix adding same chunk to atomicUpdateGroup

* resolving todos

* Fix segments to lock

* fix segments to lock

* fix kill task

* resolving todos

* resolving todos

* fix teamcity

* remove unused class

* fix single map

* resolving todos

* fix build

* fix SQLMetadataSegmentManager

* fix findInputSegments

* adding more tests

* fixing task lock checks

* add SegmentTransactionalOverwriteAction

* changing publisher

* fixing something

* fix for perfect rollup

* fix test

* adjust package-lock.json

* fix test

* fix style

* adding javadocs

* remove unused classes

* add more javadocs

* unused import

* fix test

* fix test

* Support forceTimeChunk context and force timeChunk lock for parallel index task if intervals are missing

* fix travis

* fix travis

* unused import

* spotbug

* revert getMaxVersion

* address comments

* fix tc

* add missing error handling

* fix backward compatibility

* unused import

* Fix perf of versionedIntervalTimeline

* fix timeline

* fix tc

* remove remaining todos

* add comment for parallel index

* fix javadoc and typos

* typo

* address comments
2019-07-24 17:35:46 -07:00
Fokko Driesprong 714a2726a7 Spotbugs: RC_REF_COMPARISON_BAD_PRACTICE_BOOLEAN (#8076)
* Spotbugs: RC_REF_COMPARISON_BAD_PRACTICE_BOOLEAN

* Use Objects instead
2019-07-24 16:17:02 -07:00
Magnus Henoch c87b47e0fa More documentation formatting fixes (#8149)
Add empty lines before bulleted lists and code blocks, to ensure that
they show up properly on the web site.  See also #8079.
2019-07-24 15:26:03 -07:00
Clint Wylie 0695e487e7 fix issue with CuratorLoadQueuePeon shutting down executors it does not own (#8140)
* fix issue with CuratorLoadQueuePeon shutting down executors it does not own

* use lifecycled executors

* maybe this
2019-07-24 10:59:43 -07:00
Eugene Sevastianov 799d20249f Response context refactoring (#8110)
* Response context refactoring

* Serialization/Deserialization of ResponseContext

* Added java doc comments

* Renamed vars related to ResponseContext

* Renamed empty() methods to createEmpty()

* Fixed ResponseContext usage

* Renamed multiple ResponseContext static fields

* Added PublicApi annotations

* Renamed QueryResponseContext class to ResourceIOReaderWriter

* Moved the protected method below public static constants

* Added createEmpty method to ResponseContext with DefaultResponseContext creation

* Fixed inspection error

* Added comments to the ResponseContext length limit and ResponseContext
http header name

* Added a comment of possible future refactoring

* Removed .gitignore file of indexing-service

* Removed a never-used method

* VisibleForTesting method reducing boilerplate

Co-Authored-By: Clint Wylie <cjwylie@gmail.com>

* Reduced boilerplate

* Renamed the method serialize to serializeWith

* Removed unused import

* Fixed incorrectly refactored test method

* Added comments for ResponseContext keys

* Fixed incorrectly refactored test method

* Fixed IntervalChunkingQueryRunnerTest mocks
2019-07-24 18:29:03 +03:00
Clint Wylie 0388581493
Revert "Spotbugs: SS_SHOULD_BE_STATIC (#8073)" (#8145)
This reverts commit 04a180a5fb.
2019-07-23 22:57:19 -07:00
Clint Wylie b8b22b7aaa fix references to bin/supervise in tutorial docs (#8087) 2019-07-23 15:05:01 -07:00
Clint Wylie cb82d72547
fix forking task runner task shutdown to be more graceful (#8085)
* fix forking task runner shutdown to be more graceful

* javadoc
2019-07-23 11:47:23 -07:00
Clint Wylie 83514958db remove unnecessary lock in ForegroundCachePopulator leading to a lot of contention (#8116)
* remove unecessary lock in ForegroundCachePopulator leading to a lot of contention

* mutableboolean, javadocs,document some cache configs that were missing

* more doc stuff

* adjustments

* remove background documentation
2019-07-23 10:57:59 -07:00
Fokko Driesprong 04a180a5fb Spotbugs: SS_SHOULD_BE_STATIC (#8073) 2019-07-23 18:18:49 +08:00
ilhanadiyaman b2867c9d06 web console added to docker-compose (#8117) 2019-07-22 12:40:05 -06:00
Fokko Driesprong e1a745717e Spotbugs: NP_STORE_INTO_NONNULL_FIELD (#8021) 2019-07-21 21:23:47 +08:00
Clint Wylie f24e2f16af fix npe with sql metadata manager polling and empty database (#8106)
* fix npe with sql metadata manager polling and empty database

* treat null segments separately

* use preconditions check

* add test
2019-07-20 19:09:02 -07:00
Vadim Ogievetsky 2b2fcc0371 Web console: cleanup build, check licenses in test (#8113)
* cleanup build, check licenses

* cleanup filterMap type

* don't run license check twice
2019-07-19 22:59:07 -07:00
Sashidhar Thallam ea4bad7836 Druid SQL EXTRACT time function - adding support for additional Time Units (#8068)
* 1. Added TimestampExtractExprMacro.Unit for MILLISECOND 2. expr eval for MILLISECOND 3. Added a test case to test extracting millisecond from expression. #7935

* 1. Adding DATASOURCE4 in tests. 2. Adding test TimeExtractWithMilliseconds

* Fixing testInformationSchemaTables test

* Fixing failing tests in DruidAvaticaHandlerTest

* Adding cannotVectorize() call before the test

* Extract time function - Adding support for MICROSECOND, ISODOW, ISOYEAR and CENTURY time units, documentation changes.

* Adding MILLISECOND in test case

* Adding support DECADE and MILLENNIUM, updating test case and documentation

* Fixing expression eval for DECADE and MILLENIUM
2019-07-19 20:38:32 -07:00
Samarth Jain ceb3a891bb Fix druid sql group by queries returning complex aggregation type (#8099)
* Fix druid sql group by queries returning complex aggregation type

* Remove unnecessary check
2019-07-19 13:52:14 -07:00
mcbrewster 01f3da6fda Web Console: add Group by interval to segments view (#7986)
* add group by

* update snapshot

* fix spacing

* change design pattern

* rename interface

* add static function

* save

* add whereClause"

* add default page size

* add where to inner query

* snapshots
2019-07-19 11:25:25 -07:00
Himanshu 54a7b54d2d avoid 'must return non-void type' warning (#8105) 2019-07-18 15:02:27 -07:00
Jihoon Son c7eb7cd018
Add intermediary data server for shuffle (#8088)
* Add intermediary data server for shuffle

* javadoc

* adjust timeout

* resolved todo

* fix test

* style

* address comments

* rename to shuffleDataLocations

* Address comments

* bit adjustment StorageLocation

* fix test

* address comment & fix test

* handle interrupted exception
2019-07-18 14:46:47 -07:00
Clint Wylie 03e55d30eb
add CachingClusteredClient benchmark, refactor some stuff (#8089)
* add CachingClusteredClient benchmark, refactor some stuff

* revert WeightedServerSelectorStrategy to ConnectionCountServerSelectorStrategy and remove getWeight since felt artificial, default mergeResults in toolchest implementation for topn, search, select

* adjust javadoc

* adjustments

* oops

* use it

* use BinaryOperator, remove CombiningFunction, use Comparator instead of Ordering, other review adjustments

* rename createComparator to createResultComparator, fix typo, firstNonNull nullable parameters
2019-07-18 13:16:28 -07:00
Sashidhar Thallam 72496d3712 #7858 Throwing UnsupportedOperationException from ImmutableDrui… (#7933)
* #7858 Throwing UnsupportedOperationException from ImmutableDruidDataSource's equals() and hashCode() methods.

* 1. Turning ImmutableDruidDataSource into a data container. 2. Adding a Util method to be used in tests for checking equality of ImmutableDruidDataSource objects.

* Removing unused method

* Fixing assert equals

* Fixing assert equals in TestUtils.java

* Adding java doc comments, Using ExpectedException in tests

* Fixing test cases

* Fixed expected exception message in tests

* fixed line width

* line width fix

* code style fixes

* code indentation fixes

* fixing method name
2019-07-18 22:35:19 +03:00
Surekha da16144495 Refactoring to use `CollectionUtils.mapValues` (#8059)
* doc updates and changes to use the CollectionUtils.mapValues utility method

* Add Structural Search patterns to intelliJ

* refactoring from PR comments

* put -> putIfAbsent

* do single key lookup
2019-07-17 23:02:22 -07:00
Roman Leventov 2d6d1c17a0 Add more exempt tags for stalebot; Make stalebot to close PRs/issues in 28 days rather than in 7/14 days (#8084)
* Add more exempt tags for stalebot; Make stalebot to close PRs/issues in 21 days rather than in 7/14 days

* Exempt labels only for issues, not PRs; Add 'Uncategorized problem report' tag
2019-07-17 22:57:41 -07:00
Roman Leventov ceb969903f
Refactor SQLMetadataSegmentManager; Change contract of REST met… (#7653)
* Refactor SQLMetadataSegmentManager; Change contract of REST methods in DataSourcesResource

* Style fixes

* Unused imports

* Fix tests

* Fix style

* Comments

* Comment fix

* Remove unresolvable Javadoc references; address comments

* Add comments to ImmutableDruidDataSource

* Merge with master

* Fix bad web-console merge

* Fixes in api-reference.md

* Rename in DruidCoordinatorRuntimeParams

* Fix compilation

* Residual changes
2019-07-17 17:18:48 +03:00
Vadim Ogievetsky b80f20f769 add missing dynamic coordinator configs (#8090) 2019-07-16 17:40:41 -07:00
Clint Wylie 15fbf5983d add Class.getCanonicalName to forbidden-apis (#8086)
* add checkstyle to forbid unecessary use of Class.getCanonicalName

* use forbiddin-api instead of checkstyle

* add space
2019-07-16 15:21:50 -07:00