Commit Graph

9504 Commits

Author SHA1 Message Date
Sashidhar Thallam 3bee6adcf7 Use map.putIfAbsent() or map.computeIfAbsent() as appropriate instead of containsKey() + put() (#7764)
* https://github.com/apache/incubator-druid/issues/7316 Use Map.putIfAbsent() instead of containsKey() + put()

* fixing indentation

* Using map.computeIfAbsent() instead of map.putIfAbsent() where appropriate

* fixing checkstyle

* Changing the recommendation text

* Reverting auto changes made by IDE

* Implementing recommendation: A ConcurrentHashMap on which computeIfAbsent() is called should be assigned into variables of ConcurrentHashMap type, not ConcurrentMap

* Removing unused import
2019-06-14 17:59:36 +02:00
Vadim Ogievetsky 05346a9e0c allow semicolon with comment (#7892) 2019-06-14 07:53:19 -07:00
Xue Yu 456a3654ce add PolygonBound and missing extentions list doc (#7885) 2019-06-13 12:03:58 -07:00
mcbrewster 0763585208 Web-console: update supervisors table (#7799)
* add new columns

* fix syling

* fix spaces

* update snapshots

* fix Spelling

* fix capitalization

* reorder action dialog

* set color using state

* fix snapshots

* fix array

* update snapshots

* remove extra columns

* update snapshots

* update snapshots

* fixes

* update snapshots

* use cell

* fix spacing

* update snapshot
2019-06-11 13:06:35 -07:00
Clint Wylie 8117222da3 use right port for kafka tutorial, reinfoce that tutorials assume you are using micro-quickstart single-server configuration (#7862) 2019-06-11 08:50:52 -07:00
Xue Yu ce591d1457 Support var_pop, var_samp, stddev_pop and stddev_samp etc in sql (#7801)
* support var_pop, stddev_pop etc in sql

* fix sql compatible

* rebase on master

* update doc
2019-06-10 09:40:09 -07:00
Vadim Ogievetsky c612ddc0f4 Support more JODA time formats (#7857) 2019-06-09 20:08:29 -07:00
Vadim Ogievetsky c3d230b354 Web console: make the data loader more responsive when sampling data with many columns (#7856)
* Don't rerender data loader tables

* removed unused imports

* removed extra console.log

* fix typo
2019-06-09 10:44:13 -07:00
Vadim Ogievetsky 53b9fc4eb0 switch to use exInterlop (#7853) 2019-06-07 22:27:19 -07:00
Clint Wylie 3fbb0a5e00 Supervisor list api with states and health (#7839)
* allow optionally listing all supervisors with their state and health

* docs

* add state to full

* clean

* casing

* format

* spelling
2019-06-07 16:26:33 -07:00
Vadim Ogievetsky 248e075e24 Web console: Simplify task and supervisor detail displays (#7848)
* simplify detail display

* update tests
2019-06-07 11:02:45 -07:00
Surekha ea752ef562 Optimize overshadowed segments computation (#7595)
* Move the overshadowed segment computation to SQLMetadataSegmentManager's poll

* rename method in MetadataSegmentManager

* Fix tests

* PR comments

* PR comments

* PR comments

* fix indentation

* fix tests

*  fix test

*  add test for SegmentWithOvershadowedStatus serde format

* PR comments

* PR comments

* fix test

* remove snapshot updates outside poll

* PR comments

* PR comments

* PR comments

*  removed unused import
2019-06-07 19:15:54 +02:00
Vadim Ogievetsky 061b465d34 Web console: Normalize ingestion spec type (#7847)
* Normalize ingestion spec types

* also normalize on direct spec change
2019-06-07 08:32:40 -07:00
Vadim Ogievetsky df01335789 Web console: prevent the time parse view from going into a bad state (#7846)
* prevent the time parse view from going into a bad state

* add test
2019-06-07 08:32:10 -07:00
Vadim Ogievetsky ca5afd29b8 switch the web console to use react pure components (#7845) 2019-06-07 08:31:46 -07:00
Vadim Ogievetsky 0ca867efbf less aggressive default value (#7844) 2019-06-07 08:31:21 -07:00
Clint Wylie ee0d4ea589 add bloom filter fallback aggregator when types are unknown (#7719) 2019-06-06 14:39:32 -07:00
Alexander Saydakov 4dd446bfdd sketches-core-0.13.4 (#7666) 2019-06-06 14:36:52 -07:00
Clint Wylie 12a1ecfc2b allow sql lookup function to take advantage of injective lookups (#7655) 2019-06-06 14:36:10 -07:00
Eugene Sevastyanov 080270283a Druid basic authentication class composition config (#7789)
* Druid basic authentication class composition config.

* Added comments

* Reduced nulls

* Used noop implementations to get rid of null

* Added docs for no-op metadata storage updaters

* Fixed BasicAuthClassCompositionConfig javadoc

* Removed incorrect comments
2019-06-06 15:51:37 +02:00
Gian Merlino 1de1a02e49 Kinesis: Fix getPartitionIds, should be checking isHasMoreShards. (#7830) 2019-06-04 16:26:22 -07:00
Himanshu 0493780799
discard filter when processing subtotalsSpec (#7827) 2019-06-04 10:59:22 -07:00
Xue Yu d482da6e9b fix timestamp ceil lower bound bug (#7823) 2019-06-04 01:16:31 -07:00
Fokko Driesprong f2b00023f8 Bump Checkstyle to 8.21 (#7826) 2019-06-04 01:02:46 -07:00
Nishant Bangarwa fdc03bd336 [druid-kerberos] Fix checking of host URI when reading cookies from cookie store (#7825)
Reading of auth cookie was not checking URI of the server where request was being sent.  This was causing cookie set for one server to be sent to another one and extra authentication round trips between internal druid services.
2019-06-03 19:32:50 -07:00
Vadim Ogievetsky 0de7983db5 Add minor CSS to make long value copy even easier (#7829) 2019-06-03 19:28:57 -07:00
Jihoon Son 61ec521135
Remove keepSegmentGranularity option for compaction (#7747)
* Remove keepSegmentGranularity option from compaction

* fix it test

* clean up

* remove from web console

* fix test
2019-06-03 12:59:15 -07:00
Jihoon Son e289820bbd Add a script to find missing backports (#7817) 2019-06-03 07:56:52 -07:00
Vadim Ogievetsky 9d3dfd168a Web console: Truncate and allow easy copy of large values in query table (#7816)
* table-column-selection -> table-column-selector

* null-table-cell -> table-cell

* make action icon

* made copyable

* updated snapshots

* more tests

* normalizing tests

* tslint fixes

* fix up snapshot desciptions

* nicer edit icons

* remove unused

* better language

* remove unused

* undo unwanted change

* normalize new lines

* truncate long arrays also

* removed double space
2019-06-01 17:37:54 -07:00
Vadim Ogievetsky 4ad058c429 Fix spaces and make linting more strict (#7812) 2019-06-01 10:14:13 -07:00
mcbrewster 21d3d0cde8 Web-console: prevent unnecessary loading of disabled data sources (#7804)
* prevent loading disabled datasources

* prevent double loading

* remove complexity

* rename function

* remove space
2019-06-01 02:08:34 -07:00
litao91 55af692b56 Fix repeated expr parsing in ExpressionPostAggregation (#7791)
* Fix repeatedly expr parsing in ExpressionPostAggregation

Change-Id: Ib739fb1cbc460afeb59a255f635305441dc6997b

* Style fix and avoid code copying

Change-Id: I2d6ba3d1ae37f1fb84b6f7eaab5dab817e1980ec

* Lazilly parse expressions in ExpressionVirtualColumn and ExpressionDimFilter

Change-Id: I5ae2bb3ef9a18fbbfb5e0780c86f6bc0039edc83
2019-05-31 20:56:31 -07:00
Eyal Yurman 69e9b8a464 Enables SQL by default. (#7808) 2019-05-31 20:53:42 -07:00
Justin Borromeo 8032c4add8 Add errors and state to stream supervisor status API endpoint (#7428)
* Add state and error tracking for seekable stream supervisors

* Fixed nits in docs

* Made inner class static and updated spec test with jackson inject

* Review changes

* Remove redundant config param in supervisor

* Style

* Applied some of Jon's recommendations

* Add transience field

* write test

* implement code review changes except for reconsidering logic of markRunFinishedAndEvaluateHealth()

* remove transience reporting and fix SeekableStreamSupervisorStateManager impl

* move call to stateManager.markRunFinished() from RunNotice to runInternal() for tests

* remove stateHistory because it wasn't adding much value, some fixes, and add more tests

* fix tests

* code review changes and add HTTP health check status

* fix test failure

* refactor to split into a generic SupervisorStateManager and a specific SeekableStreamSupervisorStateManager

* fixup after merge

* code review changes - add additional docs

* cleanup KafkaIndexTaskTest

* add additional documentation for Kinesis indexing

* remove unused throws class
2019-05-31 17:16:01 -07:00
Fokko Driesprong c8e1511f12 Bump Joda time to 2.10.2 (#7809) 2019-05-31 14:25:35 -07:00
Clint Wylie 94b3591171 friendly printed arrays for json responses (#7805) 2019-05-31 17:34:12 +08:00
Jihoon Son 7abfbb066a Bump up snapshot version to 0.16.0 (#7802) 2019-05-30 17:17:33 -07:00
Clint Wylie aaefdb3386 fix group-by v2 BufferArrayGrouper for empty multi-value dimension row (#7794)
* fix groupby v2 BufferArrayGrouper

* better name test

* fix sql compatible null handling array grouper bug

* another test
2019-05-30 12:59:59 -07:00
Gian Merlino 58a571ccda
SQL: Use SegmentId instead of DataSegment as set/map keys. (#7796)
Recently we've been talking about using SegmentIds as map keys rather than
DataSegments, because its sense of equality is more well-defined. This is
a refactor that does this in the druid-sql module, which mostly involves
DruidSchema and some related classes. It should have no user-visible effects.
2019-05-30 12:58:36 -07:00
Gian Merlino 8649b8ab4c
SQL: Allow select-sort-project query shapes. (#7769)
* SQL: Allow select-sort-project query shapes.

Fixes #7768.

Design changes:

- In PartialDruidQuery, allow projection after select + sort by removing
  the SELECT_SORT query stage and instead allowing the SORT and
  SORT_PROJECT stages to apply either after aggregation or after a plain
  non-aggregating select. This is different from prior behavior, where
  SORT and SORT_PROJECT were only considered valid after aggregation
  stages. This logic change is in the "canAccept" method.
- In DruidQuery, represent either kind of sorting with a single "Sorting"
  class (instead of DefaultLimitSpec). The Sorting class is still
  convertible into a DefaultLimitSpec, but is also convertible into the
  sorting parameters accepted by a Scan query.
- In DruidQuery, represent post-select and post-sorting projections with
  a single "Projection" class. This obsoletes the SortProject and
  SelectProjection classes, and simplifies the DruidQuery by allowing us
  to move virtual-column and post-aggregator-creation logic into the
  new Projection class.
- Split "DruidQuerySignature" into RowSignature and VirtualColumnRegistry.
  This effectively means that instead of having mutable and immutable
  versions of DruidQuerySignature, we instead of RowSignature (always
  immutable) and VirtualColumnRegistry (always mutable, but sometimes
  null). This change wasn't required, but IMO it this makes the logic
  involving them easier to follow, and makes it more clear when the
  virtual column registry is active and when it's not.

Other changes:

- ConvertBoundsToSelectors now just accepts a RowSignature, but we
  use the VirtualColumnRegistry.getFullRowSignature() method to get
  a signature that includes all columns, and therefore allows us to
  simplify the logic (no need to special-case virtual columns).
- Add `__time` to the Scan column list if the query is ordering by time.

* Remove unused import.
2019-05-30 12:56:29 -07:00
mcbrewster 245eded350 Web-console: Add run keyboard short cut to sql-control (#7787)
* add run keyboard shortcut

* add allow input
2019-05-29 16:11:59 -07:00
Roman Leventov 782863ed0f Fix some problems reported by PVS-Studio (#7738)
* Fix some problems reported by PVS-Studio

* Address comments
2019-05-29 11:20:45 -07:00
Vadim Ogievetsky b051d6688d bump deps (#7788) 2019-05-29 08:50:06 -07:00
Jihoon Son f9528d6476 Add missing protobuf license (#7785) 2019-05-28 18:09:05 -07:00
Vadim Ogievetsky 96b9b3073c Web console: Adding a server view that can display all servers (#7770)
* adding a server view that can see all servers

* fixing action-cell snapshot test

* dynamic resize on sql view

* remove extra word

* update home view server tile also

* offer more context options

* pass context to explain also
2019-05-28 17:12:50 -07:00
Vadim Ogievetsky 6fdc78abbc fix supervisor resume route (#7782) 2019-05-28 13:22:42 -07:00
Xavier Léauté 58a6f0d5d0 Enable compiling against Java 9+ (tests disabled)
This change only enables compilation to ensure code compiles against
recent Java versions going forward. Tests are still disabled in this
profile until test failures are addressed.
2019-05-27 18:40:19 -07:00
Xavier Léauté 484afe0afb exclude hadoop system dependency
ambari depends on hadoop-annotation. Under linux, it sometimes activates a
maven profile that includes an explicit dependency on ${java.home}/../lib/tools.jar

This change excludes this dependency, since this jar is no longer
included with Java 9. It also seems seems unlikely that we would require
this dependency at runtime, since it is only enabled on some platforms.
2019-05-27 18:40:19 -07:00
Fokko Driesprong e46bdf082e Remove Codehaus references from the tests (#7773) 2019-05-27 10:51:14 -07:00
BIGrey 42cf078843 Fix memory problem (OOM/FGC) when expression is used in metricsSpec (#7716)
* AggregatorUtil should cache parsed expression to avoid memory problem (OOM/FGC) when Expression is used in metricsSpec

* remove debug log check in Parser.parse

* remove cache and use suppliers.memorize
2019-05-27 09:46:17 -07:00