Commit Graph

9393 Commits

Author SHA1 Message Date
Surekha ea752ef562 Optimize overshadowed segments computation (#7595)
* Move the overshadowed segment computation to SQLMetadataSegmentManager's poll

* rename method in MetadataSegmentManager

* Fix tests

* PR comments

* PR comments

* PR comments

* fix indentation

* fix tests

*  fix test

*  add test for SegmentWithOvershadowedStatus serde format

* PR comments

* PR comments

* fix test

* remove snapshot updates outside poll

* PR comments

* PR comments

* PR comments

*  removed unused import
2019-06-07 19:15:54 +02:00
Vadim Ogievetsky 061b465d34 Web console: Normalize ingestion spec type (#7847)
* Normalize ingestion spec types

* also normalize on direct spec change
2019-06-07 08:32:40 -07:00
Vadim Ogievetsky df01335789 Web console: prevent the time parse view from going into a bad state (#7846)
* prevent the time parse view from going into a bad state

* add test
2019-06-07 08:32:10 -07:00
Vadim Ogievetsky ca5afd29b8 switch the web console to use react pure components (#7845) 2019-06-07 08:31:46 -07:00
Vadim Ogievetsky 0ca867efbf less aggressive default value (#7844) 2019-06-07 08:31:21 -07:00
Clint Wylie ee0d4ea589 add bloom filter fallback aggregator when types are unknown (#7719) 2019-06-06 14:39:32 -07:00
Alexander Saydakov 4dd446bfdd sketches-core-0.13.4 (#7666) 2019-06-06 14:36:52 -07:00
Clint Wylie 12a1ecfc2b allow sql lookup function to take advantage of injective lookups (#7655) 2019-06-06 14:36:10 -07:00
Eugene Sevastyanov 080270283a Druid basic authentication class composition config (#7789)
* Druid basic authentication class composition config.

* Added comments

* Reduced nulls

* Used noop implementations to get rid of null

* Added docs for no-op metadata storage updaters

* Fixed BasicAuthClassCompositionConfig javadoc

* Removed incorrect comments
2019-06-06 15:51:37 +02:00
Gian Merlino 1de1a02e49 Kinesis: Fix getPartitionIds, should be checking isHasMoreShards. (#7830) 2019-06-04 16:26:22 -07:00
Himanshu 0493780799
discard filter when processing subtotalsSpec (#7827) 2019-06-04 10:59:22 -07:00
Xue Yu d482da6e9b fix timestamp ceil lower bound bug (#7823) 2019-06-04 01:16:31 -07:00
Fokko Driesprong f2b00023f8 Bump Checkstyle to 8.21 (#7826) 2019-06-04 01:02:46 -07:00
Nishant Bangarwa fdc03bd336 [druid-kerberos] Fix checking of host URI when reading cookies from cookie store (#7825)
Reading of auth cookie was not checking URI of the server where request was being sent.  This was causing cookie set for one server to be sent to another one and extra authentication round trips between internal druid services.
2019-06-03 19:32:50 -07:00
Vadim Ogievetsky 0de7983db5 Add minor CSS to make long value copy even easier (#7829) 2019-06-03 19:28:57 -07:00
Jihoon Son 61ec521135
Remove keepSegmentGranularity option for compaction (#7747)
* Remove keepSegmentGranularity option from compaction

* fix it test

* clean up

* remove from web console

* fix test
2019-06-03 12:59:15 -07:00
Jihoon Son e289820bbd Add a script to find missing backports (#7817) 2019-06-03 07:56:52 -07:00
Vadim Ogievetsky 9d3dfd168a Web console: Truncate and allow easy copy of large values in query table (#7816)
* table-column-selection -> table-column-selector

* null-table-cell -> table-cell

* make action icon

* made copyable

* updated snapshots

* more tests

* normalizing tests

* tslint fixes

* fix up snapshot desciptions

* nicer edit icons

* remove unused

* better language

* remove unused

* undo unwanted change

* normalize new lines

* truncate long arrays also

* removed double space
2019-06-01 17:37:54 -07:00
Vadim Ogievetsky 4ad058c429 Fix spaces and make linting more strict (#7812) 2019-06-01 10:14:13 -07:00
mcbrewster 21d3d0cde8 Web-console: prevent unnecessary loading of disabled data sources (#7804)
* prevent loading disabled datasources

* prevent double loading

* remove complexity

* rename function

* remove space
2019-06-01 02:08:34 -07:00
litao91 55af692b56 Fix repeated expr parsing in ExpressionPostAggregation (#7791)
* Fix repeatedly expr parsing in ExpressionPostAggregation

Change-Id: Ib739fb1cbc460afeb59a255f635305441dc6997b

* Style fix and avoid code copying

Change-Id: I2d6ba3d1ae37f1fb84b6f7eaab5dab817e1980ec

* Lazilly parse expressions in ExpressionVirtualColumn and ExpressionDimFilter

Change-Id: I5ae2bb3ef9a18fbbfb5e0780c86f6bc0039edc83
2019-05-31 20:56:31 -07:00
Eyal Yurman 69e9b8a464 Enables SQL by default. (#7808) 2019-05-31 20:53:42 -07:00
Justin Borromeo 8032c4add8 Add errors and state to stream supervisor status API endpoint (#7428)
* Add state and error tracking for seekable stream supervisors

* Fixed nits in docs

* Made inner class static and updated spec test with jackson inject

* Review changes

* Remove redundant config param in supervisor

* Style

* Applied some of Jon's recommendations

* Add transience field

* write test

* implement code review changes except for reconsidering logic of markRunFinishedAndEvaluateHealth()

* remove transience reporting and fix SeekableStreamSupervisorStateManager impl

* move call to stateManager.markRunFinished() from RunNotice to runInternal() for tests

* remove stateHistory because it wasn't adding much value, some fixes, and add more tests

* fix tests

* code review changes and add HTTP health check status

* fix test failure

* refactor to split into a generic SupervisorStateManager and a specific SeekableStreamSupervisorStateManager

* fixup after merge

* code review changes - add additional docs

* cleanup KafkaIndexTaskTest

* add additional documentation for Kinesis indexing

* remove unused throws class
2019-05-31 17:16:01 -07:00
Fokko Driesprong c8e1511f12 Bump Joda time to 2.10.2 (#7809) 2019-05-31 14:25:35 -07:00
Clint Wylie 94b3591171 friendly printed arrays for json responses (#7805) 2019-05-31 17:34:12 +08:00
Jihoon Son 7abfbb066a Bump up snapshot version to 0.16.0 (#7802) 2019-05-30 17:17:33 -07:00
Clint Wylie aaefdb3386 fix group-by v2 BufferArrayGrouper for empty multi-value dimension row (#7794)
* fix groupby v2 BufferArrayGrouper

* better name test

* fix sql compatible null handling array grouper bug

* another test
2019-05-30 12:59:59 -07:00
Gian Merlino 58a571ccda
SQL: Use SegmentId instead of DataSegment as set/map keys. (#7796)
Recently we've been talking about using SegmentIds as map keys rather than
DataSegments, because its sense of equality is more well-defined. This is
a refactor that does this in the druid-sql module, which mostly involves
DruidSchema and some related classes. It should have no user-visible effects.
2019-05-30 12:58:36 -07:00
Gian Merlino 8649b8ab4c
SQL: Allow select-sort-project query shapes. (#7769)
* SQL: Allow select-sort-project query shapes.

Fixes #7768.

Design changes:

- In PartialDruidQuery, allow projection after select + sort by removing
  the SELECT_SORT query stage and instead allowing the SORT and
  SORT_PROJECT stages to apply either after aggregation or after a plain
  non-aggregating select. This is different from prior behavior, where
  SORT and SORT_PROJECT were only considered valid after aggregation
  stages. This logic change is in the "canAccept" method.
- In DruidQuery, represent either kind of sorting with a single "Sorting"
  class (instead of DefaultLimitSpec). The Sorting class is still
  convertible into a DefaultLimitSpec, but is also convertible into the
  sorting parameters accepted by a Scan query.
- In DruidQuery, represent post-select and post-sorting projections with
  a single "Projection" class. This obsoletes the SortProject and
  SelectProjection classes, and simplifies the DruidQuery by allowing us
  to move virtual-column and post-aggregator-creation logic into the
  new Projection class.
- Split "DruidQuerySignature" into RowSignature and VirtualColumnRegistry.
  This effectively means that instead of having mutable and immutable
  versions of DruidQuerySignature, we instead of RowSignature (always
  immutable) and VirtualColumnRegistry (always mutable, but sometimes
  null). This change wasn't required, but IMO it this makes the logic
  involving them easier to follow, and makes it more clear when the
  virtual column registry is active and when it's not.

Other changes:

- ConvertBoundsToSelectors now just accepts a RowSignature, but we
  use the VirtualColumnRegistry.getFullRowSignature() method to get
  a signature that includes all columns, and therefore allows us to
  simplify the logic (no need to special-case virtual columns).
- Add `__time` to the Scan column list if the query is ordering by time.

* Remove unused import.
2019-05-30 12:56:29 -07:00
mcbrewster 245eded350 Web-console: Add run keyboard short cut to sql-control (#7787)
* add run keyboard shortcut

* add allow input
2019-05-29 16:11:59 -07:00
Roman Leventov 782863ed0f Fix some problems reported by PVS-Studio (#7738)
* Fix some problems reported by PVS-Studio

* Address comments
2019-05-29 11:20:45 -07:00
Vadim Ogievetsky b051d6688d bump deps (#7788) 2019-05-29 08:50:06 -07:00
Jihoon Son f9528d6476 Add missing protobuf license (#7785) 2019-05-28 18:09:05 -07:00
Vadim Ogievetsky 96b9b3073c Web console: Adding a server view that can display all servers (#7770)
* adding a server view that can see all servers

* fixing action-cell snapshot test

* dynamic resize on sql view

* remove extra word

* update home view server tile also

* offer more context options

* pass context to explain also
2019-05-28 17:12:50 -07:00
Vadim Ogievetsky 6fdc78abbc fix supervisor resume route (#7782) 2019-05-28 13:22:42 -07:00
Xavier Léauté 58a6f0d5d0 Enable compiling against Java 9+ (tests disabled)
This change only enables compilation to ensure code compiles against
recent Java versions going forward. Tests are still disabled in this
profile until test failures are addressed.
2019-05-27 18:40:19 -07:00
Xavier Léauté 484afe0afb exclude hadoop system dependency
ambari depends on hadoop-annotation. Under linux, it sometimes activates a
maven profile that includes an explicit dependency on ${java.home}/../lib/tools.jar

This change excludes this dependency, since this jar is no longer
included with Java 9. It also seems seems unlikely that we would require
this dependency at runtime, since it is only enabled on some platforms.
2019-05-27 18:40:19 -07:00
Fokko Driesprong e46bdf082e Remove Codehaus references from the tests (#7773) 2019-05-27 10:51:14 -07:00
BIGrey 42cf078843 Fix memory problem (OOM/FGC) when expression is used in metricsSpec (#7716)
* AggregatorUtil should cache parsed expression to avoid memory problem (OOM/FGC) when Expression is used in metricsSpec

* remove debug log check in Parser.parse

* remove cache and use suppliers.memorize
2019-05-27 09:46:17 -07:00
Gian Merlino cf09fbf4ce
Fix LookupSerdeModule double-binding in Coordinator-as-Overlord mode. (#7765)
Fixes #7762.
2019-05-26 15:14:50 -07:00
Jonathan Wei c97eb93e16 Fix dead code in IndexTask.collectIntervalsAndShardSpecs() (#7746) 2019-05-25 10:12:14 -07:00
mcbrewster 7eac685e47 Web-console: add hostname to tasks view (#7758)
* add host name

* change to location

* remove space
2019-05-24 23:18:18 -07:00
Himanshu 2b7bb064b5 remove unused ObjectMapper from DatasourcePathSpec (#7754) 2019-05-24 23:15:40 -07:00
mcbrewster 3a77a3e112 Web-Console: add actions icon to lookups (#7725)
* add lookup-table-action-dialog

* reorder arguments

* remove lookup-table-action-dialog

* fix supervisors table action dialog

* fix imports

* Add host column to task view

* reorder buttons

* remove log

* update snapshots

* remove host

* update snapshots

* remove console log

* remove fragment

* update snapshots and remove space
2019-05-24 22:53:12 -07:00
mcbrewster 5f50f357a4 Web-Console: add go to editor button to tasks and supervisors view (#7705)
* add go to editor button to tasks and supervisors

* fix package.json

* remove intent

* quuick fixes

* fixes

* add getsupervisorjson and gettaskjson

* remove space

* remove gotoloaddata

* fixes

* add error handling

* save

* add loader

* remove initspec

* fixup! add loader

* update snapshots

* remove gotoloaddataview form headerbar.spec
2019-05-24 22:52:26 -07:00
mcbrewster 2b34e1b710 Web console: add log tailing to task log view (#7703)
* add tail dialog checkbox to tasks show-log-component for webconsole

* add check box button css

* add 2 second delay

* left align

* change tail formatting

* use async/await syntax

* add space

* add public prefix

* update jest to reflect changes in show-log

* use sentence case

* update snapshots
2019-05-24 15:34:29 -07:00
mcbrewster d7e3fa510e Web-console: add resizable split screen layout to tasks and servers views (#7723)
* add split scroll layout

* fix dev dependancies

* clean up

* adjust margin

* update jest to reflect ui changes

* fix css

* fix css issues, update npm

* remove scrollable
2019-05-24 15:01:45 -07:00
Jihoon Son 2bf44eb0b4 Missing licenses and notices for source distribution (#7760)
* Add missing licenses and notices

* address comment
2019-05-24 14:47:41 -07:00
Jonathan Wei 83152a7a00 Fix performance-faq and remove insert-segment-to-db redirects (#7759) 2019-05-24 13:20:02 -07:00
Gian Merlino 7ec7257e1d Fix lookup serde on node types that don't load lookups. (#7752)
This includes the router, overlord, middleManager, and coordinator.
Does the following things:

- Loads LookupSerdeModule on MM, overlord, and coordinator.
- Adds LookupExprMacro to LookupSerdeModule, which allows these node
  types to understand that the 'lookup' function exists.
- Adds a test to make sure that LookupSerdeModule works for virtual
  columns, filters, transforms, and dimension specs.

This is implementing the technique discussed on these two issues:

- https://github.com/apache/incubator-druid/issues/7724#issuecomment-494723333
- https://github.com/apache/incubator-druid/pull/7082#discussion_r264888771
2019-05-24 12:30:49 -07:00