9673 Commits

Author SHA1 Message Date
Xavier Léauté
8e0c307e54
Do not assume system classloader is URLClassLoader in Java 9+ (#8392)
* Fallback to parsing classpath for hadoop task in Java 9+
In Java 9 and above we cannot assume that the system classloader is an
instance of URLClassLoader. This change adds a fallback method to parse
the system classpath in that case, and adds a unit test to validate it matches
what JDK8 would do.

Note: This has not been tested in an actual hadoop setup, so this is mostly
to help us pass unit tests.

* Remove granularity test of dubious value
One of our granularity tests relies on system classloader being a URLClassLoaders to
catch a bug related to class initialization and static initializers using a subclass (see
#2979)
This test was added to catch a potential regression, but it assumes we would add back
the same type of static initializers to this specific class, so it seems to be of dubious value
as a unit test and mostly serves to illustrate the bug.

relates to #5589
2019-08-24 20:47:54 -04:00
Xavier Léauté
20f7db5d22
Fix ConcurrentModificationException in JDK11 (#8391)
When building column/dimension selectors, calling computeIfAbsent can
cause the applied function to modify the same cache through virtual
column references. The JDK11 map implementation detects this change and
will throw an exception.

This fix – while not as elegant – breaks the single call into two
steps to avoid this problem.
2019-08-24 18:24:50 -04:00
Xavier Léauté
5c7803fe6b fix powermock classloader issues with Java 9 and above 2019-08-24 18:20:52 -04:00
Vadim Ogievetsky
c948422f32 Web console: Better build experience (#8389)
* rename sql docs

* rm old file

* add old files
2019-08-24 11:35:30 -07:00
Vadim Ogievetsky
efbf1a5822 Restore old console (#8387) 2019-08-23 17:48:11 -07:00
Jihoon Son
95fa609615 Fix wrong partitionsSpec type names in the document (#8297)
* Fix wrong type names for partitionsSpec

* add unit tests; add json properties for backward compatibility

* beautify conf names

* remove maxRowsPerSegment from hashed partitionsSpec

* fix doc build
2019-08-23 13:44:58 -07:00
Clint Wylie
7749571a7f order and add more ports to hadoop docker container in hadoop indexing tutorial (#8329)
LGTM
2019-08-23 15:43:06 -05:00
Vadim Ogievetsky
20ea90a5a6 Web console: more explicit limit on run button (#8378)
* update sql doc parsing

* keyword fixes

* fix header default

* tidy

* fix tests
2019-08-23 13:40:48 -07:00
Jonathan Wei
368ace4e87
Fix ClassCastException for TopN with long-type dimension (#8349)
* Fix ClassCastException for TopN with long-type dimension

* Add DimValHolderTest
2019-08-23 14:55:31 -05:00
Chi Cao Minh
2383d9e522 Disable coveralls (#8382)
The coveralls code coverage reports inaccurate coverage for our parallel
builds. Disable it until it can be fixed or a better alternative can be
found.
2019-08-23 08:05:37 -07:00
SandishKumarHN
33f0753a70 Add Checkstyle for constant name static final (#8060)
* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* merging with upstream

* review-1

* unknow changes

* unknow changes

* review-2

* merging with master

* review-2 1 changes

* review changes-2 2

* bug fix
2019-08-23 13:13:54 +03:00
David Lim
d117bfb149 Handle exception thrown in log while trying to call sun.misc.VM.maxDirectMemory() which is not available in Java 11 (#8352)
* handle exception thrown while trying to call sun.misc.VM.maxDirectMemory() which is not available in Java 11

* fixup String.format -> StringUtils.format
2019-08-23 01:22:41 -07:00
Atul Mohan
661976f266 Reset sketch combiner in AggregatorCombiner (#8368)
* Reset union in AggregateCombiner

* Use newer sketch objects for test

* Add empty sketch objects
2019-08-23 00:22:40 -07:00
mcbrewster
bfbae76031 Web-console: small fixes (#8376)
* fix column tree

* move remove filter

* fix lgtm

* remove from state

* stop mutating state diirectly

* fix column-tree

* remove component did mount

* remove expanded node
2019-08-22 21:54:11 -07:00
Surekha
cf2a2dd917
Add group_id to the sys.tasks table (#8304)
* Add group_id to overlord tasks API and sys.tasks table

* adjust test

* modify docs

* Make groupId nullable

* fix integration test

* fix toString

* Remove groupId from TaskInfo

* Modify docs and tests

* modify TaskMonitorTest
2019-08-22 15:28:23 -07:00
Jihoon Son
fba92ae469 Fix to always use end sequenceNumber for reset (#8305)
* Fix to always use end sequenceNumber for reset

* fix checkstyle

* fix style and add log
2019-08-22 16:51:25 -05:00
Clint Wylie
71676cd16d add checkbox for licenses.yaml in PR template, mention it in CONTRIBUTING.md (#8367) 2019-08-22 14:14:24 -07:00
Clint Wylie
e3e85e8f1a use alternative public domain text instead of lorem ipsum text in compression test (#8299)
* use apache license text instead of lorem ipsum text to be unambiguous with regards to licensing

* feed your head
2019-08-22 12:53:38 -07:00
Chi Cao Minh
64f3c6588b Add coveralls parallel builds webhook (#8374)
Since we use parallel builds for unit tests, follow the instructions for
properly merging the results on coveralls:
https://docs.coveralls.io/parallel-build-webhook
2019-08-22 12:50:00 -07:00
Jonathan Wei
96e2142ea3 Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager (#8287)
* Cleanup Appenderators in UnifiedIndexerAppenderatorsManager

* PR comments

* More PR comments

* Fix test
2019-08-22 12:18:46 -07:00
Alexandre Yang
167a0e7ae1 [opentsdb-emitter] Fix ingest persists metrics typo (#8233) 2019-08-22 10:58:19 -07:00
Vadim Ogievetsky
441616919b Put readme in a good order (#8372) 2019-08-22 09:50:55 -07:00
mcbrewster
5274c5ab73 Web-Console: Replace from clause (#8371)
* add replace from

* add failture state

* fix uninential downgrade

* use uppercase for SQL keywords
2019-08-22 09:09:37 -07:00
Clint Wylie
b95607d31c
remove YeOldePlumberSchool.java, unused (#8347) 2019-08-21 18:15:51 -07:00
Aaron Bossert
a4d1219184 Removed hard-coded Kafka Deserializer in Web-UI Kafka data import such that users can supply a custom deserializer in the UI as well as in hand-built ingestion specs. (#8364) 2019-08-21 18:03:57 -07:00
Vadim Ogievetsky
aa51668ea7 Web console: LGTM alert squash + final fixes (#8351)
* mv old console

* update dep

* reduce card size

* fix tuning configs for forceGuaranteedRollup

* fix tests

* revert console mv

* fix some old console issues

* revert build change
2019-08-21 13:52:48 -07:00
Benedict Jin
e1f94a5e26 Reduce the size of images with lossless compression (#8358) 2019-08-21 13:29:30 -07:00
Clint Wylie
010f70b371
autogenerate NOTICE.BINARY from NOTICE and licenses.yaml (#8306)
* migrate binary notice entries to live in licenses.yaml, use licenses.yaml and NOTICE to generate NOTICE.BINARY at distribution time

* +x

* move release scripts to distribution/bin, fixup notice script, trim dependencies for avro and kerberos in licenses.yaml

* add missing hdfs-storage dependencies

* revert to old syntax, fixes

* formatting

* update notices for recently updated dependencies
2019-08-21 12:46:27 -07:00
Jihoon Son
22d6384d36
Fix unrealistic test variables in KafkaSupervisorTest and tidy up unused variable in checkpointing process (#7319)
* Fix unrealistic test arguments in KafkaSupervisorTest

* remove currentCheckpoint from checkpoint action

* rename variable
2019-08-21 10:58:22 -07:00
Gian Merlino
d007477742
Docusaurus build framework + ingestion doc refresh. (#8311)
* Docusaurus build framework + ingestion doc refresh.

* stick to npm instead of yarn

* fix typos

* restore some _bin

* Adjustments.

* detect and fix redirect anchors

* update anchor lint

* Web-console: remove specific column filters (#8343)

* add clear filter

* update tool kit

* remove usless check

* auto run

* add %

* Fix resource leak (#8337)

* Fix resource leak

* Patch comments

* Enable Spotbugs NP_NONNULL_RETURN_VIOLATION (#8234)

* Fixes from PR review.

* Fix more anchors.

* Preamble nix.

* Fix more anchors, headers

* clean up placeholder page

* add to website lint to travis config

* better broken link checking

* travis fix

* Fixed more broken links

* better redirects

* unfancy catch

* fix LGTM error

* link fixes

* fix md issues

* Addl fixes
2019-08-20 21:48:59 -07:00
Benedict Jin
c4db83608f
Add grade shield (#8344) 2019-08-21 12:20:19 +08:00
mcbrewster
e4aa7fb268 Web-console: fix alerts from lgtm (#8346)
* fix alerts from lgtm

* remove unordered imports

* fix introduced alert

* move getExpanded to renderSchemaSelector

* use getDerivedStateFromProps

* use prevState

* add semi colons, remove unused imports

* fixes
2019-08-20 20:40:47 -07:00
Clint Wylie
c87b68d2a4 use Number instead of long for response context (#8342)
* use Number instead of long for response context to be forgiving of json serde to int or long

* test that encounters issue without fix

* now with more test

* is ints
2019-08-20 19:05:49 -07:00
Fokko Driesprong
d5a19675dd Remove fromPigAvroStorage from the docs (#8340)
This one has been deprecated a while ago
2019-08-20 16:34:55 -07:00
Benedict Jin
14a4238381 Bump JUnitParams from 1.0.4 to 1.1.1 (#8017) 2019-08-20 16:15:12 -07:00
Fokko Driesprong
8821ac330d Bump opencsv from 4.2 to 4.6 (#8294)
* Bump opencsv from 4.2 to 4.6

* Fix transitive dependencies
2019-08-20 16:12:03 -07:00
Fokko Driesprong
3a58431bff Bump jackson-jq from 0.0.7 to 0.0.10 (#8293)
* Bump jackson-jq from 0.0.7 to 0.0.10

For the changelog:
https://github.com/eiiches/jackson-jq/releases

* Update dependent licenses
2019-08-20 16:09:04 -07:00
Chi Cao Minh
6fa22f6939 Enable code coverage (#8303)
* Enable code coverage

Code coverage was disabled via
https://github.com/apache/incubator-druid/pull/3122 due to an issue with
cobertura in Travis CI. Switch code coverage tool from cobertura to
jacoco to avoid issue and re-enable coveralls for Travis CI.

* Exclude non-production code

* Exclude benchmark generated code

* Exclude DruidTestRunnerFactory
2019-08-20 15:36:19 -07:00
Fokko Driesprong
cb1339e19a Bump derby from 10.11.1.1 to 10.14.2.0 (#8292)
* Bump derby from 10.11.1.1 to 10.15.1.3

* Update server/pom.xml as well

* Move to derby 10.14.2.0

10.15.* is Java9+
https://db.apache.org/derby/derby_downloads.html
2019-08-20 14:03:32 -07:00
Jonathan Wei
e2a25fb51e
Add logging for LZ4Factory instance type (#8341) 2019-08-20 15:24:53 -05:00
Benedict Jin
5e35500e0e Fix unused format argument (#8345) 2019-08-20 12:45:46 -07:00
pphust
ffcbd1ecb8 Ensure ReferenceCountingSegment.decrement() is invoked correctly (#8323)
* fix issue8291. Make sure ReferenceCountingSegment.decrement() is invoked correctly

* add some comments in SinkQuerySegmentWalker.java

* extracting perSegmentRunners and perHydrantRunners to reduce the level of nesting
2019-08-20 22:01:16 +03:00
Fokko Driesprong
818bf4990c Enable Spotbugs NP_NONNULL_RETURN_VIOLATION (#8234) 2019-08-20 17:23:46 +03:00
Benedict Jin
781873ba53 Fix resource leak (#8337)
* Fix resource leak

* Patch comments
2019-08-20 12:55:41 +03:00
mcbrewster
e64070b8bf Web-console: remove specific column filters (#8343)
* add clear filter

* update tool kit

* remove usless check

* auto run

* add %
2019-08-19 19:54:18 -07:00
Jonathan Wei
dd2e53baf4
Clarify Avro decoder docs (#8302) 2019-08-19 15:37:18 -05:00
Vadim Ogievetsky
56e440383f Web console: data loader native batch shuffle UI (#8229)
* data loader shuffle UI

* better support for inline firehose

* feedback changes

* make numShards mandatory sometimes

* edit context dialog fixes

* make auto form less agro

* download full log

* improve auto column detection

* add sample for example
2019-08-19 13:27:34 -07:00
Benedict Jin
566dc8c719
Fix missing format argument (#8331) 2019-08-19 16:19:44 +08:00
mcbrewster
cae1361c32 Web-console: gate auto complete on current table and schema (#8322)
* gate auto complete on current table and schema

* reset defaults

* add static functions

* move completions to state
2019-08-17 09:34:36 -07:00
Jihoon Son
31af4eb9ad
Rename maxNumSubTasks to maxNumConcurrentSubTasks for native parallel index task (#8324) 2019-08-16 15:57:13 -07:00