Commit Graph

9987 Commits

Author SHA1 Message Date
Himanshu d5d170f866
skip unnecessary aggregate(..) calls with LimitedBufferHashGrouper (#8412)
* skip unnecessary aggregate(..) calls with LimitedBufferHashGrouper

* remove unused bucketWasUsed arg from canSkipAggregate(..)
2019-08-27 15:01:07 -07:00
Himanshu 4d87a19547
Logging emitter to publish query and other metric events as valid json objects (#8359)
* LoggingEmitter: print event as json

* use DefaultRequestLogEventBuilderFactory in emitting request logger by default

* print context in query metric as json

* removed unused jsonMapper from DefaultQueryMetrics

* add comment

* remove change to DefaultRequestLogEventBuilderFactory.java
2019-08-27 15:00:23 -07:00
Clint Wylie 44dd5b5f0d add jaxb-runtime to fix exception with newer versions of java (#8409)
* add jaxb-runtime to fix exception with jdk9+

* fix licenses

* oops
2019-08-27 14:25:05 -06:00
Chi Cao Minh d8b81f4fd9 Speed up javascript Travis CI jobs (#8361)
* Speed up javascript Travis CI jobs

Skip mvn install for javascript CI jobs since it is not needed.

* Specify base filepath for source files

* Remove coveralls
2019-08-27 12:03:48 -07:00
Jihoon Son e5ef5ddafa Fix the shuffle with TLS enabled for parallel indexing; add an integration test; improve unit tests (#8350)
* Fix shuffle with tls enabled; add an integration test; improve unit tests

* remove debug log

* fix tests

* unused import

* add javadoc

* rename to getContent
2019-08-26 19:27:41 -07:00
Benedict Jin 95f727f66c
Add docker shield (#8403) 2019-08-27 10:04:51 +08:00
Chi Cao Minh a4b842ac2e Speed up ExpressionSelectors.makeExprEvalSelector (#8373)
* Speed up ExpressionSelectors.makeExprEvalSelector

Addresses performance bottlenecks observed when running a search query
with an expression filter and granularity set to none by caching and
changing streams to for-loops. After changes, the query was observed to
run 2-3x faster.

Also fixes various IntelliJ inspection warnings.

* Fix static analysis errors
2019-08-26 14:34:16 -07:00
Dylan Wylie b2821a8371 do not exclude client core jar (#8339)
make indexing service depend on hadoop client
2019-08-26 13:48:24 -07:00
Vadim Ogievetsky 894b4efb36 Web console: Fix segment timeline styling (#8402)
* Fix segment timeline styling

* update snapshot
2019-08-25 21:31:20 -07:00
Vadim Ogievetsky 9254dc8b80 Web console: Save query context also (#8395)
* tidy up menus

* fix query output API

* save context also

* pull out auto run into a switch

* better copy

* add group_id

* support FLOAT also

* use built in time logic

* fix trunc direction

* add skipCache

* add manifest url

* remove depricated props
2019-08-25 16:56:27 -07:00
Xavier Léauté 496dfa3b15
fix JVMMonitor initialization with JDK11 (#8397)
JVMMonitor requires access to jdk.internal.perf.Perf to enable GC
counters, which requires additional JVM arguments with JDK11.

This change adds a fallback in case GC counters cannot be initialized,
and logs a warning message explaining how GC counters can be enabled.
2019-08-25 08:26:58 -04:00
Jonathan Wei c626452b47 Add nano-quickstart single server example configuration (#8390)
* Add nano-quickstart single server example configuration

* Use two workers

* Shrink processing buffers
2019-08-24 22:07:20 -07:00
Furkan KAMACI 02fe3db911 Zookeeper version is updated. (#8363)
* Zookeeper version is updated.

* Zookeeper version is updated at licenses.yaml

* licenses.yaml is updated and dependencies are fixed to make the project successfully build.

* Zookeeper versions are fixed at licenses.yaml
2019-08-24 22:00:43 -07:00
Xavier Léauté d5e3c53e74 workaround for Guava 16 bug using Java 9 and above 2019-08-25 00:58:59 -04:00
Xavier Léauté 8e0c307e54
Do not assume system classloader is URLClassLoader in Java 9+ (#8392)
* Fallback to parsing classpath for hadoop task in Java 9+
In Java 9 and above we cannot assume that the system classloader is an
instance of URLClassLoader. This change adds a fallback method to parse
the system classpath in that case, and adds a unit test to validate it matches
what JDK8 would do.

Note: This has not been tested in an actual hadoop setup, so this is mostly
to help us pass unit tests.

* Remove granularity test of dubious value
One of our granularity tests relies on system classloader being a URLClassLoaders to
catch a bug related to class initialization and static initializers using a subclass (see
#2979)
This test was added to catch a potential regression, but it assumes we would add back
the same type of static initializers to this specific class, so it seems to be of dubious value
as a unit test and mostly serves to illustrate the bug.

relates to #5589
2019-08-24 20:47:54 -04:00
Xavier Léauté 20f7db5d22
Fix ConcurrentModificationException in JDK11 (#8391)
When building column/dimension selectors, calling computeIfAbsent can
cause the applied function to modify the same cache through virtual
column references. The JDK11 map implementation detects this change and
will throw an exception.

This fix – while not as elegant – breaks the single call into two
steps to avoid this problem.
2019-08-24 18:24:50 -04:00
Xavier Léauté 5c7803fe6b fix powermock classloader issues with Java 9 and above 2019-08-24 18:20:52 -04:00
Vadim Ogievetsky c948422f32 Web console: Better build experience (#8389)
* rename sql docs

* rm old file

* add old files
2019-08-24 11:35:30 -07:00
Vadim Ogievetsky efbf1a5822 Restore old console (#8387) 2019-08-23 17:48:11 -07:00
Jihoon Son 95fa609615 Fix wrong partitionsSpec type names in the document (#8297)
* Fix wrong type names for partitionsSpec

* add unit tests; add json properties for backward compatibility

* beautify conf names

* remove maxRowsPerSegment from hashed partitionsSpec

* fix doc build
2019-08-23 13:44:58 -07:00
Clint Wylie 7749571a7f order and add more ports to hadoop docker container in hadoop indexing tutorial (#8329)
LGTM
2019-08-23 15:43:06 -05:00
Vadim Ogievetsky 20ea90a5a6 Web console: more explicit limit on run button (#8378)
* update sql doc parsing

* keyword fixes

* fix header default

* tidy

* fix tests
2019-08-23 13:40:48 -07:00
Jonathan Wei 368ace4e87
Fix ClassCastException for TopN with long-type dimension (#8349)
* Fix ClassCastException for TopN with long-type dimension

* Add DimValHolderTest
2019-08-23 14:55:31 -05:00
Chi Cao Minh 2383d9e522 Disable coveralls (#8382)
The coveralls code coverage reports inaccurate coverage for our parallel
builds. Disable it until it can be fixed or a better alternative can be
found.
2019-08-23 08:05:37 -07:00
SandishKumarHN 33f0753a70 Add Checkstyle for constant name static final (#8060)
* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* check ctyle for constant field name

* merging with upstream

* review-1

* unknow changes

* unknow changes

* review-2

* merging with master

* review-2 1 changes

* review changes-2 2

* bug fix
2019-08-23 13:13:54 +03:00
David Lim d117bfb149 Handle exception thrown in log while trying to call sun.misc.VM.maxDirectMemory() which is not available in Java 11 (#8352)
* handle exception thrown while trying to call sun.misc.VM.maxDirectMemory() which is not available in Java 11

* fixup String.format -> StringUtils.format
2019-08-23 01:22:41 -07:00
Atul Mohan 661976f266 Reset sketch combiner in AggregatorCombiner (#8368)
* Reset union in AggregateCombiner

* Use newer sketch objects for test

* Add empty sketch objects
2019-08-23 00:22:40 -07:00
mcbrewster bfbae76031 Web-console: small fixes (#8376)
* fix column tree

* move remove filter

* fix lgtm

* remove from state

* stop mutating state diirectly

* fix column-tree

* remove component did mount

* remove expanded node
2019-08-22 21:54:11 -07:00
Surekha cf2a2dd917
Add group_id to the sys.tasks table (#8304)
* Add group_id to overlord tasks API and sys.tasks table

* adjust test

* modify docs

* Make groupId nullable

* fix integration test

* fix toString

* Remove groupId from TaskInfo

* Modify docs and tests

* modify TaskMonitorTest
2019-08-22 15:28:23 -07:00
Jihoon Son fba92ae469 Fix to always use end sequenceNumber for reset (#8305)
* Fix to always use end sequenceNumber for reset

* fix checkstyle

* fix style and add log
2019-08-22 16:51:25 -05:00
Clint Wylie 71676cd16d add checkbox for licenses.yaml in PR template, mention it in CONTRIBUTING.md (#8367) 2019-08-22 14:14:24 -07:00
Clint Wylie e3e85e8f1a use alternative public domain text instead of lorem ipsum text in compression test (#8299)
* use apache license text instead of lorem ipsum text to be unambiguous with regards to licensing

* feed your head
2019-08-22 12:53:38 -07:00
Chi Cao Minh 64f3c6588b Add coveralls parallel builds webhook (#8374)
Since we use parallel builds for unit tests, follow the instructions for
properly merging the results on coveralls:
https://docs.coveralls.io/parallel-build-webhook
2019-08-22 12:50:00 -07:00
Jonathan Wei 96e2142ea3 Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager (#8287)
* Cleanup Appenderators in UnifiedIndexerAppenderatorsManager

* PR comments

* More PR comments

* Fix test
2019-08-22 12:18:46 -07:00
Alexandre Yang 167a0e7ae1 [opentsdb-emitter] Fix ingest persists metrics typo (#8233) 2019-08-22 10:58:19 -07:00
Vadim Ogievetsky 441616919b Put readme in a good order (#8372) 2019-08-22 09:50:55 -07:00
mcbrewster 5274c5ab73 Web-Console: Replace from clause (#8371)
* add replace from

* add failture state

* fix uninential downgrade

* use uppercase for SQL keywords
2019-08-22 09:09:37 -07:00
Clint Wylie b95607d31c
remove YeOldePlumberSchool.java, unused (#8347) 2019-08-21 18:15:51 -07:00
Aaron Bossert a4d1219184 Removed hard-coded Kafka Deserializer in Web-UI Kafka data import such that users can supply a custom deserializer in the UI as well as in hand-built ingestion specs. (#8364) 2019-08-21 18:03:57 -07:00
Vadim Ogievetsky aa51668ea7 Web console: LGTM alert squash + final fixes (#8351)
* mv old console

* update dep

* reduce card size

* fix tuning configs for forceGuaranteedRollup

* fix tests

* revert console mv

* fix some old console issues

* revert build change
2019-08-21 13:52:48 -07:00
Benedict Jin e1f94a5e26 Reduce the size of images with lossless compression (#8358) 2019-08-21 13:29:30 -07:00
Clint Wylie 010f70b371
autogenerate NOTICE.BINARY from NOTICE and licenses.yaml (#8306)
* migrate binary notice entries to live in licenses.yaml, use licenses.yaml and NOTICE to generate NOTICE.BINARY at distribution time

* +x

* move release scripts to distribution/bin, fixup notice script, trim dependencies for avro and kerberos in licenses.yaml

* add missing hdfs-storage dependencies

* revert to old syntax, fixes

* formatting

* update notices for recently updated dependencies
2019-08-21 12:46:27 -07:00
Jihoon Son 22d6384d36
Fix unrealistic test variables in KafkaSupervisorTest and tidy up unused variable in checkpointing process (#7319)
* Fix unrealistic test arguments in KafkaSupervisorTest

* remove currentCheckpoint from checkpoint action

* rename variable
2019-08-21 10:58:22 -07:00
Gian Merlino d007477742
Docusaurus build framework + ingestion doc refresh. (#8311)
* Docusaurus build framework + ingestion doc refresh.

* stick to npm instead of yarn

* fix typos

* restore some _bin

* Adjustments.

* detect and fix redirect anchors

* update anchor lint

* Web-console: remove specific column filters (#8343)

* add clear filter

* update tool kit

* remove usless check

* auto run

* add %

* Fix resource leak (#8337)

* Fix resource leak

* Patch comments

* Enable Spotbugs NP_NONNULL_RETURN_VIOLATION (#8234)

* Fixes from PR review.

* Fix more anchors.

* Preamble nix.

* Fix more anchors, headers

* clean up placeholder page

* add to website lint to travis config

* better broken link checking

* travis fix

* Fixed more broken links

* better redirects

* unfancy catch

* fix LGTM error

* link fixes

* fix md issues

* Addl fixes
2019-08-20 21:48:59 -07:00
Benedict Jin c4db83608f
Add grade shield (#8344) 2019-08-21 12:20:19 +08:00
mcbrewster e4aa7fb268 Web-console: fix alerts from lgtm (#8346)
* fix alerts from lgtm

* remove unordered imports

* fix introduced alert

* move getExpanded to renderSchemaSelector

* use getDerivedStateFromProps

* use prevState

* add semi colons, remove unused imports

* fixes
2019-08-20 20:40:47 -07:00
Clint Wylie c87b68d2a4 use Number instead of long for response context (#8342)
* use Number instead of long for response context to be forgiving of json serde to int or long

* test that encounters issue without fix

* now with more test

* is ints
2019-08-20 19:05:49 -07:00
Fokko Driesprong d5a19675dd Remove fromPigAvroStorage from the docs (#8340)
This one has been deprecated a while ago
2019-08-20 16:34:55 -07:00
Benedict Jin 14a4238381 Bump JUnitParams from 1.0.4 to 1.1.1 (#8017) 2019-08-20 16:15:12 -07:00
Fokko Driesprong 8821ac330d Bump opencsv from 4.2 to 4.6 (#8294)
* Bump opencsv from 4.2 to 4.6

* Fix transitive dependencies
2019-08-20 16:12:03 -07:00