04ea3c9f8c
Update license headers. ( #5976 )
...
* Update license headers.
For compliance with http://www.apache.org/legal/src-headers.html .
* More license adjustments.
* Fix mistakenly edited package line.
2018-07-11 09:55:18 -07:00
8ec2d2fe18
Use unique segment paths for Kafka indexing ( #5692 )
...
* support unique segment file paths
* forbiddenapis
* code review changes
* code review changes
* code review changes
* checkstyle fix
2018-04-29 21:59:48 -07:00
5ab17668c0
CompressionUtils: Add support for decompressing xz, bz2, zip. ( #5586 )
...
Also switch various firehoses to the new method.
Fixes #5585 .
2018-04-06 08:06:45 -07:00
723f7ac550
Add support for task reports, upload reports to deep storage ( #5524 )
...
* Add support for task reports, upload reports to deep storage
* PR comments
* Better name for method
* Fix report file upload
* Use TaskReportFileWriter
* Checkstyle
* More PR comments
2018-04-02 12:10:56 -07:00
8878a7ff94
Replace guava Charsets with native java StandardCharsets ( #5545 )
2018-03-28 21:00:08 -07:00
693e3575f9
Remove unused code and exception declarations ( #5461 )
...
* Remove unused code and exception declarations
* Address comments
* Remove redundant Exception declarations
* Make FirehoseFactoryV2.connect() to throw IOException again
2018-03-16 22:11:12 +01:00
be38b18a85
Support Hadoop batch ingestion for druid-azure-extensions ( #5221 )
...
* Support Hadoop batch ingestion for druid-azure-extensions #5181
* Fix indentation issues
* Fix forbidden-apis violation
* Code & doc improvements for azure-extensions
* Rename version to binaryVersion where appropriate to avoid confusion
* Set default protocol to wasbs://, as recommended by the Azure docs
* Add link to Azure documentation for wasb(s):// path
* Remove any colons from the dataSegment.getVersion()
* Added test for dataSegment.getVersion colon is replaced
* Use StringUtils.format for String concatenation
* remove empty lines
* Remove unneeded StringUtils.format from log.info
2018-02-15 16:45:18 +01:00
7e02408510
Update versions to 0.13.0-SNAPSHOT. ( #5323 )
2018-02-02 12:06:38 -06:00
8877ce38d6
Enforce modifier order with Checkstyle ( #5246 )
2018-01-11 09:50:42 +01:00
5d0619f5ce
Support retrying for PrefetchableTextFilesFirehoseFactory when prefetch is disabled ( #5162 )
...
* Add RetryingInputStream
* unnecessary exception
* fix PrefetchableTextFilesFirehoseFactoryTest
* Fix retrying on connection reset
* fix start offset
* fix checkstyle
* fix check connection reset
* address comments
* fix compile
* address comments
* address comments
2018-01-10 17:37:19 +01:00
a7967ade4d
Support replaceExisting parameter for segments pushers ( #5187 )
...
* support replaceExisting parameter for segments pushers
* code review changes
* code review changes
2018-01-03 16:13:21 -08:00
5787d04fad
Bump Druid version to 0.12.0 ( #5138 )
2017-12-15 07:37:01 -08:00
3541b7544b
Prohibit and remove unused declarations in the processing module ( #4930 )
...
* Prohibit and remove unused declarations in the processing module
* Fix tests
* Fix integration tests
* Suppress unused
* Try to remove SuppressWarnings unused in VirtualColumn
* Remove reset 'false positives'
* Annotate CliCommandCreator as ExtensionPoint
* Unused import warning instead of error in IntelliJ
* Fixes
* Add comment
* Fix AzureBlob
* Fix CloudFilesBlob
* Address comments
* Add Project SDK section to INTELLIJ_SETUP.md
* Fix image
2017-11-09 09:27:27 -08:00
8d9902831e
Refactoring PrefetchableTextFilesFirehoseFactory ( #4836 )
...
* Refactoring prefetchable firehose
* Fix to read cache when prefetch is disabled
* More tests
* Cleanup codes
* Add Fetcher
* Fix test failure
* Count file size
* Fix test
* rename generic parameter
* address comments
* address comments
* reuse buffer
* move Execs to java-util
* use execs
* Fix build
2017-10-13 21:39:28 -05:00
d95915f8d2
Implement get methods for PrefetchableFirehose ( #4948 )
2017-10-12 16:14:45 +09:00
1f2074c247
Bump versions in master to 0.11.1-SNAPSHOT. ( #4878 )
...
* Bump versions in master to 0.11.1-SNAPSHOT.
* Missed a few.
2017-09-28 17:09:51 -05:00
cbd1902db8
Add forbidden-apis plugin; prohibit using system time zone ( #4611 )
...
* Forbidden APIs WIP
* Remove some tests
* Restore io.druid.math.expr.Function
* Integration tests fix
* Add comments
* Fix in SimpleWorkerProvisioningStrategy
* Formatting
* Replace String.format() with StringUtils.format() in RemoteTaskRunnerTest
* Address comments
* Fix GroupByMultiSegmentTest
2017-08-21 13:02:42 -07:00
aa7e4ae5e4
Enforce correct spacing with Checkstyle ( #4651 )
2017-08-05 10:18:25 -07:00
c0beb78ffd
Enforce brace formatting with Checkstyle ( #4564 )
2017-07-21 10:26:59 -05:00
441ee56ba9
DataSegmentPusher: Add allowed hadoop property prefixes. ( #4562 )
...
* DataSegmentPusher: Add allowed hadoop property prefixes.
* Fix dots.
2017-07-18 10:16:12 -07:00
9ae457f7ad
Avoid using the default system Locale and printing to System.out in production code ( #4409 )
...
* Avoid usages of Default system Locale and printing to System.out or System.err in production code
* Fix Charset in DruidKerberosUtil
* Remove redundant string format in GenericIndexed
* Rename StringUtils.safeFormat() to unimportantSafeFormat(); add StringUtils.format() which fails as well as String.format()
* Fix testSafeFormat()
* More fixes of redundant StringUtils.format() inside ISE
* Rename unimportantSafeFormat() to nonStrictFormat()
2017-06-29 14:06:19 -07:00
ae900a4934
Update versions to 0.11.0-SNAPSHOT ( #4483 )
2017-06-28 17:05:58 -07:00
2cd91b64d0
Uncompress streams without having to download to tmp first ( #4364 )
...
* Uncompress streams without having to download to tmp first
* Remove unused file
2017-06-08 18:08:38 -07:00
31d33b333e
Make using implicit system Charset an error ( #4326 )
...
* Make using implicit system charset an error
* Use StringUtils.toUtf8() and fromUtf8() instead of String.getBytes() and new String()
* Use English locale in StringUtils.safeFormat()
* Restore comment
2017-06-05 23:57:25 -07:00
a2584d214a
Delagate creation of segmentPath/LoadSpec to DataSegmentPushers and add S3a support ( #4116 )
...
* Adding s3a schema and s3a implem to hdfs storage module.
* use 2.7.3
* use segment pusher to make loadspec
* move getStorageDir and makeLoad spec under DataSegmentPusher
* fix uts
* fix comment part1
* move to hadoop 2.8
* inject deep storage properties
* set version to 2.7.3
* fix build issue about static class
* fix comments
* fix default hadoop default coordinate
* fix create filesytem
* downgrade aws sdk
* bump the version
2017-06-04 00:55:09 -06:00
733dfc9b30
Add PrefetchableTextFilesFirehoseFactory for cloud storage types ( #4193 )
...
* Add PrefetcheableTextFilesFirehoseFactory
* fix comment
* exception handling
* Fix wrong json property
* Remove ReplayableFirehoseFactory and fix misspelling
* Defer object initialization
* Add a temporaryDirectory parameter to FirehoseFactory.connect()
* fix when cache and fetch are disabled
* Address comments
* Add more test
* Increase timeout for test
* Add wrapObjectStream
* Move methods to Firehose from PrefetchableFirehoseFactory
* Cleanup comment
* add directory listing to s3 firehose
* Rename a variable
* Addressing comments
* Update document
* Support disabling prefetch
* Fix race condition
* Add fetchLock
* Remove ReplayableFirehoseFactoryTest
* Fix compilation error
* Fix test failure
* Address comments
* Add default implementation for new method
2017-05-18 15:37:18 +09:00
2ca7b00346
Update versions to 0.10.1-SNAPSHOT. ( #4191 )
2017-04-20 18:12:28 -07:00
81a5f9851f
TmpFileIOPeons to create files under the merging output directory, instead of java.io.tmpdir ( #3990 )
...
* In IndexMerger and IndexMergerV9, create temporary files under the output directory/tmpPeonFiles, instead of java.io.tmpdir
* Use FileUtils.forceMkdir() across the codebase and remove some unused code
* Fix test
* Fix PullDependencies.run()
* Unused import
2017-03-02 14:05:12 -08:00
8854ce018e
File.deleteOnExit() ( #3923 )
...
* Less use of File.deleteOnExit()
* removed deleteOnExit from most of the tests/benchmarks/iopeon
* Made IOpeon closable
* Formatting.
* Revert DeterminePartitionsJobTest, remove cleanup method from IOPeon
2017-02-13 15:12:14 -08:00
12317fd001
Bump version to 0.10.0-SNAPSHOT. ( #3913 )
2017-02-06 17:54:35 -08:00
4ca3b7f1e4
overlord helpers framework and tasklog auto cleanup ( #3677 )
...
* overlord helpers framework and tasklog auto cleanup
* review comment changes
* further review comments addressed
2016-12-21 15:18:55 -08:00
b76b3f8d85
reset-cluster command to clean up druid state stored on metadata and deep storage ( #3670 )
2016-11-09 11:07:01 -06:00
4b3bd8bd63
Migrating java-util from Metamarkets. ( #3585 )
...
* Migrating java-util from Metamarkets.
* checkstyle and updated license on java-util files.
* Removed unused imports from whole project.
* cherry pick metamx/java-util@826021f .
* Copyright changes on java-util pom, address review comments.
2016-10-21 14:57:07 -07:00
5dc95389f7
Add Checkstyle framework ( #3551 )
...
* Add Checkstyle framework
* Avoid star import
* Need braces for control flow statements
* Redundant imports
* Add NewLineAtEndOfFile check
2016-10-13 13:37:47 -07:00
40f2fe7893
Bump versions to 0.9.3-SNAPSHOT ( #3524 )
2016-09-29 13:53:32 -07:00
485e381387
remove datasource from hadoop output path ( #3196 )
...
fixes #2083 , follow-up to #1702
2016-06-29 08:53:45 -07:00
45f553fc28
Replace the deprecated usage of NoneShardSpec ( #3166 )
2016-06-25 10:27:25 -07:00
ebf890fe79
Update master version to 0.9.2-SNAPSHOT. ( #3133 )
2016-06-13 13:10:38 -07:00
7e7a886f65
Move druid-api into the druid repo.
...
This is from druid-api-0.3.17, as of commit 51884f1d05d5512cacaf62cedfbb28c6ab2535cf
in the druid-api repo.
2016-03-24 11:04:34 -07:00
738dcd8cd9
Update version to 0.9.1-SNAPSHOT.
...
Fixes #2462
2016-03-17 10:34:20 -07:00
ba1185963b
Fix a bunch of dependencies
...
* Eliminate exclusion groups from pull-deps
* Only consider dependency nodes in pull-deps if they are not in the following scopes
* provided
* test
* system
* Fix a bunch of `<scope>provided</scope>` missing tags
* Better exclusions for a couple of problematic libs
2016-03-10 10:18:08 -08:00
e3e932a4d4
refactor extensions into core and contrib
2016-03-08 17:12:09 -08:00
Gian Merlino
David Lim
Jonathan Wei
Kirill Kozlov
Roman Leventov
Spyros Kapnissis
Jihoon Son
Niketh Sabbineni
Slim
Roman Leventov
Akash Dwivedi
Himanshu
Xavier Léauté
Hyukjin Kwon
Nishant
fjy