Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,801 Commits 111 Branches 503 Tags 728 MiB
Commit Graph

46 Commits

Author SHA1 Message Date
fjy c89b5a385b cleanup s3 pulling logs 2014-07-29 16:17:44 -07:00
Xavier Léauté f847a46ccf fix typo 2014-07-22 15:53:50 -07:00
fjy 2be4c04b1a Merge pull request #629 from metacret/ISSUE-628 
AWSCredentialsProvider for s3-extentions
 2014-07-18 12:01:19 -06:00
Jae Hyeon Bae 9cec7107a9 Properties.load instead of manual population 2014-07-18 09:50:15 -07:00
Xavier Léauté 12bc3ac27b fix broken S3 moving test 2014-07-15 10:58:53 -07:00
Xavier Léauté f56d60b451 fix storage class npe 2014-07-14 16:24:01 -07:00
Jae Hyeon Bae 193c7259f6 refresh thread should be daemon 2014-07-09 15:52:17 -07:00
Jae Hyeon Bae 3a849b0341 update refresh manually every hour 2014-07-07 23:11:08 -07:00
Jae Hyeon Bae 4ab9af36b0 License header and utf-8 2014-07-03 16:52:47 -07:00
Jae Hyeon Bae 5e5fa1f357 A few fixes 
- EasyMock instead of Mockito
- FileSessionCredentialsProvider fields should be volatile
- getRestS3Service should create AWSCredentials not a AWSSessionCredentials with fixed credentials
 2014-07-02 15:53:39 -07:00
Jae Hyeon Bae 31f7bd8bcb AWSCredentialsProvider for s3-extentions 
- fileSessionCredentials property is added
- AWSSessionCredentialsAdapter is adapter class between aws sdk AWSCredentialsProvider to jets3t Credentials
- FileSessionCredentialsProvider will refresh its session credentials from the file
- S3StorageDruidModule should return different AWSCredentialsProvider on the config settings
 2014-07-02 14:44:02 -07:00
fjy a63cda3281 Merge branch 'master' into new-guava 
Conflicts:
	server/src/main/java/io/druid/server/QueryResource.java
 2014-06-13 10:08:10 -07:00
Xavier Léauté ae0e36f3eb warn glacier objects cannot be moved 2014-06-10 11:22:10 -07:00
fjy 77ec4df797 update guava, java-util, and druid-api 2014-06-03 13:43:38 -07:00
Gian Merlino 8a6384862f Better task log errors. 2014-06-02 12:55:17 -07:00
fjy 76e0a48527 Merge branch 'master' into new-schema 
Conflicts:
	indexing-hadoop/src/main/java/io/druid/indexer/DbUpdaterJob.java
	indexing-hadoop/src/test/java/io/druid/indexer/HadoopDruidIndexerConfigTest.java
	indexing-service/src/main/java/io/druid/indexing/common/task/HadoopIndexTask.java
	server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumber.java
	server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumberSchool.java
 2014-04-25 14:03:28 -07:00
Xavier Léauté c46a39b13e fix missing arg 2014-04-04 15:13:27 -07:00
Xavier Léauté cfe2466e49 enforce acl on object move 2014-04-04 14:28:54 -07:00
Xavier Léauté b21820c68a make index file acl consistent with descriptor 2014-04-04 14:20:57 -07:00
fjy 46b9ac78e7 Merge branch 'master' into new-schema 
Conflicts:
	indexing-hadoop/src/test/java/io/druid/indexer/HadoopDruidIndexerConfigTest.java
	pom.xml
	publications/whitepaper/druid.pdf
	publications/whitepaper/druid.tex
 2014-03-03 14:48:15 -08:00
Xavier Léauté 2f61035585 add restore task 2014-02-25 13:41:40 -08:00
fjy 4b7c76762d unit tests passingn at this point, finished rt port maybe 2014-02-18 15:14:38 -08:00
Gian Merlino 6478c9a162 S3DataSegmentMover: No need to move files onto themselves 2013-12-18 18:36:10 -08:00
Gian Merlino b7a184cb06 S3DataSegmentMover: Retries for S3 failures 2013-12-18 16:00:37 -08:00
Xavier Léauté c8be38fe40 back to baseKey 2013-12-18 14:57:14 -08:00
Xavier Léauté 178c26f9f7 fix naming and docs 2013-12-16 16:50:31 -08:00
Xavier Léauté f7f5ffc880 add S3 segment move test and fix bug caught by test 2013-12-16 15:12:15 -08:00
Xavier Léauté f3b8d9c047 safely move files 2013-12-16 14:00:56 -08:00
Xavier Léauté ac2ca0e46c separate move and archive tasks 2013-12-16 14:00:55 -08:00
Xavier Léauté 6b90372002 separate segment mover and segment archiver 2013-12-16 13:59:16 -08:00
Xavier Léauté 3af6e49cd4 throw exception instead of just printing a warning 2013-12-16 13:59:16 -08:00
Xavier Léauté 3ae48a8191 fix cut-n-paste typo 2013-12-16 13:59:16 -08:00
Xavier Léauté cd7a941f83 consolidate path functions 2013-12-16 13:59:15 -08:00
Xavier Léauté a417cd5df2 add archive task 2013-12-16 13:59:15 -08:00
Xavier Léauté e38f2877fb default to archiving segments + docs 2013-12-16 13:59:15 -08:00
Xavier Léauté 1f7a089fa8 S3 storage: enable archiving in addition to deleting unused segments 2013-12-16 13:59:15 -08:00
Gian Merlino 36756e611b S3Utils: Fix retry predicate 2013-12-16 08:27:26 -08:00
Gian Merlino 1ec6d5b0ef S3Utils: Use RetryUtils from java-util 2013-12-11 15:05:16 -08:00
fjy 6cc0860657 fix s3 module problems and prepare for next release 2013-12-09 20:05:55 -08:00
Xavier Léauté 89f821900c add imports 2013-11-13 11:51:17 -08:00
fjy 51aa7a2284 doesnt work yet 2013-11-11 14:04:03 -08:00
Gian Merlino 50a8c806c3 S3Utils: Replace S.o.println with log.warn 2013-11-02 15:24:05 -07:00
Gian Merlino f8e564469f S3Utils: More aggressive retries, S3 can be pretty unreliable sometimes 2013-11-02 08:57:58 -07:00
fjy 1bd1bc98d1 update s3 calls to not use deprecated fns 2013-10-16 13:21:54 -07:00
fjy 9796a40b92 port docs over to 0.6 and a bunch of misc fixes 2013-10-11 18:38:53 -07:00
cheddar c47fe202c7 Fix HadoopDruidIndexer to work with the new way of things 
There are multiple and sundry changes in here.

First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.

In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module.  I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.

In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api.  They are all API-level things, however, so they really belong in druid-api instead.

Lastly, I removed the druid-realtime module and put it all in druid-server.
 2013-10-09 15:15:44 -05:00