Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,416 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4895 Commits

Author SHA1 Message Date
Steve Loughran 0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-09 16:00:34 +00:00
Praveen Krishna 2b94e51a8f
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host. 
Author:    Praveen Krishna <praveenkrishna@tutanota.com>
 2019-03-07 11:06:34 +00:00
Vivek Ratnavel Subramanian a55fc36299 HDDS-1093. Configuration tab in OM/SCM ui is not displaying the correct values. 2019-03-06 17:43:57 -08:00
Sahil Takiar 618e009ac0 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 15:04:06 -08:00
Wei-Chiu Chuang 6192c1fe3b Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar." 
This reverts commit f5a4b43a49.
 2019-03-06 15:02:18 -08:00
Sahil Takiar f5a4b43a49 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 14:58:45 -08:00
Eric Yang 3c5b7136e2 HADOOP-16150. Added concat method to ChecksumFS as unsupported operation. 
Contributed by Steve Loughran

(cherry picked from commit 8b517e7ad6)
 2019-03-05 13:32:00 -05:00
Stephen O'Donnell 686c0141ef
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately. 
Contributed by Stephen O'Donnell.

Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 14:09:00 +00:00
Prabhu Joseph e40e2d6ad5
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-03-05 14:10:08 +09:00
coder-chenzhi fe7551f21b
HADOOP-16162. Remove unused Job Summary Appender configurations from log4j.properties 
This closes #551

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-03-05 13:41:33 +09:00
David Mollitor 9fcd89ab93
HADOOP-16148. Cleanup LineReader Unit Test. 
Contributed by David Mollitor.

Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-04 23:08:12 +00:00
Ajay Kumar 0d61facd37 HADOOP-15889. Add hadoop.token configuration parameter to load tokens. Contributed by Íñigo Goiri 2019-02-28 10:34:28 -08:00
Eric Yang feccd282fe HADOOP-16107. Update ChecksumFileSystem createFile/openFile API to generate checksum. 
Contributed by Steve Loughran
 2019-02-27 15:53:41 -05:00
Tsz Wo Nicholas Sze 9192f71e21 HADOOP-16127. In ipc.Client, put a new connection could happen after stop. 2019-02-26 15:14:21 -08:00
Abhishek Modi 52b2eab575
HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util. 
Contributed by Abhishek Modi
 2019-02-26 17:10:41 +00:00
Surendra Singh Lilhore 59ba3552d3 HDFS-14299. ViewFs: Correct error message for read only operations. Contributed by hu xiaodong. 2019-02-26 12:40:32 +05:30
Konstantin V Shvachko a6ab37192a HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko. 2019-02-25 14:35:02 -08:00
Inigo Goiri ba4e7bd192 HADOOP-16125. Support multiple bind users in LdapGroupsMapping. Contributed by Lukas Majercak. 2019-02-25 13:39:13 -08:00
Tsz Wo Nicholas Sze 0edb0c51dc HADOOP-16126. ipc.Client.stop() may sleep too long to wait for all connections. 2019-02-25 13:15:28 -08:00
Yongjun Zhang f7a27cdee4 HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li. 2019-02-23 09:35:36 -08:00
Daryn Sharp a87e458432 HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-20 18:17:04 -08:00
Shweta Yakkali 371a6db59a HDFS-14273. Fix checkstyle issues in BlockLocation's method javadoc 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I546aa4a0fe7f83b53735acd9925f366b2f1a00e2
 2019-02-20 15:36:53 -08:00
Akira Ajisaka dabfeab785
YARN-9308. fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240. Contributed by Wilfred Spiegelenburg. 2019-02-15 14:49:49 +09:00
Erik Krogen 64f28f9efa HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 2019-02-14 11:22:04 -08:00
Siyao Meng fd026863b1 HDFS-14241. Provide feedback on successful renameSnapshot and deleteSnapshot. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-13 19:48:02 -08:00
Chen Liang 024c87291c HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-02-13 12:40:31 -08:00
Vinayakumar B 00c5ffaee2 HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena. 2019-02-13 16:44:32 +05:30
Yiqun Lin 7b11b404a3 HADOOP-16097. Provide proper documentation for FairCallQueue. Contributed by Erik Krogen. 2019-02-13 11:16:04 +08:00
Steve Loughran 668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth." 
(accidentally mixed in two patches)

This reverts commit fa8cd1bf28.
 2019-02-07 21:57:22 +00:00
Wangda Tan 308f3168fa Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
 2019-02-05 14:03:18 -08:00
Da Zhou fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-05 19:23:15 +00:00
Steve Loughran f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00
Steve Loughran 7f46d13dac
HADOOP-16079. Token.toString faulting if any token listed can't load. 
Contributed by Steve Loughran.
 2019-02-01 14:31:47 +00:00
Inigo Goiri bcc3a79f58 HADOOP-16084. Fix the comment for getClass in Configuration. Contributed by Fengnan Li. 2019-01-31 10:06:05 -08:00
Akira Ajisaka 1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Hanisha Koneru b3bc94ebfd HDFS-14236. Lazy persist copy/ put fails with ViewFs. 2019-01-29 16:45:44 -08:00
Wei-Chiu Chuang d1714c20e9 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit 1d523279da.
 2019-01-29 15:43:09 -08:00
Pranay Singh 1d523279da HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-25 09:02:57 -08:00
Eric Yang 0dd35e218f HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user. 
Contributed by He Xiaoqiao
 2019-01-22 18:59:36 -05:00
Akira Ajisaka a463cf75a0
HADOOP-15787. [JDK11] TestIPC.testRTEDuringConnectionSetup fails. Contributed by Zsolt Venczel. 2019-01-22 10:19:05 +09:00
Sunil G 2e2508b8e3 Make 3.2.0 aware to other branches 2019-01-21 21:24:51 +05:30
Surendra Singh Lilhore dacc1a759e HDFS-14175. EC: Native XOR decoder should reset the output buffer before using it. Contributed by Ayush Saxena. 2019-01-18 13:55:12 +05:30
Akira Ajisaka 01405df87f
HADOOP-15941. Addendum patch. Contributed by Takanobu Asanuma. 2019-01-16 10:28:08 +09:00
luhuachao ccc43621ab
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. 
Contributed by luhuachao.
 2019-01-15 13:15:18 +00:00
Steve Loughran 6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran and Daryn Sharp.
 2019-01-14 17:59:27 +00:00
Inigo Goiri 35fa3bd685 HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 2019-01-11 18:07:19 -08:00
Chen Liang bf08f4abae HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-11 14:01:23 -08:00
Chen Liang 01cb958af4 HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen. 2019-01-11 12:51:07 -08:00
Giovanni Matteo Fumarola fb8932a727 HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena. 2019-01-11 10:54:49 -08:00
Akira Ajisaka d4ca907da6
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 2019-01-11 11:24:05 +09:00