Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,926 Commits 326 Branches 371 Tags
Commit Graph

14926 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kihwal Lee
500dc05924 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b6493d315c8910fbc06eaba96273ee335c)
(cherry picked from commit fd8b8c7494d19467485423ce2f3f7459a4fddc29)
 2019-05-30 17:37:38 -05:00
Ayush Saxena
c63049dfd4 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f27e3e4f58ff1b5ef3e93efc56bbecbe)
(cherry picked from commit 2cd7577a5a58db42a14a659b85fbd475091d0cb2)
(cherry picked from commit 90cc5ddd4957fcb27ef5a1e7ee1a67c4eca9a3f5)
(cherry picked from commit 942c5b04f2f7bc33595c4df5c66104ce3c4c2c69)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
 2019-05-29 21:11:15 -07:00
Siyao Meng
cecba551aa HDFS-14514. Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2. Contributed by Siyao Meng, Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Stephen O'Donnell <sodonnell@cloudera.com>
 2019-05-29 17:28:49 -07:00
Takanobu Asanuma
e3d4f6bfed HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:41:21 +09:00
Akira Ajisaka
cea1b0000c
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:54 +09:00
Erik Krogen
f024d27cd9 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f002f74cb4a0360b6682a1b6796d1598)
(cherry-picked from c26cf22d6b3949a4798d8133669b451660a0f050)
(cherry-picked from 9295e95174ad8627874b81a9d43183384a81fb16)
 2019-05-24 13:39:52 -07:00
Konstantin V Shvachko
2afc770443 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:43:12 -07:00
Wei-Chiu Chuang
380289a167 HDFS-13087. Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge, Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Siyao Meng <smeng@cloudera.com>
 2019-05-24 18:36:53 +02:00
Alexis Daboville
b70fdc64d3
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,

(cherry picked from commit 4cb3da6ac703b10de59a8ca371fe957a4a338601)
 2019-05-17 15:27:47 +01:00
Weiwei Yang
11fb89bd68 HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 
(cherry picked from commit 2d8282bb8248e6984878626c4cdc7148aa2e7202)
 2019-05-14 14:09:12 -07:00
Haibo Chen
dc28c86270 YARN-9529. Log correct cpu controller path on error while initializing CGroups. (Contributed by Jonathan Hung) 
(cherry picked from commit 597fa47ad125c0871f5c4deb3a883e5b3341c67b)
(cherry picked from commit c6573562cbc43832e9332989996e5d07b8bce9b0)
(cherry picked from commit ea1f0f282bee7ca64ce9fbe1a59bb5e231420d29)
(cherry picked from commit aa7ff8552e0f1b9439926ae3476a3227dcf339f3)
 2019-05-06 12:01:26 -07:00
Eric E Payne
5bf40f592e YARN-9285: RM UI progress column is of wrong type. Contributed by Ahmed Hussein. 2019-05-03 16:17:43 +00:00
Yuan Gao
8900c7f4f4
HADOOP-16205 Backport ABFS driver from trunk to branch 2. 
Contributed by Yuan Gao.
 2019-04-25 18:37:30 +01:00
Yuan Gao
18f66a05cf
HADOOP-16205 Backport ABFS driver from trunk to branch 2.0: Fix build and test failures. 
Contributed by Yuan Gao.
 2019-04-25 18:36:14 +01:00
Vishwajeet Dusane
662a75d65b HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file. 
Contributed by Vishwajeet Dusane.
 2019-04-24 21:46:25 +00:00
Da Zhou
024a694684 HADOOP-16169. ABFS: Bug fix for getPathProperties. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2019-04-24 21:46:25 +00:00
Da Zhou
c2d7429691 HADOOP-16136. ABFS: Should only transform username to short name 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
80c8b77cec HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-04-24 21:46:25 +00:00
Shweta Yakkali
0bbbb9c4b9 HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali. 
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
 2019-04-24 21:46:25 +00:00
Da Zhou
f799c4644d HADOOP-16048. ABFS: Fix Date format parser. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
00b990d0b6 HADOOP-16044. ABFS: Better exception handling of DNS errors followup 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
34dbac8347 HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME. 
Contributed by Da Zhou.

(cherry picked from commit 8b5fbe7a125f9d08cbb9f5e5ae28dc984e0d73d8)
 2019-04-24 21:46:25 +00:00
Da Zhou
a0b99f42e5 HADOOP-15662. Better exception handling of DNS errors. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
4a1761067d HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Sean Mackrory
175a69e863 HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). Contributed by Shweta Yakkali. 2019-04-24 21:46:25 +00:00
Da Zhou
eaaf2839a5 HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
38a36d4dec HADOOP-15972 ABFS: reduce list page size to to 500. 
Contributed by Da Zhou.

(cherry picked from commit d09dbcc8fb6889e51d4d4c914eb6159e40845e9f)
 2019-04-24 21:46:25 +00:00
Da Zhou
06e6a677f3 HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
6f200a61b9 HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
9cf1fb6e95 HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename. 
Contributed by Da Zhou <da.zhou@microsoft.com>

(cherry picked from commit 1a3a4960d91aa594283e447963fb2e407d4c6af5)
 2019-04-24 21:46:25 +00:00
Da Zhou
afa5705ff7 HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2. 
Contributed by Junhua Gu and Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
9e1d39bbae HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
9f220c41c4 HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2019-04-24 21:46:25 +00:00
Junhua Gu
d37098196d HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. 
Contributed by Junhua Gu.

(cherry picked from commit 66715005f9e8f4f25faa352a06d142b75a029f0e)
 2019-04-24 21:46:25 +00:00
Daniel Templeton
8330f182dd HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI 
(Contributed by Da Zhou via Daniel Templeton)

Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
 2019-04-24 21:46:25 +00:00
Steve Loughran
885c726d38 HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
73bbada59a HADOOP-15809. ABFS: better exception handling when making getAccessToken call. 
Contributed by Da Zhou
 2019-04-24 21:46:25 +00:00
Steve Loughran
1b08e8d537 HADOOP-15795. Make HTTPS the default protocol for ABFS. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
e3033a5ba8 HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld. 
Contributed by Abhishek Modi.
 2019-04-24 21:46:25 +00:00
Steve Loughran
d5e4c40760 HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore. 
Contributed by Da Zhou
 2019-04-24 21:46:25 +00:00
Steve Loughran
0c5074a228 HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization 
Contributed by Yuan Gao.
 2019-04-24 21:46:25 +00:00
Steve Loughran
ee0fa0d15c HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
bde00d56df HADOOP-15723. ABFS: Ranger Support. 
Contributed by Yuan Gao.
 2019-04-24 21:46:25 +00:00
Steve Loughran
4f3cc1d417 HADOOP-15778. ABFS: Fix client side throttling for read. 
Contributed by Sneha Varma.
 2019-04-24 21:46:25 +00:00
Steve Loughran
6c7c8b83f7 HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable. 
Contributed by Steve Loughran.
 2019-04-24 21:46:25 +00:00
Sean Mackrory
920b77c8d7 HADOOP-15773. Fixing checkstyle and other issues raised by Yetus. 2019-04-24 21:46:25 +00:00
Steve Loughran
10e149d882 HADOOP-15769. ABFS: distcp tests are always skipped. 
Contributed by Steve Loughran
 2019-04-24 21:46:25 +00:00
Steve Loughran
1efa1b73a9 HADOOP-15719. Fail-fast when using OAuth over http. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
8acab6c6aa HADOOP-15714. Tune abfs/wasb parallel and sequential test execution. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
26ce4e9301 HADOOP-15715. ITestAzureBlobFileSystemE2E timing out with non-scale timeout of 10 min. 
Contributed by Da Zhou
 2019-04-24 21:46:25 +00:00