Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,303 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4772 Commits

Author SHA1 Message Date
Wei-Chiu Chuang ce93a2b84b HDFS-12619. Do not catch and throw unchecked exceptions if IBRs fail to process. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 4ab0c8f96a)
 2017-10-19 06:18:40 -07:00
Zhe Zhang 87f4407396 HDFS-12502. nntop should support a category based on FilesInGetListingOps. 
(cherry picked from commit 60bfee270e)
(cherry picked from commit aecf6c50d3)
 2017-10-19 00:05:34 -07:00
Arpit Agarwal 0aa1b62b39 HDFS-12420. Add an option to disallow 'namenode format -force'. Contributed by Ajay Kumar. 
(cherry picked from commit b6942cbe9b)
(cherry picked from commit 5897095d53)
 2017-10-17 23:44:13 -07:00
Wei-Chiu Chuang a596c6f5eb HDFS-12485. expunge may fail to remove trash from encryption zone. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 8dbc8909c9)
 2017-10-16 12:59:41 -07:00
Xiao Chen b2f27641dd HDFS-12642. Log block and datanode details in BlockRecoveryWorker. 
(cherry picked from commit 971b3ca1acc15d9a1f8e774291813dafd16ef704)
 2017-10-16 10:34:24 -07:00
Xiao Chen d82edb49c7 HDFS-12603. Enable async edit logging by default. Contributed by Andrew Wang. 
(cherry picked from commit d0311dfc5fe92a0e0f3fb8ae92e412edce609740)
 2017-10-16 09:51:27 -07:00
Xiao Chen 4b734fe386 HDFS-12659. Update TestDeadDatanode#testNonDFSUsedONDeadNodeReReg to increase heartbeat recheck interval. Contributed by Ajay Kumar. 
(cherry picked from commit 11a7729c1474aa379a62c0440c8c1a11be261a75)
 2017-10-15 19:28:38 -07:00
Andrew Wang 1ddc49527a Revert "HDFS-12603. Enable async edit logging by default." 
This reverts commit 42d3229e6f.
 2017-10-09 13:29:52 -07:00
Andrew Wang 42d3229e6f HDFS-12603. Enable async edit logging by default. 2017-10-09 11:21:43 -07:00
Xiao Chen dd0113d777 HADOOP-14521. KMS client needs retry logic. Contributed by Rushabh S Shah. 
(cherry picked from commit 212342d6db14673f8f5652a5209e1c4fcc0f0ede)
 2017-10-05 19:47:04 -07:00
Yiqun Lin 2e2f8e42cf HDFS-12495. TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks fails intermittently. Contributed by Eric Badger. 
(cherry picked from commit e9b790db9d)
 2017-09-27 10:45:44 +08:00
Daryn Sharp 7f3ed5f61e [HDFS-12386] Add fsserver defaults call to WebhdfsFileSystem. (Rushabh Shah via daryn) 2017-09-26 15:21:37 -05:00
Anu Engineer de5744c903 HDFS-12531. Fix conflict in the javadoc of UnderReplicatedBlocks.java in branch-2. Contributed by Bharat Viswanadham. 2017-09-23 11:16:50 -07:00
Arpit Agarwal 09290659e7 HDFS-12530. Processor argument in Offline Image Viewer should be case insensitive. Contributed by Hanisha Koneru. 2017-09-22 09:48:10 -07:00
Arpit Agarwal 18e5f2068b HDFS-12496. Make QuorumJournalManager timeout properties configurable. Contributed by Ajay Kumar. 2017-09-21 08:53:27 -07:00
Kihwal Lee 9640bfb808 HDFS-12371. BlockVerificationFailures and BlocksVerified show up as 0 in Datanode JMX. Contributed by Hanisha Koneru. 
(cherry picked from commit 6bf921a5c3)
 2017-09-21 08:48:04 -05:00
Akira Ajisaka 3527327f34
HDFS-12526. FSDirectory should use Time.monotonicNow for durations. Contributed by Bharat Viswanadham. 
(cherry picked from commit 50849ec9eb)
 2017-09-21 19:28:49 +09:00
Ming Ma 3bb23f4be9 HDFS-11035. Better documentation for maintenace mode and upgrade domain. 
(cherry picked from commit ce943eb17a)
 2017-09-20 09:42:22 -07:00
Ming Ma 7dd662eafd HDFS-12473. Change hosts JSON file format. 2017-09-20 09:08:41 -07:00
Brahma Reddy Battula eddc581860 HDFS-11799. Introduce a config to allow setting up write pipeline with fewer nodes than replication factor. Contributed by Brahma Reddy Battula 
(cherry picked from commit fda1221c55)
 2017-09-19 11:31:22 +05:30
Arpit Agarwal f42705b957 HDFS-12472. Add JUNIT timeout to TestBlockStatsMXBean. Contributed by Bharat Viswanadham. 2017-09-16 10:14:39 -07:00
Erik Krogen 791e20c244 HDFS-12323. NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout. Contributed by Erik Krogen. 
(cherry picked from commit 90894c7262)
 2017-09-15 14:03:10 -07:00
Nathan Roberts bd14e8d8b0 HDFS-12441. Supress UnresolvedPathException in namenode log. Contributed by Kihwal Lee. 
(cherry picked from commit de197fc156)
 2017-09-15 14:47:07 -05:00
Kihwal Lee 07dee7fd63 HDFS-12424. Datatable sorting on the Datanode Information page in the Namenode UI is broken. Contributed by Shawna Martell. 
(cherry picked from commit 78bdf10ae4)
 2017-09-15 09:55:59 -05:00
John Zhuge eaef8446f6 HADOOP-14864. FSDataInputStream#unbuffer UOE should include stream class name. Contributed by Bharat Viswanadham. 
(cherry picked from commit 7ee02d1065)
 2017-09-14 11:29:34 -07:00
Arpit Agarwal ecf352d645 HADOOP-14867. Update HDFS Federation setup document, for incorrect property name for secondary name node http address. Contributed by Bharat Viswanadham. 2017-09-13 12:54:21 -07:00
Xiao Chen 0a8ed025e4 Revert "HADOOP-14521. KMS client needs retry logic. Contributed by Rushabh S Shah." 
This reverts commit 5337c2511d.
 2017-09-12 19:53:14 -07:00
Arpit Agarwal ac20d7e246 HDFS-12407. Journal node fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start. Contributed by Ajay Kumar. 2017-09-12 16:18:56 -07:00
Manoj Govindassamy 796404c3ed HDFS-12404. Rename hdfs config authorization.provider.bypass.users to attributes.provider.bypass.users. 
(cherry picked from commit 3b3be355b3)
 2017-09-07 17:21:33 -07:00
Xiao Chen c6a1e3facb HDFS-12369. Edit log corruption due to hard lease recovery of not-closed file which has snapshots. 
(cherry picked from commit 3964b131f3f0131a663377685950d1b7bd3fe63d)
(cherry picked from commit 8b7d597d3a8d728aa05302ccd769af84a1c9ec0f)
 2017-09-07 16:30:38 -07:00
Yongjun Zhang 32e8c84614 HDFS-12357. Let NameNode to bypass external attribute provider for configured users. Contributed by Yongjun Zhang, Arun Suresh. 
(cherry picked from commit d77ed238a9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
 2017-09-07 10:22:37 -07:00
Wei-Chiu Chuang 3a2f3e78ff HDFS-12182. BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks. Contributed by Wellington Chevreuil. 2017-09-05 14:58:08 -07:00
Xiao Chen 858715d054 HDFS-12336. Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory. Contributed by Wellington Chevreuil. 2017-08-30 09:43:12 -07:00
Yongjun Zhang 870ef0b2fd HDFS-12191. Provide option to not capture the accessTime change of a file to snapshot if no other modification has been done to this file. Contributed by Yongjun Zhang. 
(cherry picked from commit cf93d60d3f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirAttrOp.java
 2017-08-29 21:57:25 -07:00
Arpit Agarwal e19f004f65 HDFS-11998. Enable DFSNetworkTopology as default. Contributed by Chen Liang. 2017-08-28 16:22:42 -07:00
Arpit Agarwal 98ba68b05d HDFS-12293. DataNode should log file name on disk error. Contributed by Ajay Kumar. 2017-08-28 15:50:06 -07:00
Yiqun Lin d63e98ad5f HDFS-11530. Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault. Contributed by Chen Liang. 2017-08-28 20:47:38 +08:00
Lei Xu c22b5108df HDFS-12215. DataNode.transferBlock does not create its daemon in the xceiver thread group. (Lei Xu) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
 2017-08-25 16:06:34 -07:00
Kihwal Lee 5a83ffa396 HDFS-12299. Race Between update pipeline and DN Re-Registration. Contributed by Brahma Reddy Battula. 2017-08-25 14:53:53 -05:00
Arpit Agarwal 2cae387402 HDFS-12319. DirectoryScanner will throw IllegalStateException when Multiple BP's are present. Contributed by Brahma Reddy Battula. 2017-08-25 10:41:59 -07:00
Arpit Agarwal ae69c99068 HDFS-11577. Combine the old and the new chooseRandom for better performance. Contributed by Chen Liang. 2017-08-24 11:17:36 -07:00
Arpit Agarwal 8b2d5a4faa HDFS-11514. DFSTopologyNodeImpl#chooseRandom optimizations. Contributed by Chen Liang. 2017-08-23 12:49:13 -07:00
John Zhuge c54c3500ed HDFS-11738. Hedged pread takes more time when block moved from initial locations. Contributed by Vinayakumar B. 
(cherry picked from commit b6bfb2fcb2)
 2017-08-23 19:44:52 +05:30
Arpit Agarwal b3ea11dfdb HDFS-11482. Add storage type demand to into DFSNetworkTopology#chooseRandom. Contributed by Chen Liang. 2017-08-21 14:07:59 -07:00
Manoj Govindassamy 89fc7fe670 HDFS-11988. Verify HDFS Snapshots with open files captured are consistent across truncates and appends to current version file. 2017-08-21 11:14:44 -07:00
John Zhuge 76fbeed997 HDFS-12117. HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST Interface. Contributed by Wellington Chevreuil. 2017-08-18 22:06:02 -07:00
Arpit Agarwal d8d2a221f8 HDFS-11450. HDFS specific network topology classes with storage type info included. Contributed by Chen Liang. 2017-08-18 11:43:17 -07:00
Manoj Govindassamy a5bc30170c HDFS-12316. Verify HDFS snapshot deletion doesn't crash the ongoing file writes. 
(cherry picked from commit 4230872dd6)
 2017-08-17 16:25:15 -07:00
Mingliang Liu 03d15fa58e HDFS-11430. Separate class InnerNode from class NetworkTopology and make it extendable. Contributed by Tsz Wo Nicholas Sze 2017-08-16 16:06:13 -07:00
Ravi Prakash 895a35eac0 HDFS-12301. NN File Browser UI: Navigate to a path when enter is pressed 
(cherry picked from commit f34646d652)
 2017-08-15 15:45:37 -07:00
John Zhuge ed4d6aa2c1 HDFS-11303. Hedged read might hang infinitely if read data from all DN failed . Contributed by Chen Zhang, Wei-chiu Chuang, and John Zhuge. 
(cherry picked from commit 8b242f09a6)
 2017-08-11 21:55:23 -07:00
Kihwal Lee 306abf4980 HDFS-12278. LeaseManager operations are inefficient in 2.8. Contributed by Rushabh S Shah. 
(cherry picked from commit b5c02f95b5)
 2017-08-09 16:48:55 -05:00
Kihwal Lee 74bcc8d1f2 HDFS-12157. Do fsyncDirectory(..) outside of FSDataset lock. Contributed by Vinayakumar B. 2017-08-09 09:31:53 -05:00
Haohui Mai 3e2753daa1 HADOOP-14598. Blacklist Http/HttpsFileSystem in FsUrlStreamHandlerFactory. Contributed by Steve Loughran. 2017-08-08 16:33:54 -07:00
Haohui Mai 21d76f710e HDFS-10326. Disable setting tcp socket send/receive buffers for write pipelines. Contributed by Daryn Sharp. 2017-08-08 14:58:44 -07:00
Andrew Wang 2da20df1dd HDFS-12131. Add some of the FSNamesystem JMX values as metrics. Contributed by Erik Krogen. 2017-08-03 15:45:21 -07:00
Manoj Govindassamy 065a906be7 HDFS-9388. Decommission related code to support Maintenance State for datanodes. 2017-08-02 16:58:54 -07:00
Arpit Agarwal 45cde760b0 HDFS-12043. Add counters for block re-replication. Contributed by Chen Liang. 2017-08-02 16:45:08 -07:00
Manoj Govindassamy a77ae7abdb HDFS-12217. HDFS snapshots doesn't capture all open files when one of the open files is deleted. 
(cherry picked from commit 52d7bafcf4)
 2017-08-02 10:16:30 -07:00
Jason Lowe f34c01b92a Revert "HDFS-12217. HDFS snapshots doesn't capture all open files when one of the open files is deleted." 
This reverts commit b0330c15da.
 2017-08-02 11:15:58 -05:00
Manoj Govindassamy b0330c15da HDFS-12217. HDFS snapshots doesn't capture all open files when one of the open files is deleted. 
(cherry picked from commit 52d7bafcf4)
 2017-08-01 17:15:29 -07:00
Manoj Govindassamy 70c511598f HDFS-11402. HDFS Snapshots should capture point-in-time copies of OPEN files. 2017-08-01 17:11:27 -07:00
Akira Ajisaka 72bde1dad5
HADOOP-14245. Use Mockito.when instead of Mockito.stub. Contributed by Andras Bokor. 
(cherry picked from commit b38a1eea8e)
 2017-08-01 15:16:21 +09:00
Lei Xu f09d20cffb HADOOP-14397. Pull up the builder pattern to FileSystem and add AbstractContractCreateTest for it. (Lei (Eddy) Xu) 
(cherry picked from commit 667ee003bf47e44beb3fdff8d06a7264a13dd22c)
 2017-07-31 20:07:13 -07:00
Weiwei Yang 60ae10b14f HDFS-12154. Incorrect javadoc description in StorageLocationChecker#check. Contributed by Nandakumar. 2017-07-31 17:02:54 -07:00
Arpit Agarwal 8c63580981 HDFS-12082. BlockInvalidateLimit value is incorrectly set after namenode heartbeat interval reconfigured. Contributed by Weiwei Yang. 2017-07-31 11:34:00 -07:00
Brahma Reddy Battula 8bfb9971ca HADOOP-14455. ViewFileSystem#rename should support be supported within same nameservice with different mountpoints. Contributed by Brahma Reddy Battula. 2017-07-30 14:01:41 +08:00
Yongjun Zhang c3e8c7d56e HDFS-12190. Enable 'hdfs dfs -stat' to display access time. Contributed by Yongjun Zhang. 
(cherry picked from commit c6330f22a5)
 2017-07-27 17:05:30 -07:00
Brahma Reddy Battula 1c5095b3cb HDFS-11896. Non-dfsUsed will be doubled on dead node re-registration. Contributed by Brahma Reddy Battula. 
(cherry picked from commit c4a85c694f)
 2017-07-27 12:30:57 -07:00
Kihwal Lee 3d8c7d701b HDFS-12171. Reduce IIP object allocations for inode lookup. Contributed by Daryn Sharp. 2017-07-25 11:07:43 -05:00
Kihwal Lee 9fe5dd098d HDFS-12143. Improve performance of getting and removing inode features. Contributed by Daryn Sharp. 
(cherry picked from commit 1a79dcfc45)
 2017-07-25 10:30:27 -05:00
Akira Ajisaka c108372777
HDFS-12176. dfsadmin shows DFS Used%: NaN% if the cluster has zero block. Contributed by Weiwei Yang. 
(cherry picked from commit 770cc46228)
 2017-07-24 16:24:50 +09:00
Kihwal Lee c12bf9a128 HDFS-11742. Improve balancer usability after HDFS-8818. Contributed by Kihwal Lee 
(cherry picked from commit 8e3a992ecc)
 2017-07-21 09:19:01 -05:00
Konstantin V Shvachko af2fb6a909 HDFS-11472. Fix inconsistent replica size after a data pipeline failure. Contributed by Erik Krogen and Wei-Chiu Chuang. 
(cherry picked from commit 2a5a313539)
 2017-07-20 15:57:34 -07:00
Yongjun Zhang 10a6b6b12c HDFS-12139. HTTPFS liststatus returns incorrect pathSuffix for path of file. Contributed by Yongjun Zhang. 
(cherry picked from commit 3556e36be3)
 2017-07-19 11:44:35 -07:00
Anu Engineer e0297ffbc8 HDFS-12158. Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection. Contributed by Mukul Kumar Singh. 
(cherry picked from commit 413b23eb04)
 2017-07-19 10:42:28 -07:00
Brahma Reddy Battula 979d37ae21 HDFS-12067. Correct dfsadmin commands usage message to reflects IPC port. Contributed by steven-wugang. 
(cherry picked from commit f8cd55fe33)
 2017-07-19 23:22:40 +08:00
Tsz-Wo Nicholas Sze 02627f8c5f HDFS-12138. Remove redundant 'public' modifiers from BlockCollection. Contributed by Chen Liang 2017-07-17 13:59:52 -07:00
Brahma Reddy Battula a33e89931b HDFS-12112. TestBlockManager#testBlockManagerMachinesArray sometimes fails with NPE. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit b778887af5)
 2017-07-15 10:49:11 +08:00
Xiao Chen 5337c2511d HADOOP-14521. KMS client needs retry logic. Contributed by Rushabh S Shah. 
(cherry picked from commit aa395c1ec9a170f2da732c6e6114c60412161d40)
 2017-07-14 22:23:07 -07:00
Tsz-Wo Nicholas Sze bff39ca718 HDFS-12130. Optimizing permission check for getContentSummary. Contributed by Chen Liang 2017-07-14 14:40:11 -07:00
Tsz-Wo Nicholas Sze d51395363a Revert "HDFS-12130. Optimizing permission check for getContentSummary." to fix commit message. 
This reverts commit 2ba41729ab.
 2017-07-14 14:38:27 -07:00
Kihwal Lee 0fd86de128 HDFS-12140. Remove BPOfferService lock contention to get block pool id. Contributed by Daryn Sharp. 
(cherry picked from commit e7d187a1b6)
 2017-07-14 16:12:28 -05:00
Kihwal Lee a47820d72a HDFS-12137. DN dataset lock should be fair. Contributed by Daryn Sharp. 2017-07-14 15:45:26 -05:00
Tsz-Wo Nicholas Sze 2ba41729ab HDFS-12130. Optimizing permission check for getContentSummary. 2017-07-14 13:42:02 -07:00
Xiaoyu Yao af8a227a68 HDFS-11502. Datanode UI should display hostname based on JMX bean instead of window.location.hostname. Contributed by Jeffrey E Rodriguez and Yuanbo Liu. 
(cherry picked from commit e15e2713e1)
 2017-07-12 16:38:56 -07:00
Tsz-Wo Nicholas Sze b4a108fa9f HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS. Contributed by Weiwei Yang 2017-07-12 14:18:22 -07:00
Akira Ajisaka 2653cf4466
HADOOP-14629. Improve exception checking in FileContext related JUnit tests. Contributed by Andras Bokor. 
(cherry picked from commit 9144fd9e9b)
 2017-07-12 11:40:48 +09:00
Mingliang Liu b803641523 HDFS-12089. Fix ambiguous NN retry log message in WebHDFS. Contributed by Eric Badger 
(cherry picked from commit 6436768baf)
 2017-07-05 11:12:15 -07:00
Akira Ajisaka 063f2132ef
HDFS-12078. Add time unit to the description of property dfs.namenode.stale.datanode.interval in hdfs-default.xml. Contributed by Weiwei Yang. 
(cherry picked from commit 186650d21d)
 2017-07-04 14:52:39 +09:00
Akira Ajisaka 003b4a1675
HDFS-12079. Description of dfs.block.invalidate.limit is incorrect in hdfs-default.xml. Contributed by Weiwei Yang. 
(cherry picked from commit b0560e0624)
 2017-07-04 14:03:07 +09:00
Wei-Chiu Chuang 94bc5cdbb3 HDFS-12042. Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption. Contributed by Misha Dmitriev. 
(cherry picked from commit bcba844d11)
 2017-06-30 11:42:51 -07:00
Wei-Chiu Chuang 6917c19167 HDFS-11881. NameNode consumes a lot of memory for snapshot diff report generation. Contributed by Manoj Govindassamy. 
(cherry picked from commit 16c8dbde57)
 2017-06-29 06:40:24 -07:00
Akira Ajisaka a896c0f11e
HDFS-12040. TestFsDatasetImpl.testCleanShutdownOfVolume fails. Contributed by hu xiaodong. 
(cherry picked from commit 07defa4c09)
 2017-06-27 20:49:56 +09:00
Ravi Prakash ef491f4637 HDFS-11993. Add log info when connect to datanode socket address failed. Contributed by chencan 
(cherry picked from commit a9d3412b4c)
 2017-06-26 13:25:32 -07:00
Akira Ajisaka b59b96deee
HDFS-11992. Replace commons-logging APIs with slf4j in FsDatasetImpl. Contributed by hu xiaodong. 2017-06-27 03:28:09 +09:00
Kihwal Lee 8f533ca76b HDFS-11960. Successfully closed files can stay under-replicated. Contributed by Kihwal Lee. 2017-06-20 09:16:37 -05:00
Brahma Reddy Battula 933fad2e17 HADOOP-14533. Size of args cannot be less than zero in TraceAdmin#run as its linkedlist. Contributed by Weisen Han. 
(cherry picked from commit 2e9daa2e27)
 2017-06-20 20:45:33 +08:00
Akira Ajisaka 6b3c13d543
HDFS-11345. Document the configuration key for FSNamesystem lock fairness. Contributed by Erik Krogen. 
(cherry picked from commit 2c769167db)
 2017-06-20 16:36:47 +09:00
Akira Ajisaka 71626fd0b9
HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin. 2017-06-20 13:55:42 +09:00
Masatake Iwasaki b19ca498ae HDFS-11995. HDFS Architecture documentation incorrectly describes writing to a local temporary file. Contributed by Nandakumar. 
(cherry picked from commit d954a64730)
 2017-06-20 08:09:09 +09:00
Andrew Wang 1ec1aebd35 HDFS-10480. Add an admin command to list currently open files. Contributed by Manoj Govindassamy. 2017-06-19 09:52:07 -07:00
Lei Xu 33afa1fdca HADOOP-14395. Provide Builder pattern for DistributedFileSystem.append. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 6460df21a0)

    Conflicts:
    hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
 2017-06-16 17:32:29 -07:00
Arpit Agarwal 9529513f18 HADOOP-14503. Make RollingAverages a mutable metric. Contributed by Hanisha Koneru. 2017-06-16 16:41:08 -07:00
Xiao Chen 2afe9722af HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up proactively. 2017-06-16 09:49:41 -07:00
Lei Xu abac844c90 HADOOP-14394. Provide Builder pattern for DistributedFileSystem.create. (lei) 
(cherry picked from commit 5fbec46525d6d49837d934556b59ba77bd2301a8)

    Conflicts:
    hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java
 2017-06-15 10:59:11 -07:00
Akira Ajisaka 0fd0612896
Revert "HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin." 
This reverts commit 88fa17c238.
 2017-06-16 00:21:54 +09:00
Akira Ajisaka 88fa17c238
HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin. 
(cherry picked from commit 315f07700d)
 2017-06-15 13:47:22 +09:00
Tsz-Wo Nicholas Sze 670224ae59 HDFS-11947. When constructing a thread name, BPOfferService may print a bogus warning message. Contributed by Weiwei Yang 2017-06-13 09:47:14 +08:00
Arpit Agarwal eda4bf5196 HDFS-11967. TestJMXGet fails occasionally. Contributed by Arpit Agarwal. 2017-06-12 16:04:08 -07:00
Arpit Agarwal 097b1f74e5 HDFS-11907. Add metric for time taken by NameNode resource check. Contributed by Chen Liang. 2017-06-12 15:22:36 -07:00
Mingliang Liu facb6812d9 HDFS-11945. Internal lease recovery may not be retried for a long time. Contributed by Kihwal Lee 2017-06-08 14:46:31 -07:00
Brahma Reddy Battula fca08f8362 HDFS-11711. DN should not delete the block On "Too many open files" Exception. Contributed by Brahma Reddy Battula. 2017-06-07 16:54:29 +08:00
Yiqun Lin 72f0fdfb59 HDFS-11929. Document missing processor of hdfs oiv_legacy command. Contributed by LiXin Ge. 
(cherry picked from commit 5ec71639cf)
 2017-06-07 13:44:33 +08:00
Vinayakumar B 9a26120809 HDFS-11708. Positional read will fail if replicas moved to different DNs after stream is opened. Contributed by Vinayakumar B. 2017-06-07 10:57:47 +05:30
Andrew Wang 5c819585f2 HDFS-11840. Log HDFS Mover exception message of exit to its own log. Contributed by LiXin Ge. 
(cherry picked from commit c31cb879a3)
 2017-06-06 14:57:51 -07:00
Arpit Agarwal 226688a12c HDFS-11932. BPServiceActor thread name is not correctly set. Contributed by Chen Liang. 2017-06-06 13:51:25 -07:00
Yongjun Zhang 1e52314117 HDFS-11914. Add more diagnosis info for fsimage transfer failure. Contributed by Yongjun Zhang. 
(cherry picked from commit 6a28a2b914)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java
 2017-06-06 12:31:11 -07:00
Kihwal Lee e2ea601231 HDFS-10816. TestComputeInvalidateWork#testDatanodeReRegistration fails due to race between test and replication monitor. Contributed by Eric Badger. 2017-06-05 15:24:54 -05:00
Arpit Agarwal baff06f019 HDFS-11928. Segment overflow in FileDistributionCalculator. Contributed by LiXin Ge. 2017-06-05 13:21:35 -07:00
Yiqun Lin 2e8557de12 HDFS-11899. ASF License warnings generated intermittently in trunk. Contributed by Yiqun Lin. 
(cherry picked from commit 367da9b7b8)
 2017-06-03 22:10:01 +08:00
Brahma Reddy Battula 8e119f1e90 HADOOP-14436. Remove the redundant colon in ViewFs.md. Contributed by maobaolong. 
(cherry picked from commit 056cc72885)
 2017-06-02 22:42:23 +08:00
Brahma Reddy Battula ac2af66e77 HDFS-11905. Fix license header inconsistency in hdfs. Contributed by Yeliang Cang. 2017-06-02 22:22:43 +08:00
Yiqun Lin 870b5e39d2 HDFS-11359. DFSAdmin report command supports displaying maintenance state datanodes. Contributed by Yiqun Lin. 2017-06-02 12:53:31 +08:00
Andrew Wang d0a4e7e95d HDFS-11383. Intern strings in BlockLocation and ExtendedBlock. Contributed by Misha Dmitriev. 
(cherry picked from commit 7101477e47)
 2017-06-01 15:20:22 -07:00
Xiao Chen eb98773ac8 HDFS-11904. Reuse iip in unprotectedRemoveXAttrs calls. 
(cherry picked from commit 219f4c199e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java
 2017-06-01 14:16:58 -07:00
Xiao Chen 91b72a8aa5 HDFS-11741. Long running balancer may fail due to expired DataEncryptionKey. Contributed by Wei-Chiu Chuang and Xiao Chen. 
(cherry picked from commit 068e23b896c63b0f817e6b91d73c994be1551eb2)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2017-06-01 14:05:43 -07:00
Brahma Reddy Battula 5cbdefbcc0 HDFS-11893. Fix TestDFSShell.testMoveWithTargetPortEmpty failure. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0dcf843c00)
 2017-06-01 22:39:22 +08:00
Kihwal Lee 212a566088 HDFS-11856. Ability to re-add Upgrading nodes to pipeline for future pipeline updates. Contributed by Vinayakumar B. 2017-05-31 12:45:33 -05:00
Kihwal Lee 5b0baeab5e HDFS-5042. Completed files lost after power failure. Contributed by Vinayakumar B. 2017-05-31 11:02:34 -05:00
Lei Xu a3ad1a39c5 HDFS-11659. TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten fail due to no DataNode available for pipeline recovery. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 91d6fe151f)
 2017-05-30 11:11:00 -07:00
Yiqun Lin d36e64b8b2 HDFS-11446. TestMaintenanceState#testWithNNAndDNRestart fails intermittently. Contributed by Yiqun Lin. 
(cherry picked from commit 31058b243e)
 2017-05-28 11:27:09 +08:00
Inigo Goiri e336d92b99 Fix NPE in LazyPersistFileScrubber. Contributed by Inigo Goiri. 
(cherry picked from commit 303c8dc9b6)
 2017-05-26 13:40:16 -07:00
Xiao Chen 54971c4195 HDFS-11421. Make WebHDFS' ACLs RegEx configurable. Contributed by Harsh J. 2017-05-25 21:01:39 -07:00
Kihwal Lee 2cb63433ab HDFS-11817. A faulty node can cause a lease leak and NPE on accessing data. Contributed by Kihwal Lee. 2017-05-25 17:21:56 -05:00
Brahma Reddy Battula 7bad748091 HDFS-11445. FSCK shows overall health stauts as corrupt even one replica is corrupt. Contributed by Brahma Reddy Battula. 2017-05-25 22:38:12 +08:00
Yongjun Zhang 49dc547252 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-05-24 18:54:51 -07:00
Yongjun Zhang 97317b1aaa Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li." 
This reverts commit c411adac32.
 2017-05-24 18:54:51 -07:00
Wei-Chiu Chuang bd0138ea0a Revert "HDFS-10797. Disk usage summary of snapshots causes renamed blocks to get counted twice. Contributed by Sean Mackrory." 
This reverts commit 5c96ef365d.
 2017-05-24 18:08:30 -07:00
Wei-Chiu Chuang fe2c28df70 Revert "HDFS-11515. -du throws ConcurrentModificationException. Contributed by Istvan Fajth, Wei-Chiu Chuang." 
This reverts commit 4f1f16f1a6.
 2017-05-24 18:08:10 -07:00
Sunil G ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Brahma Reddy Battula ec2468ced1 HDFS-11849. JournalNode startup failure exception should be logged in log file. Contributed By Surendra Singh Lilhore. 
(cherry picked from commit 9cab42cc79)
 2017-05-23 02:40:11 +08:00
John Zhuge 85f7b7e8e4 HDFS-11579. Make HttpFS Tomcat SSL property sslEnabledProtocols and clientAuth configurable. Contributed by John Zhuge. 2017-05-18 18:04:05 -07:00
Arpit Agarwal 3db695d400 HDFS-11842. TestDataNodeOutlierDetectionViaMetrics UT fails intermittently. Controbuted by Hanisha Koneru. 2017-05-17 19:38:54 -07:00
Arpit Agarwal 0e95ff78c3 HDFS-11827. NPE is thrown when log level changed in BlockPlacementPolicyDefault#chooseRandom() method. Contributed by xupeng. 2017-05-16 19:51:17 -07:00
Kihwal Lee 82ea3f4545 HDFS-11641. Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus. Contributed by Daryn Sharp. 2017-05-16 15:45:41 -05:00
Akira Ajisaka c17cb03a23
HDFS-11833. HDFS architecture documentation descibes outdated placement policy. Contributed by Chen Liang. 
(cherry picked from commit 1d1c52b42f)
 2017-05-16 11:53:55 -04:00
John Zhuge da7c900fe3 HDFS-11816. Update default SSL cipher list for HttpFS. Contributed by John Zhuge. 2017-05-15 16:26:39 -07:00
Andrew Wang 18b89e902b HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-13 21:41:11 -07:00
Jason Lowe aeca471ec1 HDFS-11818. TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently. Contributed by Nathan Roberts 2017-05-12 17:47:17 -05:00