Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,431 Commits 326 Branches 371 Tags
Commit Graph

3855 Commits

Author SHA1 Message Date
Akira Ajisaka
d1fb4948d2
HADOOP-14629. Improve exception checking in FileContext related JUnit tests. Contributed by Andras Bokor. 
(cherry picked from commit 9144fd9e9b5d84d71158451428341746a6567152)
(cherry picked from commit 2653cf4466ab99960ea65a937ea01fc4d4ca2f00)
 2017-07-12 11:44:15 +09:00
Akira Ajisaka
9f9d554edf
HDFS-12078. Add time unit to the description of property dfs.namenode.stale.datanode.interval in hdfs-default.xml. Contributed by Weiwei Yang. 
(cherry picked from commit 186650d21d482e2f3bc4523ae989ebe76081b0e3)
(cherry picked from commit 063f2132ef54436e8a537d437e7a39256a74214f)
 2017-07-04 14:53:02 +09:00
Wei-Chiu Chuang
8027246306 HDFS-12042. Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption. Contributed by Misha Dmitriev. 
(cherry picked from commit bcba844d1144cc334e2babbc34c9d42eac1c203a)
(cherry picked from commit 94bc5cdbb3d0fd9133a8cab9d3daac6798e8d8dd)
 2017-06-30 11:43:41 -07:00
Wei-Chiu Chuang
7c076f917f HDFS-11881. NameNode consumes a lot of memory for snapshot diff report generation. Contributed by Manoj Govindassamy. 
(cherry picked from commit 16c8dbde574f49827fde5ee9add1861ee65d4645)
(cherry picked from commit 6917c19167060df7879b0e0ead4db1ded62714b3)
 2017-06-29 06:44:37 -07:00
Kihwal Lee
a560104fab HDFS-11960. Successfully closed files can stay under-replicated. Contributed by Kihwal Lee. 
(cherry picked from commit 8f533ca76b3f3592dc8ec6e318382a54becf10f5)
 2017-06-20 12:12:33 -05:00
Kihwal Lee
e87bf17129 HDFS-9754. Avoid unnecessary getBlockCollection calls in BlockManager. Contributed by Jing Zhao. 
(cherry picked from commit 972782d9568e0849484c027f27c1638ba50ec56e)
(cherry picked from commit f8c9c0ff0e2d977fc0f69bde4cdbb03371c0bac4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMetadataConsistency.java
 2017-06-20 12:10:43 -05:00
Kihwal Lee
1f9255672d Revert "HDFS-11960. Successfully closed files can stay under-replicated. Contributed by Kihwal Lee." 
This reverts commit 40e3cd24b068d08be8cf05668290b908f6b43c11.
 2017-06-20 09:22:25 -05:00
Kihwal Lee
40e3cd24b0 HDFS-11960. Successfully closed files can stay under-replicated. Contributed by Kihwal Lee. 
(cherry picked from commit 8f533ca76b3f3592dc8ec6e318382a54becf10f5)
 2017-06-20 09:20:02 -05:00
Brahma Reddy Battula
a526c3b0bf HADOOP-14533. Size of args cannot be less than zero in TraceAdmin#run as its linkedlist. Contributed by Weisen Han. 
(cherry picked from commit 2e9daa2e27536484846d9c5f020c6afaf27932ce)
 2017-06-20 20:46:21 +08:00
Akira Ajisaka
189e39a90d
HDFS-11345. Document the configuration key for FSNamesystem lock fairness. Contributed by Erik Krogen. 
(cherry picked from commit 2c769167dbdb66c52d2ba7b7193a686444085570)
(cherry picked from commit 6b3c13d543ddaba3b4a2e6a5649420982a107603)
 2017-06-20 16:41:10 +09:00
Akira Ajisaka
6a4a63b0fd
HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin. 
(cherry picked from commit 71626fd0b92ee70ee113fde237352266aa599bfb)
 2017-06-20 13:56:20 +09:00
Masatake Iwasaki
6032511239 HDFS-11995. HDFS Architecture documentation incorrectly describes writing to a local temporary file. Contributed by Nandakumar. 
(cherry picked from commit d954a64730c00346476322743462cde857164177)
 2017-06-20 09:25:05 +09:00
Akira Ajisaka
4eb577c9f0
Revert "HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin." 
This reverts commit a52315d2c3f758fff2be75e42928576752616bcd.
 2017-06-16 00:22:24 +09:00
Akira Ajisaka
a52315d2c3
HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin. 
(cherry picked from commit 315f07700dcfe32513b87f63070db854ecb3cf48)
(cherry picked from commit 88fa17c2383baafbfb6b43b69e08cf40493c774a)
 2017-06-15 13:47:40 +09:00
Arpit Agarwal
6ca195997b HDFS-11967. TestJMXGet fails occasionally. Contributed by Arpit Agarwal. 2017-06-12 16:04:02 -07:00
Mingliang Liu
72d895a11f HDFS-11945. Internal lease recovery may not be retried for a long time. Contributed by Kihwal Lee 
(cherry picked from commit facb6812d9004ae509b2564150bbe28317a293f1)
 2017-06-08 14:49:37 -07:00
wenxin he
2c12102edd
HADOOP-14208. Fix typo in the top page in branch-2.8 
This closes #230

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2017-06-08 11:36:24 +09:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
396de0fd8f Preparing for 2.8.3 development 2017-06-07 15:38:20 -07:00
Brahma Reddy Battula
8a16846f5d HDFS-11711. DN should not delete the block On "Too many open files" Exception. Contributed by Brahma Reddy Battula. 
(cherry picked from commit fca08f8362f0332b064f28c2625fd535004ef85d)
 2017-06-07 16:59:26 +08:00
Vinayakumar B
4a391c72d4 HDFS-11708. Positional read will fail if replicas moved to different DNs after stream is opened. Contributed by Vinayakumar B. 2017-06-07 10:59:04 +05:30
Arpit Agarwal
5fc4b8567b HDFS-11932. BPServiceActor thread name is not correctly set. Contributed by Chen Liang. 2017-06-06 13:52:20 -07:00
Kihwal Lee
1d3fa87011 HDFS-10816. TestComputeInvalidateWork#testDatanodeReRegistration fails due to race between test and replication monitor. Contributed by Eric Badger. 
(cherry picked from commit e2ea6012311c927cd7bc1524f811411fda4fd6cc)
 2017-06-05 15:28:01 -05:00
Xiao Chen
ec10a81bdb HDFS-11741. Long running balancer may fail due to expired DataEncryptionKey. Contributed by Wei-Chiu Chuang and Xiao Chen. 
(cherry picked from commit 068e23b896c63b0f817e6b91d73c994be1551eb2)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
(cherry picked from commit 6f2391e0ad6fbaa6786cc9f0e3a09b955e0cb21c)
 2017-06-01 14:05:31 -07:00
Kihwal Lee
89d59c292f HDFS-5042. Completed files lost after power failure. Contributed by Vinayakumar B. 2017-06-01 12:25:27 -05:00
Brahma Reddy Battula
0cba282261 HDFS-11893. Fix TestDFSShell.testMoveWithTargetPortEmpty failure. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0dcf843c008f2b9cece8c0a0cef78140398ac464)
 2017-06-01 22:40:43 +08:00
Kihwal Lee
fe6de5e594 HDFS-11856. Ability to re-add upgrading nodes to pipeline for future pipeline updates. Contributed by Vinayakumar B. 2017-05-31 13:08:51 -05:00
Inigo Goiri
a20789f299 Fix NPE in LazyPersistFileScrubber. Contributed by Inigo Goiri. 
(cherry picked from commit 303c8dc9b6c853c0939ea9ba14388897cc258071)
 2017-05-26 13:42:55 -07:00
Kihwal Lee
bbef16b84e HDFS-11817. A faulty node can cause a lease leak and NPE on accessing data. Contributed by Kihwal Lee. 
(Updated TestBlockUnderConstruction for nextGenerationStamp method)
(cherry picked from commit 2cb63433abb21cd2b74bd266b70a682caf9e2d98)
 2017-05-25 17:34:56 -05:00
Brahma Reddy Battula
4f5846f1e3 HDFS-11445. FSCK shows overall health stauts as corrupt even one replica is corrupt. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 7bad748091b6a5f90e919c781a915e13000e1be6)
 2017-05-25 22:40:11 +08:00
Wei-Chiu Chuang
caad191c0e Revert "HDFS-10797. Disk usage summary of snapshots causes renamed blocks to get counted twice. Contributed by Sean Mackrory." 
This reverts commit 998e44a972ad06b48675afac89656ffba8c4d595.
 2017-05-24 18:12:03 -07:00
Wei-Chiu Chuang
6c50341ea8 Revert "HDFS-11515. -du throws ConcurrentModificationException. Contributed by Istvan Fajth, Wei-Chiu Chuang." 
This reverts commit f9d3bb3b8335889b30691baca4331f6f9ed28f69.
 2017-05-24 18:11:42 -07:00
Brahma Reddy Battula
a7f063576c HDFS-11849. JournalNode startup failure exception should be logged in log file. Contributed By Surendra Singh Lilhore. 
(cherry picked from commit 9cab42cc797986081fef184748044f1790a4f039)
 2017-05-24 23:45:57 +08:00
Yiqun Lin
94a80974d3 HDFS-11377. Balancer hung due to no available mover threads. Contributed by yunjiong zhao. 
(cherry picked from commit 9cbbd1eae893b21212c9bc9e6745c6859317a667)
 2017-05-22 17:11:52 -07:00
Kihwal Lee
95e8f4c80d HDFS-11641. Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus. Contributed by Daryn Sharp. 
(cherry picked from commit 82ea3f4545c88f2dc106e63afd6fcd616bb120be)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2017-05-16 15:55:53 -05:00
Akira Ajisaka
9b595ba40b
HDFS-11833. HDFS architecture documentation descibes outdated placement policy. Contributed by Chen Liang. 
(cherry picked from commit 1d1c52b42feae5a4271ef4b771d0d8de43e83c15)
(cherry picked from commit c17cb03a23f2ea9f1af8f9c147ac68d8441be935)
 2017-05-16 11:54:27 -04:00
Jason Lowe
f42222a75b HDFS-11818. TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently. Contributed by Nathan Roberts 
(cherry picked from commit aeca471ec13135064f9605174cd6daa120007b18)
 2017-05-12 17:48:00 -05:00
Vinayakumar B
b121e047b4 HDFS-11674. reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered (Contributed by Vinayakumar B) 
(cherry picked from commit 1411612aa4e70c704b941723217ed4efd8a0125b)
(cherry picked from commit 53d9f56a18214ff351384589ebfadd6afae2c62f)
 2017-05-12 07:47:57 +05:30
Kihwal Lee
a8f209dee7 HDFS-11755. Underconstruction blocks can be considered missing. Contributed by Nathan Roberts. 2017-05-10 14:22:17 -05:00
Jason Lowe
ef120bed7b HDFS-11745. Increase HDFS test timeouts from 1 second to 10 seconds. Contributed by Eric Badger 
(cherry picked from commit fab73748a1ce982904d332e6ddd46169f186ad54)
 2017-05-10 13:03:48 -05:00
Lei Xu
5b8564f986 HDFS-11687. Add new public encryption APIs required by Hive. (lei) 
Change-Id: I4a23a00de63ad18022312ceb1f306a87d032d07c
(cherry picked from commit 25f5d9ad5ee5ead349d259a99b49541a70b1604d)
(cherry picked from commit 1f30a8a105d178e0e554e85d713320bf5356e035)
 2017-05-09 12:11:48 -07:00
Kihwal Lee
9d8d3a84bd HDFS-11702. Remove indefinite caching of key provider uri in DFSClient. Contributed by Rushabh S Shah. 
(cherry picked from commit cef2815cf48154fe82f44082dcbdce6373c81284)
 2017-05-08 08:31:18 -05:00
Xiaoyu Yao
a80ac0822b HADOOP-14207. "dfsadmin -refreshCallQueue" fails with DecayRpcScheduler. Contributed by Surendra Singh Lihore. 
(cherry picked from commit 918dee10f56861d15bb6644edcefb4246a3e00a4)
(cherry picked from commit 81883e3150c42e866ba9e68d68b734f2772344b2)
 2017-05-04 16:03:39 -07:00
Erik Krogen
bcafaf2c61 HDFS-11717. Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen. 
(cherry picked from commit d9014bda93760f223789d2ec9f5e35f40de157d4)
(cherry picked from commit 9d6b873d3c249aa76420ab03ceefc6180295915f)
 2017-05-02 18:44:26 -07:00
Ming Ma
d1ae1fb449 HDFS-9922. Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes. (Chris Trezzo via mingma) 
(cherry picked from commit b48f27e794e42ba90836314834e872616437d7c9)
(cherry picked from commit 32b115da1d75b9b5b6a770371fbecfda43928bb9)
 2017-05-02 12:48:51 -07:00
Ming Ma
305a9d886a HDFS-9016. Display upgrade domain information in fsck. 2017-05-02 09:52:16 -07:00
Ming Ma
c4c5533216 HDFS-9005. Provide configuration support for upgrade domain. 2017-05-02 06:53:32 -07:00
Kihwal Lee
c9bf21b0f3 HDFS-11714. Newly added NN storage directory won't get initialized and cause space exhaustion. Contributed by Kihwal Lee. 2017-05-01 17:34:19 -05:00
Kihwal Lee
d1e6b6db7c HDFS-11609. Some blocks can be permanently lost if nodes are decommissioned while dead. Contributed by Kihwal Lee. 2017-05-01 14:28:27 -05:00
Konstantin V Shvachko
cb12ada03f HDFS-11384. Balancer disperses getBlocks calls to avoid NameNode's rpc queue saturation. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 28eb2aabebd15c15a357d86e23ca407d3c85211c)
 2017-04-27 11:42:48 -07:00
Zhe Zhang
70f18911c7 HDFS-11709. StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen. 
(cherry picked from commit d8a33098309f17dfb0e3a000934f68394de44bf7)
(cherry picked from commit fc46f398d9192e83c7197115589eca7805c49943)
 2017-04-27 09:54:10 -07:00