Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,844 Commits 326 Branches 371 Tags
Commit Graph

19844 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ayush Saxena
162923e799 HADOOP-17594. DistCp: Expose the JobId for applications executing through run method (#2786). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2021-03-19 14:21:02 +05:30
Ayush Saxena
8ec256db04 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:14:11 +05:30
Viraj Jasani
503101c1f1 HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348bea7f89f7c0fc86f2813fdf2d1929f)
 2021-03-17 15:14:29 +09:00
Takanobu Asanuma
b898058e45 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8e68766aec49e50ec4c86519c6f9889)
 2021-03-17 14:18:54 +09:00
touchida
b4c07209fc
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc74ab8a6bbc53305b7a49df2437406f2)
 2021-03-17 13:12:40 +09:00
Jim Brennan
a92300e0a9 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 
(cherry picked from commit 38f86cc8c72201b081a083cd32bc3a322c6d9398)
 2021-03-16 19:19:01 +00:00
Eric Payne
2439e07708 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7af122163789a731ced05f740562e45)
 2021-03-15 19:55:13 +00:00
He Xiaoqiao
0224dd87af HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0695fedd8269a2a30a74da43e1d0ae7)
 2021-03-14 15:20:16 +00:00
Ayush Saxena
77aedb7d4c HADOOP-17586. Upgrade org.codehaus.woodstox:stax2-api to 4.2.1. (#2769). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 Conflicts:
	LICENSE-binary
 2021-03-13 15:20:50 +05:30
Jim Brennan
a6c2fced53 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f304df673d709bbc6faa9a635651c404)
 2021-03-12 17:47:43 +00:00
Viraj Jasani
6ecf829e79
HADOOP-17571 : Bump up woodstox-core to 5.3.0 due to security concerns (#2757) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-11 12:28:02 -08:00
Hui Fei
7250f0e327 HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee46f61686bd7fd8b62d141399e9af4dc)
 2021-03-11 10:45:57 +08:00
Akira Ajisaka
be253c2fd0
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. 
(cherry picked from commit 63aa142b770eda31e83ca6c6e5e96cb802b8b6a9)
 2021-03-09 11:11:32 +09:00
tomscut
7b925bc486 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106802c78b1400fba9f1d1c7e772dd5e7)
 2021-03-06 14:18:20 -08:00
Akira Ajisaka
5dda43c061
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa58610c04bdc1ac7570476accdc25b7)
 2021-03-04 14:57:41 +09:00
litao
c503130d8b HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380bce2ce7d8918e0772a5762fdba3122)
 2021-03-03 22:38:04 +09:00
Eric Badger
14dea40154 [MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file. 
Contributed by Eric Badger.

(cherry picked from commit a17dc4e8218b77e7fb5b5e253cb2d20014073b49)
 2021-03-02 21:50:12 +00:00
Konstantin V Shvachko
78bd68a0b8 HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4789188f744f3a1fba7f1a9bb9b8aa4)
 2021-03-01 17:27:38 -08:00
S O'Donnell
53f46214e4 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee475e5f37f85946be1a42a1b1e622a5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java

(cherry picked from commit 707806092f64f709814e18cb0d1b4f58731a5716)
 2021-03-01 12:26:41 +00:00
Jim Brennan
4d323cd47c MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 21:39:04 +00:00
Eric Badger
5862543c89 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec45ded55e8b7fdc29f8caf85547cca3)
 2021-02-26 19:36:55 +00:00
Surendra Singh Lilhore
379a7a5726 HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3959220c6f985f940e2e731f99d8e30)
(cherry picked from commit 7856af2cc65377352db5ebc93a778f373f71d215)
(cherry picked from commit 5315de3d437f78d81dc5fa9f43849de6198220e6)
 2021-02-26 10:49:59 +08:00
Jonathan Hung
eb6c08e423 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
(cherry picked from commit be6e99963ded94adf6f447ff53f2ba66b99120ca)
(cherry picked from commit 6863a5bb8ace591de3374102920bba916dbebfda)
 2021-02-25 15:35:01 -08:00
Jim Brennan
0a45b5460d [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit 3795f66364e39d9e7bafacd54d16bb60ab1d2188)
 2021-02-25 20:09:27 +00:00
S O'Donnell
bf229135ca HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5d0024ab052429ef2f0c0b20e1ea258)
 2021-02-24 11:33:43 +00:00
Renukaprasad C
2cf469eb74 HADOOP-17510. Hadoop prints sensitive Cookie information. (#2673) 
(cherry picked from commit 924fa87010ac193a8242d59b14f9fa9d17957fe5)
(cherry picked from commit 9951ea9fa8427c134861f047acc164b5cee4eef0)
 2021-02-24 17:34:26 +08:00
Eric Badger
ea73b6ad5c YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3edb9076d3aac7775de8712a7685dc135)
 2021-02-22 19:15:51 +00:00
Xieming Li
d6797520b9
HADOOP-16052. Remove Subversion and Forrest from Dockerfile 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit bdbca0e53b4e0e728c71bc51f99e035d9081c7b1)
 2021-02-22 23:34:17 +09:00
Akira Ajisaka
4cbab66547
MAPREDUCE-7323. Remove job_history_summary.py. (#2712) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655d91d949f8aa8daae656f34ed950e78)
 2021-02-22 23:16:53 +09:00
Wang Yuxuan
09bbbc8b86 HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2fa83ad4119954b32a3a8daac9d08134)
(cherry picked from commit 52a46f064a70505034367cffc32dfa1fabc7134b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
(cherry picked from commit fe822b525e4b9d5ebe58eb5f6d81af717f6290f3)
 2021-02-22 18:16:12 +08:00
jianghuazhu
cab9172d2e HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8bf96cf5a1ec75134fb67fdeb955e5b6)
(cherry picked from commit ee0e32abdcfd08582e4545c30e7dca120e5dafd3)
(cherry picked from commit efb796f45fb22a83587da941c5ba70efd34af1b9)
 2021-02-22 15:25:08 +08:00
Kengo Seki
7bcb1a83a3
HADOOP-16764. Rewrite Python example codes using Python3 (#1762) 
(cherry picked from commit fd7de2b82a6b9d600b43909a0770a6b0afea899a)
 2021-02-20 19:16:07 +09:00
Eric Badger
9327039fb8 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b2bdf36589da1174d4d8ee29acc57f6)
 2021-02-19 23:44:28 +00:00
Jim Brennan
1b5de609ec [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 
(cherry picked from commit e6f5dbbe7f556e3525293814ad2239e4f04d88ad)
 2021-02-16 17:20:52 +00:00
Zehao Chen
e162ef84dc HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd27c69b86b837e8da320bad0031f7895)
 2021-02-15 19:41:30 +00:00
Jim Brennan
d8d8546678 YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) Contributed by Masatake Iwasaki 
(cherry picked from commit 2117ab6b7125bfcbdae907015001a7599e2f0c4c)
 2021-02-11 21:47:41 +00:00
Akira Ajisaka
3675f60354
HADOOP-17516. Upgrade ant to 1.10.9. (#2683) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 933fbeffee1e9e2b9d78b5be31466d4d640768c5)
 2021-02-11 08:35:50 +09:00
Eric Badger
9c620c6302 MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log. 
Contributed by Jim Brennan.

(cherry picked from commit 462561654bb37dc1d453bfe15c3e75c274bbfb02)
 2021-02-09 19:23:23 +00:00
Jim Brennan
0661977821 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 23:01:50 +00:00
fermi
e4d5ac227c HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c8932c54ced1f0d4b892a54d22d7cfaf3)
 2021-02-05 11:18:09 +08:00
Hui Fei
6e5cc45a16 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc02af81e89e9fb96691f6cdf19ceff5)
 2021-02-04 10:01:55 +08:00
crossfire
69635e09d2 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e204d105fb05807d33387a048e9ddb762)
 2021-02-02 10:44:59 +00:00
Akira Ajisaka
5302731ac4
HADOOP-17354. Move Jenkinsfile outside of the root directory. (#2647) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 84b154ebc054843234e9a3c4c27748f87582807f)
 2021-02-01 18:53:42 +09:00
He Xiaoqiao
3a835753ee
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 20:00:41 +08:00
Jim Brennan
dfe23ab6ba [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939332c2b17a0f470ac650ab55c6c0f64)
 2021-01-28 18:41:09 +00:00
Jim Brennan
7943b31c29 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f670c181d1fda99e76729e920fff287e)
 2021-01-28 17:14:27 +00:00
Akira Ajisaka
fb5b97305c
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a2e9f48050e0c41440b36c759ec068f)
 2021-01-24 22:58:06 +09:00
Akira Ajisaka
0eee9bda61
HDFS-15731. Reduce threadCount for unit tests to reduce the memory usage (#2556) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9cf408d6baf2fa9071729cac57516d366e928a9f)
 2021-01-24 22:53:42 +09:00
Anton Kutuzov
397ff302b3 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5b85379303ac8fb2a1a03ba10b07d4e)
 2021-01-22 18:16:21 -08:00
He Xiaoqiao
d1d79488fd
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:45:17 +08:00