Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,721 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

21721 Commits

Author SHA1 Message Date
Akira Ajisaka 6a24757718
HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de2550)
(cherry picked from commit 0c1ba49c37)
 2021-03-12 01:25:20 +09:00
Akira Ajisaka 0b278502e2
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml

(cherry picked from commit de2904f123)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
	hadoop-cloud-storage-project/hadoop-cos/pom.xml
	hadoop-common-project/hadoop-registry/pom.xml
	hadoop-project/pom.xml
 2021-03-11 15:37:56 +09:00
Hui Fei 01f9d73dad HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee4)
 2021-03-11 10:22:34 +08:00
Akira Ajisaka 8d71ee12d2
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. 
(cherry picked from commit 63aa142b77)
 2021-03-09 11:11:10 +09:00
Eric Badger f30d1092bc YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.
 2021-03-08 20:20:52 +00:00
Peter Bacsko 0b8bfc50c6 YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth. 2021-03-08 14:32:13 +01:00
Peter Bacsko 0d032d7b68 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-03-08 14:22:14 +01:00
tomscut 03ac2e41c0 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106)
 2021-03-06 14:17:10 -08:00
Takanobu Asanuma 7a8542b833 HADOOP-17563. Update Bouncy Castle to 1.68. (#2740) 
(cherry picked from commit 0774116756)
 2021-03-05 23:01:49 +09:00
Akira Ajisaka 4fa57485e7
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa)
 2021-03-04 14:57:19 +09:00
litao 92e29d57d9 HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380)
 2021-03-03 22:37:23 +09:00
Eric Badger 86b387622a [MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file. 
Contributed by Eric Badger.

(cherry picked from commit a17dc4e821)
 2021-03-02 21:49:40 +00:00
Konstantin V Shvachko 1d48f3b44c HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4)
 2021-03-01 17:24:21 -08:00
S O'Donnell 707806092f HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java
 2021-03-01 12:24:31 +00:00
Ahmed Hussein e4c638eb21 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a06)
 2021-02-26 20:38:13 +00:00
Eric Badger 49375c0b53 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec)
 2021-02-26 19:17:24 +00:00
Surendra Singh Lilhore 929411acce HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3)
(cherry picked from commit 7856af2cc65377352db5ebc93a778f373f71d215)
 2021-02-26 10:49:48 +08:00
Jonathan Hung 6863a5bb8a YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
(cherry picked from commit be6e99963d)
 2021-02-25 15:26:57 -08:00
Jim Brennan 3795f66364 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 20:07:30 +00:00
S O'Donnell f98bc24ab6 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5)
 2021-02-24 11:22:13 +00:00
Renukaprasad C 9951ea9fa8 HADOOP-17510. Hadoop prints sensitive Cookie information. (#2673) 
(cherry picked from commit 924fa87010)
 2021-02-24 17:34:14 +08:00
Akira Ajisaka 08652eeee7
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (branch-3.2) (#2717) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile

(cherry picked from commit ff8b79f890)
 2021-02-24 11:05:58 +09:00
Masatake Iwasaki 165eaafaba HADOOP-17538. Add kms-default.xml and httpfs-default.xml to site index. (#2716) 
(cherry picked from commit e9a3c2950e)
 2021-02-23 22:56:35 +00:00
Eric Badger 4ef5ed382b YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3ed)
 2021-02-22 19:07:51 +00:00
Xieming Li b1085bc874
HADOOP-16052. Remove Subversion and Forrest from Dockerfile 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit bdbca0e53b)
 2021-02-22 23:28:36 +09:00
Akira Ajisaka 0172ce763a
MAPREDUCE-7323. Remove job_history_summary.py. (#2712) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655)
 2021-02-22 23:16:32 +09:00
Wang Yuxuan fe822b525e HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f)
(cherry picked from commit 52a46f064a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
 2021-02-22 18:15:07 +08:00
Akira Ajisaka fa5c9a1886
HADOOP-16054. Update Dockerfile to use Bionic (#2713) 
(cherry picked from commit 81d8b71534)

 Conflicts:
	dev-support/docker/Dockerfile
 2021-02-22 17:08:25 +09:00
jianghuazhu efb796f45f HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b)
(cherry picked from commit ee0e32abdc)
 2021-02-22 15:24:57 +08:00
Kengo Seki 5abf19a773
HADOOP-16764. Rewrite Python example codes using Python3 (#1762) 
(cherry picked from commit fd7de2b82a)
 2021-02-20 19:15:16 +09:00
Eric Badger 2c25f80a9c YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b)
 2021-02-19 23:37:13 +00:00
Jim Brennan e6f5dbbe7f [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 2021-02-16 17:19:15 +00:00
Zehao Chen 49480b87e0 HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd2)
 2021-02-15 17:46:09 +00:00
Jim Brennan 2117ab6b71 RN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) Contributed by Masatake Iwasaki 2021-02-11 21:32:07 +00:00
Akira Ajisaka d465fa8fcd
HADOOP-17516. Upgrade ant to 1.10.9. (#2683) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 933fbeffee)
 2021-02-11 08:23:43 +09:00
Eric Badger dbd4ec5e2d MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log. 
Contributed by Jim Brennan.

(cherry picked from commit 462561654b)
 2021-02-09 19:11:45 +00:00
Jim Brennan d74e9453d9 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 
(cherry picked from commit 62389a5a04)
 2021-02-05 22:18:52 +00:00
Jim Brennan 6cc0eb3e30 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 
(cherry picked from commit c22c77af43)
 2021-02-05 17:52:16 +00:00
fermi 05f08c8482 HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c89)
 2021-02-05 11:15:50 +08:00
Akira Ajisaka 710a24b6cd
YARN-10036. Install yarnpkg and upgrade nodejs in Dockerfile (#1772) 
(cherry picked from commit bc825b9628)
 2021-02-04 14:28:25 +09:00
Hui Fei b043e966e4 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc)
 2021-02-04 10:00:08 +08:00
Kihwal Lee 7c18c77d16 HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 
(cherry picked from commit 182623d2bc)
 2021-02-03 09:03:31 -06:00
crossfire ba03f84812 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e20)
 2021-02-02 10:06:08 +00:00
Akira Ajisaka c62f90b4c5
HADOOP-17354. Move Jenkinsfile outside of the root directory. (#2647) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 84b154ebc0)
 2021-02-01 18:53:17 +09:00
Gautham B A 084b56c616 HADOOP-17506. Fix typo in BUILDING.txt (#2662) 
(cherry picked from commit 9801b78edf)

 Conflicts:
	BUILDING.txt
 2021-01-31 14:26:44 +00:00
He Xiaoqiao c7cdcbfe2b
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 20:10:08 +08:00
Jim Brennan 8b9c883ea4 [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939)
 2021-01-28 18:32:01 +00:00
Jim Brennan 6c564f0bc1 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f6)
 2021-01-28 17:04:39 +00:00
Akira Ajisaka 52004affa8
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a)
 2021-01-24 22:57:48 +09:00
Akira Ajisaka a5647b99da
HDFS-15731. Reduce threadCount for unit tests to reduce the memory usage (#2556) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9cf408d6ba)
 2021-01-24 22:53:20 +09:00