Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
23,743 Commits 326 Branches 371 Tags
Commit Graph

23743 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wilfred Spiegelenburg
542c478a8c
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c8562a9e29b00e60d5bde1773c09188)
 2020-05-28 13:44:04 +10:00
Eric Yang
54bd2424f7 YARN-10228. Relax restriction of file path character in yarn.service.am.java.opts. 
Contributed by Bilwa S T via eyang
 2020-05-20 09:29:21 -07:00
Chen Liang
940a422525
HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 
(cherry picked from commit 7bb902bc0d0c62d63a8960db444de3abb0a6ad22)
(cherry picked from commit e452163a06daa6bbebc571127754962d8776a925)
 2020-05-19 09:52:50 +09:00
Surendra Singh Lilhore
02d04e5b0f YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng 2020-05-14 15:23:05 +05:30
Ayush Saxena
69eeb673d8 YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng. 2020-05-14 00:37:30 +05:30
Akira Ajisaka
0263e61076
Revert "SPNEGO TLS verification" 
This reverts commit 3bf7cd0030f217ae2c751a1dee68417e6ebc38bd.
 2020-05-13 17:15:17 +09:00
Akira Ajisaka
140bb86d97
HDFS-15343. TestConfiguredFailoverProxyProvider is failing. (#2001) 
(cherry picked from commit c784ba370eeb3e48c1c2c471ca72674c62a4edb7)
(cherry picked from commit 763a79916d1d5dba4d77f8b42d7fe9bddbcb53ca)
 2020-05-08 17:20:24 +09:00
Wei-Chiu Chuang
3d8e392eb4 HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998) 
Reviewed-by: Arpit Agarwal <arp@apache.org>
(cherry picked from commit adecdb8b534c536354f4f47789467ffb82dd1496)
(cherry picked from commit f4c05f1d93718c5cd574abf40c14510886289353)
 2020-05-05 21:14:55 -07:00
Szilard Nemeth
adfb68cb22 YARN-10257. FS-CS converter: skip increment properties for mem/vcores and fix DRF check. Contributed by Peter Bacsko 2020-05-05 16:01:52 +02:00
Konstantin V Shvachko
89f1095454 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab991c242b5089a18881aa10abf318ea0)
(cherry picked from commit 8809b43fb97147df0b8088adb6fa237d378c1b30)
 2020-05-05 01:30:20 +05:30
Szilard Nemeth
7e632d5470 YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 
(cherry picked from commit 410c605aec308a2ccd903f60aade3aaeefcaa610)
(cherry picked from commit 8ffe1f313c9719ea550ac524fee84320c4aff63c)
 2020-05-04 09:50:58 +05:30
Eric Yang
3bf7cd0030
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454a5f6ea84f2cf7ac0082c555e2954a7)
(cherry picked from commit 0f27c04c23c53f625ecea75ed914375eaf6b9538)
 2020-05-02 21:53:47 +09:00
Ayush Saxena
80fec40499
HADOOP-17007. hadoop-cos fails to build. Contributed by Yang Yu. 
(cherry picked from commit 9e79cca171384676037fe349b0511515c0b944b1)
 2020-04-30 11:14:00 +09:00
Akira Ajisaka
855e9acc8a
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f696aba7fc209874d232baf2a50d547)
(cherry picked from commit d504574d829170f61b2d6fda3cf1a09b9cf2d66d)
 2020-04-30 11:13:12 +09:00
Aryan Gupta
1272418f35 HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.(#1980) 
(cherry picked from commit 816042e62bf472a58d9f6dbce1123e9af6d06fb0)
 2020-04-29 11:51:11 +05:30
Szilard Nemeth
5f2e0d5729
YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 
(cherry picked from commit f91e21ac109e753e76d19c5c872c59a767b7b837)
(cherry picked from commit c0b7b38e228451f826aec8f4a92e645761dabd03)
 2020-04-28 05:51:26 +09:00
Akira Ajisaka
472385e4e6
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 
(cherry picked from commit 6ef01646ba9540248ffaf322f4678fbdac28fcd4)
(cherry picked from commit 62ee5970f8cb85e83d93ff0c3bf9b7109b4feefe)
 2020-04-28 05:45:49 +09:00
Szilard Nemeth
7a3f190d89 YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-24 17:09:35 +02:00
Szilard Nemeth
61ca459c74 YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-24 11:28:52 +02:00
Akira Ajisaka
497c7a1680
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939) 
(cherry picked from commit 9827ff2961e249e8be9023f39d91e1c0fbd23d61)
 2020-04-24 10:27:39 +09:00
Mehakmeet Singh
2471ba8b5c HADOOP-16914 Adding Output Stream Counters in ABFS (#1899) 
Contributed by Mehakmeet Singh.There
 2020-04-23 14:38:15 +01:00
Sneha Vijayarajan
32fb174da2 Hadoop 16857. ABFS: Stop CustomTokenProvider retry logic to depend on AbfsRestOp retry policy 
Contributed by Sneha Vijayarajan
 2020-04-23 14:37:25 +01:00
Szilard Nemeth
1340518cd8 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-23 14:55:50 +02:00
Mingliang Liu
47b330dc34
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:41:04 -07:00
Mingliang Liu
5b92d73a74
HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi 2020-04-22 12:48:39 -07:00
bilaharith
f53ded6185 HADOOP-16922. ABFS: Change User-Agent header (#1938) 
Contributed by Bilahari T H.
 2020-04-22 14:32:30 +01:00
Mukund Thakur
98fdbb820e HADOOP-16965. Refactor abfs stream configuration. (#1956) 
Contributed by Mukund Thakur.
 2020-04-22 14:32:01 +01:00
Mehakmeet Singh
f74a571fdf HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters on IO. (#1918). Contributed by Mehakmeet Singh. 2020-04-22 14:30:25 +01:00
Akira Ajisaka
54a64e542e
HADOOP-16944. Use Yetus 0.12.0 in GitHub PR (#1917) 
(cherry picked from commit 5576915236aba172cb5ab49b43111661590058af)
 2020-04-21 21:12:02 +09:00
Akira Ajisaka
5459dd64fd
HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky 
(cherry picked from commit e069a0613710926905de991d1d563cea5b4ee6df)
 2020-04-21 10:27:27 +09:00
Steve Loughran
0982f56f3a
HADOOP-16953. tuning s3guard disabled warnings (#1962) 
Contributed by Steve Loughran.

The S3Guard absence warning of HADOOP-16484 has been changed
so that by default the S3A connector only logs at debug
when the connection to the S3 Store does not have S3Guard
enabled.

The option to control this log level is now
fs.s3a.s3guard.disabled.warn.level
and can be one of: silent, inform, warn, fail.

On a failure, an ExitException is raised with exit code 49.

For details on this safety feature, consult the s3guard documentation.

Change-Id: If868671c9260977c2b03b3e475b9c9531c98ce79
 2020-04-20 15:07:00 +01:00
Steve Loughran
de9a6b4588
HADOOP-16986. S3A to not need wildfly on the classpath. (#1948) 
Contributed by Steve Loughran.

This is a successor to HADOOP-16346, which enabled the S3A connector
to load the native openssl SSL libraries for better HTTPS performance.

That patch required wildfly.jar to be on the classpath. This
update:

* Makes wildfly.jar optional except in the special case that
"fs.s3a.ssl.channel.mode" is set to "openssl"

* Retains the declaration of wildfly.jar as a compile-time
dependency in the hadoop-aws POM. This means that unless
explicitly excluded, applications importing that published
maven artifact will, transitively, add the specified
wildfly JAR into their classpath for compilation/testing/
distribution.

This is done for packaging and to offer that optional
speedup. It is not mandatory: applications importing
the hadoop-aws POM can exclude it if they choose.

Change-Id: I7ed3e5948d1e10ce21276b3508871709347e113d
 2020-04-20 14:42:36 +01:00
Sammi Chen
9c81b17153 HADOOP-16959. Resolve hadoop-cos dependency conflict. Contributed by Yang Yu. 
(cherry picked from commit 82ff7bc9abc8f3ad549db898953d98ef142ab02d)
 2020-04-20 18:57:49 +08:00
Surendra Singh Lilhore
a6c718fd0f HDFS-15218. RBF: MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 8e6227441ab31f3ad37998f76faa131e11b0cfe9)
 2020-04-18 20:21:01 +05:30
Masatake Iwasaki
de5d43300a HADOOP-16972. Ignore AuthenticationFilterInitializer for KMSWebServer. (#1961) 
(cherry picked from commit ac40daece17e9a6339927dbcadab76034bd7882c)
 2020-04-18 06:40:18 +09:00
Jonathan Hung
49ae9b2137 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 2020-04-17 10:34:59 -07:00
Mukund Thakur
96d7ceb39a
HADOOP-13873. log DNS addresses on s3a initialization. 
Contributed by Mukund Thakur.

If you set the log org.apache.hadoop.fs.s3a.impl.NetworkBinding
to DEBUG, then when the S3A bucket probe is made -the DNS address
of the S3 endpoint is calculated and printed.

This is useful to see if a large set of processes are all using
the same IP address from the pool of load balancers to which AWS
directs clients when an AWS S3 endpoint is resolved.

This can have implications for performance: if all clients
access the same load balancer performance may be suboptimal.

Note: if bucket probes are disabled, fs.s3a.bucket.probe = 0,
the DNS logging does not take place.

Change-Id: I21b3ac429dc0b543f03e357fdeb94c2d2a328dd8
 2020-04-17 14:20:54 +01:00
Stephen O'Donnell
aaad947c74 HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a9ea9f6e217a0ba158c48da89b3faf2)
 2020-04-16 20:19:00 -07:00
Vinayakumar B
b4ba9bed7c
HADOOP-16985. Handle release package related issues (#1957) 
(cherry picked from commit 4d24d99e854e1718270a75807f7779a623758247)
 2020-04-15 23:09:41 +05:30
Mukund Thakur
94da630cd2
HADOOP-16465 listLocatedStatus() optimisation (#1943) 
Contributed by Mukund Thakur

Optimize S3AFileSystem.listLocatedStatus() to perform list
operations directly and then fallback to head checks for files

Change-Id: Ia2c0fa6fcc5967c49b914b92f41135d07dab0464
 2020-04-15 17:04:55 +01:00
Szilard Nemeth
4bd37f2283 YARN-5277. When localizers fail due to resource timestamps being out, provide more diagnostics. Contributed by Siddharth Ahuja 2020-04-15 07:54:53 +02:00
Szilard Nemeth
f473473355 YARN-10234. FS-CS converter: don't enable auto-create queue property for root. Contributed by Peter Bacsko 
(cherry picked from commit 55fcbcb5c2a096f98f273fda52ae25ecaa1d8bb6)
 2020-04-15 07:27:47 +02:00
sunlisheng
7aad965198 HADOOP-16982. Update Netty to 4.1.48.Final. Contributed by Lisheng Sun. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit ae474e1ae587569e7ccc2526537a0e6a90ebdfa9)
 2020-04-15 07:22:54 +09:00
bilaharith
6bae8c46a8 HADOOP-16855. Changing wildfly dependency scope in hadoop-azure to compile 
Contributed by Biliharith
 2020-04-14 19:18:15 +01:00
Steve Loughran
68a9562848 HADOOP-16941. ITestS3GuardOutOfBandOperations.testListingDelete failing on versioned bucket (#1919) 
Contributed by Steve Loughran.

Removed the failing probe and replacing with two probes which will fail
on both versioned and unversioned buckets.
 2020-04-14 10:58:13 +01:00
Eric E Payne
b397a3a875 MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit 11d17417ceba0f1a2944e0c8b4286515c4883889)
 2020-04-13 18:51:00 +00:00
Jonathan Hung
e1dd78143b YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 
(cherry picked from commit 57659422abbf6d9bf52e6e27fca775254bb77a56)
 2020-04-09 11:06:49 -07:00
Steve Loughran
e4331a73c9
HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 (#1936) 
Contributed by Steve Loughran.

This strips out all the -p preservation options which have already been
processed when uploading a file before deciding whether or not to query
the far end for the status of the (existing/uploaded) file to see if any
other attributes need changing.

This will avoid 404 caching-related issues in S3, wherein a newly created
file can have a 404 entry in the S3 load balancer's cache from the
probes for the file's existence prior to the upload.

It partially addresses a regression caused by HADOOP-8143,
"Change distcp to have -pb on by default" that causes a resurfacing
of HADOOP-13145, "In DistCp, prevent unnecessary getFileStatus call when
not preserving metadata"

Change-Id: Ibc25d19e92548e6165eb8397157ebf89446333f7
 2020-04-09 18:23:47 +01:00
Wei-Chiu Chuang
749a5b81da HDFS-15269. NameNode should check the authorization API version only … (#1945) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 061afcdf30ce10d04986672a0583d925d3f8f741)
 2020-04-09 09:45:05 -07:00
Szilard Nemeth
061a7a8a41 YARN-9995. Code cleanup in TestSchedConfCLI. Contributed by Bilwa S T. 
(cherry picked from commit c04a484aedb09bdac7484908be1b8fcd1add2e30)
 2020-04-09 15:47:57 +02:00