Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,090 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

21090 Commits

Author SHA1 Message Date
Eric E Payne 5e2be81fcb YARN-7387: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit b1e07d27cc)
 2020-01-08 19:30:09 +00:00
Eric E Payne b20ce118da YARN-10072: TestCSAllocateCustomResource failures. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 6899be5a17)
 2020-01-08 17:42:41 +00:00
Ahmed Hussein ee7c56c6de MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit cdd6efd3ab)
 2020-01-08 11:09:58 -06:00
Steve Loughran e21cb8f96e HADOOP-16785. Improve wasb and abfs resilience on double close() calls. 
This hardens the wasb and abfs output streams' resilience to being invoked
in/after close().

wasb:
  Explicity raise IOEs on operations invoked after close,
  rather than implicitly raise NPEs.
  This ensures that invocations which catch and swallow IOEs will perform as
  expected.

abfs:
  When rethrowing an IOException in the close() call, explicitly wrap it
  with a new instance of the same subclass.
  This is needed to handle failures in try-with-resources clauses, where
  any exception in closed() is added as a suppressed exception to the one
  thrown in the try {} clause
  *and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b
 2020-01-08 12:04:11 +00:00
Rakesh Radhakrishnan d7f9a90e2c HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He. 
(cherry picked from commit 7030722e5d)
 2020-01-08 14:29:05 +05:30
Masatake Iwasaki 3e17b471b6 HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 
(cherry picked from commit aba3f6c3e1)
 2020-01-08 17:05:20 +09:00
Masatake Iwasaki 3e32f5476a
HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. (#1775) 
(cherry picked from commit a43c177f1d)
 2020-01-08 11:29:23 +09:00
Tamás Pénzes 250cd9f575 HADOOP-16772. Extract version numbers to head of pom.xml (addendum) (#1774). Contributed by Tamas Penzes. 
Follow up task of HADOOP-16729, extract even more version numbers for branch-3.2.

Change-Id: I0546fc8ec8f68baa4bcfa9db0df4d8293b078913
 2020-01-06 14:12:28 +01:00
Sergey Pogorelov d7cc897bdc
MAPREDUCE-7255. Fix typo in MapReduce documentaion example (#1793) 
(cherry picked from commit b343e1533b)
 2020-01-06 12:40:29 +09:00
Ayush Saxena 3a624cad37 HADOOP-16784. Update the year to 2020. Contributed by Ayush Saxena. 2020-01-02 21:38:49 +05:30
Steve Loughran 5410732cff
HADOOP-16775. DistCp reuses the same temp file within the task for different files. 
Contributed by Amir Shenavandeh.

This avoids overwrite consistency issues with S3 and other stores -though
given S3's copy operation is O(data), you are still best of using -direct
when distcp-ing to it.

Change-Id: I8dc9f048ad0cc57ff01543b849da1ce4eaadf8c3
 2020-01-02 15:37:55 +00:00
Prabhu Joseph ad98a30810 YARN-10053. Use Shared Group Mapping Service in Placement Rules. Contributed by Wilfred Spiegelenburg. 
(cherry Picked from commit 217b56ffdd)
 2020-01-02 14:30:01 +05:30
Rakesh Radhakrishnan a27f9bd28e HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He. 
(cherry picked from commit d79cce20ab)
 2020-01-02 11:57:02 +05:30
Surendra Singh Lilhore 5d9cbca31e HDFS-15074. DataNode.DataTransfer thread should catch all the expception and log it. Contributed by hemanthboyina. 2019-12-29 11:21:33 +05:30
Takanobu Asanuma ad11b25df2 HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena. 
(cherry picked from commit dc32f583af)
 2019-12-28 21:34:19 +09:00
Chandra Sanivarapu 8ec1461b25
HADOOP-16042. Update the link to HadoopJavaVersion (#1784) 
(cherry picked from commit d80d86264d)
 2019-12-27 17:20:26 +09:00
Duo Zhang 440b7abc04
HADOOP-16610. Upgrade to yetus 0.11.1 and use emoji vote on github pre commit (#1527). Contributed by Duo Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3cecb2a469)
 2019-12-25 19:48:15 +09:00
Chandra Sanivarapu 0d94e0b1a0
HDFS-15073. Replace curator-shaded guava import with the standard one (#1782) 
(cherry picked from commit d8cd7098b4)
 2019-12-25 11:12:03 +09:00
Konstantin V Shvachko 82957562ea HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b98ac2a3af)
 2019-12-24 12:50:57 -08:00
Akira Ajisaka 85da5cb870
YARN-10055. bower install fails. (#1778) 
(cherry picked from commit 34ff7dbaf5)
 2019-12-23 20:26:40 +09:00
Eric Badger 355ec33416 YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne 2019-12-20 19:32:36 +00:00
Eric Badger 11c5396b4a Revert "HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui." 
This reverts commit 59cdf18d5a.
 2019-12-20 19:04:45 +00:00
Akira Ajisaka 7201384e1b
HADOOP-16771. Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0. Contributed by Andras Bokor. 
(cherry picked from commit f777cd398f)
 2019-12-20 13:12:02 +09:00
Inigo Goiri 59cdf18d5a HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui. 
(cherry picked from commit 52d7b745c6)
 2019-12-19 09:43:40 -08:00
Jonathan Hung 0707d0a0ae YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 
(cherry picked from commit 7b93575b92)
 2019-12-18 13:18:22 -08:00
Jonathan Hung 01e2c996ff YARN-10039. Allow disabling app submission from REST endpoints 
(cherry picked from commit fddc3d55c3)
 2019-12-18 10:51:57 -08:00
Shashikant Banerjee afe00a1ca5 HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee. 
(cherry picked from commit fdd96e46d1)
 2019-12-18 23:10:00 +05:30
Eric Badger 18b515322c YARN-10033. TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resources. Contributed by Eric Payne. 
(cherry picked from commit f47dcf2d4c)
 2019-12-17 17:36:52 +00:00
Chen Liang d4a6901c42 HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang. 
(cherry picked from commit 65c4660bcd)
 2019-12-12 15:57:15 -08:00
He Xiaoqiao 8ba4e13dcc HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 99af49878cc47f5750b6c0edadc8a4c2fcd7a7c6)
 2019-12-11 16:34:28 -08:00
Mingliang Liu d19981fe48
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml (#1753) 
Contributed by Mingliang Liu
 2019-12-11 11:54:12 -08:00
Erik Krogen 3402c87353 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 
(cherry picked from c174d50b30)
 2019-12-11 09:40:36 -08:00
Kevin Su 0f25cbbb25 HADOOP-16754. Fix docker failed to build yetus/hadoop. 
Contributed by Kevin Su.

Change-Id: Ie9cf27b7a7ba5923405f24d39b95b0248f3718b7
 2019-12-11 16:57:30 +00:00
Akira Ajisaka a3cdba7713
YARN-9985. Unsupported transitionToObserver option displaying for rmadmin command. Contributed by Ayush Saxena. 
(cherry picked from commit dc66de7448)
 2019-12-09 18:38:47 +09:00
Sneha Vijayarajan aa9cd0a2d6
HADOOP-16660. ABFS: Make RetryCount in ExponentialRetryPolicy Configurable. 
Contributed by Sneha Vijayarajan.
 2019-12-08 21:32:13 -08:00
bilaharith c225efe237
HADOOP-16455. ABFS: Implement FileSystem.access() method. 
Contributed by Bilahari T H.
 2019-12-08 21:32:02 -08:00
Jeetesh Mangwani b1e748f45b
HADOOP-16612. Track Azure Blob File System client-perceived latency 
Contributed by Jeetesh Mangwani.

This add the ability to track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency in the Hadoop ABFS driver.
The latency information is sent back to the ADLS Gen 2 REST API endpoints in the subsequent requests.
 2019-12-08 21:31:51 -08:00
bilaharith ffeb6d8ece
HADOOP-16587. Make ABFS AAD endpoints configurable. 
Contributed by Bilahari T H.

This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate
in China.

Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c
 2019-12-08 21:31:39 -08:00
Sneha Vijayarajan 8b2c7e0c4d
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists 2019-12-08 21:31:24 -08:00
Sneha Vijayarajan 546db6428e
HADOOP-16548 : Disable Flush() over config 2019-12-08 21:31:08 -08:00
dependabot[bot] 26367b6cc7
Bump nimbus-jose-jwt from 4.41.1 to 7.9 (#1682) 
Bumps [nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt) from 4.41.1 to 7.9.
- [Changelog](https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt)
- [Commits](https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/7.9..4.41.1)

Signed-off-by: dependabot[bot] <support@github.com>
(cherry picked from commit c1d393a156)
 2019-12-09 10:19:13 +09:00
Jonathan Hung 9228e3f0ad YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 
(cherry picked from commit 92bce918dc)
 2019-12-08 16:36:08 -08:00
Jonathan Hung 8badcd989e Revert "YARN-10012. Guaranteed and max capacity queue metrics for custom resources" 
This reverts commit b5235f1ed0.
 2019-12-08 16:36:00 -08:00
Jonathan Hung b5235f1ed0 YARN-10012. Guaranteed and max capacity queue metrics for custom resources 
(cherry picked from commit 92bce918dc)
 2019-12-08 16:03:34 -08:00
Sunil G 69dc329acc YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 
(cherry picked from commit 002dcc4ebf)
 2019-12-08 14:40:45 -08:00
Masatake Iwasaki e83684eae3 HADOOP-16744. Fix building instruction to enable zstd. (#1736) 
(cherry picked from commit 18059acb6a)
 2019-12-06 15:27:46 +09:00
Mingliang Liu 8a60429e0b
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:50:28 -08:00
Tamás Pénzes dcb0454bb9 HADOOP-16729. Extract version numbers to head of pom.xml (#1732). Contributed by Tamas Penzes. 2019-12-04 10:35:21 +01:00
Aravindan Vijayan f0c1403ec3 HADOOP-16718. Allow disabling Server Name Indication (SNI) for Jetty. Contributed by Aravindan Vijayan. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Siyao Meng <smeng@cloudera.com>
(cherry picked from commit f1ab7f18c4)
 2019-12-03 15:48:15 -08:00
Erik Krogen c1ed247459 HADOOP-16700. Track response time separately in Server.Call to avoid negative RpcQueueTime values. Contributed by xuzq. 
(cherry picked from be77231452)
 2019-12-03 14:01:16 -08:00