7732 Commits

Author SHA1 Message Date
Kihwal Lee
02e0b6e306 HDFS-7788. Post-2.6 namenode may not start up with an image containing inodes created with an old release. Contributed by Rushabh Shah.
(cherry picked from commit 7ae5255a1613ccfb43646f33eabacf1062c86e93)

(cherry picked from commit b9157f92fc3e008e4f3029f8feeaf6acb52eb76f)

Conflicts:
  hadoop-hdfs-project/hadoop-hdfs/src/site/resources/image-with-zero-block-size.tar.gz
  hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java

(cherry picked from commit 1faa44d8f4d7b944e99dd0470ea2638c7653a131)
2015-08-31 18:32:00 -07:00
Kihwal Lee
4ec7b6174d HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is full. Contributed by Ming Ma.
(cherry picked from commit 685af8a3d0504724fe588daf3722519fedc45b01)
(cherry picked from commit 6c01e586198a3c3ebaa7561778c124ae62553246)
2015-08-31 18:10:00 -07:00
Xuan
95edb6e64f YARN-3207. Secondary filter matches entites which do not have the key
being filtered for. Contributed by Zhijie Shen

(cherry picked from commit 57db50cbe3ce42618ad6d6869ae337d15b261f4e)
(cherry picked from commit ba18adbb27c37a8fa92223a412ce65eaa462d18b)
(cherry picked from commit 9fd18e94849600ec66832df5ae424eeb0116330c)
2015-08-31 17:44:42 -07:00
Zhijie Shen
28160a0bd6 YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. Contributed by Devaraj K.
(cherry picked from commit d5855c0e46404cfc1b5a63e59015e68ba668f0ea)
(cherry picked from commit fd75b8c9cadd069673afc80a0fc5661d779897bd)
(cherry picked from commit a62891971380e5f8e4a645ed36bd88aa6fe0e47a)
2015-08-31 17:38:51 -07:00
Vinayakumar B
6154abfeda HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.(Contributed by Vinayakumar B)
(cherry picked from commit 3d15728ff5301296801e541d9b23bd1687c4adad)
(cherry picked from commit a1bf7aecf7d018c5305fa3bd7a9e3ef9af3155c1)
(cherry picked from commit c1e65de57e8ef760586e28cd37397ea9a7ac7944)
2015-08-31 17:26:28 -07:00
Jian He
a703952d39 YARN-3094. Reset timer for liveness monitors after RM recovery. Contributed by Jun Gong
(cherry picked from commit 0af6a99a3fcfa4b47d3bcba5e5cc5fe7b312a152)

(cherry picked from commit 61466809552f96a83aa19446d4d59cecd0d2cad5)
(cherry picked from commit ab654746fbad2da12b24b13425dc9bf17c46b50c)
2015-08-31 17:17:47 -07:00
Gera Shegalov
bb8a1551c3 HADOOP-11506. Configuration variable expansion regex expensive for long values. (Gera Shegalov via gera)
(cherry picked from commit 644548f201743408904dfe24b9f5b515b2c96713)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java

(cherry picked from commit 3d36d4737c160d7dc8829e9dd6b801ef6726c0c0)
2015-08-31 16:00:32 -07:00
Kihwal Lee
a1c3e6a058 HDFS-7707. Edit log corruption due to delayed block removal again. Contributed by Yongjun Zhang
(cherry picked from commit 843806d03ab1a24f191782f42eb817505228eb9f)

(cherry picked from commit c428d303f67bef3a7df12153947c6b0199a0938b)
(cherry picked from commit 21d8b22feef2fc9f6793d9c157391b142368e588)
2015-08-31 15:55:07 -07:00
Jian He
e917ef3cfe MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken service name properly. Contributed by Jason Lowe
(cherry picked from commit cff05bff1fe24628677d41a0d537f2c383b44faf)

(cherry picked from commit 43b3b43cea1f620ce66521bcc1c4b6aec264aa9a)
(cherry picked from commit 013e271036554020e3d26de4fcbb5b6970f5a9c0)
2015-08-31 15:49:24 -07:00
Jian He
8658945b3a YARN-3103. AMRMClientImpl does not update AMRM token properly. Contributed by Jason Lowe
(cherry picked from commit 6d2bdbd7dab179dfb4f19bb41809e97f1db88c6b)

(cherry picked from commit 12522fd9cbd8da8c040a5b7bb71fcdaa256daf89)
(cherry picked from commit f50f5ad49d3b70448647384fc5f020214cb58f10)
2015-08-31 15:42:03 -07:00
Jian He
994c3d049a YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService might lead NM to crash. Contributed by Varun Saxena
(cherry picked from commit 4e15fc08411318e11152fcd5a4648ed1d6fbb480)

(cherry picked from commit 8100c8a68c32978a177af9a3e6639f6de533886d)
(cherry picked from commit 10a6c4f349e6f32ed2a520bf669a0cbfff31c824)
2015-08-31 15:38:45 -07:00
Andrew Wang
24da944241 HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy user. Contributed by Arun Suresh.
(cherry picked from commit 4b00935643f6c3656ccbd7eeb54884738bc12c2e)
(cherry picked from commit 7b69719455a1a374c9417417ef0c8d7ba6bf593f)
2015-08-31 15:14:41 -07:00
Arpit Agarwal
f81f97eb7c HDFS-7575. Upgrade should generate a unique storage ID for each volume. (Contributed by Arpit Agarwal)
(cherry picked from commit 1d9d166c0beb56aa45e65f779044905acff25d88)
(cherry picked from commit ca8e1b0739b6653833f9bc8990ab126420703f66)
2015-08-31 15:08:42 -07:00
Haohui Mai
31c99a3dbf HADOOP-11350. The size of header buffer of HttpServer is too small when HTTPS is enabled. Contributed by Benoy Antony.
(cherry picked from commit 2e4df8710435c8362506fe944a935e74ad5919c0)
2015-08-31 12:59:50 -07:00
Kihwal Lee
ae288f7291 HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade shutdown command. Contributed by Eric Payne.
(cherry picked from commit 6bbf9fdd041d2413dd78e2bce51abae15f3334c2)

(cherry picked from commit 33534a0c9aef5024aa6f340e7ee24930c8fa8ed5)
(cherry picked from commit e9a28251ee46e64e1b99b2dd54b0432bdc0b9578)
2015-08-31 11:53:24 -07:00
cnauroth
8382e52c94 HDFS-7596. NameNode should prune dead storages from storageMap. Contributed by Arpit Agarwal.
(cherry picked from commit ef3c3a832c2f0c1e5ccdda2ff8ef84902912955f)
(cherry picked from commit 75e4e55e12b2faa521af7c23fddcba06a9ce661d)
(cherry picked from commit cc637d6ece64dfeb89e78c7e9766836149e098be)
2015-08-31 11:26:48 -07:00
Jing Zhao
c8ce11d067 HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma.
(cherry picked from commit 4b589e7cfa27bd042e228bbbcf1c3b75b2aeaa57)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java

(cherry picked from commit 96f0813c5d6140aabe7b2837f30971936276e689)
2015-08-30 21:21:05 -07:00
cnauroth
e6b52fc3e9 HDFS-7579. Improve log reporting during block report rpc failure. Contributed by Charles Lamb.
(cherry picked from commit 7e2d9a32426d04b5f08c2835f61882b053612a20)
(cherry picked from commit f0acb7c2a284db61640efee15a1648c6c26d24f5)
(cherry picked from commit 33fb7b45195a89b6464e4a1cb3fbf6bbad2bcecb)
2015-08-30 21:17:34 -07:00
Jian He
3f8da2a9eb YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed container statuses on heartbeat. Contributed by Chengbing Liu
(cherry picked from commit cc2a745f7e82c9fa6de03242952347c54c52dccc)

(cherry picked from commit e7e6173049adca2a2ae0e1231adcaca8168bec27)
(cherry picked from commit 3c4ed2497b14140f09b3cae4959be6474c4cdc99)
2015-08-30 20:45:45 -07:00
Tsuyoshi Ozawa
03f9ac2de7 YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. Contributed by Rohith Sharmaks.
(cherry picked from commit ddc5be48fc35868abf7f59088f747c636e76a42a)
(cherry picked from commit c116743bdda2b1792bf872020a5e2b14d772ac60)
(cherry picked from commit 3c9d26ae14625de3e9437c07eceda0d05f1985b2)
2015-08-30 20:43:14 -07:00
Jian He
e7fc071906 YARN-2992. ZKRMStateStore crashes due to session expiry. Contributed by Karthik Kambatla
(cherry picked from commit 1454efe5d4fe4214ec5ef9142d55dbeca7dab953)

(cherry picked from commit ca0349b87ab1b2d0d2b9dc93de7806d26713165c)
(cherry picked from commit 2f6be218fa41fd0f39633ec5ed0df6e0fa0f54b6)
2015-08-30 20:42:06 -07:00
Jian He
8cdcb792a1 YARN-2340. Fixed NPE when queue is stopped during RM restart. Contributed by Rohith Sharmaks
(cherry picked from commit 0d89859b51157078cc504ac81dc8aa75ce6b1782)

(cherry picked from commit 8b398a66ca3728f47363fc8b2fcf7e556e6bbf5a)
(cherry picked from commit 37f1789f419fa741c2fbae9855549d405e856bff)
2015-08-30 20:40:09 -07:00
Jian He
6dfdcf094d YARN-2952. Fixed incorrect version check in StateStore. Contributed by Rohith Sharmaks
(cherry picked from commit 808cba3821d5bc4267f69d14220757f01cd55715)

(cherry picked from commit 9180d11b3bbb2a49127d5d25f53b38c5113bf7ea)
2015-08-30 20:34:14 -07:00
Jason Lowe
17a23b8191 YARN-1984. LeveldbTimelineStore does not handle db exceptions properly. Contributed by Varun Saxena
(cherry picked from commit 1ce4d33c2dc86d711b227a04d2f9a2ab696a24a1)

(cherry picked from commit 89ef49fb0814baea3640798fdf66d2ae3a550896)
2015-08-30 20:25:39 -07:00
Colin Patrick Mccabe
c2e00e0012 HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin P. McCabe)
(cherry picked from commit 3b173d95171d01ab55042b1162569d1cf14a8d43)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

(cherry picked from commit dda1fc169db2e69964cca746be4ff8965eb8b56f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit ba28192f9d5a8385283bd717bca494e6981d378f)
2015-08-30 20:11:41 -07:00
Jason Lowe
570d52e53c YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie). Contributed by Jian He
(cherry picked from commit 0402bada1989258ecbfdc437cb339322a1f55a97)

(cherry picked from commit 173664d70f0ed3b1852b6703d32e796778fb1c78)
(cherry picked from commit 04e71db1ce9572ae0641234a02b7db5d174668fd)
2015-08-30 20:06:07 -07:00
Gera Shegalov
1931fa5f4b MAPREDUCE-6166. Reducers do not validate checksum of map outputs when fetching directly to disk. (Eric Payne via gera)
(cherry picked from commit af006937e8ba82f98f468dc7375fe89c2e0a7912)
(cherry picked from commit 7c7bccfc31ede6f4afc043d81c8204046148c02e)
(cherry picked from commit 2e9dca584ba25809d9c6269e5f9326e09f55ed99)
2015-08-30 19:50:43 -07:00
Colin Patrick Mccabe
1698110acc HADOOP-11238. Update the NameNode's Group Cache in the background when possible (Chris Li via Colin P. McCabe)
(cherry picked from commit e5a692519956aefb3a540ed0137b63cf598ac10d)

(cherry picked from commit b521d91c0f5b6d114630b6727f6a01db56dba4f1)
2015-08-30 19:47:15 -07:00
Jian He
3d5627b563 YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks
(cherry picked from commit 614b6afea450ebb897fbb2519c6f02e13b9bd12d)

(cherry picked from commit 41f0d20fcb4fec5b932b8947a44f93345205222c)
(cherry picked from commit aa4f8184ee7aa1817a9b23b3b4e7e78da95dc031)
2015-08-30 13:49:25 -07:00
Karthik Kambatla
6e954bc25c YARN-2910. FSLeafQueue can throw ConcurrentModificationException. (Wilfred Spiegelenburg via kasha)
(cherry picked from commit a2e07a54561a57a83b943628ebbc53ed5ba52718)
(cherry picked from commit 1986ea8dd223267ced3e3aef69980b46e2fef740)
(cherry picked from commit 2b827a18d7b4eb41dc0095ea7277239273e7e396)
2015-08-30 13:44:52 -07:00
Andrew Wang
2a2888f2f3 HADOOP-11368. Fix SSLFactory truststore reloader thread leak in KMSClientProvider. Contributed by Arun Suresh.
(cherry picked from commit 74d4bfded98239507511dedb515bc6a54958d5a8)
(cherry picked from commit deaa172e7a2ab09656cc9eb431a3e68a73e0bd96)
2015-08-30 13:41:40 -07:00
Andrew Wang
65180617fb HADOOP-11343. Overflow is not properly handled in caclulating final iv for AES CTR. Contributed by Jerry Chen.
(cherry picked from commit 0707e4eca906552c960e3b8c4e20d9913145eca6)
(cherry picked from commit dabdd2d746d1e1194c124c5c7fe73fcc025e78d2)
2015-08-30 13:31:43 -07:00
Karthik Kambatla
7d686eccc3 YARN-2874. Dead lock in DelegationTokenRenewer which blocks RM to execute any further apps. (Naganarasimha G R via kasha)
(cherry picked from commit 799353e2c7db5af6e40e3521439b5c8a3c5c6a51)

(cherry picked from commit 25be97808b99148412c0efd4d87fc750db4d6607)
(cherry picked from commit d82bf536d44c6e7ba06a01105545b3979b731d80)
2015-08-30 13:25:37 -07:00
Jian He
3600f30c35 YARN-2894. Fixed a bug regarding application view acl when RM fails over. Contributed by Rohith Sharmaks
(cherry picked from commit 392c3aaea8e8f156b76e418157fa347256283c56)

(cherry picked from commit d6f3d4893d750f19dd8c539fe28eecfab2a54576)
(cherry picked from commit 61efbc1cba0c4a81b8aafb1d45c2f7b3cf7857d8)
2015-08-30 13:08:42 -07:00
Colin Patrick Mccabe
70f33f5eff HDFS-7035. Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe)
(cherry picked from commit a9331fe9b071fdcdae0c6c747d7b6b306142e671)

(cherry picked from commit ec2621e907742aad0264c5f533783f0f18565880)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java

(cherry picked from commit d79a5849999cdb0bc315938b80ed71b4f2dcb720)
2015-08-30 12:13:33 -07:00
Arpit Agarwal
46e9926ce3 HDFS-8486. Addendum patch to fix backporting bug.
(cherry picked from commit 6c9e26609ceac5d16434cc7f4a1013e6faacd92f)
2015-08-30 12:02:23 -07:00
Kihwal Lee
8549891cb3 HDFS-7213. processIncrementalBlockReport performance degradation.
Contributed by Eric Payne.
(cherry picked from commit e226b5b40d716b6d363c43a8783766b72734e347)

(cherry picked from commit 946463efefec9031cacb21d5a5367acd150ef904)
(cherry picked from commit f94aa4d25c2f96faf5164e807c2c3eb031e9a1fe)
2015-08-30 11:46:16 -07:00
Zhijie Shen
8a47d1aa55 YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai.
(cherry picked from commit 51af8d367de94689770f57c64bea3b244d7755f6)
(cherry picked from commit d21ef79707a0f32939d9a5af4fed2d9f5fe6f2ec)
(cherry picked from commit db325c053c6ef3ee8731d7273d1d92da7e5deee7)
2015-08-27 19:47:43 -07:00
Jason Lowe
2f0a34b5ba YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted. Contributed by Varun Saxena
(cherry picked from commit 0f9528b99addbb0fd9a19d84db22a8c8e934b05f)

(cherry picked from commit 38ea1419f60d2b8176dba4931748f1f0e52ca84e)
(cherry picked from commit 3877166754956bf66a8b1c81440dba2d279a1e03)
2015-08-27 19:27:24 -07:00
Colin Patrick Mccabe
bfc0a19d6d HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification pipe is full (zhaoyunjiong via cmccabe)
(cherry picked from commit 86e3993def01223f92b8d1dd35f6c1f8ab6033f5)

(cherry picked from commit f6d1bf5ed1cf647d82e676df15587de42b1faa42)
2015-08-27 19:18:45 -07:00
Jian He
888ab4a6e7 YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. Contributed by Jason Lowe
(cherry picked from commit 8a7ca13b13c0c3f008a6490cc96d4d48a051d1f7)

(cherry picked from commit ae35b0e14d3438237f4b5d3b5d5268d45e549846)
(cherry picked from commit 65acee3e19a147e5c5a8688319ab75357bdf51b5)
2015-08-27 19:10:43 -07:00
Jian He
7f97189bcf YARN-2865. Fixed RM to always create a new RMContext when transtions from StandBy to Active. Contributed by Rohith Sharmaks
(cherry picked from commit 9cb8b75ba57f18639492bfa3b7e7c11c00bb3d3b)

(cherry picked from commit db31ef7e7f55436bbf88c6d93e2273c4463ca9f0)
(cherry picked from commit e669974ae94c03914c9181a4481b4879fd4acc0d)
2015-08-27 18:56:22 -07:00
Andrew Wang
0898c21e24 HDFS-7225. Remove stale block invalidation work when DN re-registers with different UUID. (Zhe Zhang and Andrew Wang)
(cherry picked from commit 406c09ad1150c4971c2b7675fcb0263d40517fbf)
(cherry picked from commit 2e15754a92c6589308ccbbb646166353cc2f2456)
(cherry picked from commit 014d07de2e9b39be4b6793f0e09fcf8548570ad5)
2015-08-27 18:48:17 -07:00
Jason Lowe
f307b426f3 YARN-2414. RM web UI: app page will crash if app is failed before any attempt has been created. Contributed by Wangda Tan
(cherry picked from commit 81c9d17af84ed87b9ded7057cb726a3855ddd32d)

(cherry picked from commit 242fd0e39ad1c5d51719cd0f6c197166066e3288)
(cherry picked from commit a9d5acd898b34e1050a78f2d70ed62fdb82948a6)
2015-08-27 18:36:50 -07:00
Jason Lowe
f83d898944 YARN-2816. NM fail to start with NPE during container recovery. Contributed by Zhihai Xu
(cherry picked from commit 49c38898b0be64fc686d039ed2fb2dea1378df02)

(cherry picked from commit ad140d1fc831735fb9335e27b38d2fc040847af1)
(cherry picked from commit 85b23c323c80c5303bd0b7bdb066258792ca67d8)
2015-08-27 18:32:59 -07:00
Jian He
81ba30211e YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state on app recovery. Contributed by Rohith Sharmaks
(cherry picked from commit d005404ef7211fe96ce1801ed267a249568540fd)

(cherry picked from commit beb184ac580b0d89351a3f3a7201da34a26db1c1)
(cherry picked from commit 325bb33988743d60cb333002f9da60314241632e)
2015-08-27 18:29:39 -07:00
Vinayakumar B
d6666e8d26 HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu.
Moved CHANGES.txt entry to 2.6.1

(cherry picked from commit d6050f06a3b7e049541b1cb4597c388abf00a5be)
2015-08-27 18:18:32 -07:00
Haohui Mai
7405de4970 HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. Contributed by Stephen Chu.
(cherry picked from commit 9e63cb4492896ffb78c84e27f263a61ca12148c8)
(cherry picked from commit 7adb6f9501e8efaa712fb70fe5a97e233622e3e1)
2015-08-27 18:18:25 -07:00
Tao Luo
6d2f80107f HDFS-7263. Snapshot read can reveal future bytes for appended files. Contributed by Tao Luo.
(cherry picked from commit 8bfef590295372a48bd447b1462048008810ee17)

(cherry picked from commit 3827a1acdbc4f9fec3179dcafa614734b5fa31bc)
2015-08-27 17:53:06 -07:00
Colin Patrick Mccabe
d3a7769bff HDFS-7235. DataNode#transferBlock should report blocks that don't exist using reportBadBlock (yzhang via cmccabe)
(cherry picked from commit ac9ab037e9a9b03e4fa9bd471d3ab9940beb53fb)

(cherry picked from commit 842a54a5f66e76eb79321b66cc3b8820fe66c5cd)
(cherry picked from commit 1aa9e34c5106c496ffd390f6b2c822d387fb1908)
2015-08-27 17:47:39 -07:00