Commit Graph

4800 Commits

Author SHA1 Message Date
Andrew Wang d447152d49 Update CHANGES, RELEASENOTES, jdiff for 3.0.0 release. 2017-12-13 13:05:32 -08:00
Chris Douglas 693169ef34 HDFS-12882. Support full open(PathHandle) contract in HDFS 2017-12-11 20:14:15 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) f19638333b HADOOP-15059. Undoing the switch of Credentials to PB format as default - done via HADOOP-12563 for supporting 2.x to 3.x upgrades. 2017-12-08 08:00:21 -08:00
Xiao Chen 19e0894209 HADOOP-15056. Fix TestUnbuffer#testUnbufferException failure. Contributed by Jack Bearden. 2017-12-07 21:05:55 -08:00
Sunil G daa1cdd062 YARN-7564. Cleanup to fix checkstyle issues of YARN-5881 branch. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Wangda Tan 5e798b1a0d YARN-6471. Support to add min/max resource configuration for a queue. (Sunil G via wangda)
Change-Id: I9213f5297a6841fab5c573e85ee4c4e5f4a0b7ff
2017-12-07 18:56:54 -08:00
Sean Mackrory acb92904d0 HADOOP-15098. TestClusterTopology#testChooseRandom fails intermittently. Contributed by Zsolt Venczel. 2017-12-07 12:30:58 -07:00
Lei Xu 67662e2ac9 HDFS-12840. Creating a file with non-default EC policy in a EC zone is not correctly serialized in the editlog. Contributed by Lei (Eddy) Xu. 2017-12-07 11:15:40 -08:00
Arpit Agarwal e00c7f78c1 HADOOP-14976. Set HADOOP_SHELL_EXECNAME explicitly in scripts. 2017-12-04 21:02:04 -08:00
Xiao Chen 404eab4dc0 HDFS-12396. Webhdfs file system should get delegation token from kms provider. Contributed by Rushabh S Shah. 2017-12-04 10:40:11 -08:00
Chris Douglas 42307e3c3a HDFS-11576. Block recovery will fail indefinitely if recovery time > heartbeat interval. Contributed by Lukas Majercak 2017-12-01 22:34:30 -08:00
Chris Douglas f9d195dfe9 HADOOP-14600. LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions. Contributed by Ping Liu 2017-12-01 22:12:26 -08:00
Chris Douglas 53bbef3802 Revert "HDFS-11576. Block recovery will fail indefinitely if recovery time > heartbeat interval. Contributed by Lukas Majercak"
This reverts commit 5304698dc8.
2017-12-01 11:19:38 -08:00
Chris Douglas 5304698dc8 HDFS-11576. Block recovery will fail indefinitely if recovery time > heartbeat interval. Contributed by Lukas Majercak 2017-12-01 10:29:30 -08:00
Chris Douglas 0780fdb1eb HDFS-12877. Add open(PathHandle) with default buffersize 2017-11-30 15:13:16 -08:00
Robert Kanter 75a3ab88f5 HADOOP-13493. Compatibility Docs should clarify the policy for what takes precedence when a conflict is found (templedf via rkanter) 2017-11-30 07:39:15 -08:00
Chris Douglas 0e560f3b8d HDFS-12681. Make HdfsLocatedFileStatus a subtype of LocatedFileStatus 2017-11-29 20:28:06 -08:00
Xiao Chen d162252d7a HADOOP-15067. GC time percentage reported in JvmMetrics should be a gauge, not counter. Contributed by Misha Dmitriev. 2017-11-23 09:01:28 -08:00
Steve Loughran de8b6ca5ef HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints.
Contributed by Steve Loughran and Ryan Blue.
2017-11-22 15:28:12 +00:00
Robert Kanter f2efaf013f HADOOP-14982. Clients using FailoverOnNetworkExceptionRetry can go into a loop if they're used without authenticating with kerberos in HA env (pbacsko via rkanter) 2017-11-16 11:11:19 -08:00
Chris Douglas 675e9a8f57 Revert "HDFS-12681. Fold HdfsLocatedFileStatus into HdfsFileStatus."
This reverts commit b85603e3f8.
2017-11-15 19:20:37 -08:00
Xiao Chen b1941b200d HADOOP-15023. ValueQueue should also validate (int) (lowWatermark * numValues) > 0 on construction. 2017-11-15 16:44:06 -08:00
Daniel Templeton 69043ba8b5 HADOOP-14876. Create downstream developer docs from the compatibility guidelines 2017-11-15 10:03:29 -08:00
Arun Suresh b07e68b02a Addendum patch for Configuration fix. (Jason Lowe via asuresh) 2017-11-13 14:03:50 -08:00
Eric Yang 1b68b8ff2c HADOOP-15008. Fixed period unit calculation for Hadoop Metrics V2. (Contribute by Erik Krogen) 2017-11-13 12:42:43 -05:00
Akira Ajisaka 975a57a688
HADOOP-15031. Fix javadoc issues in Hadoop Common. Contributed by Mukul Kumar Singh. 2017-11-13 23:12:23 +09:00
Chris Douglas 796a0d3a5c HADOOP-8522. ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. Contributed by Mike Percy 2017-11-10 17:41:29 -08:00
Xiao Chen 3c6adda291 HADOOP-14960. Add GC time percentage monitor/alerter. Contributed by Misha Dmitriev. 2017-11-09 21:06:06 -08:00
Arun Suresh 5eb7dbe9b3 Fixing Job History Server Configuration parsing. (Jason Lowe via asuresh) 2017-11-09 15:15:51 -08:00
John Zhuge 6c32ddad30 HADOOP-14872. CryptoInputStream should implement unbuffer. Contributed by John Zhuge. 2017-11-09 10:16:12 -08:00
John Zhuge bf6a660232 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 2017-11-09 10:16:12 -08:00
Mingliang Liu b6d8f87b8e HADOOP-15015. TestConfigurationFieldsBase to use SLF4J for logging. Contributed by Steve Loughran 2017-11-03 21:01:56 -07:00
Xiaoyu Yao 59d78a5088 HADOOP-14987. Improve KMSClientProvider log around delegation token checking. Contributed by Xiaoyu Yao and Xiao Chen. 2017-11-03 16:10:37 -07:00
Chris Douglas b85603e3f8 HDFS-12681. Fold HdfsLocatedFileStatus into HdfsFileStatus. 2017-11-03 14:30:57 -07:00
Chris Douglas d015e0bbd5 HDFS-7878. API - expose a unique file identifier. 2017-10-31 09:44:01 -07:00
Sean Mackrory 81225430d0 HDFS-206. Support for head in FSShell. Contributed by Gabor Bota. 2017-10-31 08:34:01 -06:00
Jason Lowe 2fae63aa60 HADOOP-14919. BZip2 drops records when reading data in splits. Contributed by Jason Lowe 2017-10-31 09:30:13 -05:00
Junping Du a25b5aa0cf Add several jdiff xml files for 2.8.2 release. 2017-10-26 19:30:48 -07:00
Junping Du 1ad863e220 Update CHANGES.md and RELEASENOTES for 2.8.2 release. 2017-10-26 19:13:58 -07:00
Xiao Chen f66ad1fab9 HADOOP-14957. ReconfigurationTaskStatus is exposing guava Optional in its public api. Contributed by Haibo Chen, Xiao Chen. 2017-10-25 14:11:30 -07:00
Akira Ajisaka 4c970088af
HADOOP-14030. PreCommit TestKDiag failure. Contributed by Wei-Chiu Chuang. 2017-10-25 17:17:21 +09:00
Varun Saxena 67e7673750 HADOOP-9657. NetUtils.wrapException to have special handling for 0.0.0.0 addresses and :0 ports. Contributed by Varun Saxena. 2017-10-25 03:06:12 +05:30
John Zhuge c904d60c38 Revert "HADOOP-14954. MetricsSystemImpl#init should increment refCount when already initialized. Contributed by John Zhuge."
This reverts commit 14b3c2695b.
2017-10-23 15:32:25 -07:00
Naganarasimha 6b8122458e HADOOP-14966. Handle JDK-8071638 for hadoop-common. Contributed by Bibin A Chundatt. 2017-10-23 23:42:06 +05:30
Arpit Agarwal 4b00c9a47f HDFS-12683. DFSZKFailOverController re-order logic for logging Exception. Contributed by Bharat Viswanadham. 2017-10-23 10:47:16 -07:00
John Zhuge 14b3c2695b HADOOP-14954. MetricsSystemImpl#init should increment refCount when already initialized. Contributed by John Zhuge. 2017-10-20 16:13:41 -07:00
Kai Zheng ce7cf66e5e HDFS-12448. Make sure user defined erasure coding policy ID will not overflow. Contributed by Huafeng Wang 2017-10-20 09:42:04 +08:00
Wei-Chiu Chuang 97c70c7ac6 HADOOP-14880. [KMS] Document&test missing KMS client side configs. Contributed by Gabor Bota. 2017-10-19 06:02:13 -07:00
Junping Du b016f08f67 HADOOP-14958. Fix source-level compatibility after HADOOP-11252. Contributed by Junping Du. 2017-10-18 15:06:30 -07:00
Xiao Chen 86ee0c5e4e HADOOP-14944. Add JvmMetrics to KMS. 2017-10-17 19:06:45 -07:00
Lei Xu 31ebccc962 HDFS-12613. Native EC coder should implement release() as idempotent function. (Lei (Eddy) Xu) 2017-10-16 19:44:30 -07:00
Wei-Chiu Chuang e906108fc9 HADOOP-14948. Document missing config key hadoop.treat.subject.external. Contributed by Ajay Kumar. 2017-10-16 16:42:59 -07:00
Xiao Chen b7ff624c76 HADOOP-14949. TestKMS#testACLs fails intermittently. 2017-10-16 14:19:31 -07:00
Manoj Govindassamy 133d7ca76e HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem 2017-10-13 17:43:21 -07:00
Robert Kanter e163f41850 HADOOP-14938. Configuration.updatingResource map should be initialized lazily (misha@cloudera.com via rkanter) 2017-10-13 13:52:58 -07:00
Anu Engineer 075358eb6f HADOOP-13102. Update GroupsMapping documentation to reflect the new changes. Contributed by Esther Kundin. 2017-10-11 15:58:20 -07:00
Arun Suresh b6c2c9058e HADOOP-13556. Change Configuration.getPropsWithPrefix to use getProps instead of iterator. (Larry McCay via asuresh) 2017-10-11 15:21:21 -07:00
Steve Loughran 3de574413c HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB.
Contributed by Varada Hemeswari.
2017-10-11 18:06:43 +01:00
Jason Lowe 1123f8f0b6 HADOOP-14912. FairCallQueue may defer servicing calls. Contributed by Daryn Sharp 2017-10-10 14:47:25 -05:00
Jitendra Pandey d6602b5f39 HDFS-11575. Supporting HDFS NFS gateway with Federated HDFS. Contributed by Mukul Kumar Singh. 2017-10-10 10:38:05 -07:00
John Zhuge 9676ed19be HADOOP-14184. Remove service loader config entry for ftp fs. Contributed by Sen Zhao. 2017-10-09 00:33:17 -07:00
Xiaoyu Yao 2b08a1fc64 HADOOP-14920. KMSClientProvider won't work with KMS delegation token retrieved from non-Java client. Contributed by Xiaoyu Yao. 2017-10-06 10:12:24 -07:00
Xiao Chen 25f31d9fc4 HADOOP-14521. KMS client needs retry logic. Contributed by Rushabh S Shah. 2017-10-05 19:46:47 -07:00
Akira Ajisaka fe84cdc9da
HADOOP-14616. Client fails to read a block with erasure code (XOR, native) when one of the data block is lost. Contributed by Huafeng Wang. 2017-10-05 11:06:09 +09:00
Daniel Templeton 20e9ce3ab3 HADOOP-14459. SerializationFactory shouldn't throw a NullPointerException if the serializations list is not defined
(Contributed by Nandor Kollar via Daniel Templeton)
2017-10-04 15:04:02 +02:00
Andrew Wang b34b3ff94b Add 3.0.0-beta1 CHANGES and RELEASENOTES. 2017-10-03 17:14:40 -07:00
Akira Ajisaka c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
Xiaoyu Yao 107c177782 HDFS-12455. WebHDFS - Adding "snapshot enabled" status to ListStatus query result. Contributed by Ajay Kumar. 2017-10-03 13:02:12 -07:00
Arpit Agarwal b91305119b HDFS-11968. ViewFS: StoragePolicies commands fail with HDFS federation. Contributed by Mukul Kumar Singh. 2017-10-03 11:23:40 -07:00
Allen Wittenauer 4d5dd75b60
HADOOP-14908. CrossOriginFilter should trigger regex on more input (Johannes Alberti via aw) 2017-10-03 10:58:28 -07:00
Anu Engineer 442ea85545 HADOOP-14752. TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky. Contributed by Andras Bokor. 2017-10-02 13:11:47 -07:00
Chen Liang 563dcdfc1d HADOOP-14915. method name is incorrect in ConfServlet. Contributed by Bharat Viswanadham. 2017-10-02 10:58:19 -07:00
Xiao Chen 06df6ab254 HADOOP-14095. Document caveats about the default JavaKeyStoreProvider in KMS. 2017-09-29 19:34:29 -07:00
Jason Lowe 6f789fe057 HADOOP-14902. LoadGenerator#genFile write close timing is incorrectly calculated. Contributed by Hanisha Koneru 2017-09-28 16:38:30 -05:00
Akira Ajisaka d08b8c801a
HADOOP-14893. WritableRpcEngine should use Time.monotonicNow. Contributed by Chetna Chaudhari. 2017-09-26 16:46:03 +09:00
Anu Engineer 0889e5a8b7 HDFS-12529. Get source for config tags from file name. Contributed by Ajay Kumar. 2017-09-25 13:55:09 -07:00
Jason Lowe ac05a51bbb HADOOP-14881. LoadGenerator should use Time.monotonicNow() to measure durations. Contributed by Bharat Viswanadham 2017-09-25 15:35:44 -05:00
Akira Ajisaka 3a10367a17
HADOOP-14892. MetricsSystemImpl should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari. 2017-09-26 01:47:02 +09:00
Daniel Templeton 0807470808 HADOOP-14897. Loosen compatibility guidelines for native dependencies 2017-09-25 14:12:25 +02:00
Anu Engineer ecc85620e8 HDFS-12339. NFS Gateway on Shutdown Gives Unregistration Failure. Does Not Unregister with rpcbind Portmapper. Contributed by Mukul Kumar Singh. 2017-09-23 12:27:34 -07:00
Anu Engineer 1c4a798505 HADOOP-14894. ReflectionUtils should use Time.monotonicNow to mesaure duration. Contributed by Bharat Viswanadham. 2017-09-23 11:56:34 -07:00
Anu Engineer 423ebcaf51 HDFS-12320. Add quantiles for transactions batched in Journal sync. Contributed by Hanisha Koneru. 2017-09-23 10:34:13 -07:00
Anu Engineer e1b32e0959 HADOOP-14901. ReuseObjectMapper in Hadoop Common. Contributed by Hanisha Koneru. 2017-09-22 15:45:22 -07:00
Akira Ajisaka 7e58b2478c
HADOOP-7308. Remove unused TaskLogAppender configurations from log4j.properties. Contributed by Todd Lipcon and J.Andreina. 2017-09-20 21:07:49 +09:00
Daniel Templeton 7618fa9194 HADOOP-13714. Tighten up our compatibility guidelines for Hadoop 3 2017-09-16 09:20:33 +02:00
Steve Loughran 2d2d97fa7d
HADOOP-14553. Add (parallelized) integration tests to hadoop-azure
Contributed by Steve Loughran
2017-09-15 17:03:01 +01:00
Aaron Fabbri 49467165a5
HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. Contributed by Steve Loughran. 2017-09-14 14:10:48 -07:00
John Zhuge 7ee02d1065 HADOOP-14864. FSDataInputStream#unbuffer UOE should include stream class name. Contributed by Bharat Viswanadham. 2017-09-14 11:26:49 -07:00
Andrew Wang 8277fab2be HADOOP-14857. Fix downstream shaded client integration test. Contributed by Sean Busbey. 2017-09-13 10:59:04 -07:00
Chen Liang 10b2cfa96c HADOOP-14804. correct wrong parameters format order in core-default.xml. Contributed by Chen Hongfei. 2017-09-13 10:49:34 -07:00
Xiao Chen fa6cc43edd Revert "HADOOP-14521. KMS client needs retry logic. Contributed by Rushabh S Shah."
This reverts commit 0a6d5c0cf1.
2017-09-12 19:53:07 -07:00
Andrew Wang f4b6267465 HDFS-12222. Document and test BlockLocation for erasure-coded files. Contributed by Huafeng Wang. 2017-09-12 17:35:30 -07:00
Jason Lowe 86f4d1c66c HADOOP-14843. Improve FsPermission symbolic parsing unit test coverage. Contributed by Bharat Viswanadham 2017-09-12 16:51:08 -05:00
Ray Chiang 556812c179 HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang) 2017-09-12 10:19:34 -07:00
Ray Chiang ad74691807 HADOOP-14798. Update sshd-core and related mina-core library versions. (rchiang) 2017-09-12 10:04:22 -07:00
Haibo Chen d2d8f665e6 YARN-7181. CPUTimeTracker.updateElapsedJiffies can report negative usage. (Miklos Szegedi via Haibo Chen) 2017-09-11 12:47:55 -07:00
Aaron Fabbri 180e814b08
HADOOP-14851 LambdaTestUtils.eventually() doesn't spin on Assertion failures. Contributed by Steve Loughran 2017-09-08 19:32:07 -07:00
Junping Du 3fddabc2fe YARN-7072. Add a new log aggregation file format controller. Contributed by Xuan Gong. 2017-09-08 15:16:19 -07:00
John Zhuge a4661850c1 HADOOP-14844. Remove requirement to specify TenantGuid for MSI Token Provider. Contributed by Atul Sikaria. 2017-09-08 11:51:03 -07:00
Anu Engineer e8278b02a4 HADOOP-14850. Read HttpServer2 resources directly from the source tree (if exists). Contributed by Elek, Marton. 2017-09-08 10:01:01 -07:00
Steve Loughran 5bbca80428
HADOOP-13421. Switch to v2 of the S3 List Objects API in S3A.
Contributed by Aaron Fabbri
2017-09-08 12:07:02 +01:00
Xiao Chen b3a4d7d2a0 HDFS-12400. Provide a way for NN to drain the local key cache before re-encryption. 2017-09-07 20:51:02 -07:00
Anu Engineer a4cd101934 HDFS-12350. Support meta tags in configs. Contributed by Ajay Kumar. 2017-09-07 12:40:09 -07:00
Jason Lowe dd814946f6 HADOOP-14827. Allow StopWatch to accept a Timer parameter for tests. Contributed by Erik Krogen 2017-09-06 16:04:30 -05:00
Chris Douglas 1f3bc63e67 HADOOP-12077. Provide a multi-URI replication Inode for ViewFs. Contributed by Gera Shegalov 2017-09-05 23:51:51 -07:00
Wei-Chiu Chuang ad32759fd9 HADOOP-14688. Intern strings in KeyVersion and EncryptedKeyVersion. Contributed by Xiao Chen. 2017-09-05 15:11:37 -07:00
Andrew Wang 0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Jitendra Pandey 063b6d0c93 HADOOP-14674. Correct javadoc for getRandomizedTempPath. Contributed by Mukul Kumar Singh. 2017-09-01 10:17:12 -07:00
Allen Wittenauer a3fee475f7
HADOOP-14414. Calling maven-site-plugin directly for docs profile is unnecessary (Andras Bokor via aw) 2017-09-01 08:38:30 -07:00
Steve Loughran 621b43e254
HADOOP-13345 HS3Guard: Improved Consistency for S3A.
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
2017-09-01 14:13:41 +01:00
Allen Wittenauer 27359b7139
HADOOP-14781. Clarify that HADOOP_CONF_DIR shouldn't actually be set in hadoop-env.sh
Signed-off-by: Andrew Wang <wang@apache.org>
2017-08-31 21:10:52 -07:00
Allen Wittenauer 0adc3a0533
YARN-6721. container-executor should have stack checking
Signed-off-by: Chris Douglas <cdouglas@apache.org>
2017-08-31 19:39:31 -07:00
Allen Wittenauer 190410085b
HADOOP-14364. refresh changelog/release notes with newer Apache Yetus build
Signed-off-by: Andrew Wang <wang@apache.org>
2017-08-31 19:06:49 -07:00
Yiqun Lin ac12e153a8 HDFS-12317. HDFS metrics render error in the page of Github. Contributed by Yiqun Lin. 2017-08-31 17:23:17 +08:00
Allen Wittenauer 71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms
Signed-off-by: Chris Douglas <cdouglas@apache.org>
2017-08-30 21:47:24 -07:00
Junping Du 41480233a9 HADOOP-14814. Fix incompatible API change on FsServerDefaults to HADOOP-14104. Contributed by Junping Du. 2017-08-30 13:30:14 -07:00
Rohith Sharma K S 32188d3295 YARN-7038. [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Steve Loughran 021974f4cb
HADOOP-14802. Add support for using container saskeys for all accesses.
Contributed by Sivaguru Sankaridurg
2017-08-29 19:02:43 +01:00
Arpit Agarwal 8b7cbe3840 HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar. 2017-08-25 09:55:46 -07:00
Steve Loughran 8196a07c32
HDFS-12344. LocatedFileStatus regression: no longer accepting null FSPermission.
Contributed by Ewan Higgs
2017-08-24 16:17:30 +01:00
John Zhuge 7e6463d2fb HADOOP-14251. Credential provider should handle property key deprecation. Contributed by John Zhuge. 2017-08-23 11:37:18 -07:00
Subru Krishnan 4249172e14 YARN-7053. Move curator transaction support to ZKCuratorManager. (Jonathan Hung via Subru). 2017-08-22 19:20:57 -07:00
Jason Lowe c379310212 HADOOP-14687. AuthenticatedURL will reuse bad/expired session cookies. Contributed by Daryn Sharp 2017-08-22 16:50:01 -05:00
Xiao Chen 4ec5acc704 HADOOP-14705. Add batched interface reencryptEncryptedKeys to KMS. 2017-08-22 07:47:39 -07:00
Arpit Agarwal 736ceab2f5 HDFS-12325. SFTPFileSystem operations should restore cwd. Contributed by Chen Liang. 2017-08-21 11:48:51 -07:00
Arpit Agarwal 2d105a2068 Revert "HADOOP-14732. ProtobufRpcEngine should use Time.monotonicNow to measure durations. Contributed by Hanisha Koneru."
This reverts commit 8bef4eca28.
2017-08-18 10:15:52 -07:00
Lei Xu 99e558b13b HADOOP-14398. Modify documents for the FileSystem Builder API. (Lei (Eddy) Xu) 2017-08-17 18:06:23 -07:00
Andrew Wang dd7916d3cd HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas. 2017-08-17 09:35:36 -07:00
Alexandr Krasheninnikov 1f04cb45f7 HADOOP-14560. Make HttpServer2 backlog size configurable. Contributed by Alexander Krasheninnikov.
This closes #242.
2017-08-17 01:05:19 -07:00
Andrew Wang 96b3a6b972 HDFS-11082. Provide replicated EC policy to replicate files. Contributed by SammiChen. 2017-08-16 22:17:06 -07:00
Akira Ajisaka 08aaa4b36f
HDFS-12269. Better to return a Map rather than HashMap in getErasureCodingCodecs. Contributed by Huafeng Wang. 2017-08-17 13:20:27 +09:00
Subru Krishnan 75dd866bfb HADOOP-14773. Extend ZKCuratorManager API for more reusability. (Íñigo Goiri via Subru). 2017-08-15 16:53:59 -07:00
Chris Douglas 645a8f2a4d HADOOP-14726. Mark FileStatus::isDir as final 2017-08-14 21:57:20 -07:00
Arpit Agarwal 0446511398 HADOOP-14673. Remove leftover hadoop_xml_escape from functions. Contributed by Ajay Kumar. 2017-08-14 16:22:10 -07:00
Arpit Agarwal 8bef4eca28 HADOOP-14732. ProtobufRpcEngine should use Time.monotonicNow to measure durations. Contributed by Hanisha Koneru. 2017-08-14 15:53:35 -07:00
John Zhuge 7769e96149 HADOOP-14627. Support MSI and DeviceCode token provider in ADLS. Contributed by Atul Sikaria. 2017-08-13 00:22:34 -07:00
Subru Krishnan bbbf0e2a41 HADOOP-14741. Refactor curator based ZooKeeper communication into common library. (Íñigo Goiri via Subru). 2017-08-11 13:58:45 -07:00
Xiao Chen 07fff43f4a HADOOP-14760. Add missing override to LoadBalancingKMSClientProvider. 2017-08-11 11:41:41 -07:00
John Zhuge 582648befa HADOOP-14260. Configuration.dumpConfiguration should redact sensitive information. Contributed by John Zhuge. 2017-08-11 10:16:08 -07:00
Allen Wittenauer 4222c97108
HADOOP-10392. Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem) (ajisakaa via aw) 2017-08-11 09:25:56 -07:00
Steve Loughran d964062f66
HADOOP-14754. TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties missing in classes.
Contributed by John Zhuge.
2017-08-11 10:18:17 +01:00
Wei-Chiu Chuang a8b75466b2 HADOOP-14743. CompositeGroupsMapping should not swallow exceptions. Contributed by Wei-Chiu Chuang. 2017-08-10 09:35:27 -07:00
Steve Loughran 54356b1e83
HADOOP-14183. Remove service loader config file for wasb fs.
Contributed by Esfandiar Manii.
2017-08-10 16:46:33 +01:00
Haohui Mai 1db4788b7d HADOOP-14598. Blacklist Http/HttpsFileSystem in FsUrlStreamHandlerFactory. Contributed by Steve Loughran. 2017-08-08 16:33:18 -07:00
Jian He acf9bd8b1d YARN-6890. Not display killApp button on UI if UI is unsecured but cluster is secured. Contributed by Junping Du 2017-08-08 11:09:38 -07:00
Arpit Agarwal bc206806da HDFS-12264. DataNode uses a deprecated method IoUtils#cleanup. Contributed by Ajay Yadav. 2017-08-07 15:05:10 -07:00
Xiao Chen a3a9c976c3 HADOOP-14727. Socket not closed properly when reading Configurations with BlockReaderRemote. Contributed by Jonathan Eagles. 2017-08-07 10:25:52 -07:00
Akira Ajisaka a4eb7016cb
HDFS-12198. Document missing namenode metrics that were added recently. Contributed by Yiqun Lin. 2017-08-07 18:47:33 +09:00
Lei Xu fe3341786a HDFS-12251. Add document for StreamCapabilities. (Lei (Eddy) Xu) 2017-08-04 11:21:58 -07:00
Arpit Agarwal a6fdeb8a87 HADOOP-13963. /bin/bash is hard coded in some of the scripts. Contributed by Ajay Yadav. 2017-08-04 10:40:52 -07:00
wenxinhe 691bf5ec5d
HADOOP-14706. Adding a helper method to determine whether a log is Log4j implement.
This closes #257

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-08-04 14:35:57 +09:00
Andrew Wang f4c6b00a9f HDFS-12131. Add some of the FSNamesystem JMX values as metrics. Contributed by Erik Krogen. 2017-08-03 15:44:51 -07:00
Chris Douglas 12e44e7bda HDFS-6984. Serialize FileStatus via protobuf. 2017-08-02 12:12:48 -07:00
Sean Mackrory 1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Steve Loughran 5e4434f628
HADOOP-14709. Fix checkstyle warnings in ContractTestUtils.
Contributed by Thomas Marquardt.

(cherry picked from commit cb46792610)
2017-08-02 12:48:06 +01:00
Masatake Iwasaki 3a0b84d9c9 HADOOP-14702. Fix formatting issue and regression caused by conversion from APT to Markdown. Contributed by Doris Gu. 2017-08-02 06:06:25 +00:00
Junping Du cf6794ebe9 HADOOP-14683. FileStatus.compareTo binary compatible issue. Contributed by Akira Ajisaka. 2017-08-01 17:13:02 -07:00
Lei Xu 02cd71ba9d HADOOP-14495. Add set options interface to FSDataOutputStreamBuilder. (Lei (Eddy) Xu) 2017-08-01 16:00:53 -07:00
Andrew Wang a11c230236 HADOOP-14701. Configuration can log misleading warnings about an attempt to override final parameter. Contributed by Andrew Sherman. 2017-08-01 15:13:29 -07:00
Steve Loughran 778d4edd9a
HADOOP-14660 wasb: improve throughput by 34% when account limit exceeded.
Contributed by Thomas Marquardt.
2017-08-01 21:36:28 +01:00
Arpit Agarwal 72eba25e09 HDFS-14676. Wrong default value for fs.df.interval. Contributed by xiangguang zheng. 2017-08-01 13:31:44 -07:00
Lei Xu 9586b0e24f HADOOP-14397. Pull up the builder pattern to FileSystem and add AbstractContractCreateTest for it. (Lei (Eddy) Xu) 2017-07-31 20:12:40 -07:00
Andras Bokor abbf4129a2
HADOOP-14343. Wrong pid file name in error message when starting secure daemon
Signed-off-by: Allen Wittenauer <aw@apache.org>
2017-07-31 20:03:43 -07:00
Akira Ajisaka f14be0d241
HADOOP-14690. RetryInvocationHandler should override toString(). Contributed by Yeliang Cang. 2017-07-31 14:08:30 +09:00
Ravi Prakash 746189ad8c HADOOP-14229. hadoop.security.auth_to_local example is incorrect in the documentation. Contributed by Andras Bokor. 2017-07-28 11:43:36 -07:00
Yongjun Zhang c6330f22a5 HDFS-12190. Enable 'hdfs dfs -stat' to display access time. Contributed by Yongjun Zhang. 2017-07-27 16:48:24 -07:00
Kihwal Lee 27a1a5fde9 HADOOP-14578. Bind IPC connections to kerberos UPN host for proxy users. Contributed by Daryn Sharp. 2017-07-26 13:12:39 -05:00
Brahma Reddy Battula 6d983cca52 HADOOP-14455. ViewFileSystem#rename should support be supported within same nameservice with different mountpoints. Contributed by Brahma Reddy Battula. 2017-07-25 23:51:53 +08:00
Akira Ajisaka cca51e916b
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 2017-07-25 15:24:56 +09:00
Jitendra Pandey f2921e51f0 HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov. 2017-07-24 23:01:01 -07:00
Ravi Prakash 94ca52ae9e HADOOP-14597. Native compilation broken with OpenSSL-1.1.0. Contributed by Ravi Prakash. 2017-07-24 16:01:45 -07:00
Jason Lowe 24853bf32a YARN-6768. Improve performance of yarn api record toString and fromString. Contributed by Jonathan Eagles 2017-07-24 10:19:20 -05:00
Wei-Chiu Chuang 465c213439 HDFS-12094. Log torrent when none isa-l EC is used. Contributed by LiXin Ge. 2017-07-21 12:25:03 -07:00
Kihwal Lee b3269f7cc1 HADOOP-14659. UGI getShortUserName does not need to search the Subject. Contributed by Daryn Sharp. 2017-07-20 16:12:21 -05:00
Daniel Templeton c21c260392 HADOOP-14666. Tests use assertTrue(....equals(...)) instead of assertEquals() 2017-07-19 13:58:55 -07:00
Jason Lowe df180259b0 HADOOP-14669. GenericTestUtils.waitFor should use monotonic time. Contributed by Daniel Templeton 2017-07-19 09:41:22 -05:00
Jitendra Pandey 2843c688bc HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg. 2017-07-19 00:13:06 -07:00
Jason Lowe 5aa2bf231f HADOOP-14637. GenericTestUtils.waitFor needs to check condition again after max wait time. Contributed by Daniel Templeton 2017-07-18 16:23:41 -05:00
Akira Ajisaka ccaf036662
HADOOP-14539. Move commons logging APIs over to slf4j in hadoop-common. Contributed by Wenxin He. 2017-07-18 13:32:37 +09:00
Anu Engineer 02b141ac60 HDFS-11786. Add support to make copyFromLocal multi threaded. Contributed by Mukul Kumar Singh. 2017-07-16 10:59:34 -07:00
Xiao Chen 0a6d5c0cf1 HADOOP-14521. KMS client needs retry logic. Contributed by Rushabh S Shah. 2017-07-14 22:22:53 -07:00
Masatake Iwasaki b61ab8573e HADOOP-14646. FileContextMainOperationsBaseTest#testListStatusFilterWithSomeMatches never runs. Contributed by Andras Bokor. 2017-07-13 21:41:43 +09:00
Akira Ajisaka 9144fd9e9b
HADOOP-14629. Improve exception checking in FileContext related JUnit tests. Contributed by Andras Bokor. 2017-07-12 11:35:50 +09:00
Steve Loughran d670c3a4da
HADOOP-14535 wasb: implement high-performance random access and seek of block blobs.
Contributed by Thomas Marquardt
2017-07-11 21:34:27 +01:00
Yiqun Lin 3a7f02b815 HDFS-12085. Reconfigure namenode heartbeat interval fails if the interval was set with time unit. Contributed by Weiwei Yang. 2017-07-11 20:40:11 +08:00
wenxinhe fce7951014
HADOOP-14638. Replace commons-logging APIs with slf4j in StreamPumper.
This closes #247

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-07-11 13:30:13 +09:00
Jitendra Pandey f1efa14fc6 HADOOP-10829. Iteration on CredentialProviderFactory.serviceLoader is thread-unsafe. Contributed by Benoy Antony and Rakesh R. 2017-07-10 17:48:27 -07:00
Steve Loughran 09653ea098
HADOOP-14634. Remove jline from main Hadoop pom.xml.
Contributed by Ray Chiang.
2017-07-10 11:53:13 +01:00
Andrew Wang f10864a820 Add release notes, changes, jdiff for 3.0.0-alpha4 2017-07-07 11:01:59 -07:00
Akira Ajisaka 7cd095272c
HADOOP-14587. Use GenericTestUtils.setLogLevel when available in hadoop-common. Contributed by Wenxin He. 2017-07-08 02:54:24 +09:00
Akira Ajisaka 8fc5dcc2a1
Revert "HADOOP-14587. Use GenericTestUtils.setLogLevel when available in hadoop-common. Contributed by Wenxin He."
This reverts commit 82cb2a6497.
2017-07-08 02:53:18 +09:00
Wei-Chiu Chuang 8153fe2bd3 HADOOP-14563. LoadBalancingKMSClientProvider#warmUpEncryptedKeys swallows IOException. Contributed by Rushabh S Shah. 2017-07-07 06:13:10 -07:00
Akira Ajisaka 82cb2a6497
HADOOP-14587. Use GenericTestUtils.setLogLevel when available in hadoop-common. Contributed by Wenxin He. 2017-07-07 15:00:47 +09:00
John Zhuge 946dd25675 HADOOP-14608. KMS JMX servlet path not backwards compatible. Contributed by John Zhuge. 2017-07-05 11:16:56 -07:00
Vinayakumar B a180ba4081 HADOOP-13414. Hide Jetty Server version header in HTTP responses. Contributed by Surendra Singth Lilhore. 2017-07-05 16:05:18 +05:30
Steve Loughran b17e655b70 HADOOP-14617. Add ReflectionUtils.logThreadInfo that accept slf4j logger API.
Contributed by Wenxin He.
2017-07-04 11:41:07 +01:00
Wenxin He f2aba1da30
HADOOP-14571. Deprecate public APIs relate to log4j1
This closes #244

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-07-04 18:55:20 +09:00
Steve Loughran 1aaa7f1eac
HADOOP-14615. Add ServiceOperations.stopQuietly that accept slf4j logger API.
Contributed by Wenxin He.
2017-07-04 10:48:02 +01:00
Andrew Wang af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Daniel Templeton 5a75f73893 HADOOP-14611. NetworkTopology.DEFAULT_HOST_LEVEL is unused
(Contributed by Chen Liang via Daniel Templeton)
2017-06-29 12:28:43 -07:00
Kai Zheng ea1da39b19 HADOOP-14479. Erasurecode testcase failures with native enabled. Contributed by Sammi Chen 2017-06-29 11:26:36 +08:00
Lei Xu d6df0fdbbd Add -E option in 'ls' to list erasure coding policy of each file and directory if applicable. Contributed by luhuichun via lei. 2017-06-28 13:47:23 -07:00
Xiao Chen fb5ee3fafb HADOOP-14515. Addendum. Specifically configure zookeeper-related log levels in KMS log4j. 2017-06-27 23:49:09 -07:00
Kai Zheng 379f19a2c7 HDFS-11943. [Erasure coding] Warn log frequently print to screen in doEncode/doDecode functions. Contributed by liaoyuxiangqin 2017-06-26 16:26:09 +08:00
Akira Ajisaka a594f96fb1
HADOOP-14549. Use GenericTestUtils.setLogLevel when available in hadoop-tools. Contributed by wenxin he. 2017-06-26 16:41:00 +09:00
Arpit Agarwal 0111711d8b HADOOP-14543. ZKFC should use getAversion() while setting the zkacl. Contributed by Brahma Reddy Battula. 2017-06-23 16:50:47 -07:00
Steve Loughran e5db9af8a0
HADOOP-14568. GenericTestUtils#waitFor missing parameter verification.
Contributed by Yiqun Lin
2017-06-23 14:39:58 +01:00
Akira Ajisaka 092ebdf885
HADOOP-12940. Fix warnings from Spotbugs in hadoop-common. 2017-06-23 10:28:58 +09:00
Arpit Agarwal 8ceaac1877 HDFS-12010. TestCopyPreserveFlag fails consistently because of mismatch in access time. Contributed by Mukul Kumar Singh. 2017-06-22 15:42:50 -07:00
Vinayakumar B 8dbd53ef9f HDFS-11067. DFS#listStatusIterator(..) should throw FileNotFoundException if the directory deleted before fetching next batch of entries. Contributed by Vinayakumar B. 2017-06-22 17:37:08 +05:30
Akira Ajisaka b64951905e
HADOOP-14542. Add IOUtils.cleanupWithLogger that accepts slf4j logger API. Contributed by Chen Liang. 2017-06-22 17:42:59 +09:00
Daryn Sharp e806c6e0ce HADOOP-14146. KerberosAuthenticationHandler should authenticate with SPN in AP-REQ. Contributed by Daryn Sharp 2017-06-21 11:03:41 -05:00
Akira Ajisaka 1a598479a9
HDFS-11992. Replace commons-logging APIs with slf4j in FsDatasetImpl. Contributed by hu xiaodong. 2017-06-21 11:19:48 +09:00
Yongjun Zhang 5157f6c46e HADOOP-14429. FTPFileSystem#getFsAction always returns FsAction.NONE. (Hongyuan Li via Yongjun Zhang) 2017-06-20 16:51:13 -07:00
Lei Xu 45ff4d38e6 HDFS-11647. Add -E option in hdfs "count" command to show erasure policy summarization. Contributed by luhuichun. 2017-06-20 11:55:09 -07:00
Brahma Reddy Battula 2e9daa2e27 HADOOP-14533. Size of args cannot be less than zero in TraceAdmin#run as its linkedlist. Contributed by Weisen Han. 2017-06-20 20:44:31 +08:00
Xiao Chen 15964ef147 HADOOP-14515. Specifically configure zookeeper-related log levels in KMS log4j. 2017-06-19 23:12:02 -07:00
Akira Ajisaka 300804595e
HADOOP-14540. Replace MRv1 specific terms in HostsFileReader. Contributed by hu xiaodong. 2017-06-19 19:39:36 +09:00
Akira Ajisaka 7ade5124b8
HADOOP-14538. Fix TestFilterFileSystem and TestHarFileSystem failures after DistributedFileSystem.append API. Contributed by Lei (Eddy) Xu. 2017-06-19 19:09:18 +09:00
Masatake Iwasaki 75043d3429 HADOOP-14424. Add CRC32C performance test. Contributed by LiXin Ge. 2017-06-19 09:16:45 +09:00
Lei Xu 6460df21a0 HADOOP-14395. Provide Builder pattern for DistributedFileSystem.append. Contributed by Lei (Eddy) Xu. 2017-06-16 17:24:00 -07:00
Xiao Chen 4ebc23ba7b HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up proactively. 2017-06-16 09:49:33 -07:00
Steve Loughran 942ecbbc98
HADOOP-14486 TestSFTPFileSystem#testGetAccessTime test failure using openJDK 1.8.0.
Contributed by Hongyuan Li.
2017-06-16 09:45:22 +01:00
Akira Ajisaka e855cc4ed4
HADOOP-14289. Move log4j APIs over to slf4j in hadoop-common. 2017-06-16 13:17:10 +09:00
Lei Xu 5e7cfdca7b HADOOP-14394. Provide Builder pattern for DistributedFileSystem.create. (lei) 2017-06-15 10:59:24 -07:00
Xiao Chen ef8edab930 HADOOP-14523. OpensslAesCtrCryptoCodec.finalize() holds excessive amounts of memory. Contributed by Misha Dmitriev. 2017-06-15 10:40:59 -07:00
Steve Loughran d780a67864
HADOOP-14506. Add create() contract test that verifies ancestor dir creation.
Contributed by Sean Mackrory.
2017-06-15 14:49:34 +01:00
Arpit Agarwal 8633ef8e10 HADOOP-14503. Make RollingAverages a mutable metric. Contributed by Hanisha Koneru. 2017-06-12 20:43:43 -07:00
Jonathan Eagles a81916ea89 HADOOP-14501. Switch from aalto-xml to woodstox to handle odd XML features (jeagles) 2017-06-12 17:07:53 -05:00
Daniel Templeton 86368cc766 HADOOP-14310. RolloverSignerSecretProvider.LOG should be @VisibleForTesting
(Contributed by Arun Shanmugam Kumar via Daniel Templeton)
2017-06-12 09:42:16 -07:00
Mingliang Liu a2121cb0d9 HADOOP-14465. LdapGroupsMapping - support user and group search base. Contributed by Shwetha G S and Mingliang Liu 2017-06-09 14:55:07 -07:00
Xiao Chen c4b5c32669 HADOOP-13854. KMS should log error details in KMSExceptionsProvider. 2017-06-08 22:35:31 -07:00
Xiao Chen 0202480742 HADOOP-13174. Add more debug logs for delegation tokens and authentication. 2017-06-08 21:34:15 -07:00
John Zhuge 5672ae7b37 HDFS-11861. ipc.Client.Connection#sendRpcRequest should log request name. Contributed by John Zhuge. 2017-06-07 15:52:52 -07:00
Carlo Curino b65100c14b YARN-6547. Enhance SLS-based tests leveraging invariant checker. 2017-06-06 17:19:15 -07:00
Brahma Reddy Battula 19ef3a81f8 HADOOP-14485. Redundant 'final' modifier in try-with-resources statement. Contributed by wenxin he. 2017-06-06 23:11:47 +08:00
Kihwal Lee 855e0477b1 HADOOP-14035. Reduce fair call queue backoff's impact on clients. Contributed by Daryn Sharp. 2017-06-06 08:35:12 -05:00
Brahma Reddy Battula 66c6fd8314 HADOOP-14431. ModifyTime of FileStatus returned by SFTPFileSystem's getFileStatus method is wrong. Contributed by Hongyuan Li. 2017-06-06 12:31:40 +08:00
Mingliang Liu 6aeda55bb8 HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu 2017-06-05 11:26:56 -07:00
Brahma Reddy Battula abdd609e51 HADOOP-14440. Add metrics for connections dropped. Contributed by Eric Badger. 2017-06-06 00:21:03 +08:00
Andrew Wang 7101477e47 HDFS-11383. Intern strings in BlockLocation and ExtendedBlock. Contributed by Misha Dmitriev. 2017-06-01 15:20:18 -07:00
Kihwal Lee 1543d0f5be HDFS-5042. Completed files lost after power failure. Contributed by Vinayakumar B. 2017-05-31 10:55:03 -05:00
Brahma Reddy Battula 62857be211 HADOOP-14456. Modifier 'static' is redundant for inner enums. Contributed by ZhangBing Lin. 2017-05-31 01:07:58 +08:00
Arpit Agarwal bd6a2172e0 HDFS-11891. DU#refresh should print the path of the directory when an exception is caught. Contributed by Chen Liang. 2017-05-26 16:02:40 -07:00
Andrew Wang 2cd612ba8e Update 3.0.0-alpha3 changes, release notes, jdiff. 2017-05-26 14:14:38 -07:00
Andrew Wang 16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Steve Loughran 8bf0e2d6b3
HADOOP-14430 the accessTime of FileStatus returned by SFTPFileSystem's
getFileStatus method is always 0.
Contributed by Hongyuan Li.
2017-05-25 15:19:58 +01:00
Steve Loughran 1ba9704eec
HADOOP-14399. Configuration does not correctly XInclude absolute file URIs.
Contributed by Jonathan Eagles
2017-05-25 14:59:33 +01:00
Akira Ajisaka 6a52b5e144
HADOOP-14180. FileSystem contract tests to replace JUnit 3 with 4. Contributed by Xiaobing Zhou and Mingliang Liu. 2017-05-25 15:36:44 +09:00
Brahma Reddy Battula dcf4559ebd HADOOP-14166. Reset the DecayRpcScheduler AvgResponseTime metric to zero when queue is not used. Contributed by Surendra Singh Lilhore. 2017-05-25 01:27:13 +08:00