8ffc356b1e
Revert "SPNEGO TLS verification"
...
This reverts commit ba66f3b454
2020-05-13 17:14:14 +09:00
936bf09c37
HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq.
2020-05-12 21:54:54 +05:30
a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) ( #2007 )
...
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
2020-05-12 10:50:04 -05:00
047d8879e7
HDFS-15345. RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442.
2020-05-12 08:31:04 -07:00
29dddb8a14
HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun.
2020-05-12 15:07:51 +01:00
928b81a533
HDFS-15350. Set dfs.client.failover.random.order to true as default. ( #2008 )
2020-05-12 09:04:03 -05:00
8dad38c0be
HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei.
2020-05-12 18:34:26 +05:30
0fe49036e5
HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu.
2020-05-12 13:11:31 +05:30
4c53fb9ce1
HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun.
2020-05-11 16:48:34 +05:30
aab9e0b16e
HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest.
2020-05-10 11:43:30 +05:30
cb64e993c2
HDFS-15347. Replace the deprecated method shaHex. Contributed by bianqi.
2020-05-10 00:11:46 +05:30
d59de27c01
HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole.
2020-05-08 00:53:49 +05:30
130f89e068
HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina.
2020-05-06 13:18:32 -07:00
7fddf4855e
HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun.
2020-05-06 16:55:04 +05:30
adecdb8b53
HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. ( #1998 )
...
Reviewed-by: Arpit Agarwal <arp@apache.org>
2020-05-05 16:04:54 -07:00
9c8236d04d
HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G.
2020-05-04 17:55:40 -07:00
1996351b0b
HDFS-15270. Account for *env == NULL in hdfsThreadDestructor ( #1951 )
...
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.
The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.
Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
2020-05-04 13:08:53 -07:00
ebb878bab9
HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko.
2020-05-04 10:29:50 -07:00
8dace8ff3a
HDFS-15328. Use DFSConfigKeys MONITOR_CLASS_DEFAULT constant. Contributed by bianqi.
2020-05-03 23:59:17 +05:30
d6fc482a54
Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida.
2020-05-03 19:09:14 +05:30
44de193bec
HDFS-15325. TestRefreshCallQueue is failing due to changed CallQueue constructor. Contributed by Fengnan Li
2020-05-02 20:56:29 -07:00
f40dacdc2e
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter ( #1992 )
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-05-02 13:02:27 -07:00
ba66f3b454
SPNEGO TLS verification
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-05-02 21:50:13 +09:00
82343790ee
HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee.
2020-05-01 12:16:02 +05:30
263c76b678
HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2020-04-30 14:15:28 -07:00
b5b45c53a4
HDFS-15309. Remove redundant String.valueOf method on ExtendedBlockId.java. Contributed by bianqi.
2020-04-30 15:46:46 +09:00
31b2f687ef
HDFS-15265. HttpFS: validate content-type in HttpFSUtils. Contributed by hemanthboyina.
2020-04-29 13:41:36 -07:00
9ca6298a9a
HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun.
2020-04-29 15:56:06 +01:00
db6252b6c3
HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore.
2020-04-29 10:58:35 +05:30
816042e62b
HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.( #1980 )
2020-04-29 09:19:59 +05:30
5e0eda5d5f
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina.
2020-04-28 06:17:14 +09:00
62c26b91fd
HDFS-15298 Fix the findbugs warnings introduced in HDFS-15217 ( #1979 )
2020-04-27 17:20:47 +09:00
74d4a18735
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena
2020-04-25 15:43:02 -07:00
453c9c9a67
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address ( #1964 )
...
Contributed by Dhiraj Hegde.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
2020-04-25 13:04:32 -07:00
453771f170
HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun.
2020-04-24 11:53:24 -07:00
5958af4dbc
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina
2020-04-22 13:31:02 -07:00
e069a06137
HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky
2020-04-20 08:41:18 -07:00
1fdfaebd98
HDFS-15275. HttpFS: Response of Create was not correct with noredirect and data are true. Contributed by hemanthboyina.
2020-04-20 23:09:33 +09:00
1824aee9da
HDFS-15217 Add more information to longest write/read lock held log
2020-04-18 13:52:07 -07:00
8e6227441a
HDFS-15218. RBF: MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by Surendra Singh Lilhore.
2020-04-18 20:07:21 +05:30
37d6582223
HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena.
2020-04-17 13:32:02 +05:30
3481895f8a
HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-04-16 20:18:25 -07:00
cc5c1da7c1
HDFS-15277. Parent directory in the explorer does not support all path formats. Contributed by Jianfei Jiang.
2020-04-15 23:01:58 +05:30
8d49229c37
HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.
2020-04-12 12:05:50 +05:30
275c478330
HDFS-15261. RBF: Add Block Related Metrics. Contributed by Ayush Saxena.
2020-04-09 23:44:36 +05:30
061afcdf30
HDFS-15269. NameNode should check the authorization API version only … ( #1945 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-04-09 09:19:35 -07:00
1189af4746
HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena.
2020-04-08 15:55:14 +05:30
c12ddbd1de
HDFS-15249 ThrottledAsyncChecker is not thread-safe. ( #1922 )
2020-04-07 14:38:09 +09:00
0b855b9f35
HDFS-15256. Fix typo in DataXceiverServer#run(). Contributed by Lisheng Sun.
2020-04-06 13:05:33 -07:00
1695d8d59c
HDFS-15258. RBF: Mark Router FSCK unstable. ( #1934 )
2020-04-04 00:13:41 +09:00
736659e0e1
HDFS-15252. HttpFS: setWorkingDirectory should not accept invalid paths. Contributed by hemanthboyina.
2020-04-02 19:21:02 +09:00
4a3eb10972
HDFS-15051. RBF: Impose directory level permissions for Mount entries. Contributed by Xiaoqiao He.
2020-04-02 11:06:01 +05:30
d3b5951572
HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2020-04-01 16:36:46 -07:00
80b877a72f
HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li.
2020-03-30 12:29:21 -07:00
960c9ebaea
HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang.
2020-03-30 02:40:25 +05:30
8914cf9167
Preparing for 3.4.0 development
2020-03-29 23:24:25 +05:30
f7a94ec0a4
HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina.
2020-03-29 08:17:02 -07:00
696a663cd0
HDFS-15193. Improving the error message for missing `dfs.namenode.rpc-address.$NAMESERVICE`. Contributed by Ctest.
2020-03-29 00:01:22 +05:30
f531a4a487
HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand.
2020-03-27 12:55:38 -07:00
cbe71eaddf
HDFS-15238. RBF: NamenodeHeartbeatService caused memory to grow rapidly. Contributed by xuzq.
2020-03-27 13:42:06 +05:30
679631b188
HDFS-13470. RBF: Add Browse the Filesystem button to the UI.
2020-03-26 08:23:18 -07:00
0fa7bf47df
HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. ( #1909 )
2020-03-25 16:03:26 -07:00
cdcb77a2c5
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He.
2020-03-25 11:30:54 -07:00
a700803a18
HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle.
...
Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
2020-03-25 10:28:30 -07:00
d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong
2020-03-24 14:50:15 -07:00
d9c4f1129c
HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error ( #1902 ). Contributed by zhengchenyu.
2020-03-24 23:17:22 +05:30
ea87d60493
HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun.
2020-03-24 22:56:09 +05:30
5250cd6db3
HDFS-15201 SnapshotCounter hits MaxSnapshotID limit ( #1870 )
2020-03-24 15:15:46 +05:30
5eddc82fb8
HDFS-15088. RBF: Correct annotation typo of RouterPermissionChecker#checkPermission. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-03-23 22:41:31 -07:00
f59f6891c8
HDFS-15232. Fix libhdfspp test failures with GCC 7. ( #1906 )
2020-03-24 11:36:14 +09:00
af64ce2f4a
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-03-23 12:43:51 -07:00
62c8015683
HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena.
2020-03-22 11:44:18 +05:30
b74e47e31a
HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina.
2020-03-22 01:44:55 +09:00
f9bb2a8cc5
HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena.
2020-03-19 22:26:34 +05:30
bb41ddaf1e
HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena.
2020-03-19 21:23:13 +05:30
654db35fa2
HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.
2020-03-18 21:00:45 +05:30
096533c2dc
HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs.
...
Contributed by Wei-Chiu Chuang.
Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
2020-03-18 13:27:13 +00:00
1d772dc542
HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.
2020-03-15 20:44:32 +05:30
74780c22eb
HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang.
2020-03-15 16:30:39 +05:30
8a7c54995a
HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina.
2020-03-15 16:16:27 +05:30
7b637a3341
HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun.
2020-03-13 19:01:23 -07:00
e9955bb8ff
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2020-03-13 18:55:06 -07:00
4b95c242ec
HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... ( #1829 )
...
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
2020-03-13 11:29:44 -07:00
fb1d7635ae
HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang.
2020-03-13 21:10:03 +05:30
25497799cb
HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree.
2020-03-13 15:24:19 +00:00
20903f72b4
HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun.
2020-03-12 19:23:12 -07:00
0695f7a538
HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang.
2020-03-12 17:22:11 -07:00
f736408a83
HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree.
2020-03-12 18:59:17 +05:30
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface
...
Contributed by Steve Loughran.
This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)
This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.
instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.
The FileSystem implementation is cut; tests updated as appropriate.
All new interfaces/classes/constants are marked as @unstable.
Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
2020-03-09 14:51:16 +00:00
d4d4c37810
HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden
...
Contributed by Steve Loughran.
Not all stores do complete validation here; in particular the S3A
Connector does not: checking up the entire directory tree to see if a path matches
is a file significantly slows things down.
This check does take place in S3A mkdirs(), which walks backwards up the list of
parent paths until it finds a directory (success) or a file (failure).
In practice production applications invariably create destination directories
before writing 1+ file into them -restricting check purely to the mkdirs()
call deliver significant speed up while implicitly including the checks.
Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0
2020-03-09 14:44:28 +00:00
4062217189
HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena.
2020-03-06 10:14:53 -08:00
3afd4cbe89
HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina.
2020-03-04 10:13:23 -08:00
1d4d0fcbe1
HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein.
2020-03-04 10:02:54 -08:00
bbd704bb82
HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. ( #1871 )
2020-03-04 11:07:31 +09:00
0dd8956f2e
HADOOP-16885. Encryption zone file copy failure leaks a temp file
...
Contributed by Xiaoyu Yao.
Contains HDFS-14892. Close the output stream if createWrappedOutputStream() fails
Copying file through the FsShell command into an HDFS encryption zone where
the caller lacks permissions is leaks a temp ._COPYING file
and potentially a wrapped stream unclosed.
This is a convergence of a fix for S3 meeting an issue in HDFS.
S3: a HEAD against a file can cache a 404,
-you must not do any existence checks, including deleteOnExit(),
until the file is written.
Hence: HADOOP-16490, only register files for deletion the create worked
and the upload is not direct.
HDFS-14892. HDFS doesn't close wrapped streams when IOEs are raised on
create() failures. Which means that an entry is retained on the NN.
-you need to register a file with deleteOnExit() even if the file wasn't
created.
This patch:
* Moves the deleteOnExit to ensure the created file get deleted cleanly.
* Fixes HDFS to close the wrapped stream on failures.
2020-03-02 13:22:00 +00:00
1a636da041
HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun.
2020-02-29 09:55:29 +05:30
97b797c314
HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun.
2020-02-28 18:47:22 -08:00
9a3a28018a
HDFS-15190. HttpFS: Add Support for Storage Policy Satisfier. Contributed by hemanthboyina.
2020-02-29 10:30:26 +09:00
fccfb02672
HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena.
2020-02-28 18:33:06 +05:30
429da635ec
HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong.
2020-02-28 00:31:55 +05:30
Akira Ajisaka
Ayush Saxena
Elixir Kook
Xiaoyu Yao
S O'Donnell
Takanobu Asanuma
Inigo Goiri
Wei-Chiu Chuang
Uma Maheswara Rao G
Babneet Singh
Konstantin V Shvachko
Mingliang Liu
Eric Yang
Shashikant Banerjee
Surendra Singh Lilhore
Aryan Gupta
Toshihiro Suzuki
Dhiraj
Stephen O'Donnell
He Xiaoqiao
Brahma Reddy Battula
Chen Liang
Arpit Agarwal
Isa Hekmatizadeh
Karthik Palanisamy
Steve Loughran
Akshay Nehe