HDFS-16368. DFSadmin supports refresh topology info without restarting namenode #3743
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
hfutatzhanghb
force-pushed
the
trunk-dev-zhb-refreshTopology
branch
from
e129bac to
12b3c37
Compare
|
💔 -1 overall
This message was automatically generated. |
hfutatzhanghb
force-pushed
the
trunk-dev-zhb-refreshTopology
branch
from
12b3c37 to
5f5375e
Compare
|
💔 -1 overall
This message was automatically generated. |
|
@ayushtkn @goiri @Hexiaoqiao @tomscut , sorry for disturbing you. could we push this PR forward? any suggestions? |
ayushtkn
reviewed
May 18, 2023
Comment on lines
1132
to
1224
| @Override | ||
| public void refreshTopology() throws IOException { | ||
| // Router not support this operation, because this maybe refresh multi namespaces | ||
| String methodName = RouterRpcServer.getMethodName(); | ||
| throw new UnsupportedOperationException( | ||
| "Operation \"" + methodName + "\" is not supported"); | ||
| } | ||
|
|
There was a problem hiding this comment.
in case of router it should delegate the request to all namespaces, that is how all other commands of similar nature work
| * refresh the network topology of this cluster based on the mapping_topology.data file. | ||
| */ | ||
| public void refreshTopology() throws IOException { | ||
| long start = System.currentTimeMillis(); |
There was a problem hiding this comment.
use Time.monotonicNow for duration calculations
There was a problem hiding this comment.
Sir, have fixed.
hfutatzhanghb
force-pushed
the
trunk-dev-zhb-refreshTopology
branch
from
6e283f2 to
3f576ca
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Not sure why this PR has been pick up again. IIRC we have discussed add tools to support refresh topology online many times. I would like to share my concern again. This is one high-risky operation, especially for one large cluster. Considering the case that change the cluster topology randomly then namenode failover or restart, there will be block transfer flood where the replications of one block should be going to have enough racks. Before all blocks have enough racks (which is cost long times) the cluster could enter high-load phase and couldn't process any requests from client. In one word, be careful to refresh topology online if there is no stability guarantee measures. |
@Hexiaoqiao Sir, thanks for your reminding. I got your concern here. I think this PR just provide a tool which can refresh topology without restarting namenode. As for how to use this tool safely to prevent causing transfer flood belongs to end-user. They can modify a small part of topologay_mappings.data every time, then refreshTopology and wait for transfering completed. Do this procedure until all parts of topologay_mappings.data updated finally. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Currently in HDFS, if we update the rack info for rack-awareness, we may need to rolling restart namenodes to let it be effective. If cluster is large, the cost time of rolling restart namenodes is very log. So, we develope a method to refresh topology info without rolling restart namenodes.
How was this patch tested?
fsckto check its block location and rack info.For code changes:
https://issues.apache.org/jira/browse/HDFS-16368