Skip to content

Commit 35d77a6

Browse files
slfan1989slfan1989
authored
YARN-11505. [Federation] Add Steps To Set up a Test Cluster. (#5751)
1 parent e14c52c commit 35d77a6

File tree

1 file changed

+264
-0
lines changed
  • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown

1 file changed

+264
-0
lines changed

‎hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md‎

Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -385,3 +385,267 @@ The output from this particular example job should be something like:
385385

386386
The state of the job can also be tracked on the Router Web UI at `routerhost:8089`.
387387
Note that no change in the code or recompilation of the input jar was required to use federation. Also, the output of this job is the exact same as it would be when run without federation. Also, in order to get the full benefit of federation, use a large enough number of mappers such that more than one cluster is required. That number happens to be 16 in the case of the above example.
388+
389+
How to build a Test Federation Cluster
390+
--------------------
391+
392+
The purpose of this document is to help users quickly set up a testing environment for YARN Federation. With this testing environment, users can utilize the core functionality of YARN Federation. This is the simplest test cluster setup (based on Linux) with only essential configurations (YARN non-HA mode). We require 3 machines, and each machine should have at least <4C, 8GB> of resources. We only cover YARN configuration in this document. For information on configuring HDFS and ZooKeeper, please refer to other documentation sources.
393+
394+
Test Environment Description:
395+
- We need to build a HDFS test environment, this part can refer to HDFS documentation. [HDFS SingleCluster](../../hadoop-project-dist/hadoop-common/SingleCluster.html)
396+
- We need two YARN clusters, each YARN cluster has one RM and one NM, The RM and NM on the same node.
397+
- We need one ZK cluster(We only need one ZooKeeper node.), this part can refer to Zookeeper documentation. [ZookeeperStarted](https://zookeeper.apache.org/doc/current/zookeeperStarted.html)
398+
- We need one Router and one Client.
399+
400+
Example of Machine-Role Mapping(Exclude HDFS):
401+
402+
| Machine | Role |
403+
|:----------|:--------------|
404+
| Machine A | RM1\NM1\ZK1 |
405+
| Machine B | RM2\NM2 |
406+
| Machine C | Router\Client |
407+
408+
### YARN-1(ClusterTest-Yarn1)
409+
410+
#### RM-1
411+
412+
- For the ResourceManager, we need to configure the following option:
413+
414+
```xml
415+
416+
<!-- YARN cluster-id -->
417+
<property>
418+
<name>yarn.resourcemanager.cluster-id</name>
419+
<value>ClusterTest-Yarn1</value>
420+
</property>
421+
422+
<!--
423+
We can choose to use FairScheduler or CapacityScheduler. Different schedulers have different configuration.
424+
FairScheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
425+
CapacityScheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
426+
-->
427+
<property>
428+
<name>yarn.resourcemanager.scheduler.class</name>
429+
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
430+
</property>
431+
432+
<!--
433+
This configuration option is used to specify the configuration file for FairScheduler.
434+
If we are using CapacityScheduler, we don't need to configure this option.
435+
-->
436+
<property>
437+
<name>yarn.scheduler.fair.allocation.file</name>
438+
<value>/path/fair-scheduler.xml</value>
439+
</property>
440+
441+
<!-- Enable YARN Federation mode -->
442+
<property>
443+
<name>yarn.federation.enabled</name>
444+
<value>true</value>
445+
</property>
446+
447+
<!-- We use ZooKeeper to query/store Federation information. -->
448+
<property>
449+
<name>yarn.federation.state-store.class</name>
450+
<value>org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore</value>
451+
</property>
452+
453+
<!-- ZK Address. -->
454+
<property>
455+
<name>hadoop.zk.address</name>
456+
<value>zkHost:zkPort</value>
457+
</property>
458+
459+
```
460+
461+
- Start RM
462+
463+
```
464+
$HADOOP_HOME/bin/yarn --daemon start resourcemanager
465+
```
466+
467+
#### NM-1
468+
469+
- For the NodeManager, we need to configure the following option:
470+
471+
```xml
472+
<!-- YARN cluster-id -->
473+
<property>
474+
<name>yarn.resourcemanager.cluster-id</name>
475+
<value>ClusterTest-Yarn1</value>
476+
</property>
477+
478+
<!-- local dir -->
479+
<property>
480+
<name>yarn.nodemanager.local-dirs</name>
481+
<value>path/local</value>
482+
</property>
483+
484+
<!-- log dir -->
485+
<property>
486+
<name>yarn.nodemanager.log-dirs</name>
487+
<value>path/logdir</value>
488+
</property>
489+
490+
<!-- Enable YARN Federation mode -->
491+
<property>
492+
<name>yarn.federation.enabled</name>
493+
<value>true</value>
494+
</property>
495+
496+
<!-- Disenable YARN Federation FailOver -->
497+
<property>
498+
<name>yarn.federation.failover.enabled</name>
499+
<value>false</value>
500+
</property>
501+
502+
<!-- Enable YARN Federation Non-HA Mode -->
503+
<property>
504+
<name>yarn.federation.non-ha.enabled</name>
505+
<value>true</value>
506+
</property>
507+
508+
<!-- We use ZooKeeper to query/store Federation information. -->
509+
<property>
510+
<name>yarn.federation.state-store.class</name>
511+
<value>org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore</value>
512+
</property>
513+
514+
<!-- ZK Address. -->
515+
<property>
516+
<name>hadoop.zk.address</name>
517+
<value>zkHost:zkPort</value>
518+
</property>
519+
520+
<!-- Enable AmRmProxy. -->
521+
<property>
522+
<name>yarn.nodemanager.amrmproxy.enabled</name>
523+
<value>true</value>
524+
</property>
525+
526+
<!-- interceptors to be run at the amrmproxy -->
527+
<property>
528+
<name>yarn.nodemanager.amrmproxy.interceptor-class.pipeline</name>
529+
<value>org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor</value>
530+
</property>
531+
```
532+
533+
- Start NM
534+
535+
```
536+
$HADOOP_HOME/bin/yarn --daemon start nodemanager
537+
```
538+
539+
### YARN-2(ClusterTest-Yarn2)
540+
541+
#### RM-2
542+
543+
The RM of the `YARN-2` cluster is configured the same as the RM of `YARN-1` except for the `cluster-id`
544+
545+
```xml
546+
<property>
547+
<name>yarn.resourcemanager.cluster-id</name>
548+
<value>ClusterTest-Yarn2</value>
549+
</property>
550+
```
551+
552+
#### NM-2
553+
554+
The NM of the `YARN-2` cluster is configured the same as the RM of `YARN-1` except for the `cluster-id`
555+
556+
```xml
557+
<property>
558+
<name>yarn.resourcemanager.cluster-id</name>
559+
<value>ClusterTest-Yarn2</value>
560+
</property>
561+
```
562+
563+
After we have finished configuring the `YARN-2` cluster, we can proceed with starting the `YARN-2` cluster.
564+
565+
### Router
566+
567+
- For the Router, we need to configure the following option:
568+
569+
```xml
570+
<!-- Enable YARN Federation mode -->
571+
<property>
572+
<name>yarn.federation.enabled</name>
573+
<value>true</value>
574+
</property>
575+
576+
<!-- We use ZooKeeper to query/store Federation information. -->
577+
<property>
578+
<name>yarn.federation.state-store.class</name>
579+
<value>org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore</value>
580+
</property>
581+
582+
<!-- ZK Address. -->
583+
<property>
584+
<name>hadoop.zk.address</name>
585+
<value>zkHost:zkPort</value>
586+
</property>
587+
588+
<!-- Configure the FederationClientInterceptor -->
589+
<property>
590+
<name>yarn.router.clientrm.interceptor-class.pipeline</name>
591+
<value>org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor</value>
592+
</property>
593+
594+
<!-- Configure the FederationInterceptorREST -->
595+
<property>
596+
<name>yarn.router.webapp.interceptor-class.pipeline</name>
597+
<value>org.apache.hadoop.yarn.server.router.webapp.FederationInterceptorREST</value>
598+
</property>
599+
600+
<!-- Configure the FederationRMAdminInterceptor -->
601+
<property>
602+
<name>yarn.router.rmadmin.interceptor-class.pipeline</name>
603+
<value>org.apache.hadoop.yarn.server.router.rmadmin.FederationRMAdminInterceptor</value>
604+
</property>
605+
```
606+
607+
- Start Router
608+
609+
```
610+
$HADOOP_HOME/bin/yarn --daemon start router
611+
```
612+
613+
### Yarn-Client
614+
615+
- For the Yarn-Client, we need to configure the following option:
616+
617+
```xml
618+
619+
<!-- Enable YARN Federation mode -->
620+
<property>
621+
<name>yarn.federation.enabled</name>
622+
<value>true</value>
623+
</property>
624+
625+
<!-- Disenable YARN Federation FailOver -->
626+
<property>
627+
<name>yarn.federation.failover.enabled</name>
628+
<value>false</value>
629+
</property>
630+
631+
<!-- Configure yarn.resourcemanager.address,
632+
We need to set it to the router address -->
633+
<property>
634+
<name>yarn.resourcemanager.address</name>
635+
<value>router-1-Host:8050</value>
636+
</property>
637+
638+
<!-- Configure yarn.resourcemanager.admin.address,
639+
We need to set it to the router address -->
640+
<property>
641+
<name>yarn.resourcemanager.admin.address</name>
642+
<value>router-1-Host:8052</value>
643+
</property>
644+
645+
<!-- Configure yarn.resourcemanager.scheduler.address,
646+
We need to set it to the AMRMProxy address -->
647+
<property>
648+
<name>yarn.resourcemanager.scheduler.address</name>
649+
<value>localhost:8049</value>
650+
</property>
651+
```

0 commit comments

Comments
 (0)