Untangling Apache Hadoop YARN, Part 5: Using FairScheduler queue properties

Previously inPart 4, we described the most commonly used FairScheduler properties in Apache Hadoop. In Part 5, we’ll provide some examples to show how properties can be used, individually and in combination, to achieve commonly desired behavior such as application prioritization and organizing queues.

Example: Best Effort Queue

Summary: Create a “best effort” queue that runs applications when the cluster is underutilized.

Implementation: In FairScheduler, a queue with weight 0.0 will only run applications if there is spare capacity in the cluster. In other words, all the jobs in the priority_jobs queue will get allocated first, and then FairScheduler will allocate any such spare capacity to the best_effort_jobs queue.

<queuename=”priority_jobs”> </queue> <queuename=”best_effort_jobs”> <weight>0.0</weight> </queue>

Notes:

Jobs in cluster may take a long time to finish if the cluster is fully utilized with running jobs in other queues.

Caveat: There is a bug in early versions of CDH where the value 0.0 won’t quite work properly, but really tiny values (e.g. 0.01) work just fine. YARN-5077 fixes the issue and is in releases CDH 5.7.3, CDH 5.8.2, CDH 5.9.0 and later.

Example: Using maxResources to Guarantee Resources for Low Latency Applications

Summary: Have a special queue which runs applications with special low latency requirements.

Implementation: Assume we have a cluster with the following resources: <memory: 20000 gb, vcores: 10000> . By setting the maxResources property on the other_jobs queue, FairScheduler leaves <memory: 4000 gb, vcores: 2000> for the low_latency queue.

<queuename=”root”> <queuename=”low_latency” /> <queuename=”other_jobs”> <maxResources>16000 gb,8000 vcores</maxResources> </queue> </queue>

Notes:

All the applications in the other_jobs queue total utilization cannot exceed 80% of the cluster. By leaving roughly 20% of the cluster for the low_latency queue, applications there can get started as quickly as possible. This case is provided purely as an example. In many cases, it will be preferable to use the “Queues for Low Latency Applications using Preemption” below.

Example: Using Preemption to Guarantee Resources to Low Latency Applications without Compromising on Utilization

Summary: Have a special queue which runs applications with special low latency requirements.

Implementation: Assume we have a cluster with the following resources: <memory: 20000 gb, vcores: 10000> . This configuration turns on preemption on the low_latency queue.

<queuename=”root”> <queuename=”low_latency”> <fairSharePreemptionThreshold>1.0</fairSharePreemptionThreshold> <fairSharePreemptionTimeout>1</fairSharePreemptionTimeout> </queue> <queuename=”other_jobs”> </queue> </queue>

Notes:

Unlike the maxResources version (see previous example), the full cluster is available to the other_jobs queue, but applications in the other_jobs can be preempted in order for the low_latency queue to start applications running. If you still wish to limit the total cluster usage of the low_latency queue, maxResources could be applied. There are two more examples of priority queues later that show how preemption can be used in a more sophisticated fashion. Reminder: To enable preemption in FairScheduler, this property must be set in yarn-site.xml: <property> <name>yarn.scheduler.fair.preemption</name> <value>true</value> </property> Example: Limiting the Size of Ad-Hoc Queues

Summary: Allow child ad-hoc queues to have a maxResources setting

Implementation: Normally, it is impossible to set properties on ad-hoc queues, since they are not defined in the fair-scheduler.xml file. By setting the maxChildResources property on the some_parent queue, any children of that queue (e.g. ad-hoc user queues or ad-hoc group queues) will have the equivalent of <maxResources>8192 mb,8 vcores</maxResources> set.

<queuename=”some_parent”> <maxChildResources>8192 mb,8 vcores</maxChildResources> </queue>

Notes:

This feature was introduced in and is new to CDH 5.9.0. Example: Organizational Queues

Summary: Give each organization a queue for their applications.

Implementation: Provide a queue for each organization, in this case sales, marketing, finance, and data science. Each group is given an equal share of Steady FairShare.

<queuename=”root”> <queuename=”sales” /> <queuename=”marketing” /> <queuename=”data_science” /> </queue>

Hierarchical Implementation: The sales organization has northamerica and europe subqueues. The marketing organization has reports and website queues. The data_science organization has priority and best_effort queues.

<queuename=”root”> <queuename=”sales”> <queuename=”northamerica” /> <queuename=”europe” /> <queuename=”asia” /> </queue> <queuename=”marketing”> <queuename=”reports” /> <queuename=”website” /> </queue> <queuename=”data_science”> <queuename=”priority”> <weight>100.0</weight> </queue> <queuename=”best_effort”> <weight>0.0</weight> </queue> </queue> </queue> Example: Strict Priority Queuing

Summary: This is an alternative approach to priority queues.

Implementation: In the previous example, FairScheduler is using preemption to enforce a container allocation of 100/10/1. In this version the root.other and root.other.other