docs: add topic guide for priority scheduler
Scheduling
############
+This topic guide covers the two different scheduling policies that are
+supported in Determined. Administrators can configure the desired
+scheduler in :ref:`master-configuration`. Given a scheduling algorithm
+configured at the cluster level, task scheduling is further dictated by
+task configuration values:
+
+- For the fair-share scheduler, ``resources.weight`` lets users inflate
+ the resource demand of a task relative to others.
+
+- For the priority scheduler, ``resources.priority`` lets users assign
+ a priority order to tasks.
+
+- Regardless of the scheduler, ``searcher.max_concurrent_trials`` lets
+ users cap the number of slots that an ``adaptive_asha``
+ hyperparameter search experiment can utilize concurrently.
+
+.. note::
+
+ Zero-slot tasks (e.g., cpu notebooks, tensorboards) are scheduled
+ independenty of tasks that require slots (e.g., experiments, gpu
+ notebooks). The fair-share scheduler schedules zero-slot tasks on a
+ FIFO basis. The priority scheduler schedules zero-slot tasks based on
+ priority.
+
+************
+ Fair-Share
+************
+
The master allocates cluster resources (slots) among the active
experiments using a fair-share scheduling policy. In other words, slots
are divided among the active experiments according to the *demand*
(number of desired concurrent tasks) of each experiment. For instance,
-in an eight-GPU cluster running two experiments with demands of three
-and six, the scheduler assigns three slots and five slots respectively.
-As new experiments become active or the resource demand of an active
+in an eight-GPU cluster running two experiments with demands of ten and
+thirty, the scheduler assigns two slots and six slots respectively. As
+new experiments become active or the resource demand of an active
experiment changes, the scheduler will adjust how slots are allocated to
experiments as appropriate.
-Scheduling behavior can be configured via the ``resources`` section of
-the experiment config file; see :ref:`experiment-configuration` for
-details.
+**********
+ Priority
+**********
+
+The master allocates cluster resources (slots) to active tasks based on
+their priority. While tasks of higher priority (lower priority number)
+are pending, no lower priority tasks will be scheduled. For instance, if
+tasks with priorities of five and forty-two are pending, the latter will
+not be scheduled until the former has been. Tasks of equal priority are
+scheduled in the order in which they were created.
+
+By default the priority scheduler will not perform any preemption. If
+preemption is enabled (:ref:`master-configuration`), in scenarios where
+a higher priority task is pending and cannot be scheduled, the scheduler
+will attempt to schedule it by preempting lower priority tasks, starting
+with the lowest priorities.
+
+An example of the priority scheduler:
+
+#. User submits priority 2 adaptive_asha experiment with max_trials 20
+ and slots_per_trial 1. 8 trials run and utilize all 8 GPUs.
+
+#. User submits priority 1 distributed training experiment with
+ slots_per_trial 4. 4 asha trials are preempted so it can run. Note
+ that if preemption were disabled, this would not get scheduled until
+ the ASHA experiment's GPU demand becomes <= 4.
+
+#. User starts priority 3 notebook with resources.slots 1. Will run as
+ soon as adaptive_asha plus distributed training experiments
+ collectively need <= 7 GPUs.
+
+#. Asha and the distributed training expriment both complete.
+
+#. User submits priority 1 distributed training experiment with
+ slots_per_trial 8. It will not be scheduled because notebooks are not
+ preemptible and only 7 slots available.
+
+#. User submits priority 2 distributed training experiment with
+ slots_per_trial 4. It will not be scheduled even though 7 slots
+ available, because it is behind a higher priority task.
+
+#. Notebook killed. P1 distributed training experiment runs. Once
+ complete, P2 distributed training experiment runs.
+
+.. note::
+
+ Notebooks, tensorboards, shells, and commands are non-preemptible.
+ These tasks will continue to occupy cluster resources until they
+ complete or are terminated.
