Task Queue Priority and Fairness

Task Queue Priority and Task Queue Fairness are two ways to manage the distribution of work within Task Queues. Priority focuses on how Tasks are prioritized within a Task Queue and Fairness aims to prevent one set of prioritized Tasks from blocking others within the same Task Queue.

Support, stability, and dependency info

Priority and Fairness is currently in pre-release. Please contact your AE or Support to enable this feature.

Task Queue priority

Task Queue priority lets you control the execution order of Workflows, Activities, and Child Workflows based on assigned priority values within a single Task Queue. Each priority level acts as a "virtual" task queue that prioritizes Tasks within a single Task Queue.

If you're using Temporal Cloud, contact Temporal Support or your Temporal account team to enable this feature for your cloud Namespaces.

If you're self-hosting Temporal, use the latest pre-release development server and set matching.useNewMatcher to true in the dynamic config on the relevant Task Queues or Namespaces.

When to use priority

If you need a way to specify the order your Tasks execute in, you can use priority to manage that. Priority lets you differentiate between your Tasks, like batch and real-time for example, so that you can allocate resources more effectively. You can also use this as a way to run urgent Tasks immediately and override others.

How to use priority

You can select a priority level by setting the priority key to a value within the integer range [1,5]. A lower value implies higher priority, so 1 is the highest priority level. The default priority, if unspecified, is 3.

When you set a priority level within your Task Queues, this means that they will all be processed in priority order. For example, all of your priority level 1 Tasks will execute before your priority level 2 Tasks and so on. So your lower priority Tasks will be blocked until the higher priority Tasks finish running. Tasks with the same priority level are scheduled to run in first-in-first-out (FIFO) order. If you need more flexibility to allocate resources to lower priority Tasks sooner, check out the fairness section.

Activities will inherit the priority level of their Workflow if a separate Activity priority level isn't set.

When you update priority levels, the Task Queues will only reflect these changes for Tasks that haven't dispatched yet, not Tasks that are already queued.

note

Sticky tasks ignore priority settings.

Choose your SDK below to see an example of setting priority for your Workflows:

Go

workflowOptions := client.StartWorkflowOptions{
  ID:        "my-workflow-id",
  TaskQueue: "my-task-queue",
  Priority:  temporal.Priority{PriorityKey: 5},
}
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow)

Java

WorkflowOptions options = WorkflowOptions.newBuilder()
  .setTaskQueue("my-task-queue")
  .setPriority(Priority.newBuilder().setPriorityKey(5).build()) 
  .build();

WorkflowClient client = WorkflowClient.newInstance(service);
MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options);
workflow.run();

Python

await client.start_workflow(
  MyWorkflow.run,
  args="hello",
  id="my-workflow-id",
  task_queue="my-task-queue",
  priority=Priority(priority_key=1),
)

TypeScript

TypeScript example coming soon.

PHP

PHP example coming soon.

.NET

var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(
    id: "my-workflow-id",
    taskQueue: "my-task-queue"
  )
  {
    Priority = new Priority(1),
  }
);

Ruby

Ruby example coming soon.

Choose your SDK below to see an example of setting priority for your Activities:

Go

ao := workflow.ActivityOptions{
  StartToCloseTimeout: time.Minute,
  Priority:            temporal.Priority{PriorityKey: 3},
}
ctx := workflow.WithActivityOptions(ctx, ao)
err := workflow.ExecuteActivity(ctx, MyActivity).Get(ctx, nil)

Java

ActivityOptions options = ActivityOptions.newBuilder()
  .setStartToCloseTimeout(Duration.ofMinutes(1))
  .setPriority(Priority.newBuilder().setPriorityKey(3).build())
  .build();

MyActivity activity = Workflow.newActivityStub(MyActivity.class, options);
activity.perform();

Python

await workflow.execute_activity(
  say_hello,
  "hi",
  priority=Priority(priority_key=3),
  start_to_close_timeout=timedelta(seconds=5),
)

TypeScript

TypeScript example coming soon.

PHP

PHP example coming soon.

.NET

await Workflow.ExecuteActivityAsync(
  () => SayHello("hi"),
    new()
    {
      StartToCloseTimeout = TimeSpan.FromSeconds(5),
      Priority = new(3),
    }
  );

Ruby

Ruby example coming soon.

Choose your SDK below to see an example of setting priority for your Child Workflows:

Go

cwo := workflow.ChildWorkflowOptions{
  WorkflowID: "child-workflow-id",
  TaskQueue:  "child-task-queue",
  Priority:   temporal.Priority{PriorityKey: 1},
}
ctx := workflow.WithChildOptions(ctx, cwo)
err := workflow.ExecuteChildWorkflow(ctx, MyChildWorkflow).Get(ctx, nil)

Java

ChildWorkflowOptions childOptions = ChildWorkflowOptions.newBuilder()
  .setTaskQueue("child-task-queue")
  .setWorkflowId("child-workflow-id")
  .setPriority(Priority.newBuilder().setPriorityKey(1).build())
  .build();

MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, childOptions);
child.run();

Python

await workflow.execute_child_workflow(
  MyChildWorkflow.run,
  args="hello child",
  priority=Priority(priority_key=1),
)

TypeScript

TypeScript example coming soon.

PHP

PHP example coming soon.

.NET

await Workflow.ExecuteChildWorkflowAsync(
  (MyChildWorkflow wf) => wf.RunAsync("hello child"),
  new() { Priority = new(1) });

Ruby

Ruby example coming soon.

Task Queue fairness

Task Queue fairness applies within each priority level. This gives you more control over the order that Tasks are dispatched from the backlog at a specific priority level. You can assign compute resources fairly so that priority levels are respected, but Tasks at lower priority levels will still be processed without waiting for all higher priority Tasks to finish first. It’s intended to address common situations like:

• Multi-tenant applications with big and small tenants where small tenants shouldn't be blocked by big ones. That means any single user should be able to use all available resources rather than being rate limited at a fixed level.
• Assigning Tasks to different capacity bands and then, for example, dispatching 80% from one band and 20% from another without limiting overall capacity when one band is empty.

Fairness is only avaliable as a paid feature in Temporal Cloud, so contact Temporal Support or your Temporal account team to enable this feature for your cloud Namespaces. Fairness can be enabled or disabled at the Namespace level and billing will be disabled at the next calendar hour after it is disabled.

danger

For Pre-release, fairness cannot be enabled for active Task Queues. The Task Queues have to be new or idle and there can't be any running Workflows. Once fairness is enabled, all existing backlog Tasks in the Task Queue will be abandoned.

How fairness works

Fairness sequences Tasks at each priority level based on fairness keys and their fairness weight within a Task Queue. The fairness keys are used to describe your Task structure. A fairness key can correspond to unique applications, tenants, or other groupings useful to your workloads. The Tasks associated with each fairness key are dispatched based on the fairness weight that has been assigned to the key. Using the fairness keys and their corresponding fairness weights lets you define tiers with weighted capacities, like high, medium, and low, so that higher priority tasks don't overwhelm your resources and block lower priority tasks.

Fairness attempts to distribute all the Tasks from a higher priority level first before distributing any from a lower priority. The fairness weights allow for unequal distribution among tenants or for allocating fractions of capacity to different levels or types of Tasks.

Fairness applies within each priority level. It sequences Tasks in the Task Queue probabilistically using a weighted round-robin, based on:

• Fairness weights you set
• The current backlog of Tasks
• A data structure that tracks how you've distributed Tasks for different fairness keys

When you update fairness keys or fairness weights, the Task Queues will only reflect these changes for Tasks that haven't dispatched yet, not Tasks that are already queued.

When to use fairness

Fairness applies to backlogged Tasks when there isn't sufficient Worker capacity to dispatch Tasks immediately. This is applicable if you want to distribute a finite resource across all of your Tasks without blockers. If all Tasks can be dispatched immediately, then you don't need to use fairness.

Fairness applies at Task dispatch time based on information about the Tasks passing through the Task Queue and considers each Task as having equal cost. It doesn't consider any Task execution that is currently being done by Workers. So if you look at Tasks being processed by Workers, you might not see "fairness" across tenants.

Limitations of fairness

When you use Worker Versioning and you're moving Workflows from one version to another, the ordering of Workflow Tasks that are moved to the next version is undefined. Tasks redirected to a new Worker version may not be treated fairly with respect to each other or Tasks that aren't redirected.

There isn't a limit on the number of fairness keys you can use, but their accuracy can degrade as you add more.

Task Queues are internally partitioned and Tasks are distributed to partitions randomly. This could interfer with fairnesss. Depending on your use case, you can reach out to Temporal Support to get your Task Queues set to a single partition.

Choose your SDK below to see an example of setting fairness for your Workflows:

Go

workflowOptions := client.StartWorkflowOptions{
  ID:        "my-workflow-id",
  TaskQueue: "my-task-queue",
  Priority:  temporal.Priority{
    PriorityKey:    1,
    FairnessKey:    "a-key",
    FairnessWeight: 3.14,
  },
}
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow)

Java

WorkflowOptions options = WorkflowOptions.newBuilder()
  .setTaskQueue("my-task-queue")
  .setPriority(Priority.newBuilder().setPriorityKey(5).setFairnessKey("a-key").setFairnessWeight(3.14).build())
  .build();
WorkflowClient client = WorkflowClient.newInstance(service);
MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options);
workflow.run();

Python

await client.start_workflow(
  MyWorkflow.run,
  args="hello",
  id="my-workflow-id",
  task_queue="my-task-queue",
  priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14),
)

TypeScript

const handle = await startWorkflow(workflows.priorityWorkflow, {
  args: [false, 1],
  priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 },
});

PHP

PHP example coming soon.

.NET

var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(
    id: "my-workflow-id",
    taskQueue: "my-task-queue"
  )
  {
    Priority = new Priority(
      priorityKey: 3,
      fairnessKey: "a-key",
      fairnessWeight: 3.14
    )
  }
);

Ruby

client.start_workflow(
  MyWorkflow, "input-arg",
  id: "my-workflow-id",
  task_queue: "my-task-queue",
  priority: Temporalio::Priority.new(
    priority_key: 3,
    fairness_key: "a-key",
    fairness_weight: 3.14
  )
)

Choose your SDK below to see an example of setting fairness for your Activities:

Go

ao := workflow.ActivityOptions{
  Stardd
}
ctx := workflow.WithActivityOptions(ctx, ao)
err := workflow.ExecuteActivity(ctx, MyActivity).Get(ctx, nil)

Java

ActivityOptions options = ActivityOptions.newBuilder()
  .setStartToCloseTimeout(Duration.ofMinutes(1))
  .setPriority(Priority.newBuilder().setPriorityKey(3).setFairnessKey("a-key").setFairnessWeight(3.14).build())
  .build();
MyActivity activity = Workflow.newActivityStub(MyActivity.class, options);
activity.perform();

Python

await workflow.execute_activity(
  say_hello,
  "hi",
  priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14),
  start_to_close_timeout=timedelta(seconds=5),
)

TypeScript

const handle = await startWorkflow(workflows.priorityWorkflow, {
  args: [false, 1],
  priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 },
});

PHP

PHP example coming soon.

.NET

var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(
    id: "my-workflow-id",
    taskQueue: "my-task-queue"
  )
  {
    Priority = new Priority(
      priorityKey: 3,
      fairnessKey: "a-key",
      fairnessWeight: 3.14
    )
  }
);

Ruby

client.start_activity(
  MyActivity, "input-arg",
  id: "my-workflow-id",
  task_queue: "my-task-queue",
  priority: Temporalio::Priority.new(
    priority_key: 3,
    fairness_key: "a-key",
    fairness_weight: 3.14
  )
)

Choose your SDK below to see an example of setting fairness for your Child Workflows:

Go

cwo := workflow.ChildWorkflowOptions{
  WorkflowID: "child-workflow-id",
  TaskQueue:  "child-task-queue",
  Priority:   temporal.Priority{
    PriorityKey:    1,
    FairnessKey:    "a-key",
    FairnessWeight: 3.14,
  },
}
ctx := workflow.WithChildOptions(ctx, cwo)
err := workflow.ExecuteChildWorkflow(ctx, MyChildWorkflow).Get(ctx, nil)

Java

ChildWorkflowOptions childOptions = ChildWorkflowOptions.newBuilder()
  .setTaskQueue("child-task-queue")
  .setWorkflowId("child-workflow-id")
  .setPriority(Priority.newBuilder().setPriorityKey(1).setFairnessKey("a-key").setFairnessWeight(3.14).build())
  .build();
MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, childOptions);
child.run();

Python

await workflow.execute_child_workflow(
  MyChildWorkflow.run,
  args="hello child",
  priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14),
)

TypeScript

const handle = await startChildWorkflow(workflows.priorityWorkflow, {
  args: [false, 1],
  priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 },
});

PHP

PHP example coming soon.

.NET

var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(
    id: "my-workflow-id",
    taskQueue: "my-task-queue"
  )
  {
    Priority = new Priority(
      priorityKey: 3,
      fairnessKey: "a-key",
      fairnessWeight: 3.14
    )
  }
);

Ruby

client.start_child_workflow(
  MyChildWorkflow, "input-arg",
  id: "my-child-workflow-id",
  task_queue: "my-task-queue",
  priority: Temporalio::Priority.new(
    priority_key: 3,
    fairness_key: "a-key",
    fairness_weight: 3.14
  )
)