If an application intends to use task queues to perform work, the app must include a configuration file that declares the queues it intends to use. For Python apps, this file is named queue.yaml.
The file specifies a value named queue set to a list of items, each of which is the configuration for a queue.
The following example defines three queues:
queue: - name: default rate: 1/s - name: mail-queue rate: 2000/d bucket_size: 10 - name: background-processing rate: 5/s
The syntax of queue.yaml is the YAML format. For more information about this syntax, see the YAML website for more information.
The app's queue configuration applies to all versions of the app. If a given version of an app enqueues a task, the queue will use the task handler for that version of the app.
All apps have a queue named default with a default rate of 5 tasks per second. You can customize the settings for this queue by defining a queue named default in queue.yaml. If you go without configuration, the default queue doesn't show up in the Administration Console until the first time it is used.
An app can only add tasks to queues defined in queue.yaml and the default queue. If you upload a new queue.yaml file that removes a queue, but that queue still has tasks, the queue is "paused" (its rate is set to 0) but not deleted. You can re-enable the deleted queue by uploading a new queue.yaml file with the queue defined.
queue.yaml has a single list element called queue. Each element in the list represents a queue for the application.
A queue element can have the following elements:
|
nameThe name of the queue. The app uses the queue name when adding a task to the queue. The queue name also appears in the Administration Console.
A queue name can contain letters, numbers and hyphens.
This element is required.
rateThe average rate at which tasks are processed on this queue. Depending on the bucket size, brief spikes to a higher rate may be possible. The value is a number followed by a slash and a unit of time, where the unit is s for seconds, m for minutes, h for hours, or d for days. For example, the value 5/m says tasks will be processed at a rate of 5 times per minute.
If the number is 0 (such as 0/s), the queue is considered "paused," and no tasks are processed.
This element is required.
bucket_sizeLimits the burstiness of the queue's processing, i.e. a higher bucket size allows bigger spikes in the queue's execution rate. For example, consider a queue with a rate of 5/s and a bucket size of 10. If that queue has been inactive for some time (allowing its "token bucket" to fill up), and 20 tasks are suddenly enqueued, it will be allowed to execute 10 tasks immediately. But in the following second, only 5 more tasks will be able to be executed because the token bucket has been depleted and is refilling at the specified rate of 5/s.
If no bucket_size is specified for a queue, the default value is 5.
For more information on the algorithm, see the Wikipedia article on token buckets.
The task queue configuration for the app is updated when you upload the application using appcfg.py update. You can update just the configuration for an app's task queues without uploading the full application. To upload the queue.yaml file, use the appcfg.py update_queues command:
appcfg.py update_queues myapp/
See Uploading an and Managing a Python App for more information.
