Tasks¶
Async¶
Use async()
from your code to quickly offload tasks to the Cluster
:
from django_q import async, result
# create the task
async('math.copysign', 2, -2)
# or with import and storing the id
import math.copysign
task_id = async(copysign, 2, -2)
# get the result
task_result = result(task_id)
# result returns None if the task has not been executed yet
# so in most cases you will want to use a hook:
async('math.modf', 2.5, hook='hooks.print_result')
# hooks.py
def print_result(task):
print(task.result)
async()
can take the following optional keyword arguments:
hook¶
The function to call after the task has been executed. This function gets passed the complete Task
object as its argument.
save¶
Overrides the result backend’s save setting for this task.
timeout¶
Overrides the cluster’s timeout setting for this task.
sync¶
Simulates a task execution synchronously. Useful for testing. Can also be forced globally via the sync configuration option.
redis¶
A redis connection. In case you want to control your own connections.
q_options¶
None of the option keywords get passed on to the task function.
As an alternative you can also put them in
a single keyword dict named q_options
. This enables you to use these keywords for your function call:
# Async options in a dict
opts = {'hook': 'hooks.print_result',
'group': 'math',
'timeout': 30}
async('math.modf', 2.5, q_options=opts)
Please not that this will override any other option keywords.
Groups¶
You can group together results by passing async()
the optional group
keyword:
# result group example
from django_q import async, result_group
for i in range(4):
async('math.modf', i, group='modf')
# after the tasks have finished you can get the group results
result = result_group('modf')
print(result)
[(0.0, 0.0), (0.0, 1.0), (0.0, 2.0), (0.0, 3.0)]
Take care to not limit your results database too much and call delete_group()
before each run, unless you want your results to keep adding up.
Instead of result_group()
you can also use fetch_group()
to return a queryset of Task
objects.:
# fetch group example
from django_q import fetch_group, count_group, result_group
# count the number of failures
failure_count = count_group('modf', failures=True)
# only use the successes
results = fetch_group('modf')
if failure_count:
results = results.exclude(success=False)
results = [task.result for task in successes]
# this is the same as
results = fetch_group('modf', failures=False)
results = [task.result for task in successes]
# and the same as
results = result_group('modf') # filters failures by default
Getting results by using result_group()
is of course much faster than using fetch_group()
, but it doesn’t offer the benefits of Django’s queryset functions.
Note
Although fetch_group()
returns a queryset, due to the nature of the PickleField , calling Queryset.values
on it will return a list of encoded results.
Use list comprehension or an iterator instead.
Synchronous testing¶
async()
can be instructed to execute a task immediately by setting the optional keyword sync=True
.
The task will then be injected straight into a worker and the result saved by a monitor instance:
from django_q import async, fetch
# create a synchronous task
task_id = async('my.buggy.code', sync=True)
# the task will then be available immediately
task = fetch(task_id)
# and can be examined
if not task.success:
print('An error occurred: {}'.format(task.result))
An error occurred: ImportError("No module named 'my'",)
Note that async()
will block until the task is executed and saved. This feature bypasses the Redis server and is intended for debugging and development.
Instead of setting sync
on each individual async
you can also configure sync as a global override.
Connection pooling¶
Django Q tries to pass redis connections around its parts as much as possible to save you from running out of connections.
When you are making individual calls to async()
a lot though, it can help to set up a redis connection to reuse for async()
:
# redis connection economy example
from django_q import async
from django_q.conf import redis_client
for i in range(50):
async('math.modf', 2.5, redis=redis_client)
Tip
If you are using django-redis , you can configure Django Q to use its connection pool.
Reference¶
-
async
(func, *args, hook=None, group=None, timeout=None, save=None, sync=False, redis=None, q_options=None, **kwargs)¶ - Puts a task in the cluster queue
Parameters: - func (object) – The task function to execute
- args (tuple) – The arguments for the task function
- hook (object) – Optional function to call after execution
- group (str) – An optional group identifier
- timeout (int) – Overrides global cluster timeout.
- save (bool) – Overrides global save setting for this task.
- sync (bool) – If set to True, async will simulate a task execution
- redis – Optional redis connection
- q_options (dict) – Options dict, overrides option keywords
- kwargs (dict) – Keyword arguments for the task function
Returns: The uuid of the task
Return type:
-
result
(task_id)¶ Gets the result of a previously executed task
Parameters: task_id (str) – the uuid or name of the task Returns: The result of the executed task
-
fetch
(task_id)¶ Returns a previously executed task
Parameters: name (str) – the uuid or name of the task Returns: The task if any Return type: Task Changed in version 0.2.0.
Renamed from get_task
-
queue_size
()¶ Returns the size of the broker queue. Note that this does not count tasks currently being processed.
Returns: The amount of task packages in the broker Return type: int
-
result_group
(group_id, failures=False)¶ Returns the results of a task group
Parameters: Returns: a list of results
Return type:
-
fetch_group
(group_id, failures=True)¶ Returns a list of tasks in a group
Parameters: Returns: a list of Tasks
Return type:
-
count_group
(group_id, failures=False)¶ Counts the number of task results in a group.
Parameters: Returns: the number of tasks or failures in a group
Return type:
-
delete_group
(group_id, tasks=False)¶ Deletes a group label from the database.
Parameters: Returns: the numbers of tasks affected
Return type:
-
class
Task
¶ Database model describing an executed task
-
id
¶
An
uuid.uuid4()
identifier-
name
¶
The name of the task as a humanized version of the
id
Note
This is for convenience and can be used as a parameter for most functions that take a task_id. Keep in mind that it is not guaranteed to be unique if you store very large amounts of tasks in the database.
-
func
¶
The function or reference that was executed
-
hook
¶
The function to call after execution.
-
args
¶
Positional arguments for the function.
-
kwargs
¶
Keyword arguments for the function.
-
result
¶
The result object. Contains the error if any occur.
-
started
¶
The moment the task was created by an async command
-
stopped
¶
The moment a worker finished this task
-
success
¶
Was the task executed without problems?
-
time_taken
()¶
Calculates the difference in seconds between started and stopped.
Note
Time taken represents the time a task spends in the cluster, this includes any time it may have waited in the queue.
-
classmethod
get_result
(task_id)¶
Gets a result directly by task uuid or name.
-
classmethod
get_result_group
(group_id, failures=False)¶
Returns a list of results from a task group. Set failures to
True
to include failed results.-
classmethod
get_task
(task_id)¶
Fetches a single task object by uuid or name.
-
classmethod
get_task_group
(group_id, failures=True)¶
Gets a queryset of tasks with this group id. Set failures to
False
to exclude failed tasks.-
classmethod
get_group_count
(group_id, failures=False)¶
Returns a count of the number of tasks results in a group. Returns the number of failures when
failures=True
-
classmethod
delete_group
(group_id, objects=False)¶
Deletes a group label only, by default. If
objects=True
it will also delete the tasks in this group from the database.-
-
class
Success
¶ A proxy model of
Task
with the queryset filtered onTask.success
isTrue
.
-
class
Failure
¶ A proxy model of
Task
with the queryset filtered onTask.success
isFalse
.