Cluster¶
Django Q uses Python’s multiprocessing module to manage a pool of workers that will handle your tasks.
Start your cluster using Django’s manage.py
command:
$ python manage.py qcluster
You should see the cluster starting
10:57:40 [Q] INFO Q Cluster-31781 starting.
10:57:40 [Q] INFO Process-1:1 ready for work at 31784
10:57:40 [Q] INFO Process-1:2 ready for work at 31785
10:57:40 [Q] INFO Process-1:3 ready for work at 31786
10:57:40 [Q] INFO Process-1:4 ready for work at 31787
10:57:40 [Q] INFO Process-1:5 ready for work at 31788
10:57:40 [Q] INFO Process-1:6 ready for work at 31789
10:57:40 [Q] INFO Process-1:7 ready for work at 31790
10:57:40 [Q] INFO Process-1:8 ready for work at 31791
10:57:40 [Q] INFO Process-1:9 monitoring at 31792
10:57:40 [Q] INFO Process-1 guarding cluster at 31783
10:57:40 [Q] INFO Process-1:10 pushing tasks at 31793
10:57:40 [Q] INFO Q Cluster-31781 running.
Stopping the cluster with ctrl-c or either the SIGTERM
and SIGKILL
signals, will initiate the Stop procedure:
16:44:12 [Q] INFO Q Cluster-31781 stopping.
16:44:12 [Q] INFO Process-1 stopping cluster processes
16:44:13 [Q] INFO Process-1:10 stopped pushing tasks
16:44:13 [Q] INFO Process-1:6 stopped doing work
16:44:13 [Q] INFO Process-1:4 stopped doing work
16:44:13 [Q] INFO Process-1:1 stopped doing work
16:44:13 [Q] INFO Process-1:5 stopped doing work
16:44:13 [Q] INFO Process-1:7 stopped doing work
16:44:13 [Q] INFO Process-1:3 stopped doing work
16:44:13 [Q] INFO Process-1:8 stopped doing work
16:44:13 [Q] INFO Process-1:2 stopped doing work
16:44:14 [Q] INFO Process-1:9 stopped monitoring results
16:44:15 [Q] INFO Q Cluster-31781 has stopped.
The number of workers, optional timeouts, recycles and cpu_affinity can be controlled via the Configuration settings.
Multiple Clusters¶
You can have multiple clusters on multiple machines, working on the same queue as long as:
- They connect to the same broker.
- They use the same cluster name. See Configuration
- They share the same
SECRET_KEY
for Django.
Using a Procfile¶
If you host on Heroku or you are using Honcho you can start the cluster from a Procfile
with an entry like this:
worker: python manage.py qcluster
Process managers¶
While you certainly can run a Django Q with a process manager like Supervisor or Circus it is not strictly necessary. The cluster has an internal sentinel that checks the health of all the processes and recycles or reincarnates according to your settings or in case of unexpected crashes. Because of the multiprocessing daemonic nature of the cluster, it is impossible for a process manager to determine the clusters health and resource usage.
An example circus.ini
[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555
pubsub_endpoint = tcp://127.0.0.1:5556
stats_endpoint = tcp://127.0.0.1:5557
[watcher:django_q]
cmd = python manage.py qcluster
numprocesses = 1
copy_env = True
Note that we only start one process. It is not a good idea to run multiple instances of the cluster in the same environment since this does nothing to increase performance and in all likelihood will diminish it.
Control your cluster using the workers
, recycle
and timeout
settings in your Configuration
Reference¶
-
class
Cluster
¶ -
start
()¶
Spawns a cluster and then returns
-
stop
()¶
Initiates Stop procedure and waits for it to finish.
-
stat
()¶
returns a
Stat
object with the current cluster status.-
pid
¶
The cluster process id.
-
host
¶
The current hostname
-
sentinel
¶
returns the
multiprocessing.Process
containing the Sentinel.-
timeout
¶
The clusters timeout setting in seconds
-
start_event
¶
A
multiprocessing.Event
indicating if the Sentinel has finished starting the cluster-
stop_event
¶
A
multiprocessing.Event
used to instruct the Sentinel to initiate the Stop procedure-
is_starting
¶
Bool. Indicating that the cluster is busy starting up
-
is_running
¶
Bool. Tells you if the cluster is up and running.
-
is_stopping
¶
Bool. Shows that the stop procedure has been started.
-
has_stopped
¶
Bool. Tells you if the cluster has finished the stop procedure
-