Developer Journal: 19 April 2020

Digging back into a piece of Conveyor that has not been visited for a while. The worker queue for running commands on a conveyor server. Working on making a prototype to test the idea of using threads instead of RQ.

Currently what has been done is refactoring the model to use a thread instead of launching an RQ task. Now when creating a task a UUID is created and stored in the Task table. Then we launch the thread and begin storing the status of the job in a Redis database.

server_provisioner.launch_server_task("Provision server",
        client.provision_droplet,
        uuid,
        droplet_id,
        server)

The launch task function will handle adding the task to the database and starting the thread

def launch_task(self, description, function, uuid, *args, **kwargs):
    task = ServerTask(id=uuid, name=function.__name__,
            description=description,
            server_id=self.id)
    db.session.add(task)

    thread = Thread(target=function, args=args).start()

Found a bug with creating a new server. If both providers are added only one of them will currently work. This was caused when I removed the use of the radio button and instead used links and url parameters to gather the form information. Will have to come back to this one later.

While working through this experiment I found that when items from redis were retrieved they were always in bytes and I was constantly decoding the responses. There is a nice little argument decode_responses=True that will decode the responses automatically.

May have thought of something else that will need to be thought about. Will using the python threading library delete the running threads if the python process is restarted?