Developer Journal: 29 April 2020

Posted on Wed 29 April 2020 in dev-journal

Updating the dev-journal script to open previous day

This morning when starting this entry I wanted to be able to check look at what I was was working on yesterday and found that what I've been doing is navigating to the file and opening the previous day. However, since I wrote a script that will open a journal entry, I can add an argument that will allow me to open a previous day. This was done by changing the way I was using the date function. Previously the line of code that I used to get the date looked like this:

DATE=$(date +20%y-%m-%d)

That worked well but then I wanted to be able to increment or decrement the day. It took me a few tries but it did allow me to learn a little bit more about the bash date function. 

# use a command line argument to change the date
if $1; then
    DATE=$(date -d "+$1 days" +20%y-%m-%d)
else
    DATE=$(date +20%y-%m-%d)
fi

So this was the first iteration that I used.. it worked great for getting previous entries however it was adding a day if you called dev-journal without any arguments. To fix this the script was again modified, the final version of this date logic looks as follows:

if [ -z "$1" ]; then
    DATE=$(date +20%y-%m-%d)
else
    DATE=$(date -d "+$1 days" +20%y-%m-%d)
fi

This uses the -z to test if $1 is an empty string. If it is than we set the date to now otherwise, apply the command line argument. This currently does not handle any validation with user input. That can be added if the script needs to be expanded further in the future.

Writing this also allowed me to recall a git command that I rarely use, the git clean command. This command can be very useful if you have created a lot of files in your git repo that have no reason to be there. In this case while I was working to git this command working the script created some files that did not match what I was looking for.

michaelabrahamsen.com git:(master) ✗ git clean -n
Would remove content/posts/dev-journal/-developer-journal.md
Would remove content/posts/dev-journal/2020-04-29+-1-developer-journal.md
Would remove content/posts/dev-journal/2020-04-29-developer-journal.md
Would remove content/posts/dev-journal/2020-04-30-developer-journal.md
Would remove content/posts/dev-journal/Sat 09 May 2020 09:17:46 AM MDT-developer-journal.md
Would remove content/posts/dev-journal/Tue 28 Apr 2020 09:21:37 AM MDT-developer-journal.md

git clean -n performs a dry run so you can see what will be deleted and git clean -f will remove the files that the previous command listed.

Shutting down a virtualbox vm via command line

VBoxManage controlvm VM-NAME poweroff

Using custom RQ workers with Conveyor.dev

In order to change Conveyor's current worker creation to allow for custom RQ workers that was shown previously Conveyor will need an extra form field. The form field will determine what command is run when configuring and starting the workers.

So where to start... yesterday I added a form field for the custom command. Let's write a couple of tests and add the field to the model.

def test_model_has_custom_worker_field(self):
    worker = SiteWorker(1, 1, queue_names="high default", worker_count=1)
    self.assertEqual(worker.worker_command, "rq worker")

def test_model_sets_custom_worker_field(self):
    worker = SiteWorker(1, 1, queue_names="high default", worker_count=1,
                        worker_command="rq-gevent")
    self.assertEqual(worker.worker_command, "rq-gevent")

Now to make these pass we just need to update the model:

class SiteWorker(db.Model):
    __tablename__ = 'site_worker'
    id = db.Column(db.Integer, primary_key=True)
    worker_count = db.Column(db.Integer)
    queue_names = db.Column(db.String)
    worker_command = db.Column(db.String)

    def __init__(self, worker_count, queue_names, worker_command=None):
        self.worker_count = worker_count
        self.queue_names = queue_names
        self.worker_command = worker_command or "rq worker"

To add the form field to the view:

{{ render_field(form.custom_worker, placeholder="rq-gevent-worker", class="w-full input") }}

So all of that was easy... now to make sure that the correct files get created on the server. More tests! This time we want to make sure the worker manager and the individual worker processes are created correctly. In this case it just means that we pass along the command and make sure that is added to the config script.

def test_formatting_with_single_queue_and_custom_worker(self):
    domain = "test.test.com"
    queues = "high"
    manager = "test.test.com-worker-1.service"
    expected_output = (
        "ExecStart=/bin/bash -c 'source "
        f"/home/conveyor/{domain}/venv/bin/activate && rq-gevent {queues}'"
    )
    config = Client.get_provision_file_contents(
        "[email protected]")
    formatted_config = Provisioner.format_provision_config(
        config, domain=domain, queues=queues, worker_manager=manager,
        command="rq-gevent")
    output_line = self.get_line_containing(formatted_config, "Exec")
    self.assertEqual(output_line, expected_output)

def test_formatting_with_multiple_queues_and_custom_worker(self):
    domain = "test.test.com"
    queues = "high default"
    manager = "test.test.com-worker-1.service"
    expected_output = (
        "ExecStart=/bin/bash -c 'source "
        f"/home/conveyor/{domain}/venv/bin/activate && rq-gevent {queues}'"
    )
    config = Client.get_provision_file_contents(
        "[email protected]")
    formatted_config = Provisioner.format_provision_config(
        config, domain=domain, queues=queues, worker_manager=manager,
        command="rq-gevent")
    output_line = self.get_line_containing(formatted_config, "Exec")
    self.assertEqual(output_line, expected_output)

This actually leads to a piece of code that is so simple yet so powerful. Part of the client has a class method to format config files. Some of these configs need to have arguments passed in. I store a base config file as a string with some string parameterization, then the following function formats the file as necessary:

@classmethod
def format_provision_config(cls, contents, **kwargs):
    return contents.format(**kwargs)

After doing some manual testing, the worker services were not being started/stopped correctly. They did not get the memo that we were adding the worker_id to the filename. The commands that are doing that are now used in multiple places so they were broken out into their own functions:

@classmethod
def get_worker_service(cls, domain, worker_id):
    return f"{domain}-worker-{worker_id}.service"

@classmethod
def get_worker_file_name(cls, domain, worker_id):
    return f"/etc/systemd/system/{domain}-worker-{worker_id}@.service"

@classmethod
def get_worker_manager_file_name(cls, domain, worker_id):
    return f"/etc/systemd/system/{domain}-worker-{worker_id}.service"

Now for the real test and the purpose of all of this. Can we deploy this using Conveyor.dev and use our custom worker class?

Resources

Hero Icons

developer journal

Previous Post Next Post