Home

Categories

About

Managing Processes with Supervisor - In Depth Tutorial

Posted on January 25, 2017 by Abhirath Mahipal
Tags  toolssystem-admin

In this post I’ll show you how to setup Supervisor on Linux and become familiar with a few handy options and settings. If you haven’t used Supervisor in the past this post should give you a decent start.

Supervisor is a Python program that makes managing other programs (processes) a breeze. It can ease the process of starting, killing and restarting applications that you develop. In this post I’ll explain various aspects that you need to keep in mind while using Supervisor.

Basics

  • Supervisor starts your application or script as it’s own child process. This way it can kill them when you want to. It daemonizes something that runs infinitely say a web server. You should not use it for a script that backups your database i.e it has a definite end.
  • The command to start a particular process and various options are given via a .conf file.
  • Supervisor looks for files ending with .conf in certain directories. It then follows the instructions that you’ve given in those files.
  • Once a proper .conf file is written starting any program is as simple as
    supervisorctl start appname.
  • It daemonizes the process that you want to run. You can save logs that your application spits out in files.
  • Say your server runs 50 processes. It would be a pain to start every one of them whenever you restart the server. Supervisor has an autostart option which will start the process when Supervisor is initialised.
  • You could make Supervisor run a program as a particular user. There are loads of such options that can help make your day to day life easier.

Installing Supervisor

  • On Ubuntu or debian you can
    sudo apt-get install supervisor

  • You can also install it using pip. Supervisor on Pypi.

Writing conf Files

Supervisor can be informed of any program that you wish to run via a .conf file.

I found this example from Digital Ocean to be very helpful. So I’m adapting a similar approach but with a different script. For the sake of this example and brevity we’ll write a small script that displays the amount of RAM that the machine sports.

Go to your home directory (~/) and save the following as displayram.sh

#!/bin/bash

counter=0
while true
do
    ((counter++))
    echo "Count is now $counter."
    cat /proc/meminfo | grep MemTotal
    sleep 1
done

Make the file you just created executable.
chmod +x displayram.sh

Save the following as displayram.conf. Supervisor will use this file to control the process. Substitute abhirath with your username.

[program:displayram]
command=/home/abhirath/displayram.sh
autostart=false
autorestart=false
stderr_logfile=NONE
stdout_logfile=/home/abhirath/displayram.out.log

Move the file that you just created to /etc/supervisor/conf.d/displayram.conf. This is the default directory that Supervisor looks for programs.

Then on the terminal supervisorctl reread followed by supervisorctl update.

On a terminal enter supervisorctl start displayram. Our script starts instantly.

You can verify that it works by opening /home/yourusername/displayram.out.log. It should be filled with lines displaying the RAM installed on the machine.

A better way would be to run
tail -f /home/yourusername/displayram.out.log
on a terminal. It will keep printing the latest entries in the file.

You will also need to familiarise yourself to spawn more than one process - An example which requires running multiple processes to start an app.

To stop the script we just started run supervisorctl stop displayram.

A Few Options Explained

  • Directory

    • The directory which Supervisor switches to before executing the command.
    • Researching about this setting to get my app to work with Supervisor was when I learnt that when you activate an environment in Python you basically are just instructing your machine to use a copy of Python situated in a particular directory instead of using the default Python installation.
  • Command

    • The way via which Supervisor starts a particular program. Give the command / path that you normally use to start your program.
    • Commands are not equivalent to bash or your typical terminal commands by default. They are fairly similar but a command that you use on your terminal may not yield the same effect here.
    • Commands can contain absolute paths or relative paths. The displayram example uses absolute paths. If you want to use relative paths you need to set the path. You can read more about it here.
    • You can ensure that your command is treated as a bash command by using bash -c "command here". Our earlier example can be rewritten as
    [program=displayram]
    directory=/home/abhirath
    command=bash -c "./displayram"
    autostart=false
    autorestart=false
    stderr_logfile=NONE
    stdout_logfile=/home/abhirath/displayram.out.log
  • Stop and Kill as Group

    • Sometimes your program doesn’t really get killed. They might have children that they don’t kill. So to actually stop them Supervisor has to be mean and kill every process that might have spawned. One possible reason is that they expect a different kill signal.
    • You don’t have to use them unless you notice that your program is still running even after you explicitly ran
      supervisorctl stop programname
    • Really killing a program.
  • Logs

    • stderr_logfile - the file to which it shall redirect stderr to. If set to NONE it will not log output from stderr.
    • stdout_logfile - the file to which it redirect stdout to. This can also be set to NONE.
    • stderr_logfile_maxbytes - maximum space that the log file will take before Supervisor uses a new file to store logs. You can input a numerical followed by GB, KB etc. By default it’s set to 50MB.
    • stdout_logfile_maxbytes - same as above but for stdout.
    • stdout_logfile_backups - the number of backup files it stores. Say you’ve set it to 3. After the first files exceeds the value of stdout_logfile_maxbytes it creates a second file then a third. If the third file exceeds the size it proceeds to delete the first file and continue logging. In all it would use a maximum of 3 * 50MB in our case. By default it has a value of 10.
    • stderr_logfile_backups - same as above but for stderr.
    • An example which uses a variety of log settings.

Supervisor saves it’s own logs as well. They are different from the logs you set for your program. These logs contain information about Supervisor. They are stored in /var/log/supervisor/supervisord.log
To see the last few logs you can
tail /var/log/supervisor/supervisord.log

A useful read - Options available under program.

Some Common Errors

  1. You get a message similar to unix:///var/run...sock no such file?

It means Supervisor isn’t running. You will have to start Supervisor.
sudo service supervisor restart

  1. Spawn errors. Supervisord is unable to start your program?

    • Check the error log and output logs. Supervisor keeps appending stuff to your program’s log file. So it’s a good idea to use tail -f /path/to/program/log/file to keep track of changes happening.
    • Check the environment and directory options under your program. Give absolute paths like /home/abhirath/so/and/so.
  2. Supervisor doesn’t start automatically after bootup?

  3. Frequently new .conf files that you write are not recognised. If that is the case do the following:-
    Check the syntax of everything in the file. If there’s an error Supervisor doesn’t even show the file. Also the files must have an extension of .conf.

  4. Any signs of the process / app that you want to start is already running?

    • They might already be running. Check the status using supervisorctl appname status.
    • I’ve seen .NET apps that don’t kill their child processes by themselves. So even after you stop a program their child process might be running. For instance you might get errors like port is already in use.
    • Read about stopasgroup, killasgroup and stopsignal over here
    • stopasgroup, killasgroup work on individual programs. They are not to be confused with group applications
    • Really killing a program.