I recently completed a Udacity course on software testing. It was a quick course, but I learned a few interesting techniques. Mostly, I got the chance to use a couple of tools I had heard about but never actually used.

Python coverage measurements

Python has a tool called Coverage.py that generates statement and branch coverage metrics for a given set of tests. For instance, we can write the following Fizz Buzz program and set of tests.

def fizzbuzz(n):
  if n % 15 == 0:
    return "FizzBuzz"
  if n % 3 == 0:
    return "Fizz"
  if n % 5 == 0:
    return "Buzz"
  return str(n)

assert fizzbuzz(3) == "Fizz"
assert fizzbuzz(4) == "4"
assert fizzbuzz(5) == "Buzz"

We can evaluate our tests with the coverage tool.

coverage erase
coverage run --branch fizzbuzz.py
coverage html
open htmlcov/index.html

This produces the following output.

Based on this, we can easily see that we don’t have any tests covering the condition where n is a multiple of 15. The red line indicates a lack of statement-level coverage, while the yellow line shows a partially covered branch. We can fix both of these issues by adding one more test to our program.

assert fizzbuzz(30) == "FizzBuzz"

Fuzzing a program

My favorite exercise of the course was writing a simple random fuzzer. Random fuzzing is surprising effective across many programs, and although I did not find any bugs I am glad to have practiced the technique.

Random fuzzing involves starting with good inputs, mutating random bytes, and feeding those modified inputs to the software under test. I decided to test feh, a photo viewer. The fuzzing code is pretty self-explanatory, which I am excited about: it turns out writing a basic fuzzer is easy!

import math
import os
import random
import string
import subprocess
import time

file_dir = "input-photos/"
file_list = os.listdir(file_dir)
app = "/usr/local/bin/feh"
output_dir = "output-photos/"

FuzzFactor = 250
num_tests = 10000

for i in range(num_tests):
  file_choice = random.choice(file_list)
  with open(file_dir + file_choice, 'rb') as f:
    buf = bytearray(f.read())
  numwrites = random.randrange(math.ceil((float(len(buf)) / FuzzFactor))) + 1

  for j in range(numwrites):
    rbyte = random.randrange(256)
    rn = random.randrange(len(buf))
    buf[rn] = rbyte

  output = output_dir + str(i) + ".tif"
  with open(output, 'w+b') as f:
    f.write(buf)

  process = subprocess.Popen([app, output], stdout=subprocess.DEVNULL,
    stderr=subprocess.DEVNULL)

  time.sleep(2)
  crashed = process.poll()
  if crashed and crashed != 0 and crashed != 1:
    print("Crash on test {}!".format(i))
  else:
    process.terminate()