Property Base Your State Machine

Sean Chalmers, Jack Kelly

Queensland Functional Programming Lab

Programs are weird

Complex, Dependent, Interactive, Insidious

Testing weird is hard

  • manual testing? [cries in Quality Assurance]
  • which processes do you test?
  • test for failure?

Cost : Benefit

  • multi-step testing can be arduous and brittle
    • input and process are static
    • coverage
    • new feature? have fun!

Do what we can

  • eat the cost for some big things
  • write unit tests for some small things
getLunch "fishy" == TunaSalad
getLunch "hungry spicy" == ExtraHot HalfChicken
  • subcontract end users to suggest regression tests

Property-based awesome

  • unleash the power of \(\forall\)
  • shrinking
  • powerful design tool
    • 'forAll text' becomes 'forAll lunchType'

Property-based state machine testing

property-based testing, lifted to the level of application behaviour.

Generate structured inputs

randomised inputs for actions

Generate list of actions

actions defined independently, package generates a list

Shrinking (OMG!)

you thought shrinking was cool before…

Useful feedback

a failure will produce:

  • the sequence of actions
  • their respective inputs

OMG

Right?

Moving parts

  • Model
  • Command

Model

  • abstract model of the thing under test
  • not the thing itself

Command

  • represents transitions
  • not states

Given

mach.png

Not this

mach_states.png

Test this

mach_transitions.png

Parts & Flow

  • generation (maybe)
  • pre-condition(s)
  • execution
  • post-condition(s)
  • update

Machines that grow

new feature ~ new Command(s)

Examples

  • Wordpress API (a tiny bit of it)
  • LevelDB

Come to the workshop!

Learn the weirding way!

Bonus Round

Use classification to understand your test-case distribution

References