Automatically end-to-end test H5P content types

From time to time, I take a day “off”. That usually means I am not coding to earn money, but to try out new things 🙂 Here’s a brief post about what I played with yesterday. It’s probably not interesting from a user perspective, but H5P developers may find this interesting.

A lot of time is currently spent with testing H5P content types. There are test cases for what they should do when a user completes tasks. For instance, if a user clicks on the “Check” button, the given answers should get evaluated, the score should be displayed, other buttons should appear (or not depending on the configuration), etc. These test cases are checked one by one. Again and again when things change, just to make sure nothing broke as a side effect of some other change. This takes time. A lot of time. Wouldn’t it be great if one could automate this?

Well, one can, of course. In theory. This kind of testing, when essentially pretending to be a user and checking whether everything works correctly on a system level in contrast to testing smaller parts, is called “system testing” (or end-to-end testing). There are different tools to help with that, and now that H5P Group brushed up the H5P CLI tool for developers, it’s not that hard to come up with a solution for testing H5P content types (or H5P plugins) automatically. I basically wrote a proof-of-concept yesterday.

My solution essentially takes an H5P content, loads it on a server that the H5P CLI tool provides, and then it makes use of the Playwright framework. That framework allows you to write the script for the play that you want to perform on stage (the browser). And you can define what you expect to happen after each step. For instance, I can simulate a click on the “Check” button and then expect the “Show solution” button to be visible. Playwright will automatically check these things according to my script and give me a report. And if something failed, then I probably broke something … If one wants to, one can even watch the whole play. Kind of fun, like a speed-run 😀

There are still some things left to be ironed out, and things could run a little smoother with further improvements to the H5P CLI tool, but it’s working already. So, hopefully this will reduce the amount of time spent on repetitive tasks and help to reduce the number of bugs at some point …