Sign in Subscribe

Integrating Sphinx doctests with Travis

Integrating Sphinx doctests with Travis

If you want to run Sphinx doc tests in Travis you will need to jump through some hoops. Fortunately it isn't that complicated, I've created some sample code on Github/doctest-travis for you to follow along.

Doc test are a way to ensure that your documentation is up to date. In principle doc tests should be largely analogous to unit tests. That's the reason we want to be able to run them from our CI job. On the other hand, doc tests are part of the documentation, so they will actually be visible to the reader. In practice, if using Sphinx, you will likely add sample code to your documentation in a doc test as part of you regular documentation. Actually executing doc tests as part of your CI ensures that what the user perceives as sample code will actually work the way they expect it to.

The Python module

Doc tests are currently only supported for Python code, that means there is no easy way to thread in command line invocation, compile C++ or run Ruby code. We'll start with a simple Python module in doctest_travis/code.py.

def stupid_sum(a, b):
    return a+b+1

def stupid_mul(a, b):
    return a*b*2

Additionally there's some support code in the __init__.py to load these functions into the package by default.

The basic unit tests

To test our stupid functions, we'll create some basic unit tests. These reside in tests/test_stupid.py and get executed by pytest. They use the Python unittest framework.

from doctest_travis import *

class TestStupidFunctions(unittest.TestCase):

	def test_stupid_sum(self):
		self.assertEqual(stupid_sum(1,2), 4)

	def test_stupid_mul(self):
        self.assertEqual(stupid_mul(2,2), 8)

Now running pytest tests/ from the top level will execute our unit tests.

Autodoc and doctest

We'll use autodoc to import function signatures from Python. Running doc tests is a lot like running regular unit tests. We'll can use a testsetup and testcleanup directive in the document and initialize and tear down the testing environment in there. Both of these directives will not generate any output.

See the doctest docs for a full description of all options available.

Here's an excerpt from mashing up these two functionalities.

doctest_travis
--------------

.. automodule:: doctest_travis
   :members:


.. autofunction:: doctest_travis.stupid_sum

.. doctest::

	>>> stupid_sum(2, 3)
	6

You provide a python expression prefixed by >>> on thefirst line and end with an expression that is parseable by Python on the second line. The Sphinx doctest module will compare both and return success, if they match.

You can find the output of doctest-travis on readthedocs.org doctest-travis rtd.

The user will generally perceive this as sample code in the HTML form. Of course we want to go beyond just providing sample code, we want to have a doc test.

Go to the project's root directory and run the following command to check if our doc tests work.

$ python -msphinx -b doctest docs foo
...
Document: index
---------------
1 items passed all tests:
   2 tests in default
2 tests in 1 items.
2 passed and 0 failed.
Test passed.

Doctest summary
===============
    2 tests
    0 failures in tests
    0 failures in setup code
    0 failures in cleanup code

There we go. Our example code now runs as a doc test.

Tying everything into Travis

We'll use tox to automate the test runs. We'll have unit tests, a sample doc build and our doc test runner all running in the test step. All will be driven by tox. Here is the sample tox.ini to make this happen.

[travis]
python =
  2.7: py27
  3.6: py36

[testenv]
deps = 
    pytest
    sphinx
commands = 
    pytest --basetemp={envtmpdir} tests/
    python -msphinx -b html -d {toxworkdir}/_build/doctrees docs {toxworkdir}/_build/html
    python -msphinx -b doctest -d {envdir}/.cache/doctrees docs {envdir}/.cache/doctest

We'll map Travis environments to Python interpreters manually, using the [travis] section and the we'll pass our dependencies into [testenv]. We need both pytest and sphinx in the virtualenv that will be generated by tox. Finally we'll specify a set of commands. pytest runs our unit tests, python -msphinx -b html... runs the HTML builder and python -msphinx -b doctest... runs the doc test runner. All the output will later show up in Travis. We can now run tox locally by calling it on the console.

$ tox

The missing piece of information is our Travis configuration file. Travis runs different Python builds in different containers and that's what we want to do. So Python 2.7 will run in a specific container and Python 3.6 will run in a different container. This is a little bit redundant with tox, which also can spawn different virtualenv environments to run different versions of Python. What we need is a Travis file that triggers different build per Python version, the tox file given above will simply pick up the current Python version and build based on that version. Here's the .travis.yml.

sudo: false
language: python
python:
  - "2.7"
  - "3.6"
install: pip install tox-travis
script: tox -vv

The [travis] section in the tox.ini is filled out by this tox-travis plug-in that we trigger.

Conclusion

You can find the build status of this project on the Travis site here.

Now we've wrapped up everything. Travis calls tox, which calls the test runners and test builds. I hope you can adapt this scheme to your own projects.

Image credits: I, Avenafatua [GFDL (http://www.gnu.org/copyleft/fdl.html), CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/) or CC BY-SA 2.5 (https://creativecommons.org/licenses/by-sa/2.5)]

Follow me on Mastodon!