Classes involved in doctesting

This module controls the various classes involved in doctesting.

AUTHORS:

• David Roe (2012-03-27) – initial version, based on Robert Bradshaw’s code.

class sage.doctest.control.DocTestController(options, args)

Bases: SageObject

This class controls doctesting of files.

After creating it with appropriate options, call the run() method to run the doctests.

add_files()

Checks for the flags ‘–all’ and ‘–new’.

For each one present, this function adds the appropriate directories and files to the todo list.

EXAMPLES:

sage: from sage.doctest.control import (DocTestDefaults,
....:                                   DocTestController)
sage: from sage.env import SAGE_SRC
sage: import tempfile
sage: with tempfile.NamedTemporaryFile() as f:
....:     DD = DocTestDefaults(all=True, logfile=f.name)
....:     DC = DocTestController(DD, [])
....:     DC.add_files()
Doctesting ...
sage: os.path.join(SAGE_SRC, 'sage') in DC.files
True

sage: DD = DocTestDefaults(new = True)
sage: DC = DocTestController(DD, [])
sage: DC.add_files()
Doctesting ...

cleanup(final=True)

Runs cleanup activities after actually running doctests.

In particular, saves the stats to disk and closes the logfile.

INPUT:

• final – whether to close the logfile

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: from sage.env import SAGE_SRC
sage: import os
sage: dirname = os.path.join(SAGE_SRC, 'sage', 'rings', 'all.py')
sage: DD = DocTestDefaults()

sage: DC = DocTestController(DD, [dirname])
sage: DC.expand_files_into_sources()
sage: DC.sources.sort(key=lambda s:s.basename)

sage: for i, source in enumerate(DC.sources):
....:     DC.stats[source.basename] = {'walltime': 0.1*(i+1)}
....:

sage: DC.run()
Running doctests with ID ...
Doctesting 1 file.
sage -t .../rings/all.py
    [... tests, ... s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: ... seconds
    cpu time: ... seconds
    cumulative wall time: ... seconds
Features detected...
0
sage: DC.cleanup()

create_run_id()

Creates the run id.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: DC = DocTestController(DocTestDefaults(), [])
sage: DC.create_run_id()
Running doctests with ID ...

expand_files_into_sources()

Expands self.files, which may include directories, into a list of sage.doctest.FileDocTestSource

This function also handles the optional command line option.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: from sage.env import SAGE_SRC
sage: import os
sage: dirname = os.path.join(SAGE_SRC, 'sage', 'doctest')
sage: DD = DocTestDefaults(optional='all')
sage: DC = DocTestController(DD, [dirname])
sage: DC.expand_files_into_sources()
sage: len(DC.sources)
12
sage: DC.sources[0].options.optional
True

sage: DD = DocTestDefaults(optional='magma,guava')
sage: DC = DocTestController(DD, [dirname])
sage: DC.expand_files_into_sources()
sage: all(t in DC.sources[0].options.optional for t in ['magma','guava'])
True

We check that files are skipped appropriately:

sage: dirname = tmp_dir()
sage: filename = os.path.join(dirname, 'not_tested.py')
sage: with open(filename, 'w') as f:
....:     _ = f.write("#"*80 + "\n\n\n\n## nodoctest\n    sage: 1+1\n    4")
sage: DC = DocTestController(DD, [dirname])
sage: DC.expand_files_into_sources()
sage: DC.sources
[]

The directory sage/doctest/tests contains nodoctest.py but the files should still be tested when that directory is explicitly given (as opposed to being recursed into):

sage: DC = DocTestController(DD, [os.path.join(SAGE_SRC, 'sage', 'doctest', 'tests')])
sage: DC.expand_files_into_sources()
sage: len(DC.sources) >= 10
True

filter_sources()

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: from sage.env import SAGE_SRC
sage: import os
sage: dirname = os.path.join(SAGE_SRC, 'sage', 'doctest')
sage: DD = DocTestDefaults(failed=True)
sage: DC = DocTestController(DD, [dirname])
sage: DC.expand_files_into_sources()
sage: for i, source in enumerate(DC.sources):
....:     DC.stats[source.basename] = {'walltime': 0.1*(i+1)}
sage: DC.stats['sage.doctest.control'] = {'failed':True,'walltime':1.0}
sage: DC.filter_sources()
Only doctesting files that failed last test.
sage: len(DC.sources)
1

load_baseline_stats(filename)

Load baseline stats.

This must be a JSON file in the same format that load_stats() expects.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: DC = DocTestController(DocTestDefaults(), [])
sage: import json
sage: filename = tmp_filename()
sage: with open(filename, 'w') as stats_file:
....:     json.dump({'sage.doctest.control':{'failed':True}}, stats_file)
sage: DC.load_baseline_stats(filename)
sage: DC.baseline_stats['sage.doctest.control']
{'failed': True}

If the file doesn’t exist, nothing happens. If there is an error, print a message. In any case, leave the stats alone:

sage: d = tmp_dir()
sage: DC.load_baseline_stats(os.path.join(d))  # Cannot read a directory
Error loading baseline stats from ...
sage: DC.load_baseline_stats(os.path.join(d, "no_such_file"))
sage: DC.baseline_stats['sage.doctest.control']
{'failed': True}

load_environment()

Return the module that provides the global environment.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: DC = DocTestController(DocTestDefaults(), [])
sage: 'BipartiteGraph' in DC.load_environment().__dict__
True
sage: DC = DocTestController(DocTestDefaults(environment='sage.doctest.all'), [])
sage: 'BipartiteGraph' in  DC.load_environment().__dict__
False
sage: 'run_doctests' in DC.load_environment().__dict__
True

load_stats(filename)

Load stats from the most recent run(s).

Stats are stored as a JSON file, and include information on which files failed tests and the walltime used for execution of the doctests.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: DC = DocTestController(DocTestDefaults(), [])
sage: import json
sage: filename = tmp_filename()
sage: with open(filename, 'w') as stats_file:
....:     json.dump({'sage.doctest.control':{'walltime':1.0r}}, stats_file)
sage: DC.load_stats(filename)
sage: DC.stats['sage.doctest.control']
{'walltime': 1.0}

If the file doesn’t exist, nothing happens. If there is an error, print a message. In any case, leave the stats alone:

sage: d = tmp_dir()
sage: DC.load_stats(os.path.join(d))  # Cannot read a directory
Error loading stats from ...
sage: DC.load_stats(os.path.join(d, "no_such_file"))
sage: DC.stats['sage.doctest.control']
{'walltime': 1.0}

log(s, end='\n')

Log the string s + end (where end is a newline by default) to the logfile and print it to the standard output.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: DD = DocTestDefaults(logfile=tmp_filename())
sage: DC = DocTestController(DD, [])
sage: DC.log("hello world")
hello world
sage: DC.logfile.close()
sage: with open(DD.logfile) as f:
....:     print(f.read())
hello world

In serial mode, check that logging works even if stdout is redirected:

sage: DD = DocTestDefaults(logfile=tmp_filename(), serial=True)
sage: DC = DocTestController(DD, [])
sage: from sage.doctest.forker import SageSpoofInOut
sage: with open(os.devnull, 'w') as devnull:
....:     S = SageSpoofInOut(devnull)
....:     S.start_spoofing()
....:     DC.log("hello world")
....:     S.stop_spoofing()
hello world
sage: DC.logfile.close()
sage: with open(DD.logfile) as f:
....:     print(f.read())
hello world

Check that no duplicate logs appear, even when forking (github issue #15244):

sage: DD = DocTestDefaults(logfile=tmp_filename())
sage: DC = DocTestController(DD, [])
sage: DC.log("hello world")
hello world
sage: if os.fork() == 0:
....:     DC.logfile.close()
....:     os._exit(0)
sage: DC.logfile.close()
sage: with open(DD.logfile) as f:
....:     print(f.read())
hello world

run()

This function is called after initialization to set up and run all doctests.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: from sage.env import SAGE_SRC
sage: import os
sage: DD = DocTestDefaults()
sage: filename = os.path.join(SAGE_SRC, "sage", "sets", "non_negative_integers.py")
sage: DC = DocTestController(DD, [filename])
sage: DC.run()
Running doctests with ID ...
Doctesting 1 file.
sage -t .../sage/sets/non_negative_integers.py
    [... tests, ... s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: ... seconds
    cpu time: ... seconds
    cumulative wall time: ... seconds
Features detected...
0

We check that github issue #25378 is fixed (testing external packages while providing a logfile does not raise a ValueError: I/O operation on closed file):

sage: logfile = tmp_filename(ext='.log')
sage: DD = DocTestDefaults(optional=set(['sage', 'external']), logfile=logfile)
sage: filename = tmp_filename(ext='.py')
sage: DC = DocTestController(DD, [filename])
sage: DC.run()
Running doctests with ID ...
Using --optional=external,sage
Features to be detected: ...
Doctesting 1 file.
sage -t ....py
    [0 tests, ... s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: ... seconds
    cpu time: ... seconds
    cumulative wall time: ... seconds
Features detected...
0

We test the --hide option (github issue #34185):

sage: from sage.doctest.control import test_hide
sage: filename = tmp_filename(ext='.py')
sage: with open(filename, 'w') as f:
....:     f.write(test_hide)
....:     f.close()
729
sage: DF = DocTestDefaults(hide='buckygen,all')
sage: DC = DocTestController(DF, [filename])
sage: DC.run()
Running doctests with ID ...
Using --optional=sage...
Features to be detected: ...
Doctesting 1 file.
sage -t ....py
    [4 tests, ... s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: ... seconds
    cpu time: ... seconds
    cumulative wall time: ... seconds
Features detected...
0

sage: DF = DocTestDefaults(hide='benzene,optional')
sage: DC = DocTestController(DF, [filename])
sage: DC.run()
Running doctests with ID ...
Using --optional=sage
Features to be detected: ...
Doctesting 1 file.
sage -t ....py
    [4 tests, ... s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: ... seconds
    cpu time: ... seconds
    cumulative wall time: ... seconds
Features detected...
0

run_doctests()

Actually runs the doctests.

This function is called by run().

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: from sage.env import SAGE_SRC
sage: import os
sage: dirname = os.path.join(SAGE_SRC, 'sage', 'rings', 'homset.py')
sage: DD = DocTestDefaults()
sage: DC = DocTestController(DD, [dirname])
sage: DC.expand_files_into_sources()
sage: DC.run_doctests()
Doctesting 1 file.
sage -t .../sage/rings/homset.py
    [... tests, ... s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: ... seconds
    cpu time: ... seconds
    cumulative wall time: ... seconds...

run_val_gdb(testing=False)

Spawns a subprocess to run tests under the control of gdb, lldb, or valgrind.

INPUT:

• testing – boolean; if True then the command to be run will be printed rather than a subprocess started.

EXAMPLES:

Note that the command lines include unexpanded environment variables. It is safer to let the shell expand them than to expand them here and risk insufficient quoting.

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: DD = DocTestDefaults(gdb=True)
sage: DC = DocTestController(DD, ["hello_world.py"])
sage: DC.run_val_gdb(testing=True)
exec gdb --eval-command="run" --args ...python... sage-runtests --serial --timeout=0 hello_world.py

sage: DD = DocTestDefaults(valgrind=True, optional="all", timeout=172800)
sage: DC = DocTestController(DD, ["hello_world.py"])
sage: DC.run_val_gdb(testing=True)
exec valgrind --tool=memcheck --leak-resolution=high --leak-check=full --num-callers=25 --suppressions="...valgrind/pyalloc.supp" --suppressions="...valgrind/sage.supp" --suppressions="...valgrind/sage-additional.supp"  --log-file=.../valgrind/sage-memcheck.%p... sage-runtests --serial --timeout=172800 --optional=all hello_world.py

save_stats(filename)

Save stats from the most recent run as a JSON file.

WARNING: This function overwrites the file.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: DC = DocTestController(DocTestDefaults(), [])
sage: DC.stats['sage.doctest.control'] = {'walltime':1.0r}
sage: filename = tmp_filename()
sage: DC.save_stats(filename)
sage: import json
sage: with open(filename) as f:
....:     D = json.load(f)
sage: D['sage.doctest.control']
{'walltime': 1.0}

second_on_modern_computer()

Return the wall time equivalent of a second on a modern computer.

OUTPUT:

Float. The wall time on your computer that would be equivalent to one second on a modern computer. Unless you have kick-ass hardware this should always be >= 1.0. Raises a RuntimeError if there are no stored timings to use as benchmark.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: DC = DocTestController(DocTestDefaults(), [])
sage: DC.second_on_modern_computer()   # not tested

sort_sources()

This function sorts the sources so that slower doctests are run first.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: from sage.env import SAGE_SRC
sage: import os
sage: dirname = os.path.join(SAGE_SRC, 'sage', 'doctest')
sage: DD = DocTestDefaults(nthreads=2)
sage: DC = DocTestController(DD, [dirname])
sage: DC.expand_files_into_sources()
sage: DC.sources.sort(key=lambda s:s.basename)
sage: for i, source in enumerate(DC.sources):
....:     DC.stats[source.basename] = {'walltime': 0.1*(i+1)}
sage: DC.sort_sources()
Sorting sources by runtime so that slower doctests are run first....
sage: print("\n".join(source.basename for source in DC.sources))
sage.doctest.util
sage.doctest.test
sage.doctest.sources
sage.doctest.reporting
sage.doctest.parsing_test
sage.doctest.parsing
sage.doctest.forker
sage.doctest.fixtures
sage.doctest.external
sage.doctest.control
sage.doctest.all
sage.doctest

source_baseline(source)

Return the baseline_stats value of source.

INPUT:

• source – a DocTestSource instance

OUTPUT:

A dictionary.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults, DocTestController
sage: from sage.env import SAGE_SRC
sage: import os
sage: filename = os.path.join(SAGE_SRC,'sage','doctest','util.py')
sage: DD = DocTestDefaults()
sage: DC = DocTestController(DD, [filename])
sage: DC.expand_files_into_sources()
sage: DC.source_baseline(DC.sources[0])
{}

class sage.doctest.control.DocTestDefaults(**kwds)

Bases: SageObject

This class is used for doctesting the Sage doctest module.

It fills in attributes to be the same as the defaults defined in sage-runtests, expect for a few places, which is mostly to make doctesting more predictable.

EXAMPLES:

sage: from sage.doctest.control import DocTestDefaults
sage: D = DocTestDefaults()
sage: D
DocTestDefaults()
sage: D.timeout
-1

Keyword arguments become attributes:

sage: D = DocTestDefaults(timeout=100)
sage: D
DocTestDefaults(timeout=100)
sage: D.timeout
100

class sage.doctest.control.Logger(*files)

Bases: object

File-like object which implements writing to multiple files at once.

EXAMPLES:

sage: from sage.doctest.control import Logger
sage: with open(tmp_filename(), "w+") as t:
....:     L = Logger(sys.stdout, t)
....:     _ = L.write("hello world\n")
....:     _ = t.seek(0)
....:     t.read()
hello world
'hello world\n'

flush()

Flush all files.

write(x)

Write x to all files.

sage.doctest.control.run_doctests(module, options=None)

Runs the doctests in a given file.

INPUT:

• module – a Sage module, a string, or a list of such.

• options – a DocTestDefaults object or None.

EXAMPLES:

sage: run_doctests(sage.rings.all)
Running doctests with ID ...
Doctesting 1 file.
sage -t .../sage/rings/all.py
    [... tests, ... s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: ... seconds
    cpu time: ... seconds
    cumulative wall time: ... seconds
Features detected...

sage.doctest.control.skipdir(dirname)

Return True if and only if the directory dirname should not be doctested.

EXAMPLES:

sage: from sage.doctest.control import skipdir
sage: skipdir(sage.env.SAGE_SRC)
False
sage: skipdir(os.path.join(sage.env.SAGE_SRC, "sage", "doctest", "tests"))
True

sage.doctest.control.skipfile(filename, tested_optional_tags, if_installed, log=False)

Return True if and only if the file filename should not be doctested.

INPUT:

• filename – name of a file

• tested_optional_tags – a list or tuple or set of optional tags to test, or False (no optional test) or True (all optional tests)

• if_installed – (boolean, default False) whether to skip Python/Cython files that are not installed as modules

• log – function to call with log messages, or None

If filename contains a line of the form "# sage.doctest: optional - xyz"), then this will return False if “xyz” is in tested_optional_tags. Otherwise, it returns the matching tag (“optional - xyz”).

EXAMPLES:

sage: from sage.doctest.control import skipfile
sage: skipfile("skipme.c")
True
sage: filename = tmp_filename(ext=".pyx")
sage: skipfile(filename)
False
sage: with open(filename, "w") as f:
....:     _ = f.write("# nodoctest")
sage: skipfile(filename)
True
sage: with open(filename, "w") as f:
....:     _ = f.write("# sage.doctest: "    # broken in two source lines to avoid the pattern
....:                 "optional - xyz")     # of relint (multiline_doctest_comment)
sage: skipfile(filename, False)
'optional - xyz'
sage: bool(skipfile(filename, False))
True
sage: skipfile(filename, ['abc'])
'optional - xyz'
sage: skipfile(filename, ['abc', 'xyz'])
False
sage: skipfile(filename, True)
False