Classes involved in doctesting#
This module controls the various classes involved in doctesting.
AUTHORS:
David Roe (2012-03-27) – initial version, based on Robert Bradshaw’s code.
- class sage.doctest.control.DocTestController(options, args)#
Bases:
SageObject
This class controls doctesting of files.
After creating it with appropriate options, call the
run()
method to run the doctests.- add_files()#
Checks for the flags ‘–all’ and ‘–new’.
For each one present, this function adds the appropriate directories and files to the todo list.
EXAMPLES:
sage: from sage.doctest.control import (DocTestDefaults, ....: DocTestController) sage: from sage.env import SAGE_SRC sage: import tempfile sage: with tempfile.NamedTemporaryFile() as f: ....: DD = DocTestDefaults(all=True, logfile=f.name) ....: DC = DocTestController(DD, []) ....: DC.add_files() Doctesting ... sage: os.path.join(SAGE_SRC, 'sage') in DC.files True
sage: DD = DocTestDefaults(new = True) sage: DC = DocTestController(DD, []) sage: DC.add_files() Doctesting ...
- cleanup(final=True)#
Runs cleanup activities after actually running doctests.
In particular, saves the stats to disk and closes the logfile.
INPUT:
final
– whether to close the logfile
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: from sage.env import SAGE_SRC sage: import os sage: dirname = os.path.join(SAGE_SRC, 'sage', 'rings', 'all.py') sage: DD = DocTestDefaults() sage: DC = DocTestController(DD, [dirname]) sage: DC.expand_files_into_sources() sage: DC.sources.sort(key=lambda s:s.basename) sage: for i, source in enumerate(DC.sources): ....: DC.stats[source.basename] = {'walltime': 0.1*(i+1)} ....: sage: DC.run() Running doctests with ID ... Doctesting 1 file. sage -t .../rings/all.py [... tests, ... s] ---------------------------------------------------------------------- All tests passed! ---------------------------------------------------------------------- Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Features detected... 0 sage: DC.cleanup()
- create_run_id()#
Creates the run id.
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: DC = DocTestController(DocTestDefaults(), []) sage: DC.create_run_id() Running doctests with ID ...
- expand_files_into_sources()#
Expands
self.files
, which may include directories, into a list ofsage.doctest.FileDocTestSource
This function also handles the optional command line option.
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: from sage.env import SAGE_SRC sage: import os sage: dirname = os.path.join(SAGE_SRC, 'sage', 'doctest') sage: DD = DocTestDefaults(optional='all') sage: DC = DocTestController(DD, [dirname]) sage: DC.expand_files_into_sources() sage: len(DC.sources) 12 sage: DC.sources[0].options.optional True
sage: DD = DocTestDefaults(optional='magma,guava') sage: DC = DocTestController(DD, [dirname]) sage: DC.expand_files_into_sources() sage: all(t in DC.sources[0].options.optional for t in ['magma','guava']) True
We check that files are skipped appropriately:
sage: dirname = tmp_dir() sage: filename = os.path.join(dirname, 'not_tested.py') sage: with open(filename, 'w') as f: ....: _ = f.write("#"*80 + "\n\n\n\n## nodoctest\n sage: 1+1\n 4") sage: DC = DocTestController(DD, [dirname]) sage: DC.expand_files_into_sources() sage: DC.sources []
The directory
sage/doctest/tests
containsnodoctest.py
but the files should still be tested when that directory is explicitly given (as opposed to being recursed into):sage: DC = DocTestController(DD, [os.path.join(SAGE_SRC, 'sage', 'doctest', 'tests')]) sage: DC.expand_files_into_sources() sage: len(DC.sources) >= 10 True
- filter_sources()#
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: from sage.env import SAGE_SRC sage: import os sage: dirname = os.path.join(SAGE_SRC, 'sage', 'doctest') sage: DD = DocTestDefaults(failed=True) sage: DC = DocTestController(DD, [dirname]) sage: DC.expand_files_into_sources() sage: for i, source in enumerate(DC.sources): ....: DC.stats[source.basename] = {'walltime': 0.1*(i+1)} sage: DC.stats['sage.doctest.control'] = {'failed':True,'walltime':1.0} sage: DC.filter_sources() Only doctesting files that failed last test. sage: len(DC.sources) 1
- load_baseline_stats(filename)#
Load baseline stats.
This must be a JSON file in the same format that
load_stats()
expects.EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: DC = DocTestController(DocTestDefaults(), []) sage: import json sage: filename = tmp_filename() sage: with open(filename, 'w') as stats_file: ....: json.dump({'sage.doctest.control':{'failed':True}}, stats_file) sage: DC.load_baseline_stats(filename) sage: DC.baseline_stats['sage.doctest.control'] {'failed': True}
If the file doesn’t exist, nothing happens. If there is an error, print a message. In any case, leave the stats alone:
sage: d = tmp_dir() sage: DC.load_baseline_stats(os.path.join(d)) # Cannot read a directory Error loading baseline stats from ... sage: DC.load_baseline_stats(os.path.join(d, "no_such_file")) sage: DC.baseline_stats['sage.doctest.control'] {'failed': True}
- load_environment()#
Return the module that provides the global environment.
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: DC = DocTestController(DocTestDefaults(), []) sage: 'BipartiteGraph' in DC.load_environment().__dict__ True sage: DC = DocTestController(DocTestDefaults(environment='sage.doctest.all'), []) sage: 'BipartiteGraph' in DC.load_environment().__dict__ False sage: 'run_doctests' in DC.load_environment().__dict__ True
- load_stats(filename)#
Load stats from the most recent run(s).
Stats are stored as a JSON file, and include information on which files failed tests and the walltime used for execution of the doctests.
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: DC = DocTestController(DocTestDefaults(), []) sage: import json sage: filename = tmp_filename() sage: with open(filename, 'w') as stats_file: ....: json.dump({'sage.doctest.control':{'walltime':1.0r}}, stats_file) sage: DC.load_stats(filename) sage: DC.stats['sage.doctest.control'] {'walltime': 1.0}
If the file doesn’t exist, nothing happens. If there is an error, print a message. In any case, leave the stats alone:
sage: d = tmp_dir() sage: DC.load_stats(os.path.join(d)) # Cannot read a directory Error loading stats from ... sage: DC.load_stats(os.path.join(d, "no_such_file")) sage: DC.stats['sage.doctest.control'] {'walltime': 1.0}
- log(s, end='\n')#
Log the string
s + end
(whereend
is a newline by default) to the logfile and print it to the standard output.EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: DD = DocTestDefaults(logfile=tmp_filename()) sage: DC = DocTestController(DD, []) sage: DC.log("hello world") hello world sage: DC.logfile.close() sage: with open(DD.logfile) as f: ....: print(f.read()) hello world
In serial mode, check that logging works even if
stdout
is redirected:sage: DD = DocTestDefaults(logfile=tmp_filename(), serial=True) sage: DC = DocTestController(DD, []) sage: from sage.doctest.forker import SageSpoofInOut sage: with open(os.devnull, 'w') as devnull: ....: S = SageSpoofInOut(devnull) ....: S.start_spoofing() ....: DC.log("hello world") ....: S.stop_spoofing() hello world sage: DC.logfile.close() sage: with open(DD.logfile) as f: ....: print(f.read()) hello world
Check that no duplicate logs appear, even when forking (github issue #15244):
sage: DD = DocTestDefaults(logfile=tmp_filename()) sage: DC = DocTestController(DD, []) sage: DC.log("hello world") hello world sage: if os.fork() == 0: ....: DC.logfile.close() ....: os._exit(0) sage: DC.logfile.close() sage: with open(DD.logfile) as f: ....: print(f.read()) hello world
- run()#
This function is called after initialization to set up and run all doctests.
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: from sage.env import SAGE_SRC sage: import os sage: DD = DocTestDefaults() sage: filename = os.path.join(SAGE_SRC, "sage", "sets", "non_negative_integers.py") sage: DC = DocTestController(DD, [filename]) sage: DC.run() Running doctests with ID ... Doctesting 1 file. sage -t .../sage/sets/non_negative_integers.py [... tests, ... s] ---------------------------------------------------------------------- All tests passed! ---------------------------------------------------------------------- Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Features detected... 0
We check that github issue #25378 is fixed (testing external packages while providing a logfile does not raise a ValueError: I/O operation on closed file):
sage: logfile = tmp_filename(ext='.log') sage: DD = DocTestDefaults(optional=set(['sage', 'external']), logfile=logfile) sage: filename = tmp_filename(ext='.py') sage: DC = DocTestController(DD, [filename]) sage: DC.run() Running doctests with ID ... Using --optional=external,sage Features to be detected: ... Doctesting 1 file. sage -t ....py [0 tests, ... s] ---------------------------------------------------------------------- All tests passed! ---------------------------------------------------------------------- Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Features detected... 0
We test the
--hide
option (github issue #34185):sage: from sage.doctest.control import test_hide sage: filename = tmp_filename(ext='.py') sage: with open(filename, 'w') as f: ....: f.write(test_hide) ....: f.close() 729 sage: DF = DocTestDefaults(hide='buckygen,all') sage: DC = DocTestController(DF, [filename]) sage: DC.run() Running doctests with ID ... Using --optional=sage... Features to be detected: ... Doctesting 1 file. sage -t ....py [4 tests, ... s] ---------------------------------------------------------------------- All tests passed! ---------------------------------------------------------------------- Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Features detected... 0 sage: DF = DocTestDefaults(hide='benzene,optional') sage: DC = DocTestController(DF, [filename]) sage: DC.run() Running doctests with ID ... Using --optional=sage Features to be detected: ... Doctesting 1 file. sage -t ....py [4 tests, ... s] ---------------------------------------------------------------------- All tests passed! ---------------------------------------------------------------------- Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Features detected... 0
- run_doctests()#
Actually runs the doctests.
This function is called by
run()
.EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: from sage.env import SAGE_SRC sage: import os sage: dirname = os.path.join(SAGE_SRC, 'sage', 'rings', 'homset.py') sage: DD = DocTestDefaults() sage: DC = DocTestController(DD, [dirname]) sage: DC.expand_files_into_sources() sage: DC.run_doctests() Doctesting 1 file. sage -t .../sage/rings/homset.py [... tests, ... s] ---------------------------------------------------------------------- All tests passed! ---------------------------------------------------------------------- Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds...
- run_val_gdb(testing=False)#
Spawns a subprocess to run tests under the control of gdb, lldb, or valgrind.
INPUT:
testing
– boolean; if True then the command to be run will be printed rather than a subprocess started.
EXAMPLES:
Note that the command lines include unexpanded environment variables. It is safer to let the shell expand them than to expand them here and risk insufficient quoting.
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: DD = DocTestDefaults(gdb=True) sage: DC = DocTestController(DD, ["hello_world.py"]) sage: DC.run_val_gdb(testing=True) exec gdb --eval-command="run" --args ...python... sage-runtests --serial --timeout=0 hello_world.py
sage: DD = DocTestDefaults(valgrind=True, optional="all", timeout=172800) sage: DC = DocTestController(DD, ["hello_world.py"]) sage: DC.run_val_gdb(testing=True) exec valgrind --tool=memcheck --leak-resolution=high --leak-check=full --num-callers=25 --suppressions="...valgrind/pyalloc.supp" --suppressions="...valgrind/sage.supp" --suppressions="...valgrind/sage-additional.supp" --log-file=.../valgrind/sage-memcheck.%p... sage-runtests --serial --timeout=172800 --optional=all hello_world.py
- save_stats(filename)#
Save stats from the most recent run as a JSON file.
WARNING: This function overwrites the file.
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: DC = DocTestController(DocTestDefaults(), []) sage: DC.stats['sage.doctest.control'] = {'walltime':1.0r} sage: filename = tmp_filename() sage: DC.save_stats(filename) sage: import json sage: with open(filename) as f: ....: D = json.load(f) sage: D['sage.doctest.control'] {'walltime': 1.0}
- second_on_modern_computer()#
Return the wall time equivalent of a second on a modern computer.
OUTPUT:
Float. The wall time on your computer that would be equivalent to one second on a modern computer. Unless you have kick-ass hardware this should always be >= 1.0. Raises a
RuntimeError
if there are no stored timings to use as benchmark.EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: DC = DocTestController(DocTestDefaults(), []) sage: DC.second_on_modern_computer() # not tested
- sort_sources()#
This function sorts the sources so that slower doctests are run first.
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: from sage.env import SAGE_SRC sage: import os sage: dirname = os.path.join(SAGE_SRC, 'sage', 'doctest') sage: DD = DocTestDefaults(nthreads=2) sage: DC = DocTestController(DD, [dirname]) sage: DC.expand_files_into_sources() sage: DC.sources.sort(key=lambda s:s.basename) sage: for i, source in enumerate(DC.sources): ....: DC.stats[source.basename] = {'walltime': 0.1*(i+1)} sage: DC.sort_sources() Sorting sources by runtime so that slower doctests are run first.... sage: print("\n".join(source.basename for source in DC.sources)) sage.doctest.util sage.doctest.test sage.doctest.sources sage.doctest.reporting sage.doctest.parsing_test sage.doctest.parsing sage.doctest.forker sage.doctest.fixtures sage.doctest.external sage.doctest.control sage.doctest.all sage.doctest
- source_baseline(source)#
Return the
baseline_stats
value ofsource
.INPUT:
source
– aDocTestSource
instance
OUTPUT:
A dictionary.
EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults, DocTestController sage: from sage.env import SAGE_SRC sage: import os sage: filename = os.path.join(SAGE_SRC,'sage','doctest','util.py') sage: DD = DocTestDefaults() sage: DC = DocTestController(DD, [filename]) sage: DC.expand_files_into_sources() sage: DC.source_baseline(DC.sources[0]) {}
- class sage.doctest.control.DocTestDefaults(**kwds)#
Bases:
SageObject
This class is used for doctesting the Sage doctest module.
It fills in attributes to be the same as the defaults defined in
sage-runtests
, expect for a few places, which is mostly to make doctesting more predictable.EXAMPLES:
sage: from sage.doctest.control import DocTestDefaults sage: D = DocTestDefaults() sage: D DocTestDefaults() sage: D.timeout -1
Keyword arguments become attributes:
sage: D = DocTestDefaults(timeout=100) sage: D DocTestDefaults(timeout=100) sage: D.timeout 100
- class sage.doctest.control.Logger(*files)#
Bases:
object
File-like object which implements writing to multiple files at once.
EXAMPLES:
sage: from sage.doctest.control import Logger sage: with open(tmp_filename(), "w+") as t: ....: L = Logger(sys.stdout, t) ....: _ = L.write("hello world\n") ....: _ = t.seek(0) ....: t.read() hello world 'hello world\n'
- flush()#
Flush all files.
- write(x)#
Write
x
to all files.
- sage.doctest.control.run_doctests(module, options=None)#
Runs the doctests in a given file.
INPUT:
module
– a Sage module, a string, or a list of such.options
– a DocTestDefaults object or None.
EXAMPLES:
sage: run_doctests(sage.rings.all) Running doctests with ID ... Doctesting 1 file. sage -t .../sage/rings/all.py [... tests, ... s] ---------------------------------------------------------------------- All tests passed! ---------------------------------------------------------------------- Total time for all tests: ... seconds cpu time: ... seconds cumulative wall time: ... seconds Features detected...
- sage.doctest.control.skipdir(dirname)#
Return True if and only if the directory
dirname
should not be doctested.EXAMPLES:
sage: from sage.doctest.control import skipdir sage: skipdir(sage.env.SAGE_SRC) False sage: skipdir(os.path.join(sage.env.SAGE_SRC, "sage", "doctest", "tests")) True
- sage.doctest.control.skipfile(filename, tested_optional_tags, if_installed, log=False)#
Return
True
if and only if the filefilename
should not be doctested.INPUT:
filename
– name of a filetested_optional_tags
– a list or tuple or set of optional tags to test, orFalse
(no optional test) orTrue
(all optional tests)if_installed
– (boolean, defaultFalse
) whether to skip Python/Cython files that are not installed as moduleslog
– function to call with log messages, orNone
If
filename
contains a line of the form"# sage.doctest: optional - xyz")
, then this will returnFalse
if “xyz” is intested_optional_tags
. Otherwise, it returns the matching tag (“optional - xyz”).EXAMPLES:
sage: from sage.doctest.control import skipfile sage: skipfile("skipme.c") True sage: filename = tmp_filename(ext=".pyx") sage: skipfile(filename) False sage: with open(filename, "w") as f: ....: _ = f.write("# nodoctest") sage: skipfile(filename) True sage: with open(filename, "w") as f: ....: _ = f.write("# sage.doctest: " # broken in two source lines to avoid the pattern ....: "optional - xyz") # of relint (multiline_doctest_comment) sage: skipfile(filename, False) 'optional - xyz' sage: bool(skipfile(filename, False)) True sage: skipfile(filename, ['abc']) 'optional - xyz' sage: skipfile(filename, ['abc', 'xyz']) False sage: skipfile(filename, True) False