Decorate interface for parallel computation#
- class sage.parallel.decorate.Fork(timeout=0, verbose=False)#
Bases:
object
A
fork
decorator class.
- class sage.parallel.decorate.Parallel(p_iter='fork', ncpus=None, **kwds)#
Bases:
object
Create a
parallel
-decorated function. This is the object created byparallel()
.
- class sage.parallel.decorate.ParallelFunction(parallel, func)#
Bases:
object
Class which parallelizes a function or class method. This is typically accessed indirectly through
Parallel.__call__()
.
- sage.parallel.decorate.fork(f=None, timeout=0, verbose=False)#
Decorate a function so that when called it runs in a forked subprocess.
This means that it will not have any in-memory side effects on the parent Sage process. The pexpect interfaces are all reset.
INPUT:
f
– a functiontimeout
– (default: 0) if positive, kill the subprocess after this many seconds (wall time)verbose
– (default:False
) whether to print anything about what the decorator does (e.g., killing the subprocess)
Warning
The forked subprocess will not have access to data created in pexpect interfaces. This behavior with respect to pexpect interfaces is very important to keep in mind when setting up certain computations. It’s the one big limitation of this decorator.
EXAMPLES:
We create a function and run it with the
fork
decorator. Note that it does not have a side effect. Despite trying to change the global variablea
below ing
, the variablea
does not get changed:sage: a = 5 sage: @fork ....: def g(n, m): ....: global a ....: a = 10 ....: return factorial(n).ndigits() + m sage: g(5, m=5) 8 sage: a 5
We use
fork
to make sure that the function terminates after 100 ms, no matter what:sage: @fork(timeout=0.1, verbose=True) ....: def g(n, m): return factorial(n).ndigits() + m sage: g(5, m=5) 8 sage: g(10^7, m=5) Killing subprocess ... with input ((10000000,), {'m': 5}) which took too long 'NO DATA (timed out)'
We illustrate that the state of the pexpect interface is not altered by forked functions (they get their own new pexpect interfaces!):
sage: gp.eval('a = 5') '5' sage: @fork() ....: def g(): ....: gp.eval('a = 10') ....: return gp.eval('a') sage: g() '10' sage: gp.eval('a') '5'
We illustrate that the forked function has its own pexpect interface:
sage: gp.eval('a = 15') '15' sage: @fork() ....: def g(): return gp.eval('a') sage: g() 'a'
We illustrate that segfaulting subprocesses are no trouble at all:
sage: cython('def f(): print(<char*>0)') # optional - sage.misc.cython sage: @fork ....: def g(): ....: os.environ["CYSIGNALS_CRASH_NDEBUG"]="yes" # skip enhanced backtrace (it is slow) ....: f() sage: print("this works"); g() # optional - sage.misc.cython this works... ------------------------------------------------------------------------ Unhandled SIG... ------------------------------------------------------------------------ 'NO DATA'
- sage.parallel.decorate.normalize_input(a)#
Convert
a
to a pair(args, kwds)
using some rules:if already of that form, leave that way.
if
a
is a tuple make(a,{})
if
a
is a dict make(tuple(),a)
otherwise make
((a,),{})
INPUT:
a
– object
OUTPUT:
args
– tuplekwds
– dictionary
EXAMPLES:
sage: sage.parallel.decorate.normalize_input( (2, {3:4}) ) ((2, {3: 4}), {}) sage: sage.parallel.decorate.normalize_input( (2,3) ) ((2, 3), {}) sage: sage.parallel.decorate.normalize_input( {3:4} ) ((), {3: 4}) sage: sage.parallel.decorate.normalize_input( 5 ) ((5,), {})
- sage.parallel.decorate.parallel(p_iter='fork', ncpus=None, **kwds)#
This is a decorator that gives a function a parallel interface, allowing it to be called with a list of inputs, whose values will be computed in parallel.
Warning
The parallel subprocesses will not have access to data created in pexpect interfaces. This behavior with respect to pexpect interfaces is very important to keep in mind when setting up certain computations. It’s the one big limitation of this decorator.
INPUT:
p_iter
– parallel iterator function or string:'fork'
– (default) use a new forked subprocess for each input'multiprocessing'
– use multiprocessing library'reference'
– use a fake serial reference implementation
ncpus
– integer, maximal number of subprocesses to use at the same timetimeout
– number of seconds until each subprocess is killed (only supported by ‘fork’; zero means not at all)
Warning
If you use anything but
'fork'
above, then a whole new subprocess is spawned, so none of your local state (variables, certain functions, etc.) is available.EXAMPLES:
We create a simple decoration for a simple function. The number of cpus (or cores, or hardware threads) is automatically detected:
sage: @parallel ....: def f(n): return n*n sage: f(10) 100 sage: sorted(list(f([1,2,3]))) [(((1,), {}), 1), (((2,), {}), 4), (((3,), {}), 9)]
We use exactly two cpus:
sage: @parallel(2) ....: def f(n): return n*n
We create a decorator that uses three subprocesses, and times out individual processes after 10 seconds:
sage: @parallel(ncpus=3, timeout=10) ....: def fac(n): return factor(2^n-1) sage: for X, Y in sorted(list(fac([101,119,151,197,209]))): print((X,Y)) (((101,), {}), 7432339208719 * 341117531003194129) (((119,), {}), 127 * 239 * 20231 * 131071 * 62983048367 * 131105292137) (((151,), {}), 18121 * 55871 * 165799 * 2332951 * 7289088383388253664437433) (((197,), {}), 7487 * 26828803997912886929710867041891989490486893845712448833) (((209,), {}), 23 * 89 * 524287 * 94803416684681 * 1512348937147247 * 5346950541323960232319657) sage: @parallel('multiprocessing') ....: def f(N): return N^2 sage: v = list(f([1,2,4])); v.sort(); v [(((1,), {}), 1), (((2,), {}), 4), (((4,), {}), 16)] sage: @parallel('reference') ....: def f(N): return N^2 sage: v = list(f([1,2,4])); v.sort(); v [(((1,), {}), 1), (((2,), {}), 4), (((4,), {}), 16)]
For functions that take multiple arguments, enclose the arguments in tuples when calling the parallel function:
sage: @parallel ....: def f(a,b): return a*b sage: for X, Y in sorted(list(f([(2,3),(3,5),(5,7)]))): print((X, Y)) (((2, 3), {}), 6) (((3, 5), {}), 15) (((5, 7), {}), 35)
For functions that take a single tuple as an argument, enclose it in an additional tuple at call time, to distinguish it as the first argument, as opposed to a tuple of arguments:
sage: @parallel ....: def firstEntry(aTuple): return aTuple[0] sage: for X, Y in sorted(list(firstEntry([((1,2,3,4),),((5,6,7,8),)]))): print((X, Y)) ((((1, 2, 3, 4),), {}), 1) ((((5, 6, 7, 8),), {}), 5)
The parallel decorator also works with methods, classmethods, and staticmethods. Be sure to apply the parallel decorator after (“above”) either the
classmethod
orstaticmethod
decorators:sage: class Foo(): ....: @parallel(2) ....: def square(self, n): ....: return n*n ....: @parallel(2) ....: @classmethod ....: def square_classmethod(cls, n): ....: return n*n sage: a = Foo() sage: a.square(3) 9 sage: sorted(a.square([2,3])) [(((2,), {}), 4), (((3,), {}), 9)] sage: Foo.square_classmethod(3) 9 sage: sorted(Foo.square_classmethod([2,3])) [(((2,), {}), 4), (((3,), {}), 9)] sage: Foo.square_classmethod(3) 9
Warning
Currently, parallel methods do not work with the multiprocessing implementation.