Fixing pickle for nested classes

As of Python 2.7, names for nested classes are set by Python in a way which is incompatible with the pickling of such classes (pickling by name):

Sage

sage: class A:
....:     class B:
....:         pass
sage: A.B.__name__
'B'

Python

>>> from sage.all import *
>>> class A:
...     class B:
...         pass
>>> A.B.__name__
'B'

instead of more natural 'A.B'. Furthermore upon pickling and unpickling a class with name 'A.B' in a module mod, the standard cPickle module searches for 'A.B' in mod.__dict__ instead of looking up 'A' and then 'B' in the result. See: https://groups.google.com/forum/#!topic/sage-devel/bHBV9KWAt64

This module provides two utilities to workaround this issue:

• nested_pickle() “fixes” recursively the name of the subclasses of a class and inserts their fullname 'A.B' in mod.__dict__

• NestedClassMetaclass is a metaclass ensuring that nested_pickle() is called on a class upon creation.

See also sage.misc.test_nested_class.

Note

In Python 3, nested classes, like any class for that matter, have __qualname__ and the standard pickle module uses it for pickling and unpickling. Thus the pickle module searches for 'A.B' first by looking up 'A' in mod, and then 'B' in the result. So there is no pickling problem for nested classes in Python 3, and the two utilities are not really necessary. However, NestedClassMetaclass is used widely in Sage and affects behaviors of Sage objects in other respects than in pickling and unpickling. Hence we keep NestedClassMetaclass even with Python 3, for now. This module will be removed when we eventually drop support for Python 2.

EXAMPLES:

Sage

sage: from sage.misc.nested_class import A1, nested_pickle

sage: A1.A2.A3.__name__
'A3'
sage: A1.A2.A3
<class 'sage.misc.nested_class.A1.A2.A3'>

sage: nested_pickle(A1)
<class 'sage.misc.nested_class.A1'>

sage: A1.A2
<class 'sage.misc.nested_class.A1.A2'>

sage: A1.A2.A3
<class 'sage.misc.nested_class.A1.A2.A3'>
sage: A1.A2.A3.__name__
'A1.A2.A3'

sage: sage.misc.nested_class.__dict__['A1.A2'] is A1.A2
True
sage: sage.misc.nested_class.__dict__['A1.A2.A3'] is A1.A2.A3
True

Python

>>> from sage.all import *
>>> from sage.misc.nested_class import A1, nested_pickle

>>> A1.A2.A3.__name__
'A3'
>>> A1.A2.A3
<class 'sage.misc.nested_class.A1.A2.A3'>

>>> nested_pickle(A1)
<class 'sage.misc.nested_class.A1'>

>>> A1.A2
<class 'sage.misc.nested_class.A1.A2'>

>>> A1.A2.A3
<class 'sage.misc.nested_class.A1.A2.A3'>
>>> A1.A2.A3.__name__
'A1.A2.A3'

>>> sage.misc.nested_class.__dict__['A1.A2'] is A1.A2
True
>>> sage.misc.nested_class.__dict__['A1.A2.A3'] is A1.A2.A3
True

All of this is not perfect. In the following scenario:

Sage

sage: class A1:
....:     class A2:
....:         pass
sage: class B1:
....:     A2 = A1.A2

sage: nested_pickle(A1)
<class '__main__.A1'>
sage: nested_pickle(B1)
<class '__main__.B1'>
sage: A1.A2
<class '__main__.A1.A2'>
sage: B1.A2
<class '__main__.A1.A2'>

Python

>>> from sage.all import *
>>> class A1:
...     class A2:
...         pass
>>> class B1:
...     A2 = A1.A2

>>> nested_pickle(A1)
<class '__main__.A1'>
>>> nested_pickle(B1)
<class '__main__.B1'>
>>> A1.A2
<class '__main__.A1.A2'>
>>> B1.A2
<class '__main__.A1.A2'>

The name for 'A1.A2' could potentially be set to 'B1.A2'. But that will work anyway.

class sage.misc.nested_class.MainClass

Bases: object

A simple class to test nested_pickle.

EXAMPLES:

Sage

sage: from sage.misc.nested_class import *
sage: loads(dumps(MainClass()))
<sage.misc.nested_class.MainClass object at 0x...>

Python

>>> from sage.all import *
>>> from sage.misc.nested_class import *
>>> loads(dumps(MainClass()))
<sage.misc.nested_class.MainClass object at 0x...>

class NestedClass

Bases: object

EXAMPLES:

Sage

sage: from sage.misc.nested_class import *
sage: loads(dumps(MainClass.NestedClass()))
<sage.misc.nested_class.MainClass.NestedClass object at 0x...>

Python

>>> from sage.all import *
>>> from sage.misc.nested_class import *
>>> loads(dumps(MainClass.NestedClass()))
<sage.misc.nested_class.MainClass.NestedClass object at 0x...>

class NestedSubClass

Bases: object

EXAMPLES:

Sage

sage: from sage.misc.nested_class import *
sage: loads(dumps(MainClass.NestedClass.NestedSubClass()))
<sage.misc.nested_class.MainClass.NestedClass.NestedSubClass object at 0x...>
sage: getattr(sage.misc.nested_class, 'MainClass.NestedClass.NestedSubClass')
<class 'sage.misc.nested_class.MainClass.NestedClass.NestedSubClass'>
sage: MainClass.NestedClass.NestedSubClass.__name__
'MainClass.NestedClass.NestedSubClass'

Python

>>> from sage.all import *
>>> from sage.misc.nested_class import *
>>> loads(dumps(MainClass.NestedClass.NestedSubClass()))
<sage.misc.nested_class.MainClass.NestedClass.NestedSubClass object at 0x...>
>>> getattr(sage.misc.nested_class, 'MainClass.NestedClass.NestedSubClass')
<class 'sage.misc.nested_class.MainClass.NestedClass.NestedSubClass'>
>>> MainClass.NestedClass.NestedSubClass.__name__
'MainClass.NestedClass.NestedSubClass'

dummy(x, r=(1, 2, 3.4), *args, **kwds)

A dummy method to demonstrate the embedding of method signature for nested classes.

class sage.misc.nested_class.NestedClassMetaclass

Bases: type

A metaclass for nested pickling.

Check that one can use a metaclass to ensure nested_pickle is called on any derived subclass:

Sage

sage: from sage.misc.nested_class import NestedClassMetaclass
sage: class ASuperClass(object, metaclass=NestedClassMetaclass):
....:     pass
sage: class A3(ASuperClass):
....:     class B():
....:         pass
sage: A3.B.__name__
'A3.B'
sage: getattr(sys.modules['__main__'], 'A3.B', 'Not found')
<class '__main__.A3.B'>

Python

>>> from sage.all import *
>>> from sage.misc.nested_class import NestedClassMetaclass
>>> class ASuperClass(object, metaclass=NestedClassMetaclass):
...     pass
>>> class A3(ASuperClass):
...     class B():
...         pass
>>> A3.B.__name__
'A3.B'
>>> getattr(sys.modules['__main__'], 'A3.B', 'Not found')
<class '__main__.A3.B'>

sage.misc.nested_class.modify_for_nested_pickle(cls, name_prefix, module, first_run=True)

Modify the subclasses of the given class to be picklable, by giving them a mangled name and putting the mangled name in the module namespace.

INPUT:

• cls – the class to modify

• name_prefix – the prefix to prepend to the class name

• module – the module object to modify with the mangled name

• first_run – boolean (default: True); whether or not this function is run for the first time on cls

NOTE:

This function would usually not be directly called. It is internally used in NestedClassMetaclass.

EXAMPLES:

Sage

sage: from sage.misc.nested_class import *
sage: class A():
....:     class B():
....:         pass
sage: module = sys.modules['__main__']
sage: A.B.__name__
'B'
sage: getattr(module, 'A.B', 'Not found')
'Not found'
sage: modify_for_nested_pickle(A, 'A', module)
sage: A.B.__name__
'A.B'
sage: getattr(module, 'A.B', 'Not found')
<class '__main__.A.B'>

Python

>>> from sage.all import *
>>> from sage.misc.nested_class import *
>>> class A():
...     class B():
...         pass
>>> module = sys.modules['__main__']
>>> A.B.__name__
'B'
>>> getattr(module, 'A.B', 'Not found')
'Not found'
>>> modify_for_nested_pickle(A, 'A', module)
>>> A.B.__name__
'A.B'
>>> getattr(module, 'A.B', 'Not found')
<class '__main__.A.B'>

Here we demonstrate the effect of the first_run argument:

Sage

sage: modify_for_nested_pickle(A, 'X', module)
sage: A.B.__name__  # nothing changed
'A.B'
sage: modify_for_nested_pickle(A, 'X', module, first_run=False)
sage: A.B.__name__
'X.A.B'

Python

>>> from sage.all import *
>>> modify_for_nested_pickle(A, 'X', module)
>>> A.B.__name__  # nothing changed
'A.B'
>>> modify_for_nested_pickle(A, 'X', module, first_run=False)
>>> A.B.__name__
'X.A.B'

Note that the class is now found in the module under both its old and its new name:

Sage

sage: getattr(module, 'A.B', 'Not found')
<class '__main__.A.B'>
sage: getattr(module, 'X.A.B', 'Not found')
<class '__main__.A.B'>

Python

>>> from sage.all import *
>>> getattr(module, 'A.B', 'Not found')
<class '__main__.A.B'>
>>> getattr(module, 'X.A.B', 'Not found')
<class '__main__.A.B'>

sage.misc.nested_class.nested_pickle(cls)

This decorator takes a class that potentially contains nested classes. For each such nested class, its name is modified to a new illegal identifier, and that name is set in the module. For example, if you have:

Sage

sage: from sage.misc.nested_class import nested_pickle
sage: module = sys.modules['__main__']
sage: class A():
....:     class B:
....:         pass
sage: nested_pickle(A)
<class '__main__.A'>

Python

>>> from sage.all import *
>>> from sage.misc.nested_class import nested_pickle
>>> module = sys.modules['__main__']
>>> class A():
...     class B:
...         pass
>>> nested_pickle(A)
<class '__main__.A'>

then the name of class 'B' will be modified to 'A.B', and the 'A.B' attribute of the module will be set to class 'B':

Sage

sage: A.B.__name__
'A.B'
sage: getattr(module, 'A.B', 'Not found')
<class '__main__.A.B'>

Python

>>> from sage.all import *
>>> A.B.__name__
'A.B'
>>> getattr(module, 'A.B', 'Not found')
<class '__main__.A.B'>

In Python 2.6, decorators work with classes; then @nested_pickle should work as a decorator:

Sage

sage: @nested_pickle    # todo: not implemented
....: class A2():
....:     class B:
....:         pass
sage: A2.B.__name__    # todo: not implemented
'A2.B'
sage: getattr(module, 'A2.B', 'Not found')    # todo: not implemented
<class __main__.A2.B at ...>

Python

>>> from sage.all import *
>>> @nested_pickle    # todo: not implemented
... class A2():
...     class B:
...         pass
>>> A2.B.__name__    # todo: not implemented
'A2.B'
>>> getattr(module, 'A2.B', 'Not found')    # todo: not implemented
<class __main__.A2.B at ...>

EXAMPLES:

Sage

sage: from sage.misc.nested_class import *
sage: loads(dumps(MainClass.NestedClass())) # indirect doctest
<sage.misc.nested_class.MainClass.NestedClass object at 0x...>

Python

>>> from sage.all import *
>>> from sage.misc.nested_class import *
>>> loads(dumps(MainClass.NestedClass())) # indirect doctest
<sage.misc.nested_class.MainClass.NestedClass object at 0x...>