Customize pickle behavior for backwards compatibility

Python's copy_reg module allows to register custom reducers and constructors. Is it correct that I can only customize the unpickle behavior of objects that I serialized after I registered the custom serializer/unserializer via copy_reg.pickle ?

Example:

import pickle, copy_reg

class C(object):
    pass

legacy_c_ser = pickle.dumps(C())

def reduce_C(obj):
    print('reduce_C called')
    tpl = obj.__reduce__()
    tpl = (load_C, ) + tpl[1:]
    return tpl

def load_C(*tpl):
    print('load_C called')
    return C()

copy_reg.constructor(load_C)
copy_reg.pickle(C, reduce_C, load_C)

new_c_ser = pickle.dumps(C())

# load_C is called
pickle.loads(new_c_ser)

# load_C is not called
pickle.loads(legacy_c_ser)

It seems that copy_reg works by simply replacing the constructor function in the pickle format:

>>> print(legacy_c_ser)
'ccopy_regn_reconstructornp0n(c__main__nCnp1[...]'
>>> print(new_c_ser)
'c__main__nload_Cnp0[...]'

Is writing my own pickle.Unpickler class the only way to customize the unpickle behavior of legacy pickle files? I dont want to do that since I rather use cPickle than pickle because of efficiency.

My problem is that I have pickled objects from a 3rd party library and the pickle format changed when I upgraded the library.

---

Pickle is supposed to be backward compatible across python releases (let's ignore python 2.x versus 3.x, for now). So when you say the pickling format changed, you mean the way that this 3rd party library registers their classes (or other objects) has changed, correct?

If that's the case… to make this work, you'd need to do some trickery. First you get the source for the old class definition, and when you grab the raw pickle you need to change the reference from the existing class to match the path to the code for the old version of the class. This should be in clear text (even in HIGHEST_PROTOCOL ), so it shouldn't be an issue to grab and edit that part of the pickle string. You then would be able to unpickle the old objects, but they'd be pointing to the old class definition. A "format converter" would be needed to convert the old objects you'd be unpicking to the new class instance objects -- basically create new class instances that take the relevant state from the old class instances.

链接地址: http://www.djcxy.com/p/64828.html

上一篇: 酸洗和取消动态生成的类

下一篇: 自定义pickle行为以实现向后兼容