Python copy-on-write behavior
-
01-07-2021 - |
Pregunta
I'm working on a problem where I'm instantiating many instances of an object. Most of the time the instantiated objects are identical. To reduce memory overhead, I'd like to have all the identical objects point to the same address. When I modify the object, though, I'd like a new instance to be created--essentially copy-on-write behavior. What is the best way to achieve this in Python?
The Flyweight Pattern comes close. An example (from http://codesnipers.com/?q=python-flyweights):
import weakref
class Card(object):
_CardPool = weakref.WeakValueDictionary()
def __new__(cls, value, suit):
obj = Card._CardPool.get(value + suit, None)
if not obj:
obj = object.__new__(cls)
Card._CardPool[value + suit] = obj
obj.value, obj.suit = value, suit
return obj
This behaves as follows:
>>> c1 = Card('10', 'd')
>>> c2 = Card('10', 'd')
>>> id(c1) == id(c2)
True
>>> c2.suit = 's'
>>> c1.suit
's'
>>> id(c1) == id(c2)
True
The desired behavior would be:
>>> c1 = Card('10', 'd')
>>> c2 = Card('10', 'd')
>>> id(c1) == id(c2)
True
>>> c2.suit = 's'
>>> c1.suit
'd'
>>> id(c1) == id(c2)
False
Update: I came across the Flyweight Pattern and it seemed to almost fit the bill. However, I'm open to other approaches.
Solución
Do you need id(c1)==id(c2)
to be identical, or is that just a demonstration, where the real objective is avoiding creating duplicated objects?
One approach would be to have each object be distinct, but hold an internal reference to the 'real' object like you have above. Then, on any __setattr__
call, change the internal reference.
I've never done __setattr__
stuff before, but I think it would look like this:
class MyObj:
def __init__(self, value, suit):
self._internal = Card(value, suit)
def __setattr__(self, name, new_value):
if name == 'suit':
self._internal = Card(value, new_value)
else:
self._internal = Card(new_value, suit)
And similarly, expose the attributes through getattr
.
You'd still have lots of duplicated objects, but only one copy of the 'real' backing object behind them. So this would help if each object is massive, and wouldn't help if they are lightweight, but you have millions of them.
Otros consejos
Impossible.
id(c1) == id(c2)
says that c1
and c2
are references to the exact same object. So
c2.suit = 's'
is exactly the same as saying c1.suit = 's'
.
Python has no way of distinguishing the two (unless you allow introspection of prior call frames, which leads to a dirty hack.)
Since the two assignments are identical, there is no way for Python to know that c2.suit = 's'
should cause the name c2
to reference a different object.
To give you an idea of what the dirty hack would look like,
import traceback
import re
import sys
import weakref
class Card(object):
_CardPool = weakref.WeakValueDictionary()
def __new__(cls, value, suit):
obj = Card._CardPool.get(value + suit, None)
if not obj:
obj = object.__new__(cls)
Card._CardPool[value + suit] = obj
obj._value, obj._suit = value, suit
return obj
@property
def suit(self):
return self._suit
@suit.setter
def suit(self, suit):
filename,line_number,function_name,text=traceback.extract_stack()[-2]
name = text[:text.find('.suit')]
setattr(sys.modules['__main__'], name, Card(self._value, suit))
c1 = Card('10', 'd')
c2 = Card('10', 'd')
assert id(c1) == id(c2)
c2.suit = 's'
print(c1.suit)
# 'd'
assert id(c1) != id(c2)
This use of traceback only works with those implementations of Python that uses frames, such as CPython, but not Jython or IronPython.
Another problem is that
name = text[:text.find('.suit')]
is extremely fragile, and would screw up, for example, if the assignment were to look like
if True: c2.suit = 's'
or
c2.suit = (
's')
or
setattr(c2, 'suit', 's')
Yet another problem is that it assumes the name c2
is global. It could just as easily be a local variable (say, inside a function), or an attribute (obj.c2.suit = 's'
).
I do not know a way to address all the ways the assignment could be made.
In any of these cases, the dirty hack would fail.
Conclusion: Don't use it. :)
This is impossible in your current form. A name (c1
and c2
in your example) is a reference, and you can not simply change the reference by using __setattr__
, not to mention all other references to the same object.
The only way this would be possible is something like this:
c1 = c1.changesuit("s")
Where c1.changesuit
returns a reference to the (newly created) object. But this only works if each object is referenced by only one name. Alternatively you might be able to do some magic with locals()
and stuff like that, but please - don't.