Dictionary populated on demand

I work with an embedded target that contains a list of commands and registers that may change from one device to another. So I need to build my commands list from metadata received from that particular target.

Unfortunately, the communication with the target is slow and the amount of commands is high. So I cannot build the full command's list at the beginning, but only when somebody requests it.

Of course, the list of commands should not be modifiable (by mistake), by the user which I assume is dumb. So I would like to prevent the instance to be easily modifiable in IPython.

For example:

device = Device()
device.commands.               # the user see all the available commands
device.commands.execute = 'oops'    # I want to prevent this to happen
device.commands['execute'] = 'oops' # Neither this.

So I wrote this design pattern:

```
import time

# The two following functions are the ones I use to communicate
# with my target.
def get_item_names():
return ['item%d' % i for i in range(100000)]

def fetch_item(item): # Time consuming function
time.sleep(0.1)
return item

# The dictionary that is populated on demand.
class Foo(dict):
__locked = False
def __init__(self):
# Populate elements with None elements.
# This is required to be able to see the available commands
# in IPython with the auto-completion turned on.
for item in get_item_names():
default = None
self[item] = default

# This is a very ugly workaround. With cProfiler, I can see
# the addition of __setattr__ is very costy.
self.__locked = True

# The item is fetched only on demand
def __getitem__(self, item):
if item in self and super(Foo, self).__getitem__(item) == None:
fetched = fetch_item(item)
super(Foo, self).__setitem__(item, fetched)
return fetched
else:
return super(Foo, self).__getitem__(item)

@pro

__missing__

I would have started to propose a simpler design using the __missing__ dunder so that, each time a user tries to access a command that has not been populated, the __missing__ method would be called and you could fetch the informations:

def Foo(dict):
    def __missing__(self, key):
        self[key] = item = fetch_item(key)
        return item

And voilà. I wouldn't try to enforce barriers to avoid users behaviour. After all, we're all consenting adults here.
Playing it nice with the interactive shell

This behaviour, however let the user clueless about all the possible commands available at the tap of the TAB key. They are still able to call get_item_names if need be. But you could automatically build them using some metaclass logic.

The class you want in the end could be:

def Foo:
    _ = None

    @property
    def (self):
        if self._ is None:
            self._ = fetch_item()
        return self._

    def __getitem__(self, key):
        return getattr(self, key)

with every ` being returned by get_item_names. Read-only of the attributes are ensured by the property.

So you need to define your base class being:

class Foo(metaclass=CommandBuilder):
    def __getitem__(self, key):
        return getattr(self, key)

And build your metaclass like

class CommandBuilder(type):
    def __new__(cls, name, bases, dct):
        for command_name in get_item_names():
            _command_name = '_{}'.format(command_name)
            dct[_command_name] = None

            def getter(self):
                command_value = getattr(self, _command_name)
                if command_value is None:
                    command_value = fetch_item(command_name)
                    setattr(self, _command_name, command_value)
                return command_value

            dct[command_name] = property(getter)

        return super(CommandBuilder, cls).__new__(cls, name, bases, dct)

If you trully need to use a subclass of dict here, it wouldn't be much more difficult:

def Foo(dict, metaclass=CommandBuilder):
    def __missing__(self, key):
        item = getattr(self, key)  # will call the property
        super(Foo, self).__setitem__(key, item)
        return item

    def __setitem__(self, key, value):
         pass  # or raise an exception of your own

    def __delitem__(self, key):
        pass  # or raise an exception of your own

Immutability of the attributes is still guarantied by property`.

StackExchange Code Review Q#144117, answer score: 6