rotifer.db.delegator.DelegatorCursor

class rotifer.db.delegator.DelegatorCursor(readers=[], writers=[], progress=True, tries=None, batch_size=None, threads=None, *args, **kwargs)

Bases: BaseCursor

Base class for cursors that dispatch queries to backend cursors.

Subclasses are expected to live in a module that defines a config dictionary with readers and writers entries mapping backend names to module paths. For each requested backend, the delegator imports the module and instantiates the class named like the delegator subclass itself.

Parameters:

readerslist of str, default []

Names of the backend modules used to retrieve data, in order of preference.

writerslist of str, default []

Names of the backend modules used to store retrieved data.

progressbool, default True

Whether to print a progress bar.

triesint, optional

Number of attempts to download each batch, shared with every backend cursor that accepts it.

batch_sizeint, optional

Number of accessions per batch, shared with every backend cursor that accepts it.

threadsint, optional

Number of simultaneous threads, shared with every backend cursor that accepts it.

Attributes:

cursorsdict

Backend cursor instances, keyed by backend name.

Notes

Attributes listed in the subclass’s _shared_attributes are propagated to every backend cursor whenever they are set on the delegator.

absorb_missing(cursor)

Take note of what a backend could not retrieve.

Called once a backend has been exhausted rather than for each result it yields, because a backend that finds nothing yields nothing and its reasons would otherwise never be recorded.

The two flags aggregate differently, because they answer different questions. retry becomes the disjunction across backends: on a delegator it means that at least one of them might still succeed on another attempt, so an unreachable server leaves the entry worth retrying even after a later backend answers “not found” for good. final is kept by update_missing(), which never downgrades it.

Parameters:

cursorrotifer.db.core.BaseCursor

The backend to copy from.

backend_arguments(name, arguments)

Adjust the arguments one backend is built with.

Shared attributes are shared because they mean the same thing everywhere, and most do. Where one does not – a path naming a directory for one backend and a file for another – this is where a delegator says so, rather than the backends guessing from what they were handed.

Parameters:

namestr

Which backend is being built.

argumentsdict

The shared attributes, as they would be passed.

Returns:

dict

What to pass instead. The same dictionary by default.

content_of(cursor)

Ask a backend to identify its data, tolerating a refusal.

Working out what may be skipped is an optimisation, so a backend that raises while answering must cost nothing worse than being consulted.

Parameters:

cursorrotifer.db.core.BaseCursor

The backend to ask.

Returns:

hashable or None

None when the backend cannot or will not say.

redundant(cursor, consulted)

Find whether a backend would repeat what another already did.

Backends are asked in order, cheapest first, and a later one is usually there to cover what the earlier ones do not have. When two of them serve the same data, though, the later one has nothing left to add, and skipping it saves whatever it would have cost – which for a backend that scans a file is the whole point.

This never suppresses a backend that might know something the others do not: only an identical content_id() counts, and a backend unable to name its data returns None and is always consulted.

Parameters:

cursorrotifer.db.core.BaseCursor

The backend about to be asked.

consulteddict

Content identifiers already seen, mapped to the name of the backend that offered them.

Returns:

str or None

Name of the backend that already served this data, or None when this backend should be consulted.

reset_cursors()

Instantiate or reinstantiate every backend cursor.

The backend class is looked up in each backend module using the delegator’s own class name. Attributes listed in _shared_attributes that are not None are passed to the backend constructors. Backends whose module does not define a matching class are skipped with an error message.