rotifer.db.parallel.GeneNeighborhoodCursor#

class rotifer.db.parallel.GeneNeighborhoodCursor(column='pid', before=7, after=7, min_block_distance=0, strand=None, fttype='same', eukaryotes=False, exclude_type=['gene', 'mRNA', 'source'], autopid=True, codontable='Bacterial', progress=False, tries=3, batch_size=None, threads=10, *args, **kwargs)[source]#

Bases: BaseCursor

Parallel transport for gene neighborhood cursors.

This class implements the genome centered download strategy shared by rotifer.db.ncbi.ftp.GeneNeighborhoodCursor and rotifer.db.ncbi.mirror.GeneNeighborhoodCursor: protein identifiers are first resolved to genome assemblies through identical protein group (IPG) reports, the best assembly is chosen for each protein, and the selected genomes are then downloaded and scanned in parallel worker processes. Subclasses provide the open_genome method that actually reaches the data source.

Parameters:

columnstr, default ‘pid’

Name of the column to scan for matches to the accessions. See rotifer.genome.data.NeighborhoodDF.

beforeint, default 7

Keep at most this number of features, of the same type as the target, before each target.

afterint, default 7

Keep at most this number of features, of the same type as the target, after each target.

min_block_distanceint, default 0

Minimum distance between two consecutive blocks.

strandstr, optional

How to evaluate rows concerning the value of the strand column. Supported values:

None : ignore strand
same : same strand as the targets
+ : positive strand features and targets only
- : negative strand features and targets only

fttype{‘same’, ‘any’}, default ‘same’

How to process feature types of neighbors. With same, only features of the same type as the target are considered. With any, all features count when setting neighborhood boundaries.

eukaryotesbool, default False

Whether to process eukaryotic genomes. When False, errors mentioning eukaryotes are treated as unrecoverable.

exclude_typelist of str, default ['gene', 'mRNA', 'source']

Feature types to ignore.

autopidbool, default True

Automatically set protein identifiers.

codontablestr or int, default ‘Bacterial’

Codon table used when the data does not define one.

progressbool, default False

Whether to print a progress bar.

triesint, default 3

Number of attempts to download each batch.

batch_sizeint, optional

Number of genomes per worker batch. When not set, genomes are divided evenly among the worker processes.

threadsint, default 10

Number of simultaneous worker processes.

__getitem__(accessions, ipgs=None)[source]#

Find gene neighborhoods around one or more target proteins.

Parameters:

accessionsstr or iterable of str: NCBI protein identifiers.
ipgspandas.DataFrame, optional: Precomputed identical protein group reports. When not given, they are downloaded with rotifer.db.ncbi.entrez.IPGCursor.

Returns:

rotifer.genome.data.NeighborhoodDF: The neighborhoods found. Empty when nothing could be retrieved.

fetcher(accession)[source]#

Open the annotation stream for one genome assembly.

When the cursor is configured to skip eukaryotes, the genome assembly report is inspected first and eukaryotic genomes are rejected.

Parameters:

accessionstr: Genome assembly accession.

Returns:

file-like: The open genome data stream.

Raises:

ValueError: If the genome is eukaryotic and eukaryotes is False, or if the genome files cannot be accessed.

fetchone(accessions, ipgs=None)[source]#

Iterate over gene neighborhoods as downloads complete.

Results are yielded in completion order, not input order.

Parameters:

accessionslist of str: NCBI protein identifiers.
ipgspandas.DataFrame, optional: Precomputed identical protein group reports, used to avoid downloading IPGs again.

Yields:

rotifer.genome.data.NeighborhoodDF: One dataframe per retrieved gene neighborhood.

Examples

Reuse previously downloaded IPGs:

>>> from rotifer.db.ncbi import entrez
>>> from rotifer.db.ncbi import ftp
>>> ic = entrez.IPGCursor(batch_size=1)
>>> gnc = ftp.GeneNeighborhoodCursor(progress=True)
>>> i = ic.fetchall(['WP_063732599.1'])
>>> n = gnc.fetchall(['WP_063732599.1'], ipgs=i)

parser(stream, accession, proteins)[source]#

Extract gene neighborhoods from a genome stream.

The genome annotation is converted to a feature table and scanned for the target proteins; only the regions around the targets are kept.

Parameters:

streamfile-like: Open genome data stream returned by fetcher().
accessionstr: Genome assembly accession.
proteinsdict: Mapping of each target protein accession to its identical protein group representative.

Returns:

rotifer.genome.data.NeighborhoodDF: The neighborhoods found in this genome, with a replaced column mapping each protein to the query it represents.

splitter(accessions, ipgs)[source]#

Group targets and their IPG rows into per-genome batches.

Parameters:

accessionsset of str: Target protein accessions.
ipgspandas.DataFrame: Identical protein group reports restricted to the selected genomes.

Returns:

list of list of tuple: Batches of (proteins, ipg_rows) pairs, one pair per genome, at most batch_size pairs per batch.

worker(chunk)[source]#

Process one batch of genomes in a worker process.

Parameters:

chunklist of tuple: Batch entries produced by splitter(). Each entry holds a set of target protein accessions and the IPG rows of one genome.

Returns:

dict: A dictionary with two keys: result, a list of neighborhood dataframes (one per gene block), and missing, the registry of entries that could not be retrieved.