Tokenization Module

hard_tokenization_main

def hard_tokenization_main(src_folder: str, dst_folder: str, universe_file: str, fraction: float = 1e-09, file_list: List[str] = None, num_workers: int = 10, bedtools_path: str = 'bedtools') -> int

Tokenizes raw BED files in parallel.

This is the main function for hard tokenization. It uses multiple processes to speed up the tokenization process.

Args: src_folder (str): The folder where raw BED files reside. dst_folder (str): The folder to store tokenized BED files. universe_file (str): The path to a universe file. fraction (float, optional): A parameter for bedtools.intersect. Defaults to 1e-9. file_list (list[str], optional): A list of files (just names not full paths) that need to be tokenized. Defaults to None and uses all BED files in src_folder. num_workers (int, optional): Number of processes used. Defaults to 10. bedtools_path (str, optional): The path to a bedtools binary. Defaults to "bedtools".

Raises: Exception: No bedtools executable found

Returns: int: 0 when the dst_folder folder has incomplete list of tokenized BED files which should be removed first; 1 when the dst_folder folder has the complete tokenized BED files or the tokenization process succeeds.

BEDToolsTokenizer

class BEDToolsTokenizer(bedtools_path: str, universe_path: str = None)

Bases: FileTokenizer

A tokenizer that uses bedtools to tokenize BED files

Initialize a BEDToolsTokenizer

Args: bedtools_path (str): Path to a bedtools binary. universe_path (str): Path to a universe BED file.

Attributes

• bedtools_path
• universe_path

Methods

tokenize

def tokenize(input_globs: list[str], universe_path: str = None) -> RegionSet

Tokenize a RegionSet using bedtools

bedtools_tokenization

def bedtools_tokenization(f: str, bedtools_path: str, data_folder: str, target_folder: str, universe: str, fraction: float) -> None

Uses bedtools to tokenize a raw BED file.

Args: f (str): File name. bedtools_path (str): Path to a bedtools binary. data_folder (str): The folder where raw BED files reside. target_folder (str): The folder that stores tokenized BED files. universe (str): Path to a universe file. fraction (float): A parameter for bedtools.intersect.

generate_tokens

def generate_tokens(raw_data_folder: str, token_folder: str, universe: str, file_list: str, bedtools: str, fraction: float) -> None

Tokenizes raw BED files specified by file_list.

Tokenizes raw BED files specified by file_list. First, checks existing files in token_folder. If token_folder has all the tokenized BED files, then does nothing. Otherwise, tokenizes raw BED files that are in file_list but not in token_folder.

Args: raw_data_folder (str): The folder where raw BED files reside. token_folder (str): The folder to store tokenized BED files. universe (str): The path to a universe file. file_list (str): The path to a file which contains selected BED files per row. bedtools (str): The path to a bedtools binary. fraction (float): A parameter for bedtools.intersect.

main

def main(args: argparse.Namespace)

Generates tokenized BED files.

Calls generate_tokens using the arguments in args. Prints status information.

Args: args (argparse.Namespace): See the definition of the ArgumentParser.

Attributes

• parser = argparse.ArgumentParser()
• args = parser.parse_args()

get_file_rows

def get_file_rows(file_path: str) -> int

Counts how many files are included.

Args: file_path (str): The path to a file.

Returns: int: The number of rows in the file.

split_file

def split_file(file_path: str, dest_folder: str, num_parts: int) -> None

Splits a list of files into num_parts non-overlapping batches.

This is a helper function for tokenization in parallel. The dest_folder must be empty.

Args: file_path (str): The path to a file with BED file names per row. dest_folder (str): The folder to store split file lists. num_parts (int): Number of batches to split.

Attributes

• parser = argparse.ArgumentParser()
• args = parser.parse_args()

Timer

class Timer()

Records the running time.

Uses Timer.s() or Timer() to record the start time. Then, calls Timer.t() to get the elapsed time in seconds.

Initializes a Timer object and starts the timer.

Attributes

• v

Methods

s

def s()

Restarts the timer.

t

def t()

Gives the elapsed time.

Returns: float: The elapsed time in seconds.

time_str

def time_str(t: float) -> str

Converts time in float to a readable format.

Converts time in float to hours, minutes, or seconds based on the value of t.

Args: t (float): Time in seconds.

Returns: str: Time in readable time.

tokenize_anndata

def tokenize_anndata(adata: sc.AnnData, tokenizer: Tokenizer)

Tokenize an AnnData object. This is more involved, so it gets its own function. Args: adata (sc.AnnData): The AnnData object to tokenize. tokenizer (Tokenizer): The tokenizer to use.