# Tokenization Module

<a id="geniml.tokenization.main"></a>

<a id="geniml.tokenization.main.hard_tokenization_main"></a>

## `hard_tokenization_main`

```python
def hard_tokenization_main(src_folder: str, dst_folder: str, universe_file: str, fraction: float = 1e-09, file_list: List[str] = None, num_workers: int = 10, bedtools_path: str = 'bedtools') -> int
```

Tokenizes raw BED files in parallel.

This is the main function for hard tokenization. It uses multiple processes to
speed up the tokenization process.

Args:
    src_folder (str): The folder where raw BED files reside.
    dst_folder (str): The folder to store tokenized BED files.
    universe_file (str): The path to a universe file.
    fraction (float, optional): A parameter for bedtools.intersect.
        Defaults to 1e-9.
    file_list (list[str], optional): A list of files (just names not full
        paths) that need to be tokenized. Defaults to None and uses all BED
        files in src_folder.
    num_workers (int, optional): Number of processes used. Defaults to 10.
    bedtools_path (str, optional): The path to a bedtools binary. Defaults
        to "bedtools".

Raises:
    Exception: No bedtools executable found

Returns:
    int: 0 when the dst_folder folder has incomplete list of tokenized BED
        files which should be removed first; 1 when the dst_folder folder
        has the complete tokenized BED files or the tokenization process
        succeeds.

<a id="geniml.tokenization.bedtools_tokenizer"></a>

<a id="geniml.tokenization.bedtools_tokenizer.BEDToolsTokenizer"></a>

## `BEDToolsTokenizer`

```python
class BEDToolsTokenizer(bedtools_path: str, universe_path: str = None)
```

Bases: `FileTokenizer`

A tokenizer that uses bedtools to tokenize BED files

Initialize a BEDToolsTokenizer

Args:
    bedtools_path (str): Path to a bedtools binary.
    universe_path (str): Path to a universe BED file.

### Attributes

- <a id="geniml.tokenization.bedtools_tokenizer.BEDToolsTokenizer.bedtools_path"></a>**`bedtools_path`**
- <a id="geniml.tokenization.bedtools_tokenizer.BEDToolsTokenizer.universe_path"></a>**`universe_path`**

### Methods

<a id="geniml.tokenization.bedtools_tokenizer.BEDToolsTokenizer.tokenize"></a>

#### `tokenize`

```python
def tokenize(input_globs: list[str], universe_path: str = None) -> RegionSet
```

Tokenize a RegionSet using bedtools

<a id="geniml.tokenization.hard_tokenization_batch"></a>

<a id="geniml.tokenization.hard_tokenization_batch.bedtools_tokenization"></a>

## `bedtools_tokenization`

```python
def bedtools_tokenization(f: str, bedtools_path: str, data_folder: str, target_folder: str, universe: str, fraction: float) -> None
```

Uses bedtools to tokenize a raw BED file.

Args:
    f (str): File name.
    bedtools_path (str): Path to a bedtools binary.
    data_folder (str): The folder where raw BED files reside.
    target_folder (str): The folder that stores tokenized BED files.
    universe (str): Path to a universe file.
    fraction (float): A parameter for bedtools.intersect.

<a id="geniml.tokenization.hard_tokenization_batch.generate_tokens"></a>

## `generate_tokens`

```python
def generate_tokens(raw_data_folder: str, token_folder: str, universe: str, file_list: str, bedtools: str, fraction: float) -> None
```

Tokenizes raw BED files specified by file_list.

Tokenizes raw BED files specified by file_list. First, checks existing files
in token_folder. If token_folder has all the tokenized BED files, then does
nothing. Otherwise, tokenizes raw BED files that are in file_list but not
in token_folder.

Args:
    raw_data_folder (str): The folder where raw BED files reside.
    token_folder (str): The folder to store tokenized BED files.
    universe (str): The path to a universe file.
    file_list (str): The path to a file which contains selected BED files per row.
    bedtools (str): The path to a bedtools binary.
    fraction (float): A parameter for bedtools.intersect.

<a id="geniml.tokenization.hard_tokenization_batch.main"></a>

## `main`

```python
def main(args: argparse.Namespace)
```

Generates tokenized BED files.

Calls generate_tokens using the arguments in args. Prints status
information.

Args:
    args (argparse.Namespace): See the definition of the ArgumentParser.

## Attributes

- <a id="geniml.tokenization.hard_tokenization_batch.parser"></a>**`parser`** = `argparse.ArgumentParser()`
- <a id="geniml.tokenization.hard_tokenization_batch.args"></a>**`args`** = `parser.parse_args()`

<a id="geniml.tokenization.split_file"></a>

<a id="geniml.tokenization.split_file.get_file_rows"></a>

## `get_file_rows`

```python
def get_file_rows(file_path: str) -> int
```

Counts how many files are included.

Args:
    file_path (str): The path to a file.

Returns:
    int: The number of rows in the file.

<a id="geniml.tokenization.split_file.split_file"></a>

## `split_file`

```python
def split_file(file_path: str, dest_folder: str, num_parts: int) -> None
```

Splits a list of files into num_parts non-overlapping batches.

This is a helper function for tokenization in parallel. The dest_folder
must be empty.

Args:
    file_path (str): The path to a file with BED file names per row.
    dest_folder (str): The folder to store split file lists.
    num_parts (int): Number of batches to split.

## Attributes

- <a id="geniml.tokenization.split_file.parser"></a>**`parser`** = `argparse.ArgumentParser()`
- <a id="geniml.tokenization.split_file.args"></a>**`args`** = `parser.parse_args()`

<a id="geniml.tokenization.utils"></a>

<a id="geniml.tokenization.utils.Timer"></a>

## `Timer`

```python
class Timer()
```

Records the running time.

Uses Timer.s() or Timer() to record the start time. Then, calls Timer.t() to get the
elapsed time in seconds.

Initializes a Timer object and starts the timer.

### Attributes

- <a id="geniml.tokenization.utils.Timer.v"></a>**`v`**

### Methods

<a id="geniml.tokenization.utils.Timer.s"></a>

#### `s`

```python
def s()
```

Restarts the timer.

<a id="geniml.tokenization.utils.Timer.t"></a>

#### `t`

```python
def t()
```

Gives the elapsed time.

Returns:
    float: The elapsed time in seconds.

<a id="geniml.tokenization.utils.time_str"></a>

## `time_str`

```python
def time_str(t: float) -> str
```

Converts time in float to a readable format.

Converts time in float to hours, minutes, or seconds based on the value of
t.

Args:
    t (float): Time in seconds.

Returns:
    str: Time in readable time.

<a id="geniml.tokenization.utils.tokenize_anndata"></a>

## `tokenize_anndata`

```python
def tokenize_anndata(adata: sc.AnnData, tokenizer: Tokenizer)
```

Tokenize an AnnData object. This is more involved, so it gets its own function.
Args:
    adata (sc.AnnData): The AnnData object to tokenize.
    tokenizer (Tokenizer): The tokenizer to use.