pipette

Unified reading and writing of data in Python.

The package provides a single read/write entry point that dispatches on file extension across a wide range of bioinformatics and general-purpose formats, plus data-sanitization helpers for NA handling and categorical conversion.

Installation

uv method

This package is hosted on PyPI as acidgenomics-pipette. The import name is unchanged: pipette. We recommend using uv to install.

uv add 'acidgenomics-pipette[extra]'

Or with pip:

pip install 'acidgenomics-pipette[extra]'

Conda method

Configure Conda to use the Bioconda channels.

# Don't install recipe into base environment.
name='pipette'
conda create --name="$name" "$name"
conda activate "$name"
python -c 'import pipette'

Conda has no equivalent of Python extras. For the optional format support that pipette[extra] provides, add the dependencies to the environment: conda create --name="$name" "$name" openpyxl pyarrow pyyaml scipy.

Read

>>> import pipette
>>> df = pipette.read("data.csv")
>>> df = pipette.read("data.tsv")
>>> df = pipette.read("data.xlsx")
>>> data = pipette.read("data.json")
>>> df = pipette.read("https://example.com/data.csv")

read dispatches on file extension, and understands a URL, a local path, or a pathlib.Path equally:

Format	Extension	Dependencies
CSV	.csv	-
TSV	.tsv, .tab	-
Excel	.xlsx, .xls	openpyxl
JSON	.json	-
YAML	.yml, .yaml	pyyaml
Pickle	.pickle, .pkl	-
Lines	.txt, .log, .list	-
GMT	.gmt	-
GMX	.gmx	-
GRP	.grp	-
GCT	.gct	-
GAF	.gaf	-
MTX	.mtx, .mtx.gz	scipy
Parquet	.parquet	pyarrow
Feather/Arrow	.feather, .arrow	pyarrow
HDF5	.h5, .hdf5	-

Write

>>> import pandas as pd
>>> df = pd.DataFrame(
...     {"sample1": [1, 2, 3], "sample2": [4, 5, 6]},
...     index=["gene1", "gene2", "gene3"],
... )
>>> pipette.write(df, "output.csv")
>>> pipette.write(df, "output.tsv")
>>> pipette.write(df, "output.csv.gz")

write supports CSV, TSV, JSON, YAML, pickle, and lines formats, with compressed output via .gz, .bz2, .xz, or .zip suffixes. It returns the written path: a pathlib.Path for a local target, or the str URI unchanged for an s3:// target.

Data sanitization

>>> df = pipette.sanitize_na(df)
>>> df = pipette.categorize(df)
>>> df = pipette.uncategorize(df)
>>> df = pipette.drop_nested_columns(df)

sanitize_na replaces NA-like strings (na_strings, aliased as NA_STRINGS) with NaN; categorize/uncategorize convert duplicated string columns to/from Categorical; drop_nested_columns keeps only scalar-valued columns.

Migrating from 0.1.x

import_data/export_data are now read/write. Several 0.1.x wrappers over R conventions have no replacement function; use the plain Python idiom instead:

# save_data(x, "obj", dir=d, ext="csv") / assign_and_save_data("obj", x, dir=d)
pipette.write(x, Path(d) / "obj.csv")

# load_data("obj", dir=d)
pipette.read(next(Path(d).glob("obj.*")))

# load_data("a", "b", dir=d)
{p.stem: pipette.read(p) for p in Path(d).glob("*")}

# load_remote_data(url)
pipette.read(pipette.cache_url(url))

# load_data_as_name(("new", "old"), dir=d)
# Python binds whatever name you write; no equivalent function is needed.
new = pipette.read(next(Path(d).glob("old.*")))

# encode(df, j=["a"]) / factorize(df)
pipette.categorize(df, columns=["a"])
pipette.categorize(df)

# metadata2(df, "genome", "GRCh38") / metadata2(df, "genome")
df.attrs["genome"] = "GRCh38"
df.attrs.get("genome")

# droplevels(df)
df["tissue"] = df["tissue"].cat.remove_unused_categories()

# remove_na(df, how="all")
df.dropna(axis=0, how="all").dropna(axis=1, how="all")

# md5(path)
with open(path, "rb") as f:
    hashlib.file_digest(f, "md5").hexdigest()

# init_dir(d)
Path(d).mkdir(parents=True, exist_ok=True)

# get_json(url)
pipette.read(url, format="json")

Optional dependencies

• openpyxl: Excel file support.

• pyarrow: Parquet and Feather file support.

• pyyaml: YAML file support.

• scipy: MTX (Matrix Market) sparse matrix support.