Bulk metadata exports

BEDbase publishes a monthly snapshot of the metadata corpus as Apache Parquet files on S3. This is the supported way to get the whole corpus in bulk; you do not need to page through the API.

Where the exports live

Artifacts are published under the snapshot/ prefix of the public storage bucket, served over HTTPS at https://data2.bedbase.org/:

https://data2.bedbase.org/snapshot/bedbase_metadata_2026_08_03.parquet
https://data2.bedbase.org/snapshot/bedbase_bedsets_2026_08_03.parquet
https://data2.bedbase.org/snapshot/bedbase_bedset_membership_2026_08_03.parquet
https://data2.bedbase.org/snapshot/manifest_2026_08_03.json

Each run publishes:

File	Contents
bedbase_metadata_<date>.parquet	One row per BED record: the bed table left-joined with bed_metadata. Denormalized descriptive metadata (genome, assay, cell type, tissue, target, sample/experiment ids, …). Per-file statistics are not included in this export.
bedbase_bedsets_<date>.parquet	One row per bedset.
bedbase_bedset_membership_<date>.parquet	The bedfile ↔ bedset relation (bedset_id, bedfile_id).
manifest_<date>.json	Verifiable build metadata (see below).

Filenames are dated and immutable

Every filename carries its build date (YYYY_MM_DD). A published file is never overwritten, and there is no latest alias. Do not construct or hardcode a filename. Instead, discover the current snapshot through the index endpoint:

GET https://api.bedbase.org/v1/exports

It returns the index newest-first, with file_path already rewritten to an absolute https://data2.bedbase.org/ URL:

{
  "count": 8,
  "results": [
    {
      "file_path": "https://data2.bedbase.org/snapshot/bedbase_metadata_2026_09_01.parquet",
      "file_type": "metadata",
      "creation_date": "2026-09-01T00:49:48.355278Z",
      "record_count": 663721,
      "file_size": 52850770,
      "checksum": "…sha256…",
      "schema_version": 1
    }
  ]
}

The newest row is first; take the first metadata entry to find the current snapshot. The same artifacts are also listed as GA4GH DRS objects at GET https://api.bedbase.org/v1/objects/exports.

Retention

Snapshots are published monthly. Monthly snapshots are retained for the trailing 12 months; older snapshots are thinned to quarterly (the 1st of January, April, July, and October), which are kept permanently. For a durable, citable reference, prefer a quarterly snapshot.

The manifest

manifest_<date>.json makes a snapshot verifiable:

{
  "schema_version": 1,
  "build_started": "2026-08-03T22:18:11.356975+00:00",
  "build_ended": "2026-08-03T22:28:48.207622+00:00",
  "source_database": "bedbase",
  "files": [
    {"name": "bedbase_metadata_2026_08_03.parquet",
     "file_type": "metadata", "rows": 663242,
     "bytes": 52754000, "sha256": "…"}
  ]
}

rows is the number of rows actually written to each file (not a separately queried count), and each sha256 matches its file on disk, so you can verify a download end-to-end.

Querying with DuckDB

Parquet plus open CORS and HTTP range requests means you can query a snapshot directly over HTTPS (no download, no server) from the DuckDB CLI or DuckDB-WASM in a browser.

Try it today with DuckDB, no download or pagination needed:

SELECT *
FROM read_parquet('https://data2.bedbase.org/snapshot/bedbase_metadata_2026_08_03.parquet')
WHERE assay = 'ChIP-seq' AND genome_alias = 'hg38';

DuckDB reads only the byte ranges it needs, so a filtered query does not transfer the whole file.

How it is produced

A monthly databio/bedbase-loader GitHub Action (export_metadata.yml) runs bedboss snapshot new, which streams the tables straight from PostgreSQL as unordered sequential scans through server-side cursors (bed left-joined with bed_metadata), writes zstd-compressed Parquet, checks row counts against a pre-scan count(*), and refuses to publish a partial artifact (more than 1% of rows missing, by default). bbconf then uploads the files and records them in the index. The read runs in a single REPEATABLE READ transaction, so each snapshot is internally consistent.