Parquet File Connector Guide

The Parquet File connector reads Apache Parquet columnar files from tenant-scoped storage. Use it to inspect table schema, page through large datasets (up to roughly 100 MB on disk), and feed rows into ProcessFlow transforms or DataPool loads.

Configuration and storage scope

Configure the integration with a tenant-relative folder and Parquet filename. The connector resolves files under storage/<tenant_id>/ and blocks directory traversal.

• filename (required): target file name ending in .parquet (for example exports/orders.parquet is invalid—use path + orders.parquet).
• path (optional): relative folder under tenant storage (for example exports).
• default_limit (optional): default row page size for read when limit is omitted (default 1000, maximum 5000).
• timeout_seconds (optional): operation timeout for large files (default 120).

Paths containing .. are rejected.

Inspect schema with schema

Call schema before reading rows to discover column names, physical/logical types, row counts, and row-group layout. This reads Parquet footer metadata only—not the full file.

const integrationId = "<INTEGRATION_ID>";

const response = await tf.connector.execute(integrationId, "schema", {
  include_row_groups: true
});

if (!response?.success) {
  throw new Error(response?.error ?? "schema failed");
}

const columns = response.data?.columns ?? [];

curl -X POST "https://api.example.com/api/v1/integrations/<ENTITY_ID>/execute" \
  -H "X-API-Key: <API_KEY>" \
  -H "X-Tenant-ID: <TENANT_ID>" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "schema",
    "data": {
      "include_row_groups": true
    }
  }'

Read rows with read

Use paginated read calls for large files. Each response returns at most limit rows (default 1000, max 5000). Loop until has_more is false.

Optional callData:

• offset — 0-based row offset (default 0)
• limit — rows per page
• columns — project top-level columns only
• row_groups — advanced: restrict to specific row-group indices

const integrationId = "<INTEGRATION_ID>";

async function readAllRows() {
  let offset = 0;
  const limit = 1000;
  const rows: Record<string, unknown>[] = [];

  for (;;) {
    const page = await tf.connector.execute(integrationId, "read", {
      offset,
      limit,
      columns: ["id", "amount", "created_at"]
    });
    if (!page?.success) throw new Error(page?.error ?? "read failed");
    rows.push(...((page.data?.rows as Record<string, unknown>[]) ?? []));
    if (!page.data?.has_more) break;
    offset = Number(page.data?.next_offset ?? offset + limit);
  }

  return rows;
}

For DataPool ingestion, insert each page inside the loop instead of accumulating all rows in memory.

Validate connectivity with test

Run test during integration setup to confirm the file exists, is readable, and contains valid Parquet metadata.

curl -X POST "https://api.example.com/api/v1/integrations/<ENTITY_ID>/execute" \
  -H "X-API-Key: <API_KEY>" \
  -H "X-Tenant-ID: <TENANT_ID>" \
  -H "Content-Type: application/json" \
  -d '{ "operation": "test" }'

Staging files in tenant storage

Parquet files often arrive through SFTP, S3, or WebApp uploads into processdata/<execution_id>/. Point the integration path and filename at the staged location, or copy the file into a stable folder (for example exports/) before scheduling reads.

Large files and compression

The connector uses range-based I/O and row-group-aware reads. Files around 100 MB are supported when you paginate read responses.

Type mapping

Reliability guidance

1. Call schema once per file version before mapping fields.
2. Keep limit at or below 1000 for wide rows.
3. Check success on every connector response.
4. Prefer inserting each page into DataPool rather than holding the full dataset in a single step variable.

Additional resources

• Apache Parquet documentation
• Integration connector usage
• File operations in ProcessFlow