# Pipelines

Ingest real time data streams and load into R2

> Links below point directly to Markdown versions of each page. Any page can also be retrieved as Markdown by sending an `Accept: text/markdown` header to the page's URL without the `index.md` suffix (for example, `curl -H "Accept: text/markdown" https://docs.ahq.lat/pipelines/`).
>
> For other Cloudflare products, see the [Cloudflare documentation directory](https://docs.ahq.lat/llms.txt).

## Overview

- [Cloudflare Pipelines](https://docs.ahq.lat/pipelines/index.md): Ingest, transform, and deliver streaming data to R2 as Apache Iceberg tables or Parquet and JSON files.

## Getting started

- [Getting started](https://docs.ahq.lat/pipelines/getting-started/index.md): Create your first pipeline to ingest streaming data and write to R2 Data Catalog as an Apache Iceberg table.

## Streams

- [Streams](https://docs.ahq.lat/pipelines/streams/index.md): Durable, buffered queues that receive and store events for processing in Cloudflare Pipelines.
- [Logpush as a source](https://docs.ahq.lat/pipelines/streams/logpush/index.md): Use Cloudflare Logpush to send logs from Cloudflare products to a Pipelines stream.
- [Manage streams](https://docs.ahq.lat/pipelines/streams/manage-streams/index.md): Create, configure, and manage streams for data ingestion
- [Writing to streams](https://docs.ahq.lat/pipelines/streams/writing-to-streams/index.md): Send data to streams via Worker bindings or HTTP endpoints

## Sinks

- [Sinks](https://docs.ahq.lat/pipelines/sinks/index.md): Configure data destinations in Cloudflare Pipelines to write to R2 or R2 Data Catalog.
- [R2](https://docs.ahq.lat/pipelines/sinks/available-sinks/r2/index.md): Write data as JSON or Parquet files to R2 object storage
- [R2 Data Catalog](https://docs.ahq.lat/pipelines/sinks/available-sinks/r2-data-catalog/index.md): Write data as Apache Iceberg tables to R2 Data Catalog
- [Manage sinks](https://docs.ahq.lat/pipelines/sinks/manage-sinks/index.md): Create, configure, and manage sinks for data storage

## Pipelines

- [Pipelines](https://docs.ahq.lat/pipelines/pipelines/index.md): Connect streams to sinks with SQL transformations to filter, enrich, and restructure events.
- [Manage pipelines](https://docs.ahq.lat/pipelines/pipelines/manage-pipelines/index.md): Create, configure, and manage SQL transformations between streams and sinks

## observability

- [Metrics and analytics](https://docs.ahq.lat/pipelines/observability/metrics/index.md): Query Pipelines metrics for data ingested, processed, and delivered via the dashboard or GraphQL API.

## platform

- [Limits](https://docs.ahq.lat/pipelines/platform/limits/index.md): Account-level limits for Cloudflare Pipelines streams, sinks, and pipelines during open beta.
- [Pricing](https://docs.ahq.lat/pipelines/platform/pricing/index.md): Cloudflare Pipelines pricing for SQL transforms, sinks, and free tier details.

## reference

- [Legacy pipelines](https://docs.ahq.lat/pipelines/reference/legacy-pipelines/index.md): Migration guide for pipelines created before September 2025 to the new Pipelines architecture.
- [REST API](https://docs.ahq.lat/api/resources/pipelines/index.md)
- [Terraform](https://docs.ahq.lat/pipelines/reference/terraform/index.md): Configure Pipelines and R2 Data Catalog with Terraform using the Cloudflare provider.
- [Wrangler commands](https://docs.ahq.lat/pipelines/reference/wrangler-commands/index.md): Wrangler CLI commands for managing Pipelines streams, sinks, and pipelines.

## sql-reference

- [Array functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/array/index.md): Scalar functions for manipulating arrays
- [Binary string functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/binary-string/index.md): Scalar functions for manipulating binary strings
- [Conditional functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/conditional/index.md): Scalar functions to implement conditional logic
- [Hashing functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/hashing/index.md): Scalar functions for hashing values
- [JSON functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/json/index.md): Scalar functions for manipulating JSON
- [Math functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/math/index.md): Scalar functions for mathematical operations
- [Other functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/other/index.md): Miscellaneous scalar functions
- [Regex functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/regex/index.md): Scalar functions for regular expressions
- [String functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/string/index.md): Scalar functions for manipulating strings
- [Struct functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/struct/index.md): Scalar functions for manipulating structs
- [Time and date functions](https://docs.ahq.lat/pipelines/sql-reference/scalar-functions/time-and-date/index.md): Scalar functions for handling times and dates
- [SELECT statements](https://docs.ahq.lat/pipelines/sql-reference/select-statements/index.md): Query syntax for data transformation in Cloudflare Pipelines SQL
- [SQL data types](https://docs.ahq.lat/pipelines/sql-reference/sql-data-types/index.md): Supported data types in Cloudflare Pipelines SQL