Skip to main content

Airbyte integration

Airbyte is an open-source data integration platform that moves data between sources and destinations using pre-built connectors, maintained by Airbyte or its community. The Apify Dataset connector lets you move data from your Apify datasets to any Airbyte-supported destination.

This guide shows you how to set up an Apify dataset as a source in Airbyte and connect it to a destination.

Help keep this page up to date

This integration uses a third-party service. If you find outdated content, please submit an issue on GitHub.

Prerequisites

Set up the Apify Dataset source in Airbyte

The Apify Dataset source reads items from a single dataset, identified by its ID. Each Actor run writes to a new default dataset. To keep syncing the results of repeated runs, have your Actor write to a named dataset and use its ID.

  1. In Apify Console, open Storage > Datasets, select your dataset, and copy its ID.
  2. Open Settings > API & Integrations and copy your API token.
  3. In Airbyte, open the Sources tab and select Apify Dataset.
  4. Enter the Dataset ID and API token you copied, and save the source.

Your Apify dataset is now available as a source in Airbyte.

Connect the source to a destination

Setting up the source only tells Airbyte where your data comes from. To move the data, create a connection: an Airbyte pipeline that links your Apify Dataset source to a destination and controls how and when the data syncs. To set one up, follow Set up a connection in the Airbyte documentation.

To sync the dataset's items, select the item_collection stream when you set up the connection.

To keep the destination up to date, set a sync schedule on the connection. If your Actor runs on a schedule, time the syncs to start after the runs finish.

⌂ 换网址