Airbyte integration
Airbyte is an open-source data integration platform that moves data between sources and destinations using pre-built connectors, maintained by Airbyte or its community. The Apify Dataset connector lets you move data from your Apify datasets to any Airbyte-supported destination.
This guide shows you how to set up an Apify dataset as a source in Airbyte and connect it to a destination.
This integration uses a third-party service. If you find outdated content, please submit an issue on GitHub.
Prerequisites
- An Apify account
- An Airbyte Cloud account or a self-hosted Airbyte instance
Set up the Apify Dataset source in Airbyte
The Apify Dataset source reads items from a single dataset, identified by its ID. Each Actor run writes to a new default dataset. To keep syncing the results of repeated runs, have your Actor write to a named dataset and use its ID.
- In Apify Console, open Storage > Datasets, select your dataset, and copy its ID.
- Open Settings > API & Integrations and copy your API token.
- In Airbyte, open the Sources tab and select Apify Dataset.
- Enter the Dataset ID and API token you copied, and save the source.
Your Apify dataset is now available as a source in Airbyte.
Connect the source to a destination
Setting up the source only tells Airbyte where your data comes from. To move the data, create a connection: an Airbyte pipeline that links your Apify Dataset source to a destination and controls how and when the data syncs. To set one up, follow Set up a connection in the Airbyte documentation.
To sync the dataset's items, select the item_collection stream when you set up the connection.
To keep the destination up to date, set a sync schedule on the connection. If your Actor runs on a schedule, time the syncs to start after the runs finish.