Skip to content

Data source

You can upload files directly to an instance or connect an external data source.

Data Source Description
Built-in storage Upload files directly to an instance. Available by default on every instance.
Website Connect a domain you own to index website pages.
R2 Bucket Connect a Cloudflare R2 bucket to index stored documents.
D1 database Beta Connect a Cloudflare D1 database to index rows returned by a SQL query.

For website data sources, Parse types covers how AI Search finds the pages to index.

A D1 data source indexes database rows, not files. For its limits, refer to D1.

Supported file types

AI Search can ingest a variety of file types. The following plain text files and rich format files are supported.

Plain text file types

Format File extensions MIME type
Text .txt, .rst text/plain, text/x-rst
Log .log, .log.gz text/plain
Config .ini, .conf, .env, .properties, .gitignore, .editorconfig, .toml text/plain, text/toml
Markdown .markdown, .md, .mdx, .mdoc text/markdown
LaTeX .tex, .latex application/x-tex, application/x-latex
Script .sh, .bat, .ps1 application/x-sh, application/x-msdos-batch, text/x-powershell
SGML .sgml text/sgml
JSON .json application/json
XML .xml application/xml
SQL .sql application/sql
YAML .yaml, .yml application/x-yaml
CSS .css text/css
JavaScript .js application/javascript
TypeScript .ts, .tsx application/typescript
PHP .php application/x-httpd-php
Python .py text/x-python
Ruby .rb text/x-ruby
Java .java text/x-java-source
C .c text/x-c
C++ .cpp, .cxx text/x-c++
C Header .h, .hpp text/x-c-header
Go .go text/x-go
Rust .rs text/rust
Swift .swift text/swift
Dart .dart text/dart
Terraform .tf text/plain
EMACS Lisp .el application/x-elisp, text/x-elisp, text/x-emacs-lisp

Rich format file types

AI Search uses Markdown Conversion to convert rich format files to markdown. The following table lists the supported formats that will be converted to Markdown:

Format

File extensions

Mime Types

PDF Documents

.pdf

application/pdf

Images 1

.jpeg, .jpg, .png, .webp, .svg, .gif, .bmp

image/jpeg, image/png, image/webp, image/svg+xml, image/gif, image/bmp

HTML Documents

.html, .htm

text/html

XML Documents

.xml

application/xml

Microsoft Office Documents

.xlsx, .xlsm, .xlsb, .xls, .et, .docx

application/vnd.openxmlformats-officedocument.spreadsheetml.sheet, application/vnd.ms-excel.sheet.macroenabled.12, application/vnd.ms-excel.sheet.binary.macroenabled.12, application/vnd.ms-excel, application/vnd.openxmlformats-officedocument.wordprocessingml.document

Open Document Format

.ods, .odt

application/vnd.oasis.opendocument.spreadsheet, application/vnd.oasis.opendocument.text

CSV

.csv

text/csv

Apple Documents

.numbers

application/vnd.apple.numbers

1 Image conversion uses two Workers AI models for object detection and summarization. See Workers AI pricing for more details.

File limits

AI Search applies these file size limits:

File type Maximum size
PDF with OCR enabled 10 MiB
PDF without OCR 4 MiB
Plain text, code, configuration, markup, and other formats listed in Plain text file types 10 MiB
Other formats converted to Markdown 4 MiB

Files that exceed these limits are not indexed. They appear in the error logs.

Images and optical character recognition

AI Search supports .jpg, .jpeg, .png, .webp, .gif, .bmp, .tif, .tiff, .heic, and .heif images. Multimodal embedding models embed supported images directly. Text-only embedding models create captions before embedding images.

Optical character recognition (OCR) extracts text from scanned PDFs and images. Set indexing_options.use_ocr to true when creating or updating an instance. OCR is disabled by default and changing this setting triggers a full reindex.

OCR is available for every account. OCR usage is billed as image-processing ingestion tokens. For pricing details, refer to Limits and pricing.

File summaries

File summaries use a large language model (LLM) to describe each file while it is indexed. For every eligible file, AI Search generates a short summary, a list of topics, and a list of keywords:

  • The summary and keywords are stored in the item's context metadata when they fit within metadata size limits. AI Search passes context to the generation model with each retrieved document. If the file already has a context value, for example from R2 custom metadata or upload metadata, AI Search keeps your value. The model is still called, and billed, to generate topics.
  • The topics are added to the context header of the file's chunks where they fit within the chunk size, so they never replace chunk content. Topics are not counted as ingestion tokens.

Turn on file summaries in the dashboard, or set indexing_options.file_summary to true when you create or update an instance. File summaries are off by default, and changing this setting triggers a full reindex.

indexing_options is replaced as a whole on every update, including updates through the Workers binding. Read the instance first and include the current value of every option. Options you leave out return to their defaults: for example, an update without file_summary turns file summaries off and triggers another full reindex. This also applies to tools that send indexing_options, such as Terraform.

File summaries apply to text documents, including converted formats such as PDFs, that produce at least three chunks at your instance's chunk size. A smaller chunk size makes more files eligible and increases cost. Smaller files are indexed without a summary and make no model calls. Images are not summarized.

Keep the following in mind:

  • Best effort: AI Search makes one attempt per file version. If the model call fails, the file is indexed without a summary. Temporary failures, such as timeouts, rate limits, or model capacity errors, are retried only if the file is processed again at least 24 hours later, for example after a reindex. Other failures are not retried until the file content changes.
  • AI Gateway and account settings apply: Guardrails or spend limits on your AI Gateway, a missing gateway, or running out of Workers AI usage, such as the daily free allocation on the Workers Free plan, stop summaries from being generated. Files indexed while this happens stay without a summary until their content changes, even after you fix the setting or reindex the instance.
  • Keyword search ranking: Topics repeat in every chunk where they fit, which increases their weight in keyword search. Test search quality on a representative instance before you turn on file summaries in production.
  • Generation tokens: The context value adds up to 1,000 characters to each retrieved document sent to the generation model, which increases generation token usage.
  • Indexing time: Summarizing adds a model call to each eligible file, so indexing takes longer.

Was this helpful?

⌂ 换网址