You can upload files directly to an instance or connect an external data source.
| Data Source | Description |
|---|---|
| Built-in storage | Upload files directly to an instance. Available by default on every instance. |
| Website | Connect a domain you own to index website pages. |
| R2 Bucket | Connect a Cloudflare R2 bucket to index stored documents. |
| D1 database Beta | Connect a Cloudflare D1 database to index rows returned by a SQL query. |
For website data sources, Parse types covers how AI Search finds the pages to index.
A D1 data source indexes database rows, not files. For its limits, refer to D1.
AI Search can ingest a variety of file types. The following plain text files and rich format files are supported.
| Format | File extensions | MIME type |
|---|---|---|
| Text | .txt, .rst |
text/plain, text/x-rst |
| Log | .log, .log.gz |
text/plain |
| Config | .ini, .conf, .env, .properties, .gitignore, .editorconfig, .toml |
text/plain, text/toml |
| Markdown | .markdown, .md, .mdx, .mdoc |
text/markdown |
| LaTeX | .tex, .latex |
application/x-tex, application/x-latex |
| Script | .sh, .bat, .ps1 |
application/x-sh, application/x-msdos-batch, text/x-powershell |
| SGML | .sgml |
text/sgml |
| JSON | .json |
application/json |
| XML | .xml |
application/xml |
| SQL | .sql |
application/sql |
| YAML | .yaml, .yml |
application/x-yaml |
| CSS | .css |
text/css |
| JavaScript | .js |
application/javascript |
| TypeScript | .ts, .tsx |
application/typescript |
| PHP | .php |
application/x-httpd-php |
| Python | .py |
text/x-python |
| Ruby | .rb |
text/x-ruby |
| Java | .java |
text/x-java-source |
| C | .c |
text/x-c |
| C++ | .cpp, .cxx |
text/x-c++ |
| C Header | .h, .hpp |
text/x-c-header |
| Go | .go |
text/x-go |
| Rust | .rs |
text/rust |
| Swift | .swift |
text/swift |
| Dart | .dart |
text/dart |
| Terraform | .tf |
text/plain |
| EMACS Lisp | .el |
application/x-elisp, text/x-elisp, text/x-emacs-lisp |
AI Search uses Markdown Conversion to convert rich format files to markdown. The following table lists the supported formats that will be converted to Markdown:
Format | File extensions | Mime Types | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
PDF Documents |
|
| ||||||||||||
Images 1 |
|
| ||||||||||||
HTML Documents |
|
| ||||||||||||
XML Documents |
|
| ||||||||||||
Microsoft Office Documents |
|
| ||||||||||||
Open Document Format |
|
| ||||||||||||
CSV |
|
| ||||||||||||
Apple Documents |
|
|
1 Image conversion uses two Workers AI models for object detection and summarization. See Workers AI pricing for more details.
AI Search applies these file size limits:
| File type | Maximum size |
|---|---|
| PDF with OCR enabled | 10 MiB |
| PDF without OCR | 4 MiB |
| Plain text, code, configuration, markup, and other formats listed in Plain text file types | 10 MiB |
| Other formats converted to Markdown | 4 MiB |
Files that exceed these limits are not indexed. They appear in the error logs.
AI Search supports .jpg, .jpeg, .png, .webp, .gif, .bmp, .tif, .tiff, .heic, and .heif images. Multimodal embedding models embed supported images directly. Text-only embedding models create captions before embedding images.
Optical character recognition (OCR) extracts text from scanned PDFs and images. Set indexing_options.use_ocr to true when creating or updating an instance. OCR is disabled by default and changing this setting triggers a full reindex.
OCR is available for every account. OCR usage is billed as image-processing ingestion tokens. For pricing details, refer to Limits and pricing.
File summaries use a large language model (LLM) to describe each file while it is indexed. For every eligible file, AI Search generates a short summary, a list of topics, and a list of keywords:
- The summary and keywords are stored in the item's
contextmetadata when they fit within metadata size limits. AI Search passescontextto the generation model with each retrieved document. If the file already has acontextvalue, for example from R2 custom metadata or upload metadata, AI Search keeps your value. The model is still called, and billed, to generate topics. - The topics are added to the context header of the file's chunks where they fit within the chunk size, so they never replace chunk content. Topics are not counted as ingestion tokens.
Turn on file summaries in the dashboard, or set indexing_options.file_summary to true when you create or update an instance. File summaries are off by default, and changing this setting triggers a full reindex.
indexing_options is replaced as a whole on every update, including updates through the Workers binding. Read the instance first and include the current value of every option. Options you leave out return to their defaults: for example, an update without file_summary turns file summaries off and triggers another full reindex. This also applies to tools that send indexing_options, such as Terraform.
File summaries apply to text documents, including converted formats such as PDFs, that produce at least three chunks at your instance's chunk size. A smaller chunk size makes more files eligible and increases cost. Smaller files are indexed without a summary and make no model calls. Images are not summarized.
Keep the following in mind:
- Best effort: AI Search makes one attempt per file version. If the model call fails, the file is indexed without a summary. Temporary failures, such as timeouts, rate limits, or model capacity errors, are retried only if the file is processed again at least 24 hours later, for example after a reindex. Other failures are not retried until the file content changes.
- AI Gateway and account settings apply: Guardrails or spend limits on your AI Gateway, a missing gateway, or running out of Workers AI usage, such as the daily free allocation on the Workers Free plan, stop summaries from being generated. Files indexed while this happens stay without a summary until their content changes, even after you fix the setting or reindex the instance.
- Keyword search ranking: Topics repeat in every chunk where they fit, which increases their weight in keyword search. Test search quality on a representative instance before you turn on file summaries in production.
- Generation tokens: The
contextvalue adds up to 1,000 characters to each retrieved document sent to the generation model, which increases generation token usage. - Indexing time: Summarizing adds a model call to each eligible file, so indexing takes longer.