# 🐙 GitHub Repository Intelligence - Repos & READMEs (`benthepythondev/github-repository-intelligence`) Actor

Returns GitHub repository data through the official REST API: name, owner, description, homepage, main language and language mix, stars, forks, watchers, open issues, topics, license, dates and the full README text. Search by keyword or list repositories. For developer research and LLM datasets.

- **URL**: https://apify.com/benthepythondev/github-repository-intelligence.md
- **Developed by:** [Ben](https://apify.com/benthepythondev) (community)
- **Categories:** AI, Developer tools, Other
- **Features:** Limited permissions
- **Stats:** 90 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $16.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## GitHub Repository Intelligence - API-Based Data & Documentation Scraper

**Extract comprehensive repository data from GitHub using the official REST API.**

Fetch repository metadata, README content, documentation, topics, language statistics, and more. Perfect for AI/LLM training data, developer research, competitive analysis, and tech stack discovery. Legal, stable, and fast API-based extraction.

### Features

✅ **Dual Scraping Modes**

- **Search Mode**: Find repositories by keywords, language, stars
- **Direct Mode**: Fetch specific repositories by URL

✅ **Comprehensive Data Extraction**

- Repository metadata (stars, forks, watchers, issues)
- README content (perfect for LLM training)
- Programming language statistics
- Repository topics/tags
- License information
- Creation/update timestamps
- Owner information

✅ **Official GitHub API**

- Uses GitHub REST API v3 (100% legal)
- No browser automation required
- Stable and reliable
- Optional authentication for higher rate limits

✅ **Built for AI & Research**

- README extraction for LLM training
- Structured JSON output
- Rich metadata for analysis
- Topic and language classification
- Dataset export (CSV, JSON, Excel)

### Use Cases

🤖 **AI/LLM Training Data**

- Extract README files for AI model training
- Gather documentation for vector databases
- Build RAG (Retrieval-Augmented Generation) pipelines
- Create code-to-text datasets

🔍 **Developer Research**

- Discover trending repositories
- Analyze tech stacks and tools
- Monitor open-source ecosystem
- Track language adoption trends

💼 **Business Intelligence**

- Competitive analysis
- Technology trend spotting
- Developer tool discovery
- Market research for dev tools

📊 **Academic Research**

- Software engineering studies
- Open-source collaboration analysis
- Programming language evolution
- Developer ecosystem research

### Input

```json
{
  "mode": "search",
  "searchQuery": "language:python stars:>1000",
  "sortBy": "stars",
  "maxResults": 50,
  "includeReadme": true,
  "includeTopics": true,
  "includeLanguages": true,
  "githubToken": "ghp_xxxxxxxxxxxx"
}
```

#### Input Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | string | `search` | Scraping mode: `search` (find repos by query) or `direct` (specific URLs) |
| `searchQuery` | string | `""` | Search query (e.g., `language:python stars:>1000`). Uses GitHub search syntax. |
| `repositoryUrls` | string | `""` | Repository URLs (one per line). Format: `https://github.com/owner/repo` |
| `sortBy` | string | `stars` | Sort search results by: `stars`, `forks`, `updated`, `help-wanted-issues` |
| `maxResults` | integer | `30` | Maximum repositories to fetch (search mode, 1-1000) |
| `includeReadme` | boolean | `true` | Extract README content (recommended for AI/LLM) |
| `includeTopics` | boolean | `true` | Fetch repository topics/tags |
| `includeLanguages` | boolean | `true` | Fetch programming language statistics |
| `githubToken` | string | `""` | Optional GitHub Personal Access Token (5,000 vs 60 requests/hour) |
| `debugMode` | boolean | `false` | Enable verbose logging |

### Output

Each repository returns comprehensive data:

```json
{
  "name": "tensorflow",
  "full_name": "tensorflow/tensorflow",
  "owner": {
    "login": "tensorflow",
    "type": "Organization",
    "url": "https://github.com/tensorflow"
  },
  "description": "An Open Source Machine Learning Framework for Everyone",
  "url": "https://github.com/tensorflow/tensorflow",
  "homepage": "https://www.tensorflow.org",
  "language": "C++",
  "stars": 185000,
  "forks": 74000,
  "watchers": 185000,
  "open_issues": 1850,
  "size": 285000,
  "topics": ["machine-learning", "deep-learning", "tensorflow", "python"],
  "license": "Apache License 2.0",
  "created_at": "2015-11-07T01:19:20Z",
  "updated_at": "2025-01-12T10:30:45Z",
  "pushed_at": "2025-01-12T09:15:22Z",
  "is_fork": false,
  "is_archived": false,
  "is_private": false,
  "default_branch": "master",
  "readme": {
    "name": "README.md",
    "path": "README.md",
    "content": "# TensorFlow...",
    "size": 12584,
    "html_url": "https://github.com/tensorflow/tensorflow/blob/master/README.md",
    "download_url": "https://raw.githubusercontent.com/tensorflow/tensorflow/master/README.md"
  },
  "languages": {
    "C++": 125847623,
    "Python": 45123456,
    "Java": 12345678
  },
  "scraped_at": "2025-01-12T15:30:00.000Z",
  "index": 1
}
```

### Example Usage

#### Search for Python Repositories

```json
{
  "mode": "search",
  "searchQuery": "language:python stars:>1000",
  "sortBy": "stars",
  "maxResults": 100
}
```

#### Search for AI/ML Projects

```json
{
  "mode": "search",
  "searchQuery": "machine learning stars:>5000",
  "sortBy": "updated",
  "maxResults": 50,
  "includeReadme": true
}
```

#### Fetch Specific Repositories

```json
{
  "mode": "direct",
  "repositoryUrls": "https://github.com/facebook/react\nhttps://github.com/tensorflow/tensorflow\nhttps://github.com/microsoft/vscode",
  "includeReadme": true,
  "includeTopics": true
}
```

#### With GitHub Token (Higher Rate Limits)

```json
{
  "mode": "search",
  "searchQuery": "language:rust stars:>500",
  "maxResults": 200,
  "githubToken": "ghp_yourtoken",
  "includeReadme": true
}
```

### GitHub Search Query Syntax

#### Search by Language

```
language:python
language:javascript
language:rust
```

#### Search by Stars/Forks

```
stars:>1000
stars:1000..5000
forks:>500
```

#### Search by Topics

```
topic:machine-learning
topic:web-development
```

#### Search by Organization

```
org:google
org:microsoft
user:torvalds
```

#### Combine Multiple Criteria

```
language:python stars:>1000 topic:machine-learning
language:go stars:>500 forks:>100
```

**Full Documentation**: [GitHub Search Syntax](https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories)

### Rate Limits & Authentication

#### Without GitHub Token

- **60 requests per hour**
- Good for: Testing, small batches (<30 repos)
- Unauthenticated access

#### With GitHub Token (Recommended)

- **5,000 requests per hour**
- Good for: Production, large batches (100s of repos)
- Required for: Frequent usage

#### Creating a GitHub Token

1. Go to [GitHub Settings → Tokens](https://github.com/settings/tokens)
2. Click "Generate new token (classic)"
3. Select scopes: `public_repo` (read public repositories)
4. Copy token and use in `githubToken` parameter

**Note**: Tokens are optional but highly recommended for production use.

### Pricing (Pay-Per-Result)

**$0.015 per repository** ($15 per 1,000 repositories)

**Example Cost Calculation:**

Fetching 1,000 repositories:

- Repository metadata: 1,000 × $0.015 = **$15.00**

💡 **No browser costs, no proxy costs** - just lightweight API calls!

### Best Practices

#### Search Optimization

1. **Use Specific Queries**: `language:python stars:>1000` > `python`
2. **Filter by Activity**: `pushed:>2024-01-01` for active projects
3. **Combine Criteria**: Use stars, language, topics together
4. **Sort Strategically**: `stars` for popular, `updated` for active

#### README Extraction for AI/LLM

1. **Enable README Fetching**: Always set `includeReadme: true`
2. **Filter Quality**: Focus on repos with stars:>100
3. **Language Filtering**: Target specific tech stacks
4. **Documentation Rich**: Search for `topic:documentation`

#### Rate Limit Management

1. **Use Authentication**: Get a GitHub token for 5,000 requests/hour
2. **Batch Requests**: Plan your searches to minimize API calls
3. **Monitor Limits**: Check rate limit in actor logs
4. **Schedule Runs**: Spread large jobs across hours

### FAQ

**Q: Is this legal?**
A: Yes! Uses GitHub's official REST API with proper permissions.

**Q: Do I need a GitHub account?**
A: No for basic usage (60 requests/hour). Yes for higher limits (5,000 requests/hour with token).

**Q: What's the rate limit without a token?**
A: 60 requests per hour (unauthenticated). 5,000 with a token.

**Q: Can I extract private repositories?**
A: Only public repositories. Private repos require different permissions.

**Q: How do I get README content for AI training?**
A: Set `includeReadme: true` and use search mode to find relevant repositories.

**Q: Can I search by multiple languages?**
A: Use `language:python OR language:javascript` in search query.

**Q: What happens if rate limit is exceeded?**
A: Actor will log a warning. Add a GitHub token to increase limits.

### Why Use This Actor?

| Feature | This Actor (GitHub API) | Web Scraping |
|---------|------------------------|--------------|
| Legal | ✅ Official API | ❌ Violates ToS |
| Stable | ✅ API rarely changes | ❌ HTML breaks often |
| Fast | ✅ Direct API calls | ❌ Browser overhead |
| Cost | ✅ $15 per 1k repos | ❌ $30+ per 1k |
| Authentication | ✅ Optional (higher limits) | ❌ Complex login |
| README Access | ✅ Direct API endpoint | ❌ Requires parsing |
| Maintenance | ✅ Minimal | ❌ Constant updates |

### Output Use Cases

#### AI/LLM Training

- Feed README content into vector databases
- Build code documentation datasets
- Create programming Q\&A pairs
- Extract technical writing samples

#### Developer Tools

- Tech stack analysis
- Framework popularity tracking
- Library comparison
- Documentation aggregation

#### Business Intelligence

- Competitor monitoring
- Technology trend analysis
- Open-source landscape mapping
- Developer ecosystem research

### Legal & Ethics

✅ **Legal Compliance:**

1. **Official API**: Uses GitHub REST API v3 with proper authentication
2. **Public Data Only**: Accesses only publicly available repositories
3. **Rate Limits**: Respects GitHub's rate limiting
4. **Terms of Service**: Complies with GitHub's API ToS
5. **No Scraping**: No HTML parsing or browser automation

This actor is **100% legal and ethical** - uses official GitHub API with proper permissions.

### Support

Need help? Have questions?

- 📧 **Support**: support@apify.com
- 📚 **Apify Docs**: https://docs.apify.com
- 💬 **Community**: https://discord.com/invite/jyEM2PRvMU
- 🐙 **GitHub API Docs**: https://docs.github.com/en/rest

***

**Built with ❤️ using [Apify](https://apify.com) and [GitHub REST API](https://docs.github.com/en/rest)**

**Perfect for:**

- 🤖 AI/LLM training data collection
- 📊 Developer research and analytics
- 💼 Competitive intelligence
- 🔍 Technology trend analysis
- 📚 Documentation aggregation

# Actor input Schema

## `mode` (type: `string`):

Search for repositories by keywords or fetch specific repositories by URL

## `searchQuery` (type: `string`):

Search query for repositories (e.g., 'language:python stars:>1000', 'react', 'machine-learning'). Supports GitHub search syntax.

## `repositoryUrls` (type: `string`):

GitHub repository URLs (one per line). Format: https://github.com/owner/repo

## `sortBy` (type: `string`):

How to sort search results

## `maxResults` (type: `integer`):

Maximum number of repositories to fetch (search mode only)

## `includeReadme` (type: `boolean`):

Extract and include README file content (recommended for AI/LLM training)

## `includeTopics` (type: `boolean`):

Fetch and include repository topics/tags

## `includeLanguages` (type: `boolean`):

Fetch programming language statistics

## `githubToken` (type: `string`):

Optional GitHub token for higher rate limits (5,000 vs 60 requests/hour). Create at: https://github.com/settings/tokens

## `debugMode` (type: `boolean`):

Enable verbose logging for troubleshooting

## Actor input object example

```json
{
  "mode": "search",
  "searchQuery": "language:python stars:>1000",
  "repositoryUrls": "https://github.com/facebook/react\nhttps://github.com/tensorflow/tensorflow",
  "sortBy": "stars",
  "maxResults": 30,
  "includeReadme": true,
  "includeTopics": true,
  "includeLanguages": true,
  "debugMode": false
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQuery": "",
    "repositoryUrls": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("benthepythondev/github-repository-intelligence").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQuery": "",
    "repositoryUrls": "",
}

# Run the Actor and wait for it to finish
run = client.actor("benthepythondev/github-repository-intelligence").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQuery": "",
  "repositoryUrls": ""
}' |
apify call benthepythondev/github-repository-intelligence --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,benthepythondev/github-repository-intelligence"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/VoHq0w6n9VgnANxZF/builds/ZetG2pEM0lwiT5g3n/openapi.json