JPX Scraper is a Go API that scrapes stock data (Kabutan), caches the JSON in MySQL, and serves it via HTTP:
GET /scrape?ticker=8306
銘柄コード(ticker)を1つ渡すだけで、株価関連の主要指標を JSONでまとめて取得 できます。
# VPS 上で 8085 にポート転送している場合の例
curl -i "http://127.0.0.1:8085/scrape?ticker=5020"{
"ticker":"5020",
"companyName":"5020 ENEOSホールディングス",
"currentPrice":"1,488.0円",
"previousClose":"1,447.5 (02/26)",
"dividendYield":"1.75%",
"per":"18.6倍",
"pbr":"1.3倍",
"marketCap":"4兆32億円",
"volume":"11,494,600"
}つまり、フロントエンドや他システムからは「/scrape?ticker=xxxx を叩けば、画面表示や分析に使いやすい最新データが返るAPI」として利用できます。
⚠️ 利用上の注意 — リクエスト間隔についてこのAPIは内部でKabutanをスクレイピングしています。スクレイピングが発生するのは キャッシュが古い(または存在しない)場合のみ ですが、短時間に大量の異なるティッカーをリクエストするとKabutanのサーバーに負荷をかけます。
- 目安: 1〜3リクエスト / 秒以内 に抑えてください
- 連続して叩く場合は
time.Sleepやtime.Tickなどで間隔を設けてください
This README focuses on what gets created in MySQL, how to verify with SQL, and which knobs control cache lifetime. It also documents the compact Docker profile you’re currently running.
環境ごとに叩くポートが違います。/scrape のパス自体は同じで、違うのはホスト側の公開ポートだけです。
| Environment | Start method | Host port | Example |
|---|---|---|---|
| VPS / port-forwarded host | VPS 側の転送設定を利用 | 8085 |
curl -i "http://127.0.0.1:8085/scrape?ticker=5020" |
| Local Docker (base compose only) | docker compose up -d --build |
8082 |
curl -i "http://127.0.0.1:8082/scrape?ticker=5020" |
| Local WSL override | make wsl-up / bash scripts/wsl-compose.sh up |
18082 |
curl -i "http://127.0.0.1:18082/scrape?ticker=5020" |
docker-compose.yml単体では8082:8081docker-compose.wsl.ymlを重ねると18082:80818085はアプリのデフォルト公開ポートではなく、VPS 側で別途ポート転送しているときの入口
Browser/Client
│
▼
HTTP (REST, JSON)
:8082 → container :8081 ───────┐
│ calls
▼
Scraper (Go) ── DB cache (MySQL 8.x)
└─> table: scrapings (1 row per ticker, JSON payload)
gRPC & KVS(Valkey) are not enabled yet. The design anticipates adding them later with minimal changes.
At startup, the app idempotently ensures the cache table exists:
CREATE TABLE IF NOT EXISTS scrapings (
ticker VARCHAR(32) NOT NULL PRIMARY KEY, -- JPX code, e.g. '8306'
jsond LONGTEXT NULL, -- raw JSON payload served by /scrape
updated DATETIME NOT NULL -- last refresh timestamp
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;- Primary key:
ticker— there is one row per ticker. New data replaces the existing row. - Write path: when the scraper refreshes data, it uses:
REPLACE INTO scrapings(ticker, jsond, updated) VALUES(?,?,NOW()); - Read path: the handler first tries the DB row; if fresh, it serves that JSON; otherwise it scrapes and updates.
The table is a cache, not a source of truth. Long-term retention is not required. (Later, you can introduce a history table or a KVS layer for faster hits.)
Compose publishes container port 3306 to host port 3307.
# root
mysql -h 127.0.0.1 -P 3307 -uroot -p
# app user
mysql -h 127.0.0.1 -P 3307 -u "$MYSQL_USER" -p"$MYSQL_PASSWORD" "$MYSQL_DATABASE"Always pass
-h 127.0.0.1when using the host client. Without-h, the client may try a UNIX socket and hit a different local server.
SHOW DATABASES;
USE jpx;
SHOW TABLES;
DESC scrapings;
SELECT ticker, LENGTH(jsond) AS bytes, updated FROM scrapings ORDER BY updated DESC LIMIT 5;
SELECT jsond FROM scrapings WHERE ticker='8306';If you delete a row to force a refresh:
DELETE FROM scrapings WHERE ticker='8306';- The service tries to serve from DB if the row is fresh (by default ~1 hour freshness window).
- If stale or missing, the service scrapes, REPLACEs the row, and returns the fresh JSON.
- Consecutive requests within the freshness window do not hit Selenium again; they read from DB.
Look for the staleness/TTL constant in the code (develop branch), typically in the “usecase” or handler layer that decides:
// pseudo: consider DB value fresh if updated within 1 hour
staleWindow := time.Hour
if time.Since(updatedAt) < staleWindow {
return fromDB
}Increase/decrease
staleWindow(e.g.,30*time.Minute,2*time.Hour) to control when the scraper re-runs.
Later (when KVS is added), the DB freshness window can be lengthened (e.g., 30 days) and a KVS TTL (e.g., 1–24 hours) can serve the hot path.
- Start the stack:
docker compose --env-file .env up -d --buildWSL Ubuntu で他プロジェクトから API として再利用するローカル運用では、VPS 用のベース構成はそのままにして docker-compose.wsl.yml を重ねます。
短いコマンドで扱うなら、次を使えます。
make wsl-up
make wsl-ps
make wsl-logs
make wsl-scrape TICKER=8306
make wsl-restart
make wsl-down既定では API は http://127.0.0.1:18082 で待ち受けます。
または:
bash scripts/wsl-compose.sh up
bash scripts/wsl-compose.sh ps
bash scripts/wsl-compose.sh logs scraper
bash scripts/wsl-compose.sh down内部的には次の compose 構成を呼びます。
docker compose \
-f docker-compose.yml \
-f docker-compose.wsl.yml \
--env-file .env \
up -d --buildこの override は以下だけをローカル向けに変えます。
- ホスト公開ポートを
13306/14444/18082に上げる - ベースの
docker-compose.ymlはそのまま使い、VPS とローカルで DB 実体を分けない
ローカル .env には少なくとも次を入れてください。
MYSQL_ROOT_PASSWORD=********
MYSQL_DATABASE=jpx
MYSQL_USER=jpx
MYSQL_PASSWORD=********
MYSQL_HOST_PORT=13306
SELENIUM_HOST_PORT=14444
SCRAPER_HOST_PORT=18082- Call the API twice (second call should be fast and DB-backed):
time curl -s "http://localhost:8082/scrape?ticker=8306" >/dev/null
time curl -s "http://localhost:8082/scrape?ticker=8306" >/dev/null- Confirm cache content:
mysql -h 127.0.0.1 -P 3307 -u "$MYSQL_USER" -p"$MYSQL_PASSWORD" "$MYSQL_DATABASE" \
-e "SELECT ticker, LENGTH(jsond) bytes, updated FROM scrapings WHERE ticker='8306'"- Start (single node is the default in this compose file):
docker compose up -d- Verify exactly one node container is running and hub is healthy:
docker ps --filter "name=selenium-node"
docker inspect --format='{{.State.Health.Status}}' jpx-scraper-selenium-hub-1- Smoke test: confirm the scraper can reach Selenium via the hub:
curl -s "http://localhost:8082/scrape?ticker=8306" >/dev/null
docker logs jpx-scraper | tail -n 50- Resource check (low-memory VPS):
free -hm
docker stats --no-streamKey variables (from .env):
# MySQL
MYSQL_ROOT_PASSWORD=********
MYSQL_DATABASE=jpx
MYSQL_USER=jpx
MYSQL_PASSWORD=********
# App DB connection
DB_HOST=mysql
DB_PORT=3306
DB_NAME=jpx
DB_USER=jpx
DB_PASSWORD=********
DB_PARAMS=parseTime=true&charset=utf8mb4&loc=Asia%2FTokyo
# Selenium (compact defaults)
SE_NODE_MAX_SESSIONS=1Health checks (Compose):
- MySQL:
mysqladmin ping --protocol=TCP -h 127.0.0.1 ... - Selenium Hub:
curl -fsS http://localhost:4444/status - Scraper waits until MySQL/Selenium are reachable.
- Connected to the wrong DB: always include
-h 127.0.0.1 -P 3307on the host, or exec into the container:docker compose exec mysql sh -lc 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "SHOW DATABASES"'
VPS 側では .env を MySQL 用に合わせたうえで、キャッシュ DB を作り直す前提なら次で再構築できます。
MYSQL_ROOT_PASSWORD=********
MYSQL_DATABASE=jpx
MYSQL_USER=jpx
MYSQL_PASSWORD=********
DB_HOST=mysql
DB_PORT=3306
DB_NAME=jpx
DB_USER=jpx
DB_PASSWORD=********PRUNE_MODE=deep ./deploy-jpx-scraper.shボリュームも含めて落としてよいなら、次でも同じです。
docker compose --env-file .env down -v --remove-orphans
docker compose --env-file .env up -d --build- Table not found: the app creates
scrapingsat start; check scraper logs:docker logs jpx-scraper | grep -i schema - Rebuild didn’t pick code changes: force a clean rebuild + recreate:
docker compose build --no-cache scraper docker compose up -d --force-recreate scraper
- Slow responses: confirm DB hit rate by watching updated times; reduce
staleWindowonly if necessary.
Does:
- Scrapes Kabutan on demand.
- Caches the JSON into
scrapings(one row per ticker). - Serves JSON via REST (
/scrape?ticker=XXXX).
Does not (yet):
- Provide gRPC (will live on a separate port later).
- Use KVS (Valkey) for hot-cache (planned).
- Store long-term history (consider a
quotestable later if needed).
- KVS(Valkey) cache-aside:
quote:latest:<ticker>TTL=1h, DB retention 30d. - SWR (stale-while-revalidate): serve old quickly, refresh async.
- gRPC on :9090 (REST remains on :8081).
- Worker/Scheduler to decouple scraping from HTTP path.