Skip to content

feat(v2)!: sync v2 with idin-core — fixes, memory MQ, presigned POST, rate limit, retry - #17

Merged
pskclub merged 7 commits into
masterfrom
feat/v2-sync-idin-core
Oct 23, 2026
Merged

pskclub merged 7 commits into
masterfrom
feat/v2-sync-idin-core

Conversation

@pskclub

@pskclub pskclub commented Oct 23, 2026

Copy link
Copy Markdown
Owner

What and why

v2 was imported from idin-core at daf4840. Since then, idin-core has fixed bugs found in a full review of v2, added several capabilities and patched published CVEs. This PR ports those six v2 commits with the module path rewritten to github.com/pskclub/mine-core/v2:

  • fix: name Sentry issues after the root cause; answer repository failures with DBError instead of the label "repository"
  • fix: correct error mapping, SQL constraint errors (409/400), transactions, Runner.Stop, S3 delete/URL, Mongo update options, mongorepo unscoped writes, jobstore, context races, Sentry scrubbing, and more. Each fix has a unit test.
  • feat: core.NewMemoryMQ(), IStorage.PresignPost, core.RateLimit, core.SecureHeaders, core.Retry, plus retries in Requester, Mailer and Slack. Probe requests can be left out of the access log with LOG_REQUEST_SKIP_PATHS.
  • deps: toolchain go1.27.2, which fixes 12 stdlib advisories; x/crypto 0.58, x/net 0.61 and grpc 1.84, which fix the Trivy findings; other direct dependencies updated. v2/.trivyignore ignores GO-2026-5932 (openpgp, which v2 never links).

v2/sentry.go keeps mine-core's existing comment on the sentry-go logs/metrics opt-out. Only the comment differs; the code is the same.

BREAKING CHANGE: visible behaviour that consumers may have relied on:

  • IStorage gained PresignPost. A type outside this module that implements IStorage itself must add this method. Callers of NewStorage/NewMemoryStorage are unaffected.
  • A SQL unique violation is now 409 DUPLICATE_KEY instead of 500 DATABASE_ERROR. A foreign-key violation is 409 FOREIGN_KEY_VIOLATED, and a check-constraint violation is 400 CHECK_CONSTRAINT_VIOLATED. To migrate, branch on the new codes.
  • Requester.Send returns 499 REQUEST_CANCELED when the caller cancels and 504 UPSTREAM_TIMEOUT when the deadline passes, instead of 500 NETWORK_ERROR.
  • GetPageOptionsWithAllowed() with no columns allows no client ordering. Use GetPageOptions() to allow ordering on any column.
  • mongorepo single-document writes without a filter return 400. Use UpdateAll/DeleteAll/Upsert/Save when you mean to write without one.
  • The tx passed by repository.Transaction no longer carries the repository's Model/Where. Build repositories on it with NewWithDB.
  • coretest.WithEnv no longer overrides APP_* variables that are already set in the process.
  • The Slack driver retries a 429 by default. Set CHAT_SLACK_RETRIES=0 to turn this off.

Module

  • v2
  • v1 — bug fix only (v1 is in maintenance mode)
  • Repository, CI or docs only

Checklist

  • go build, go vet (with and without the integration/e2e tags), gofmt and go test ./... pass locally
  • New or changed behaviour has a test that states the behaviour, not only coverage
  • Interface changes also update the memory / noop implementations
  • v2/docs/ is updated for anything a user of the framework would notice
  • Breaking change: the title has !, and the description has a BREAKING CHANGE: section with migration steps

🤖 Generated with Claude Code

pskclub and others added 6 commits September 30, 2026 15:21
…r's label

A repository failure is wrapped as "repository", so every database error
reached Sentry titled "DATABASE_ERROR: repository" — the driver's message
was only in the innermost exception. retitle now appends the root cause's
value when the root is not a framework error, read from the already-scrubbed
event. The client's message is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…y" label

mapErr wrapped every database failure with core.Wrap(err, "repository"),
which puts the label into Message — the field a response carries — so a
client read {"code":"DATABASE_ERROR","message":"repository"}.

core.WrapAs(err, sentinel, op) reports err as the sentinel (status, code,
message) and keeps op for Error(), Op(), the log and the Sentry title only.
mapErr now answers with errmsgs.DBError's message; the Sentry issue is still
titled "DATABASE_ERROR: repository: <driver error>". Wrap is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Trivy flagged nine advisories on v2/go.mod: ssh DoS in x/crypto, HTTP/2
DoS and flow-control bypasses in x/net, and a malformed-RPC DoS in grpc.
x/crypto v0.58.0, x/net v0.61.0 and grpc v1.84.0 fix all of them.

GO-2026-5932 (x/crypto/openpgp) has no fixed version and is ignored in
v2/.trivyignore: Trivy matches the whole module, but no v2 package
imports openpgp, so the code is never linked.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…r edge cases

Bugs found by a full review of v2, each with a unit test:

- repository.Transaction hands the callback a clean *gorm.DB (no Model/Where
  carried over) and returns an IError from the callback unchanged instead of
  wrapping it as DATABASE_ERROR
- core.Wrap/From keep the status, code and fields of an IError of another
  type (valid.Errors) instead of downgrading it to 500
- Sentry: a shared sentinel (errmsgs.DBError) is never marked as captured;
  scrub keys match hyphenated names (X-Api-Key); AI, Slack and DSN secrets
  are masked by value; job params are scrubbed field by field
- Runner.Stop ends RunContext for a process with no HTTP server and waits
  for the shutdown to finish
- S3 Delete reports per-key failures from DeleteObjects; PublicURL escapes
  the key; the memory storage's ETag is the MD5 like S3's
- Mongo UpdateOne/UpdateMany/ReplaceOne pass ArrayFilters, Hint and
  Collation to the driver; Find/Aggregate release their deadline when
  drained; the command monitor no longer stops logging forever when full
- mongorepo: Updates/Inc/Push/Pull/AddToSet/Unset/DeleteOne refuse an empty
  scope (UNSCOPED_UPDATE / UNSCOPED_DELETE); Pipeline.One wraps
  ErrDocumentNotFound
- jobstore: Update no longer resurrects a purged run; List checks order_by
  against the run columns and leaves the caller's options alone; the
  "trigger" column is quoted; the sqlite tests run under `make test`
- context: SetUser/GetUser are synchronised, WithContext carries the data
  bag, NewApp(nil) uses an empty environment
- memory pub/sub cannot send on a closed channel; valid.SetMessage is
  guarded by a mutex; the scheduler guards its job map
- Auth middleware clones the verifier's user before stamping the token;
  BindOnly also binds `header:` fields; truncation never splits a rune;
  CSV float32 prints at its own precision; the postgres DSN quotes values;
  the tap handler honours WithGroup; devtools hides *_pass/*_key and URLs
  carrying credentials; basic auth keeps the code's user when the password
  comes from the environment; postmangen recognises BindOnly
- coretest writes its env to a file instead of t.Setenv, so fixtures work
  under t.Parallel()

BREAKING CHANGE: visible behaviour that consumers may have relied on:

- A SQL unique-constraint violation is 409 DUPLICATE_KEY (errmsgs.DuplicateKey)
  instead of 500 DATABASE_ERROR; FK violations are 409 FOREIGN_KEY_VIOLATED
  and check constraints 400 CHECK_CONSTRAINT_VIOLATED (gorm TranslateError is
  on). Migrate: branch on the new codes, or keep a pre-insert existence check.
- Requester.Send maps a cancelled caller context to 499 REQUEST_CANCELED and
  an exceeded deadline to 504 UPSTREAM_TIMEOUT instead of 500 NETWORK_ERROR.
- GetPageOptionsWithAllowed() with no columns allows no client ordering;
  use GetPageOptions() for "any column".
- mongorepo single-document writes without a filter return 400 instead of
  changing an arbitrary document; use UpdateAll/DeleteAll/Upsert/Save on
  purpose.
- repository.Transaction's tx no longer carries the repository's Model and
  Where; build repositories on it with NewWithDB as documented.
- coretest.WithEnv values no longer override APP_* variables already set in
  the process; the process environment wins, as for a real service.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ers, retry, probe log skipping

- core.NewMemoryMQ(): an in-process broker (direct/fanout/topic/headers
  routing, bindings, dead-lettering, TTL, MaxLength, Mandatory). App.NewMQConsumer
  consumes from it when the App's publisher is a MemoryMQ, so a service's
  OnQueue/Start/Requeue wiring is tested without RabbitMQ. Messages(queue),
  Published() and Reset() are the assertion helpers. The RabbitMQ consumer's
  dispatch is shared with it (runMQHandler / logMQOutcome).
- IStorage.PresignPost: a browser upload form with a policy the bucket
  enforces (content type or prefix, size range, key prefix, metadata,
  redirect). Implemented for S3, the memory storage and the disabled one.
- core.RateLimit(cache, limit, window, opts...): fixed-window limiter on
  ICache.Incr with X-RateLimit-* and Retry-After headers, per route per key,
  fail-open. core.SecureHeaders(): X-Frame-Options, nosniff, Referrer-Policy,
  HSTS on TLS only, optional CSP/Permissions-Policy.
- core.Retry(ctx, RetryPolicy, fn): exponential backoff with full jitter,
  RetryStop, RetryIf, OnRetry; *Error.WithRetryAfter / RetryAfterOf carry a
  429's wait. Requester retries transport failures, 429 and 5xx for
  idempotent requests via HTTP_RETRY_COUNT/HTTP_RETRY_WAIT_MS/
  HTTP_RETRY_MAX_WAIT_MS, honouring Retry-After. Mailer retries transient
  send failures via EMAIL_RETRY_COUNT / WithMailRetries. The Slack driver
  waits for a short Retry-After and tries again (CHAT_SLACK_RETRIES,
  CHAT_SLACK_RETRY_MAX_WAIT).
- LOG_REQUEST_SKIP_PATHS / HTTPOptions.SkipRequestLogPaths drop the access
  line for successful requests on the listed paths (/healthz, /readyz,
  /_dev/*); a failing probe is still logged.

BREAKING CHANGE: IStorage gained the PresignPost method. A type outside this
module that implements IStorage itself must add it (delegate to
noopStorage-style StorageDisabled, or implement it); the framework's own
NewStorage/NewMemoryStorage callers are unaffected. The Slack driver now
retries a 429 by default (2 attempts, waits up to 10s); set
CHAT_SLACK_RETRIES=0 to keep the previous behaviour.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Ported from idin-core 5a1540f.

- toolchain go1.27.2 fixes the 12 stdlib vulnerabilities govulncheck reported
  against 1.27.1 (net/http, http2, crypto/tls, html/template)
- direct dependencies updated: aws-sdk-go-v2, koanf, goai 0.10.8,
  google api 0.304, plus x/net 0.61

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added module: v2 Changes the v2 module (github.com/pskclub/mine-core/v2) area: docs Documentation breaking Breaking change -- needs migration notes type: feature New capability or API -- Features in release notes labels Oct 23, 2026
govulncheck reports GO-2026-6809 (crypto/x509 trusts compromised
intermediates) against go1.27.2, reachable through the HTTP server.
go1.27.3 fixes it.

golangci-lint v2.13.2 cannot read the export data that the go1.27.2+
standard library ships ("export data version 5 is greater than maximum
supported version 4"), so every package failed typecheck. v2.15.0 reads it
and reports no issues on v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the area: ci CI, release tooling, scripts or the Makefile label Oct 23, 2026
@pskclub
pskclub merged commit 627ca06 into master Oct 23, 2026
12 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI, release tooling, scripts or the Makefile area: docs Documentation breaking Breaking change -- needs migration notes module: v2 Changes the v2 module (github.com/pskclub/mine-core/v2) type: feature New capability or API -- Features in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址