Repository navigation
fix: recover from panic when Jellyfin is unreachable during auth - #62
Merged
Merged
Conversation
hrfee/mediabrowser's Authenticate() does an unchecked type assertion on the response body (respData["AccessToken"].(string)) and panics with a nil pointer dereference when Jellyfin is unreachable and returns a non-JSON body instead of a clean connection error. Gin's global recovery middleware was catching this, but only as a generic 500, and with jellyfin_login enabled every admin login goes through this path, so a Jellyfin outage blocks admin access to the panel entirely, including for sending downtime notices to users, which defeats the point of the admin panel being separate from Jellyfin in the first place. Adds a local recover() around both call sites (validateJellyfinCredentials in auth.go, and the password-confirmation call in ChangeMyPassword) that turns the panic into a clean 503 instead.
4 tasks
jay739
added a commit
that referenced
this pull request
Jul 16, 2026
…63) app.jf.MustAuthenticate previously called Fatalf on failure, so if Jellyfin was down or unreachable at container boot, omnifin would retry a few times then die -- and with a restart-unless-stopped policy, crash-loop indefinitely until Jellyfin came back. This defeats the point of the admin panel being usable independently of Jellyfin's own uptime (e.g. sending downtime announcements to users while Jellyfin is mid-maintenance). Checked the vendored hrfee/mediabrowser client source directly: the unchecked type assertion that causes a panic (fixed at both call sites in #62) exists in exactly one place, Authenticate(). Every other MediaBrowser method returns an ordinary Go error, so failing to authenticate at startup just means later app.jf.* calls will keep returning clean errors (some still surfacing as generic 500s -- a separate, lower-urgency error-message cleanup, not a stability risk) until Jellyfin becomes reachable, instead of the whole app never starting.
3 tasks
jay739
added a commit
that referenced
this pull request
Jul 16, 2026
…64) Found live 2026-07-16 while testing #63: the housekeeping and user daemons were panicking every ~60s cycle. hrfee/mediabrowser's GetUsers/UserByID/UserByName internally call Authenticate() themselves whenever the client isn't currently authenticated -- the same unchecked type assertion that panics on an unreachable Jellyfin (fixed at its two explicit call sites in #62) is therefore reachable from any app.jf.* call, not just login/password-change, whenever Jellyfin has been unreachable since the last successful auth. That's now the normal state after #63 made a failed startup auth non-fatal, so every daemon cycle and every handler that reads Jellyfin user data was panicking silently, recovered only by gin's global middleware. Adds safeGetUsers/safeUserByID/safeUserByName wrappers on appContext that recover locally and return a clean error, and switches every call site across the app (15 files) from the raw app.jf.* method to the wrapped version. The wrappers themselves still call the real app.jf.* methods internally, so behavior when Jellyfin IS reachable is unchanged.
4 tasks
jay739
added a commit
that referenced
this pull request
Jul 16, 2026
Found live 2026-07-16 after #65 shipped and was verified deployed: panic bursts continued unchanged. Re-audited the ENTIRE hrfee/mediabrowser library source this time (every .go file, not just the ones already touched) for the "if !mb.Authenticated { Authenticate() }" lazy re-auth pattern that panics on an unreachable Jellyfin -- previous passes only checked users_shared.go and missed 3 more occurrences: GetActivityLog (system_shared.go), DeleteUser and NewUser (mediabrowser.go). That's 5 total unsafe entry points in the whole library; #62/#64/#65 covered 2 of them. GetActivityLog is the actual source of tonight's recurring panic bursts: JFActivityCache.MaybeSync calls it on every cache refresh, triggered whenever the admin panel's activity widget polls (every 30s per admin.ts) or any handler reads activity data. Adds safeGetActivityLog (jf_activity.go, since activityLogEntrySource is a locally-scoped interface, not appContext) and safeNewUser/safeDeleteUser (auth.go, alongside the existing wrappers) following the same pattern. Grepped the whole codebase once more for every one of the 5 unsafe methods -- confirmed every remaining raw call site is either a wrapper's own body or genuinely unreachable from a live request (the interactive `omnifin test` CLI tool, dead/commented code).
3 tasks
jay739
added a commit
that referenced
this pull request
Jul 16, 2026
Root cause of tonight's whole investigation, finally traced with a real stack trace instead of log-timing inference: the panic in Authenticate() (unchecked type assertion, #62) is recovered INSIDE hrfee/mediabrowser itself, by a defer mb.timeoutHandler() registered before the panic-prone line, wired up via mediabrowser.NewNamedTimeoutHandler(..., noFail=true) in main.go. It logs "Failed to authenticate with Jellyfin @ ..." via the stdlib log package directly (not app.err/app.debug, which is why this was so hard to trace across #64/#65/#66 -- none of my recover() wrappers in those PRs ever had anything to catch, because the panic never reaches them). Authenticate() returns afterward with a NIL error and a ZERO-VALUE User{}, since the panic happens before mb.Authenticated = true and before the real return statement execute. Every err != nil check downstream sees a clean success and proceeds with garbage/empty user data -- this is why /token/login returned a plain 500 (CreateToken failing on an empty userID) with zero panic trace anywhere, and why #64/#65/#66 never changed the observed behavior at all. Fixes validateJellyfinCredentials (the actual login path) to check app.authJf.Authenticated after the call, not just err -- that's the one reliable signal the library sets only on genuine success. Kept the existing defer/recover() as a backstop in case a future library version removes its own internal recovery. Scope note: #64/#65/#66's wrapper functions (safeGetUsers/safeUserByID/ etc.) likely still have latent versions of this same gap wherever they wrap a method that internally lazy-reauthenticates, but GetUsers/ UserByID/UserByName/NewUser/DeleteUser proceed to their own real HTTP call after a swallowed auth failure, which should independently surface a real error from THAT call instead of silently succeeding -- unlike Authenticate() itself, which has nothing left to do after auth. Left as a follow-up to verify/harden rather than expanding this fix further tonight.
This was referenced Jul 16, 2026
jay739
added a commit
that referenced
this pull request
Jul 16, 2026
Found live 2026-07-16, the second time in one night: the admin panel's Users tab is completely unusable during a Jellyfin outage, even after fixing the underlying panics/500s (#62-#71) -- GetUsers/SearchUsers fundamentally need a live Jellyfin connection to list users, since Jellyfin IS the user database. There's no way to make that endpoint work without Jellyfin reachable. What CAN work: app.storage's own persistent badger DB already stores an EmailAddress record (Jellyfin ID, email, label, contact/admin flags) per user, independent of Jellyfin's own availability. This was previously only accessible via internal code paths (housekeeping cleanup, migrations) with no API route exposing it -- recovering it during tonight's incident required a one-off standalone Go program reading the badger DB file directly. Adds GET /users/persisted, returning every stored EmailAddress record. Trades completeness (no online status, no Jellyfin username, no last-active -- just enough to know who to email) for actually working when Jellyfin is down. Same auth group (webAuth) as every other /users route.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Authenticate()does an unchecked type assertion on the response body (respData["AccessToken"].(string)) and panics with a nil pointer dereference when Jellyfin is unreachable and returns a non-JSON body instead of a clean connection errorjellyfin_login = true, every admin login goes through this path — a Jellyfin outage currently blocks admin access to the panel entirely, including for sending downtime notices to users, which defeats the point of having an admin panel that's meant to be usable independently of Jellyfin's own uptimeChangeMyPasswordin api-userpage.go (user-facing "change my Jellyfin password" flow) has the same panic riskChanges
auth.go:validateJellyfinCredentialsnow recovers from the panic and returns a clean 503 ("Jellyfin unreachable") insteadapi-userpage.go:ChangeMyPasswordwraps itsAuthenticatecall in the same recover-and-respond pattern (503 viarespondBool), scoped to just that call so it doesn't swallow panics from the rest of the handlerRelated
Relates to #56 (audit for remaining upstream jfa-go traces / robustness gaps) — same category of "this fork should hold up under real self-hosted failure conditions" issue.
Test plan
ChangeMyPasswordbehaves the same way with Jellyfin down