Skip to content

WebSocket pongs update last_heartbeat_timestamp, so a station whose OCPP client is stuck still looks alive #2164

Open

Description

@AlexisG-pixel

WebSocket pongs update last_heartbeat_timestamp, so a station whose OCPP client is stuck still looks alive

Problem

charge_box.last_heartbeat_timestamp is updated from three sources (master, 28e326ed):

Source Location OCPP message?
BootNotification CentralSystemService16_Service.java:130 yes
Heartbeat CentralSystemService16_Service.java:298 yes
WebSocket pong WebSocketEndpoint.java:107 (handlePongMessage) no

A pong is answered by the WebSocket stack of the station, below the OCPP layer (see #245). It proves that the network path and the WebSocket connection are alive, but not that the OCPP client of the station is working. #2121 already relies on this: a station that sends no OCPP message at all still has its last heartbeat moving.

We run a few hundred stations in production and observed a station that had sent no OCPP message for days, while it still had an open WebSocket session and still answered the server pings. During that time:

  • the station had an open session, so it looked connected,
  • its last heartbeat kept being refreshed by the pongs (every ws-ping-interval, 15 minutes by default),

so nothing in SteVe showed that its OCPP client had stopped working. The "Last Heartbeat" column of the charge points list, the "Last Heartbeat Timestamp" of the details page and the "heartbeat today" counter of the statistics (home page and StatisticsRestController) all show such a station as recently seen.

Other OCPP messages (StatusNotification, MeterValues, transactions...) do not update the timestamp either, so it currently means "last sign of life of the connection or last Heartbeat", not "last OCPP activity".

Proposal

Keep a separate timestamp of the last OCPP message received from the station, of any type, so that the cases below can be told apart:

Session Last pong Last OCPP message Meaning
open - recent communicating
open recent old OCPP client stuck, network OK
open old old connection probably dead, not detected yet (see #357, #828)
closed - - offline

Two possible approaches:

  1. Additive (no behaviour change): add a column, e.g. charge_box.last_ocpp_message_timestamp, updated when a text message is received in WebSocketEndpoint.handleTextMessage (and for SOAP in the service layer). last_heartbeat_timestamp keeps its current meaning. To avoid one database write per message, the update could be throttled per station (e.g. at most once a minute) or kept in memory and flushed periodically.
  2. Split the existing meaning: stop updating last_heartbeat_timestamp on pongs and store the pong time in its own column. Clearer naming, but it changes the behaviour for anyone who relies on the current value, so we would rather not propose it without your opinion.

The UI could then show both values. This is also relevant for #2068 / #2158, which expose connected and lastHeartbeat through the API: with the current semantics, that lastHeartbeat includes pongs. The additive approach would not change any existing semantics.

We already need this in our fork and are happy to open a PR against master for the approach you prefer. Would you accept such a change, and which approach fits SteVe best?

Activity

  1. goekay commented on Oct 26, 2026

    @goekay
    Member

    We run a few hundred stations in production and observed a station that had sent no OCPP message for days, while it still had an open WebSocket session and still answered the server pings.

    very interesting problem. thanks for bringing this to our attention.

    as extension to yours, i think i see two potential solutions:

    1. two columns: one for ping-pong, one for last_ocpp_message_timestamp. so, we are generalizing last_heartbeat_timestamp to last_ocpp_message_timestamp which i welcome, but then we lose the tracking of last actual heartbeat, which i would argue is not useful alone after this anyways. my argument is open for discussion.
    2. one column for last_ocpp_message_timestamp (as above), no column for ping-pong. here i argue: do we need to track ping-pong timestamps at all? in operations (including ours, Powerfill) where MeterValueSampleInterval values can be 30 seconds to 1 min, and ping pong intervals quite low (around 1 min as well) to keep connections open, ping-pong timestamp would not that be that informative. if i see a recent last_ocpp_message_timestamp and an open connection, i can derive that the connection is healthy.

    i am leaning more towards 2nd solution. would love to hear others' thoughts.

  2. AlexisG-pixel commented on Oct 26, 2026

    @AlexisG-pixel
    Author

    Thanks for the quick and positive feedback!

    Agreed on generalizing to last_ocpp_message_timestamp: the last actual Heartbeat is not useful on its own.

    On the ping-pong timestamp: a recent OCPP message on an open connection does mean the station is healthy. The pong matters for the opposite case, an open connection with no recent OCPP message. SteVe never checks that pongs come back (PingTask only sends the ping, and a failed send is just logged), so a connection that died without a proper close can stay open on SteVe's side, whatever the ping interval. The pong timestamp is then the only proof that the connection is alive, and it leads to two different actions:

    • recent pong: the station is reachable but its OCPP client is stuck, it needs a reboot,
    • no recent pong: the connection is probably dead, we look at the network or the site.

    With SteVe's defaults (15 min ping, 4 h heartbeat, MeterValues only during transactions), this case is common: a healthy idle station can go hours without any OCPP message.

    Since SteVe already writes a timestamp on every pong, keeping it in its own column costs nothing, so on our side we would rather go for the first option.

  3. goekay commented on Oct 26, 2026

    @goekay
    Member

    okay, i can see the usefulness of it. my new concern is the increased database load.

    so far, before proposed solution, we were hitting the database occasionally in OcppServerRepository#updateChargeboxHeartbeat due to generous ping-pong interval + only tracking actual Heartbeat.

    lets go to a big-ish scale case to better illustrate the scenario: imagine 1000s of stations with very busy/talkative OCPP communications, and a ping-pong interval of 30 seconds.

    with the 1st option, we will suddenly have 2 flows with the database that can be high frequency (low ping-pong interval, talkative OCPP comms) and high volume (we start tracking all OCPP message timestamps from all stations).

    the technical-focused proposal to this would be: having 1 job per flow in memory, that keeps all changes in memory and flushes all timestamps to database every X minutes in one go.

    but i am wondering whether we can scope the changes (i.e. new behavior) from a feature-perspective in a way such that it is cheaper to execute technically, while still providing the value.

  4. reopened this on Oct 26, 2026
  5. AlexisG-pixel commented on Oct 27, 2026

    @AlexisG-pixel
    Author

    Fair point. I think the feature can be scoped so that it does not need to hit the database per message at all.

    Both timestamps (last OCPP message, last pong) can be kept in memory per station, like the WebSocket sessions already are. Live views can read them from memory, like the session list today.

    The database is then only needed for the views that read it (charge point pages, statistics) and for stations that are no longer connected. So the timestamps could be persisted:

    • when a session closes,
    • in one batch every X minutes, which also bounds what a crash can lose.

    The write volume then depends on the number of stations and disconnections, not on how talkative they are or on the ping interval. It would even replace the current database write on every pong in updateChargeboxHeartbeat.

    The trade off is that the persisted timestamps can lag by up to X minutes, which seems acceptable for a "last sign of life" information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址