Skip to content

Recovery settles a joined delivery with the earlier answer when the process stopped before the model replied to it #896

Open

Description

@NathanDrake2406

Describe the Bug

When a host dispatches a message to a busy instance, the message joins the live response. If the process stops after the runtime records the joined message but before it records the model's reply, recovery settles the host submission and the joined submission as completed. It does not call the model again. Nobody answers the joined message, and read() on its receipt returns the answer to the earlier message.

agent-hooks-api.md L247-L248 says a joined delivery is "read by the model on its next turn" and "is never lost".

From the v2.3.0 source:

#810 fails in the same recovery check, where a completed assistant message settles the response as a success. Its trigger is an interrupted finish hook, not a joined delivery.

Expected Behavior

Recovery should treat a joined user message that has no assistant reply after it as pending input. It should resume the response so the model answers the joined message, as it would without the crash, and settle both submissions with that response.

Steps to Reproduce

  1. Run npm i @flue/runtime@2.3.0 @earendil-works/pi-ai@1.4.0. This resolves @earendil-works/pi-agent-core to 1.4.0. We ran it on macOS with Node 24.22.2, and main at deee93b has the same code.

  2. Save the script below as repro.mjs. It uses pi-ai's faux provider as the model and a local SQLite file as the store, so it makes no network calls. In crash mode the process exits while the model handles the joined message. A SIGKILL at the same point gives the same result.

    import { readFileSync, writeFileSync } from 'node:fs';
    import { fauxAssistantMessage, fauxProvider } from '@earendil-works/pi-ai/providers/faux';
    import { init, useModel } from '@flue/runtime';
    import { sqlite, start } from '@flue/runtime/node';
    
    const mode = process.argv[2]; // crash | resume
    const faux = fauxProvider();
    let modelCalls = 0;
    
    function Repro() {
      useModel('faux/faux-1');
      return 'Answer the latest user message.';
    }
    
    const runtime = await start({ agents: [Repro], providers: [faux.provider], db: sqlite('./repro.sqlite') });
    const handle = init(Repro, { id: 'repro' });
    
    if (mode === 'crash') {
      const entered = Promise.withResolvers();
      const release = Promise.withResolvers();
      faux.setResponses([
        async () => {
          entered.resolve();
          await release.promise;
          return fauxAssistantMessage('Answer to the first question.');
        },
        // The process stops while the model handles the joined message.
        () => process.exit(73),
      ]);
      const first = await handle.dispatch('First question.');
      await entered.promise;
      const second = await handle.dispatch('Second question.'); // joins the live response
      writeFileSync('./receipts.json', JSON.stringify({ first, second }));
      release.resolve();
    } else {
      faux.setResponses([
        () => {
          modelCalls++;
          return fauxAssistantMessage('Answer to the second question.');
        },
      ]);
      const { first, second } = JSON.parse(readFileSync('./receipts.json', 'utf8'));
      const [one, two] = await Promise.all([handle.read(first), handle.read(second)]);
      console.log({ modelCalls, first: one.text, second: two.text });
      await runtime.stop();
    }
  3. Run node repro.mjs crash. It exits with 73. By then the stored stream holds the first user message, the completed first answer and the joined Second question. user message.

  4. Run node repro.mjs resume. Recovery starts once the dead process's 30-second lease expires, and the script prints:

    {
      modelCalls: 0,
      first: 'Answer to the first question.',
      second: 'Answer to the first question.'
    }
    

    The expected output has modelCalls: 1 and Answer to the second question. in the replies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址