This is a submission for the Presage SmartSpectra SDK Challenge.
What I Built
Pip is a little octopus who lives in a tide pool in the corner of my desktop. I can talk to it, and it talks back.
Most AI on my desktop works for me, and it only knows what I type or say. Pip is the other half. It notices me: whether I'm smiling or frowning, how often I blink, roughly how fast my heart and breath are going. When I told it "no" with a little smile, it answered: "Funny thing is, you're smiling a little while you say that."
What Pip does:
- Reacts to your face, even when you're quiet. Smile and hearts pop up. Look surprised and it jumps. Frown and it frowns back. Nod and it nods. Blink and it blinks back. Stare at it long enough and it gets shy.
- Talks with you. You speak normally; Pip listens, thinks for a moment, and answers out loud with its mouth moving. Every reply starts with a small action (happy, shy, nod, comfort…) that Pip acts out.
- Hears more than words. Along with what you said, Pip's brain gets a short note of what the camera saw while you said it: expression, blink count, heart and breathing rate, and whether they went up or down.
- Breathes with you. The water in its pool rises and falls with your chest, and it glows faintly with each heartbeat.
It's a pet, not a tool. It can't open files, read the page I'm on, or browse, and that's on purpose. Agents that read your screen and use your computer already exist, and I didn't want to build yet another harness that I'd never use. I built the one piece I hadn't seen: an agent that can read the person in front of it. The idea is that this piece could later plug into other desktop pets and agents, rather than living only inside Pip. So this is a small demo of that one ability.
The octopus is just one look. The sensing doesn't care what the pet is: a fish, a cat, a little robot, whatever you like. Any of them gets the same new senses, your expressions, your heartbeat, your breathing, and can respond to them in its own way. What I'm really adding is more ways for desktop AI to perceive the person; Pip is one way to show it.
Demo
In the video I'm reading DEV. Pip can't see the page; it only sees me. I react to the posts, and Pip reacts to me. A few things it said, unscripted:
- "Hello hello! You look a bit surprised to see me."
- "I'm good, just bobbing about in my pool. You're blinking a lot though."
- "Funny thing is, you're smiling a little while you say that."
- "I can chat, keep you company, and sometimes notice things like a smile or a frown. I can't use your computer or look stuff up yet though."
It isn't perfect: a couple of times it didn't catch what I said, and once it told me my message got cut off. Heartbeat and breathing are there (the glow and the water) but subtle on screen.
Code
Tidepal
Pip is a desktop pet that notices you, not just your words: a little octopus who lives in a tide pool on your desktop. You can talk to it.
Voice assistants only hear your words. Pip also sees how you look while you say them: it uses the SmartSpectra SDK to read your face, blinks, breathing and heartbeat through an ordinary webcam, and passes that along with what you said. So when you say "I'm fine" with a frown, Pip can notice.
- Talk — speak normally. Pip listens (the ring around the pool follows your voice), thinks, and answers out loud, moving its mouth.
- Face — Pip blinks when you blink, blushes when you smile, jumps when you look surprised, nods along when you nod, and gets shy if you stare.
- Breath — the water in Pip's pool rises and falls with your chest; Pip glows with your heartbeat.
- …
Electron app. Setup is in the README: npm install, put your keys in .env, npm start.
How It Responds to Human Signals
The SmartSpectra Node SDK runs inside the Electron renderer and reads the webcam on the device. I request the breathing, cardio and face metric groups and decode the protobuf stream.
| Signal | What Pip does |
|---|---|
| Expressions (happy, surprise, sad, angry) | Mirrors you in real time; tells the chat model how you looked while speaking |
| Blinking | Blinks back; blink count goes to the chat model |
| Talking | Tilts its head and listens |
| Face landmarks | Eyes follow you; nod and shake detection; the "stare" reaction |
| Breathing trace and rate | The water level; rate and trend go to the chat model |
| Pulse trace and rate | The heartbeat glow; rate and trend go to the chat model (stable readings only) |
The conversation loop:
camera ─► SmartSpectra (on device) ─► face, blinks, heart, breathing ─┐
├─► DeepSeek V4.1 Flash ─► reply + action tag
mic ─► cut at pauses ─► speech-to-text (SenseVoice) ─► your words ────┘ │
Pip acts out the tag, macOS speaks the reply
Things I learned on the way:
- The chest matters. Breathing needs your shoulders and chest in view, which fights with how close we sit to a laptop. Expressions and pulse come through fine from the face alone, so Pip leans on those and only nags about the chest when it really needs breathing.
- Raw breathing is jumpy. Drawn as-is, the water jerked up and down. I remove the slow drift, let the size of the swing adapt over a few seconds, and move the water with a soft spring. Now it glides.
- "Face present" should come from expressions. Landmarks sometimes stop while the SDK waits for a better view; expression scores keep coming, so I count those as "I can see you."
- Only stable numbers. Heart rate jumps around for the first seconds. Pip shows "measuring" until the SDK marks the reading stable, and only stable readings reach the chat model.
Prize Categories
Most Creative or Delightful Use.
Consent and Data Handling
- Consent. On first launch Pip explains, in plain words, what the camera and microphone are for, and asks. "Just looking" never turns either on. A button turns the camera off any time. A small mirror shows exactly what the camera sees.
- Video stays on the computer. The SmartSpectra SDK processes frames locally. Nothing is recorded, saved or uploaded. The SDK contacts Presage only to check the key and download its models.
-
SDK telemetry is off (
enableTelemetry: false). - Voice. When you pause, the piece you just said is sent to a speech-to-text service (SiliconFlow SenseVoice) and turned into text. Audio is not saved. The mic is ignored while Pip is thinking or speaking.
- AI service. Your words, plus a one-line description of what the camera saw (for example "face while speaking: frowning; heart rate about 81/min, faster than earlier"), go to DeepSeek V4.1 Flash to write Pip's reply. Nothing else is sent.
- Retention. The conversation lives in memory and is gone when you quit. Only your consent choice and language are saved locally.
- Not medical. Pip is told never to diagnose or interpret your numbers, and to say it's just a little octopus if you ask about health.
What's next
Pip is a pet on purpose. But the same "eyes for the person" could sit next to other agents:
- Coding agents. Codex already has desktop pets: little companions that float over your windows and react to what the agent is doing, "thinking" or "task done". They look at the work. A pet with Pip's eyes could also look at the person: notice you frowning at a diff, or grinning when the tests pass, and respond to that. Coding agents like Codex or Claude Code already have the computer and the web; add how you're doing, and an agent could slow down and explain more when you look lost. I don't know of anyone doing this yet, and it seems like a fun interaction to try.
- Speaking practice. When you practise a language or a talk, the moments you frown or hesitate are the parts you haven't figured out yet. Pip could mark them and go over them with you afterwards.
- Coaching. A coach could get a small timeline of your expressions next to the transcript, to see which questions landed and which didn't.
- Calm moments. A short breathing break where the tide follows your breath. I built a version of this and set it aside for now.
Top comments (0)