Hackathon Showcase Finalist

Cactus Commander: Hybrid Voice Agent for Apple Notes

Team led by an Amazon Applied Scientist with an MS in Data Science from Michigan, specializing in NLP, ASR, and Python-based ML systems.

1 member Watch Demo

YouTube Video

Project Description

A voice assistant that turns natural language into Apple Notes actions using a confidence-gated hybrid routing architecture — on-device inference using functiongemma, and cloud fallback only when needed.

How it works:

Hold the Right Command key and speak a command
Whisper (on-device via Cactus) transcribes your speech
FunctionGemma (on-device) picks the right tool and extracts arguments
If confidence is low, Gemini Flash handles the request via cloud fallback
AppleScript executes the action on Apple Notes
A native macOS notification confirms what happened

Team

Chung Hoon Hong

Products & Tools

Cactus + FunctionGemma (Google DeepMind) — on-device tool routing Cactus + Whisper — on-device speech-to-text Cactus Compute Gemini Flash — cloud fallback for low-confidence requests Google DeepMind Python + AppleScript — tool execution layer Result: fast local inference for common cases Stage 1: FunctionGemma on-device via Cactus — handles most requests in ~1-2s Stage 2: Rule-based cross-validation — repairs known model failure modes Stage 3: Gemini Flash cloud fallback — only when on-device confidence is below threshold The stack: cloud reliability for edge cases pynput + sounddevice — push-to-talk input