Kumo

Kumo is a browser-based design editor for interfaces, diagrams, and connected product flows. Boards can link to other boards directly on the canvas, so a project can include both individual screens and a map of how they fit together. An integrated AI collaborator can build and refine native canvas objects through conversation.

I designed and built the editor, realtime collaboration, backend, workspace tools, and AI integration. The system combines visual editing with product flow analysis, shared design libraries, branch review and merging, and recovery for interrupted sessions.

Frames can describe a screen’s state, user roles, viewport, and requirement references. Kumo builds a graph from supported prototype interactions and linked boards to check for missing state variants, unreachable frames, dead ends, broken destinations, and missing recovery connections. Role and viewport checks use the frame’s annotations.

For example, a checkout flow might include payment and confirmation screens but no way to recover from a failed payment. A coverage policy can require an error-state design and a connection back to a non-error state. These checks inspect the annotated design and its connections; they do not execute the finished application or evaluate every conditional path.

Coverage policies define the required states and passing thresholds. Saved runs can be compared across revisions and exported as JSON, JUnit, or SARIF reports. Optional merge rules check a saved run against the current branch document’s checksum, with recorded owner overrides. An authenticated telemetry endpoint accepts screen and outcome events for comparison with the design.

The editor supports vector paths, boolean operations, masks, layered fills and strokes, gradients, effects, and editable text. Frames, sections, auto layout, wrapping, responsive constraints, guides, and snapping handle layout. Nested components, variants, overrides, and variables support reusable interface elements.

Diagram tools include attached connectors, freehand drawing, sticky notes, tables, and code blocks. CSV, supported SVG elements, and a subset of Mermaid flowchart syntax can become editable canvas objects. A font registry supports workspace font uploads and missing-font replacement. Inspection and SVG, PNG, rasterized PDF, and Kumo document exports support handoff.

The editor’s presentation mode supports condition checks, variables, overlays, scrolling, and component variant changes. It can navigate between frames and connected boards. Separate prototype links support device frames, passwords, expiration, and revocation, using a simpler viewer for click-through screen navigation.

Kumo AI works with the current design and the conversation around it. Users can ask it to create a layout, revise selected objects, or explain what is on the board, then refine the result through follow-up requests. Generated frames, text, shapes, and components remain editable through the same tools used for manual design.

Users choose whether the agent can work within a selection, the current board, or the workspace. Replies stream into the chat while a labeled cursor and expandable action history show applied changes. Follow can move the viewport with the agent; touching the canvas returns control to the user. Stop cancels inference and prevents new actions.

I integrated the OpenAI Responses API with a capability registry generated from 180 typed application actions. The agent discovers the actions and schemas it needs, then issues validated calls to existing editor and workspace functions. Additional tools inspect the canvas, create and patch objects, navigate boards, and check document structure. Actions return recorded results to the model before it continues.

The implementation has to account for a human editing at the same time. Commands retain their intended selection, check object values against the state the agent inspected, and respect permissions, locks, and active editing claims. Document changes and their action receipts share a Liveblocks storage batch, allowing duplicate operations to be recognized. Run undo reverses eligible changes field by field while preserving later collaborator edits and objects that other content still references.

Authenticated server endpoints keep the provider key out of the browser. PostgreSQL transactions reserve inference allowance across account, run, daily, and monthly limits before a request starts, then reconcile reported usage. Rate limits, concurrency limits, and bounded action batches constrain the work a run can perform. Uncertain paid requests are recorded for review rather than retried automatically.

Run records retain scope, execution state, and action results so interrupted work can be inspected before continuing. Local file access and actions requiring confirmation hand control back to the user. The AI integration uses a hosted language model; my work is the tool interface, execution system, collaborative editing behavior, and operational controls.

Collaborators can see each other’s cursors and selections, follow a viewport, or spotlight their own view. Comments can attach to the canvas or a layer and move with it, with mentions, replies, reactions, and resolution. Temporary cursor chat, a shared workshop timer, and canvas stamps support live sessions.

During a drag, movement stays in a local preview and commits as a property-level update on release. Shared manipulation claims help prevent two people from transforming the same object at once. Changes to separate properties can merge without replacing the entire board.

Components and design resources can be published as versioned libraries. Other boards can inspect what changed before accepting an update. Releases include descriptions, changelogs, review states, deprecation, and rollback. Reusable board templates provide starting points for new work.

Design branches have separate collaborative rooms, lists of added, changed, and removed layers, and simplified overlay or side-by-side comparisons. A branch can request review, pull in changes from the main design, and resolve conflicts before merging. Reviews are tied to the document revision, and merges record a description and create a recovery checkpoint.

Named versions and visual previews support restoring earlier work. Local recovery snapshots and an IndexedDB journal retain changes made while disconnected. The recovery panel compares the saved baseline, local edits, and current remote document. Users review conflicts and choose whether to apply the recovered changes.

Workspaces organize boards into folders, favorites, saved views, archives, and trash. Profiles, friend requests, notifications, and public board discovery support returning to shared work. Community publishing includes remix permissions and reporting.

Owners can grant viewer or editor access, approve access requests, and create expiring or domain-restricted share links. For existing collaborators, connected-board membership changes use a server-derived graph and a database transaction. The sharing scope excludes destinations the owner cannot manage. Friendship and board access are separate permissions.

React and TypeScript power the client, with Redux for local editor state. Liveblocks owns the collaborative canvas and presence; Supabase Postgres owns workspace data, permissions, snapshots, and library releases. Firebase Auth supplies identity, and authenticated Vercel functions enforce access before issuing room sessions or changing product data.

Keeping live document state separate from durable metadata avoids having two services write the same fields. Recovery, branch merges, and library updates explicitly handle the boundary between them.

The test suite covers editor geometry, concurrent manipulation, permissions, branch conflicts, offline recovery, and product coverage rules. AI-specific tests check capability schemas, selection boundaries, action ordering, streamed replies, duplicate receipts, undo, and spending reservations. A deployment canary uses two real collaborative clients with deterministic model replies to check progressive edits and undo after a human changes an object.

Browser checks exercise desktop and mobile layouts, keyboard access, and collaboration workflows. CI also verifies database migrations, bundle budgets, accessibility checks, and preview deployments before release.