Back to the journal
Introducing Loom: Looki’s Multimodal Model for Personal Context

Introducing Loom: Looki’s Multimodal Model for Personal Context

Loom connects first-person perception, experience understanding, and long-term memory to turn everyday observations into context AI can use.

Everyday life produces information that may only become useful later. For personal AI, the challenge is to understand those experiences and turn them into context that can be retrieved when needed.

Looki is introducing Loom, a multimodal model family developed in-house by Looki for first-person perception, experience understanding, and long-term memory. Loom connects fragmented visual, audio, and timing information to build personal context that AI can understand, retrieve, and use.

The goal is to help Looki users find relevant details and pick up past conversations with less searching through recordings and less need to explain the background again.

From Recording to Remembering When It Matters

Imagine discussing a proposal with colleagues over lunch. Near the end, someone says, “Let’s wait for Friday’s data before deciding.”

A few days later, you ask, “Why didn’t we decide last time?” Recovering that earlier condition would let the discussion continue from where it left off.

In a Q&A-style interaction, users typically initiate a request and provide the materials needed to answer it. Everyday life is less structured. The reason a plan was postponed, where an object was placed, or a condition attached to a decision may only become important days later.

Comparison of Q&A-style AI requiring user-provided materials and real-life scenarios requiring context to be linked across time.
Information from everyday life may become useful long after it first appears.

Loom takes its name from the weaving loom. Its purpose is to bring together threads from different senses and moments into personal context that AI can understand, retrieve, and use.

How Loom Builds Personal Context

A multimodal model processes information from different types of input. Loom brings together visual and audio observations with timing information to build personal context through three connected layers:

  • Loom Sense understands what is happening now.
  • Loom Moment organizes an experience with its timing, content, and supporting evidence.
  • Loom Memory retrieves past experiences relevant to the current question.
Loom architecture showing how Sense, Moment, and Memory turn continuous observations into personal context, with proactive intelligence as a future direction.
Sense, Moment, and Memory work together to build personal context, with proactive intelligence as a future direction.

Real-world observations are rarely clean or continuous. Movement and occlusion interrupt visual evidence. Images and audio may cover different time ranges. One experience can span several recordings, while multiple activities can overlap within the same scene.

These conditions require models to handle sparse, asynchronous, and noisy inputs while distinguishing direct evidence from inference.

Loom Sense: Understanding What’s Happening Now

Someone sitting at a table could be working, eating, or taking a break. Audio can help distinguish these activities, but hearing people speak does not necessarily mean the wearer is participating in a conversation.

Loom Sense combines recent visual and audio observations, timing information, and the previous observation window’s state. It supports concurrent events—such as eating and talking—and updates those judgments as new evidence arrives.

Defining Events Through Natural Language

Loom Sense uses natural-language questions to define the events it should detect. The model scores candidate answers directly, rather than generating a text response and then extracting a label.

For event question qj, the candidate-answer probability is:

pθ(a | Xt, Tt, qj, St−1) = softmaxa ∈ Aj ( zθ(a | Xt, Tt, qj, St−1) τ )

Here, t denotes the current observation window, and θ denotes the model parameters. Xt represents recent observations, Tt timing information, and St−1 the previous window’s state. For question qj, Aj is the candidate-answer set and a is one candidate answer, such as “yes” or “no.”

The model assigns each answer a raw score zθ. Softmax converts the scores into probabilities within that question’s candidate set, with temperature τ controlling the distribution.

Because answers are compared within each question, “Is the wearer eating?” and “Is the wearer talking?” can both receive positive predictions.

Observation encodings can also be shared across questions. Adding an event question requires incremental scoring work without repeating the full visual encoding process, helping control the cost of continuous detection.

Maintaining Continuity While Detecting Change

A brief turn away from the table should not immediately end an eating event. Leaving the table should.

Loom Sense passes prediction features and results between observation windows. New observations and elapsed time determine whether the previous state still applies.

Training balances stability within an activity with responsiveness at its boundaries:

ℒ = ℒevent + λ(1 − bt)D(pt, pt−1) + μbtℒswitch

The event loss ℒevent supervises the current event prediction. D(pt, pt−1) measures the difference between adjacent prediction distributions and can use KL divergence.

The value bt ∈ [0,1] represents how close the current window is to a true activity boundary. Within an ongoing activity, the consistency term discourages unsupported fluctuations. Near a boundary, this constraint weakens while switching supervision ℒswitch receives greater weight. The coefficients λ and μ control the two auxiliary terms.

Training data targets errors caused by changes in wording, people, environments, and activity stages. Events unseen during training still require separate validation; a natural-language interface alone does not establish recognition accuracy.

Accuracy, Speed, and Cost

On an internal business dataset covering nearly 20 event categories, Loom Sense achieved an overall F1 score of 88.6 and an average detection time of 0.51 seconds.

Model Overall F1 ↑ Average detection time, seconds ↓ Relative cost index ↓
Loom Sense 88.6 0.51 5.88
Qwen 3.8 Flash (closed-source) 85.2 2.40 100
D3-Nano (open-source) 72.8 1.29 14.87

Results are from Looki’s internal evaluation. F1 is reported on a 100-point scale.

Under the evaluation’s cost model, with the closed-source reference set to 100, Loom Sense’s cost index of 5.88 corresponds to an estimated 94.12% reduction in inference cost.

Loom Sense demo showing first-person footage, event scores, and event timelines.
Loom Sense tracks event scores across first-person observations. Illustrative example; not user data.

For users, the aim is a more coherent activity timeline, with fewer false interruptions caused by brief changes in view.

Loom Moment: Turning Observations into Experiences

Event labels alone cannot explain why a proposal remained undecided. The system must preserve what was discussed, the outcome, and the condition attached to it.

Loom Moment organizes multimodal observations into experience segments with start and end times, content descriptions, and supporting evidence.

Learning Boundaries and Content Together

Fixed-duration cuts can separate a decision from its condition. Splitting only when the image changes can miss a shift from casual conversation to a project discussion at the same table.

Loom Moment jointly models where an experience begins and ends and what it contains:

ℳ = Gϕ(V, A, T, H) = {(sk, ek, ck, Ek)}k = 1K

The model Gϕ, with parameters ϕ, receives visual observations V, audio segments A, timing information T, and context H. This context includes preceding information and relevant user memories, such as known people and entities, recent activities, and unresolved matters.

The output ℳ contains K experience segments. Each segment k includes a start time sk, end time ek, content description ck, and supporting evidence Ek.

Generating boundaries and descriptions together lets them inform one another, reducing errors introduced when segmentation is completed before content understanding.

Grounding Experiences in Time and Evidence

Timing is part of understanding. Observing an object on a table differs from seeing someone pick it up and then put it there. Visual and audio inputs must be placed on a shared timeline to establish what happened.

Loom Moment distinguishes what was seen, heard, or inferred. A scene that resembles a meeting cannot establish that a particular sentence was spoken. Prior memories can help interpret the situation, but they cannot replace current observations as evidence.

Training first focuses on boundaries and event types, then adds detailed descriptions and evidence attribution. Further data designs test temporal mismatches, missing modalities, and different sampling patterns. If the condition about Friday’s data appears only in audio, removing that audio should also remove the condition from the target description.

Evaluating Boundaries, Descriptions, and Evidence

Moment is evaluated on whether experiences are segmented correctly, important facts are preserved, and descriptions can be traced to evidence.

Segment F1 is reported at temporal intersection-over-union thresholds of 0.5, 0.7, and 0.9. Temporal intersection-over-union measures the overlap between predicted and reference time intervals relative to their combined duration. Higher thresholds require closer temporal alignment.

Model Segment F1@0.5 ↑ Segment F1@0.7 ↑ Segment F1@0.9 ↑
Qwen3-Omni-30B-A3B-Instruct (open-source) 57.0 38.3 16.3
Gemini 3.5 Flash (closed-source) 70.4 53.5 35.1
Loom Moment 74.7 62.8 46.9

Segment F1 results are from the joint segmentation-and-description evaluation.

After aligning predicted and reference segments, description evaluation checks whether the main activities, participants, and important stages have been preserved, and whether the descriptions are supported by the source material.

Model Description consistency score ↑ Evidence-attribution accuracy ↑
Qwen3-Omni-30B-A3B-Instruct (open-source) 69.1 56.0%
Gemini 3.5 Flash (closed-source) 80.9 74.2%
Loom Moment 73.6 78.6%

Description consistency is scored by an LLM judge; evidence-attribution accuracy is measured separately.

Loom Moment achieved higher segmentation scores across all three thresholds. Its description consistency exceeded the tested Qwen model but remained below Gemini Flash, while evidence-attribution accuracy was higher than both.

For users, the aim is to preserve the details that make an experience useful later: what was decided, what remained unresolved, and which observations support the record.

Loom Memory: Bringing Relevant Experiences Back

“Where did I travel over the past year, and what did I eat?” requires retrieving information scattered across different trips. A landscape may appear in an image, while the name of a dish is mentioned in conversation.

Loom Memory organizes these experiences into long-term memory, connecting information across time and choosing retrieval paths based on the question.

Organizing Memory at Different Timescales

Layer Role
Moments Specific experiences, timestamps, and supporting evidence
Summaries Daily, weekly, and monthly views of activities and changes
Pending Memory New information and user corrections awaiting integration, with sources and timing
Snapshots An updated understanding of the user, including preferences and recent priorities

Summaries provide a path from broad patterns to specific experiences. Pending Memory and Snapshots incorporate new information, resolve conflicts, and update outdated understanding. The original experiences remain available for retrieval and verification.

Users can refine that understanding through feedback. “This trip was to visit family” adds background, while “Show more photos in travel recaps” expresses a preference for future responses.

Navigating Memory Around the Question

Through Agentic Memory Navigation, Loom Memory plans how to search across different timescales.

For a year-long travel recap, it can begin with monthly summaries, identify trips, and move into daily summaries and individual Moments. A question about dinner on the first night in Kyoto can take a narrower route. A question about a menu may require returning to an original image for details absent from a description.

Search adapts to the evidence found. When a summary is insufficient, the system retrieves more detailed records. When the available information is enough, it can stop. Relevant findings are passed to Looki Agent as context for answering the user.

Evaluating Long-Term Recall

Looki built a long-sequence audiovisual memory benchmark using questions about participants’ real experiences and recordings processed to remove or mask sensitive information, with their informed consent.

The evaluation covers preferences, past events, audiovisual details, and whether memories can support judgments about appropriate moments to help.

We compared looki memory with mem0 and A-Mem, evaluating answer quality and average input and output token usage per question.

Memory system Composite score, out of 100 ↑ Average tokens processed per question ↓
mem0 61 98,617
A-Mem 64 46,828
looki memory 79 14,282

Token usage includes average input and output tokens per question in this evaluation.

looki memory achieved a composite score of 79, exceeding mem0 by 18 points and A-Mem by 15 points. Its token usage was 14.5% and 30.5% of theirs, respectively.

Toward Proactive Intelligence

Helpful assistance depends on both relevant information and appropriate timing.

We are exploring more end-to-end multimodal modeling that brings together current observations, user goals, and retrieved historical evidence. This direction aims to connect situation understanding with decisions about whether and how to help.

Possible actions include remaining quiet, continuing to observe, offering a reminder, asking a clarifying question, or calling a tool. The earlier condition about Friday’s data may be useful when the proposal comes up again. During an unrelated activity, it should remain in the background.

Sense, Moment, evidence grounding, and memory retrieval can provide supporting signals for this work. This is a direction for further development; it does not mean every form of proactive assistance is already available.

Privacy and System Boundaries

Personal context requires clear boundaries between model capabilities and system responsibilities.

Models interpret multimodal situations, connect evidence, and generate assistance. The system manages long-term storage, access permissions, retrieval infrastructure, and reliable tool execution.

Looki will never use users’ private personal data to train our foundation models.

The end-to-end architecture we are exploring is designed around the following privacy requirements: inference must process only the minimum context needed for the current task, within a user-authorized local or secure, trusted environment. The design also requires system-level encryption and access isolation for long-term records, identity associations, and sensitive information, with no model access beyond authorized permissions.

Explore Loom and Looki L1

Loom brings continuous perception, experience understanding, and long-term memory into a shared approach to personal context. We will continue developing these capabilities around real devices and everyday situations, with an emphasis on useful details, traceable evidence, and well-timed assistance.

Loom workspace with video upload controls and analysis settings.
Explore Loom in the browser-based workspace.

Explore Loom’s model capabilities or discover the Looki L1.

The Loom experience is continuing to evolve, with more functionality to come.

Have questions about Loom or Looki? Join the Looki community.

Join Discord ↗
Share this story

0 comments

Leave a comment