Contextual Chunk Retrieval Without Special Embeddings

Agentic Memory Architecture — Part 1

Updated October 16, 2026. The stored side now runs in production, and this post adds its code and results from 51 real threads.

Contextual chunk retrieval stores every small piece of text as one vector: 80% the piece's own embedding and 20% the embedding of the passage around it. Queries are built the same way. The incoming message is split into parts, each part is blended with the whole message, and each part runs its own search and keeps its own results.

Everything here runs on an embedding endpoint that returns one vector per input. We use OpenAI's text-embedding-3-small. No token-level access, no special model, and no generated context text.

This post covers both sides, the code we run, and what it measured.

The problem it solves

Our assistant missed my dogs' names in a voice turn. The names were stored, and a direct search for them worked. The same turn also carried a long engineering correction, so the one vector for the whole message pointed at engineering, and the dog facts lost the ranking.

That failure has two parts, and each needs its own fix.

Coverage. One query vector gives one search direction, and one ranked list gives every result slot to whatever dominates that direction. A subject can be in the message and still get no memories. The fix is to split the query and give each part its own search.

Meaning. Small pieces fix coverage but lose context. "We need to handle this email." can mean approve the draft, send it, confirm the recipient, or fix a tracking link. Embed that sentence alone and all four occurrences get the same vector. No query can recover a distinction that was thrown away at storage time. The fix is to carry some of the surrounding passage in each piece's vector.

The blend

Let E(text) be an embedding from one model at one dimension, and unit(v) scale a vector to length one. For a piece s inside a passage C:

text
stored = unit(0.8 * unit(E(s)) + 0.2 * unit(E(C)))

For a query part q inside a message P:

text
query = unit(0.8 * unit(E(q)) + 0.2 * unit(E(P)))

Normalize both inputs before mixing. Otherwise their lengths change the effective weight. Here is the function both sides call:

ts
export const CHUNK_FOCUS_WEIGHT = 0.8

function unit(vector: number[]): number[] {
  const norm = Math.sqrt(vector.reduce((sum, value) => sum + value * value, 0))
  return norm > 0 ? vector.map((value) => value / norm) : vector
}

export function chunkSearchVector(chunk: number[], message: number[]): number[] {
  const a = unit(chunk)
  const b = unit(message)
  return unit(a.map((value, i) => CHUNK_FOCUS_WEIGHT * value + (1 - CHUNK_FOCUS_WEIGHT) * (b[i] ?? 0)))
}

The result has the same dimension as the input, so any cosine index searches it unchanged.

When both sides are blended, the dot product of a query and a stored vector expands into four terms. Before the final normalization:

text
0.64 * dot(query_piece,   stored_piece)
+ 0.16 * dot(query_piece,   stored_context)
+ 0.16 * dot(query_context, stored_piece)
+ 0.04 * dot(query_context, stored_context)

Piece-to-piece matching carries most of the score. The two cross terms let each side's context lean on the other side's piece. The last term rewards two passages that agree. That's what lets the same sentence stored in two contexts rank differently for a detailed query.

The blend works on finished vectors. The piece's tokens never see the passage inside the model, so it isn't late chunking. It can only keep distinctions the embedding model already captures.

Query side: split the message

splitPromptChunks cuts the incoming message into at most 8 parts:

Split on line breaks and after ., ! or ?.
Cut any piece longer than 60 words into windows of 40 words. Unpunctuated voice transcripts hit this often.
Join any piece shorter than 24 characters to the piece before it. "Or anything about them?" isn't a search on its own.
If more than 8 pieces remain, join neighbours into 8 runs, so every part of a long message still gets searched.

Embed the parts and the whole message in one batch request. Then blend each part with the whole message using chunkSearchVector. The whole-message input is capped at 8,000 bytes, keeping the end, because the latest text is usually the question. Cap by bytes, not characters. A character cap lets CJK text run past the model's 8,192-token limit.

Stored side: labelled pieces

Each sealed turn of agent history becomes a set of pieces.

Split the turn by role. The user message, each assistant step, and each tool call are separate sources. A tool call's text is its arguments plus its result, and the result is kept whole.

Pack by sentence up to a byte budget. Sentences go into a piece until the next one would overflow 1,200 bytes. A sentence over the budget is cut at the last word that fits. Nothing gets dropped.

ts
export function packTextPieces(text: string, maxBytes: number): string[] {
  const pieces: string[] = []
  let current = ''
  for (const line of text.split(/\n+/)) {
    const sentences = line
      .split(/(?<=[.!?])\s+/)
      .map((s) => s.replace(/\s+/g, ' ').trim())
      .filter(Boolean)
      .flatMap((s) => cutWithinBytes(s, maxBytes)) // over-budget sentence → cut at the last word that fits
    sentences.forEach((sentence, index) => {
      const joiner = !current ? '' : index === 0 ? '\n' : ' '
      if (current && bytes(current) + bytes(joiner) + bytes(sentence) > maxBytes) {
        pieces.push(current)
        current = sentence
      } else {
        current += joiner + sentence
      }
    })
  }
  if (current) pieces.push(current)
  return pieces
}

Label every piece. Each piece starts with a header naming its turn, its source and its time, for example [Turn 4 · Tool logistics_export · Oct 16, 2:02 PM]. The header is part of the embedded text, so a tool result and a user request about the same thing still embed differently. Subtract the header's bytes from the budget before packing.

Blend with the turn. For the context vector we use the mean of the turn's piece vectors, not an embedding of the turn's text:

ts
const context = meanVector(turnPieceVectors)          // mean of unit vectors, then unit
const stored = turnPieceVectors.map((v) => chunkSearchVector(v, context))

We chose the mean for two reasons. It needs no extra embedding call. It also never hits the input limit, because a long tool-heavy turn can exceed 8,192 tokens while its pieces never do. We measured the alternative, embedding the turn's text as the context. It scored 0.5 points higher, which wasn't worth an extra call per turn and a cap on large turns.

Size requests up front. One embedding request takes at most 2,048 inputs. We cap each batch at 2,048 pieces and 300,000 bytes before sending, so no request is refused for size.

Pieces are stored at 256 dimensions. The model's dimensions setting is a truncation of its full vector, so you can cut a 1,024-dimension vector to its first 256 numbers and renormalize to get the same result.

What it replaced

The old indexer stored one vector per turn. It aimed for chunks of about 200 words but only split between turns, so any turn of 200 words or more became one chunk however large it was. Tool arguments were cut to 500 characters, and tool results over 4,000 characters were left out. A search for one detail in a big turn scored against the average of the whole turn. Turns over 8,192 tokens were refused by the embedding model and had no vector at all. That was 6% of the old chunks.

We rebuilt the existing archive once: 2,710 sections became 93,042 pieces, with no failed embeddings.

Search: every part gets its own group

For each query part i:

text
group_i = top_X(stored vectors with cosine(query_i, v) >= threshold)

Build the part's blended query vector.
Search in parallel with the other parts.
Keep up to X matches above the threshold.
Attach them under that part's group in the prompt.

Every part has the same allowance. A part with fewer qualifying matches returns fewer, and we don't pad a group with weak results. Keep the groups through prompt assembly. A final global top-X would put the parts back in competition. We don't deduplicate across groups. A memory that supports two parts appears in both.

Our graph-fact search has run this way since October 9: X = 5, cosine ≥ 0.35, at 1,024 dimensions. The 0.35 floor came from the dogs case. The dog-name fact scored 0.38.

The search over the labelled pieces still takes one whole-message vector and keeps one winner. Moving it onto the same split is the next change, and we'll measure it with real messages as queries before it ships. When it does, both sides of every comparison will have the same shape: a sentence-sized part carrying 20% of its message against a small piece carrying 20% of its turn.

Results

Controlled test

The first test made the sentence alone insufficient. We wrote 96 synthetic records across 24 sentence families. Each family used the same sentence in four different contexts. Eight families were for development and sixteen for evaluation: 128 context-specific queries and 16 generic ones.

The records came in two blocks, searched separately. The first block had 64 records using different senses of words like "key", "thread" and "session". The second had 32 records using the same subject for different purposes, for example rotating, scoping, storing or auditing an API key. Fixtures and target labels were fixed before embedding, and IDs and labels were never embedded.

We used text-embedding-3-small at 1,024 dimensions with exact cosine search. No word search, no reranker, no approximate index. Noisy conditions kept the relevant text and added five unrelated sentences.

Correct-context top-1 accuracy on the 128 evaluation queries:

Representation	Clean	Stored distractions	Query distractions	Both distracted
Isolated sentence on both sides	25.0%	25.0%	25.0%	25.0%
Whole passage and whole query	100.0%	94.5%	57.8%	36.7%
80/20 blend on both sides	100.0%	96.1%	71.9%	50.8%

The isolated-sentence row ties four ways in every family, so each tied candidate gets a quarter of a hit.

The blend helped once unrelated text competed with the intended context. With query distractions, 92 of 128 first results were correct, against 74 for whole passages. With both sides distracted it was 65 against 47. Both gaps are 14.1 points. On clean text the whole passage already scored 100%.

Controlled retrieval results comparing isolated sentences, whole-context embeddings, and weighted contextual blends, with a separate measure of generic-query coverage.

Generic queries contained only the shared sentence. With blended stored vectors, all four contexts took the top four positions in every evaluation family: 64 of 64.

The stored side matters. Blending only the query, against whole-passage stored vectors, scored 36.7% with query distractions, below the 57.8% baseline.

The 80/20 weight was picked before evaluation. A development-only sweep slightly preferred 95/5, so treat the weight as a setting to tune. The protocol, context fixtures, same-subject fixtures and full rankings are published with this post.

Production threads

After the stored side shipped, we compared it against the old whole-turn chunks on real data:

Threads: 51 production threads, all of them with sealed history.
Old chunks: rebuilt in memory from the same messages with the pre-change code, then embedded.
New pieces: read from production. A recompute matched the stored vectors exactly (cosine 1.0000).
Queries: 1,868 facts already stored for those sections. These are real sentences about what happened in those turns.
Right answer: a hit inside the section the fact came from.
Scoring: vector search only, scoped to the thread, using the same score as production recall.

Measure	Whole-turn chunks	Labelled pieces
Top hit in the right section, above the recall floor	36.8%	58.9%
Top hit in the right section	43.9%	60.6%
Mean reciprocal rank	0.579	0.697

Most of the gain comes from piece size. Small labelled pieces against one averaged vector per turn did the heavy lifting. The 80/20 blend added 1.1 points on top, and embedding the turn's text as context instead of the mean would have added 0.5 more.

One probe shows the mechanism. We buried a detail in a 400-row tool result. The piece holding it ranked first of 39 at cosine 0.59. The old whole-turn vector for the same turn scored 0.45.

A note on the floor: our vector index reports a score of (1 + cosine) / 2, so the 0.75 recall floor is a raw cosine of 0.5. The probe's piece passes. The old whole-turn vector didn't.

This eval measured vector search only. Production also runs word search alongside it, which this table doesn't include.

Where this sits next to other methods

Approach	How context gets in	What you need	Result allocation
Isolated chunk embedding	Only the chunk's text	Any embedding endpoint	Up to the retrieval system
Late chunking	Tokens encoded with the larger text, then pooled per chunk	Token vectors and pooling control, or an API that does it	Up to the retrieval system
Contextual Retrieval	Generated explanation prepended to each chunk	A generation call per chunk, then embedding and BM25	Up to the retrieval system
Weighted blend	Piece vector mixed with a context vector after embedding	Final vectors from one model and dimension	Up to the retrieval system
This post	Blended pieces and blended query parts	Batched embeddings and parallel searches	Up to X per query part

Late chunking runs the whole document through the model, then pools each chunk's token vectors, so each chunk's tokens have already seen their surroundings. The paper covers the mechanism, and Jina's follow-up separates boundary choice from context. It needs token-level output. OpenAI's endpoint returns one pooled vector per input, capped at 8,192 tokens, so we can't build it there.

Anthropic's Contextual Retrieval writes a short explanation of each chunk's place in its document and prepends it before embedding and BM25 indexing. That costs a model call per chunk. The blend reuses the passage directly.

Weighted mixes of local and wider embeddings have precedent. MCRAH-GIE (Results in Engineering, June 2026) combines chunk, paragraph and document embeddings at 50/30/20 in its section 3.3.5, using all-MiniLM-L6-v2.

On the query side, Aspect Fusion (Korikov et al., 2024) splits a query into aspects and selects an equal number of results per aspect before fusing them into one ranked list. We keep each part's group separate instead of fusing. LangChain's MultiQueryRetriever generates several queries and returns the union. RichRAG retrieves per sub-aspect and uses a learned ranker to pick the final set.

None of these share a dataset with our tests, so the table compares architecture, not accuracy.

Limits

Sentences aren't concepts. "Do you remember their names?" depends on the sentence before it. Voice transcripts run several subjects together without a full stop, which is why long pieces are windowed by word count. Our controlled test used hand-picked spans. The production eval used the automatic packer, but it didn't isolate packer quality.

More parts cost more. A message with m parts runs m searches and can attach up to m × X memories, which adds input tokens for the answering model. Our recall runs inside a 6-second limit. If groups need to shrink, lower X for every part rather than adding a global cap, which brings back the competition between parts.

Thresholds depend on the score scale. In the clean email example, the correct record scored 0.993 and the other three contexts scored 0.975 to 0.984. Blended scores sit close together, so a floor tuned for one representation won't carry over to another.

The blend has a ceiling. It compresses the context into 20% of one vector. A memory that needs several hops across relationships still needs graph traversal.

The weight isn't universal. The blend lost some individual cases in the controlled test. It mistook a conversation-sharing request for summarization, and a customer-record restriction for a correction under query noise.

Build checklist

Split stored text by source: user message, each assistant step, each tool call with its full result.
Pack sentences into pieces of at most 1,200 bytes, cutting any longer sentence at the last word that fits.
Prefix each piece with a label for its turn, source and time, and count the label against the budget.
Embed the pieces in batches sized by input count and bytes before sending.
Store each piece as chunkSearchVector(piece, meanVector(turn's pieces)).
At query time, split the message into at most 8 parts with splitPromptChunks, and embed the parts and the whole message in one request.
Blend each part with the whole message using the same chunkSearchVector.
Search each part in parallel, keep up to X results above the threshold, and keep each part's group intact into the prompt.
Measure on your own threads with stored facts as queries before you pick the piece size, X or the threshold.

Extractive Summarization with Jev

Our Most Effective Jev Use Case: Walking Agent Memory