OpenSearch vector search in production: ten lessons learned

I spent three months building a neural search and recommendation system over roughly 600,000 company documents in OpenSearch. The official documentation covers the happy path well. What it does not cover is what happens when k-nearest-neighbour search meets selective filters, why your hybrid query silently returned 2% of the candidates you expected, or how to tell an AWS proxy error from an OpenSearch rejection. This post is the collection of things I wish I had known at the start.

The system in question is a company matching platform: given a seed company or a text query, return ranked partner candidates from a large commercial database. The index holds roughly 600,000 companies, each with a multi-vector chunking layout over their web descriptions, a top-level website embedding, flat keyword fields for taxonomy and geography, and a copy_to field that aggregates everything into one analysed surface for lexical search. The combination of scale, mixed query modes and a managed AWS OpenSearch domain is what produced most of these lessons.

Lesson 1: ANN defines a candidate set, it does not score the index

This is the single most important thing to internalise about neural search in OpenSearch, because misunderstanding it breaks things silently in ways that are hard to diagnose.

A knn or neural clause is not a scoring function that runs over the whole index and produces a relevance value for every document. It is an approximate nearest-neighbour retrieval: it traverses the HNSW graph, collects the k most similar vectors it finds, and those k documents become the entire candidate set for the rest of the query. Anything outside that set does not exist to the query.

The practical consequence is brutal for selective queries. I measured a country_headquarters = "Japan" filter at k=10,000: the index contained 8,434 eligible Japanese companies, but the ANN traversal had already decided its candidate set before the filter ran, and only 689 of those companies happened to be in it. The query returned 689 results. No error. No warning. The other 7,745 eligible companies simply did not exist from the query's perspective.

This is not a bug. Approximate nearest neighbour search is approximate by design. But the gap between how the documentation presents knn (as a scored query) and what it actually does (bounded ANN retrieval) takes a while to internalise, and getting it wrong produces results that look plausible until you check them against ground truth.

Lesson 2: push filters inside the ANN clause, not beside it

Once you understand what the previous lesson says, the fix follows directly. A filter placed as a sibling to the ANN clause in a bool runs after the ANN has already decided its candidates. A filter pushed inside the clause constrains the HNSW traversal itself, so OpenSearch does filtered approximate KNN over the eligible set and ignores everything outside it.

// Wrong: filter is a sibling, post-filters the ANN candidate set
{
  "bool": {
    "must": [ { "knn": { "WEBSITE_EMBEDDING": { "vector": [...], "k": 10000 } } } ],
    "filter": [ { "term": { "COUNTRY_HEADQUARTERS": "Japan" } } ]
  }
}

// Right: filter is inside the knn clause, constrains the traversal
{
  "knn": {
    "WEBSITE_EMBEDDING": {
      "vector": [...],
      "k": 10000,
      "filter": {
        "bool": {
          "filter": [ { "term": { "COUNTRY_HEADQUARTERS": "Japan" } } ]
        }
      }
    }
  }
}

Two important notes about this fix. First, it works even for nested fields like TEXT_CHUNKS: parent-level filter fields are valid inside a nested neural clause. I verified this explicitly because it is not obvious from the documentation. Second, the outer bool.filter is worth keeping as a correctness backstop alongside the inner filter, even though it becomes redundant. ANN is approximate; the outer filter guarantees the final result set is clean regardless.

Lesson 3: hybrid queries silently collapse without pagination_depth

OpenSearch's hybrid query type is the right tool for blending neural and lexical search, but it has a behaviour that is not emphasised enough in the documentation: by default, a hybrid query ignores each sub-query's own k when collecting candidates, and instead keeps only pagination_depth documents. pagination_depth defaults to approximately the query's size. If you are asking for 20 results and omit pagination_depth, the hybrid query collects from a pool of roughly 20 documents before ranking. The final 20 you see will be the best of those 20, not the best of everything the neural and lexical clauses could find.

{
  "hybrid": {
    "pagination_depth": 10000,   // mandatory if you want real recall
    "queries": [
      { "bool": { "must": [ /* neural clause */ ], "filter": [...] } },
      { "bool": { "must": [ /* lexical clause */ ], "filter": [...] } }
    ]
  }
}

I measured this directly. With size=1000 and no pagination_depth: 2,759 unique candidates collected. With pagination_depth=10000: 115,419. That is a 97% reduction in the candidate pool from a single omitted parameter, with no query error, no warning, and results that look entirely normal because the top 1,000 out of 2,759 still appear to be reasonable matches.

The right value for pagination_depth is generally equal to your NEURAL_K: you want the hybrid query's collection phase to capture everything the neural sub-query would have returned on its own. Setting it lower is a deliberate speed-recall trade-off; leaving it absent is an accidental one.

Lesson 4: the HNSW parameter chain is a ceiling, not a setting

HNSW indexes in OpenSearch are controlled by four values, and they form a strict dependency chain. Each is the ceiling for the one to its right:

The dangerous combination is NEURAL_K=10000 with ef_search=1024. The application configuration asks for 10,000 nearest neighbours; the index silently caps traversal at 1,024 because the HNSW graph was not built or tuned to go further. No error is raised. You just get a smaller candidate set and poorer recall, and if you are not checking hit counts carefully, you may not notice for weeks.

ef_search is the one parameter you can fix without re-ingesting. Update it on the live index like this:

PUT /your-index/_settings
{
  "index.knn.algo_param.ef_search": 15000
}

A rule of thumb: ef_search should be roughly 1.5x your NEURAL_K, giving the HNSW traversal 50% headroom. For ef_construction, anything above about 4x m has severely diminishing returns on index quality, so it is not worth pushing further.

Lesson 5: build one analysed field and copy everything into it

Neural search alone cannot guarantee that a company containing a specific term will appear in the results for that term. The reason is embedding dilution: a single term buried inside a long taxonomy list barely shifts the pooled 768-dimension chunk vector. Companies that match a search term verbatim are routinely absent from that term's ANN neighbourhood.

The fix is not to add more neural clauses. It is to add a lexical clause that covers the whole document. The pattern I settled on is a single field called SEARCH_ALL defined with copy_to in the index mapping. Every human-readable field copies its tokens into it. The field itself is never stored in _source and is never sent by the ingest pipeline: copy_to fires at index time from the source field values, so the only cost is a slightly wider inverted index.

// Mapping snippet
"COMPANY_NAME": {
  "type": "text",
  "copy_to": "SEARCH_ALL",
  "fields": { "keyword": { "type": "keyword" } }
},
"WEBSITE_DESCRIPTION": {
  "type": "text",
  "copy_to": "SEARCH_ALL",
  "index": false        // not queried directly; copy_to still fires
},
"SEARCH_ALL": {
  "type": "text",
  "store": false        // tokens in the inverted index only, never in _source
}

There is one subtle trap with this pattern. Avoid copying a field into SEARCH_ALL if its value already appears inside your embedding input text. Doing so doubles its term frequency in the BM25 calculation and skews the lexical score for that field's tokens relative to everything else. In my case the web description was already embedded as-is, so it was excluded from the copy list. The company name was stripped from the embedding text before ingestion, so it needed to be explicitly added to SEARCH_ALL separately.

With this field in place, the hybrid query's lexical clause has a single match surface that covers everything: company name, location, employee band, taxonomy, web description, and any other human-readable field. A company is always reachable by any text it contains, regardless of where the ANN traversal goes. I verified this with a regression suite that searches for the same company using twelve different kinds of text it contains: HQ country, city, vendor name, employee band, technology term, company type, and so on. Every single one must return the company in the result set. Neural alone satisfies only a subset of those.

Lesson 6: de-index what you never query

OpenSearch's Lucene document count and its document count are not the same thing. Every nested object creates additional hidden Lucene documents, and on a large index with deeply nested fields this can be the dominant cost in index size, query fanout, and merge overhead.

Before the index redesign, the company index had roughly 8.2 million Lucene documents for 600,000 company records. The breakdown was sobering:

The key insight is that enabled: false on a nested mapping removes it from the inverted index while keeping it in _source. If you never query a nested field directly, just store it. The values are still available when you read documents back; they simply do not create Lucene documents.

In my case, all four technology filter dimensions (group, category, detail, scope) were moved to flat keyword arrays derived at ingest time. Each technology filter was already applied as an independent nested query with no same-row correlation requirement, so flattening is semantically equivalent while removing 5 million Lucene documents.

One caveat: once you set enabled: false, you cannot add a filter on that field without a full re-ingest. There is no inverted index to query. A terms query on a de-indexed field silently returns zero hits, which is easy to mistake for no matching documents.

Lesson 7: your 429s probably come from the ML layer, not the write queue

When ingesting with the OpenSearch neural embedding pipeline enabled, every bulk request fans out into ML inference tasks. The default max_ml_task_per_node is 10. Even a modest number of parallel ingest workers will hit that ceiling immediately and produce a stream of 429s. The write thread pool, which most documentation points you at when diagnosing 429s, may show zero rejections throughout.

// Diagnose: check the write thread pool first (the usual culprit)
GET /_nodes/stats/thread_pool/write

// But if write shows zero rejections, check ML tasks
GET /_nodes/stats/thread_pool/ml_commons

// Fix: raise the ML task limit as a persistent cluster setting
PUT /_cluster/settings
{
  "persistent": {
    "plugins.ml_commons.max_ml_task_per_node": 40
  }
}

// Verify the effective values (transient takes precedence over persistent)
GET /_cluster/settings?include_defaults=true&flat_settings=true&filter_path=**.ml_commons*

A critical distinction: use persistent, not transient, for this setting on a production domain. Transient settings are lost on a full cluster restart, including the kind that a managed service (AWS OpenSearch, Elastic Cloud) may perform during maintenance windows. If you set it transiently, the domain reverts to the default of 10 between runs and your next ingest fails with the same 429 storm you thought you had fixed.

Raising max_ml_task_per_node does not increase throughput: the ML inference is CPU-bound and the node is already saturated. What it does is stop rejections consuming your retry budget and turning a saturation event into a failure cascade. Keep the number of parallel ingest workers modest rather than treating the higher ceiling as permission to push harder.

Lesson 8: the KNN circuit breaker is entirely separate from the general one

OpenSearch has a general circuit breaker (indices.breaker.*) that protects JVM heap from query-time memory spikes. It has a separate, independent circuit breaker specifically for KNN native memory (knn.memory.circuit_breaker). The two are completely different systems with different thresholds, different diagnostic commands, and different fixes.

The symptom from the KNN breaker is: bulk indexing fails with Indexing knn vector fields is rejected as circuit breaker triggered. This looks like a write failure but it is actually a memory failure triggered by the HNSW graph growing beyond the allowed threshold (50% of JVM heap by default).

// Diagnose: check KNN-specific stats, not the general breaker
GET /_plugins/_knn/stats

// Key fields to look at:
// circuit_breaker_triggered
// graph_memory_usage_percentage   (per node)
// cache_capacity_reached

// Fix step 1: delete (not just close) old KNN indexes
// Closing an index does NOT evict its HNSW graphs from native memory.
// Deletion does.

// Fix step 2: if the breaker still trips, raise the limit transiently
PUT /_cluster/settings
{
  "transient": {
    "knn.memory.circuit_breaker.limit": "75%"
  }
}

// Restore after ingestion is complete
PUT /_cluster/settings
{
  "transient": {
    "knn.memory.circuit_breaker.limit": null
  }
}

Use transient here, unlike the ML task setting above. You want the limit to revert automatically on restart once the ingest window is over, since a permanently raised ceiling leaves less headroom for query-time memory spikes in production.

Note that plugins.ml_commons.max_ml_task_queue_size is sometimes referenced in older posts as a related setting. It no longer exists in recent ml-commons versions. A PUT /_cluster/settings with that key will succeed and report back the value, but it has no effect: the setting is silently ignored.

Lesson 9: a 502 from the AWS proxy is retryable, not a server error

On a managed AWS OpenSearch domain, requests route through an Nginx proxy before they reach the cluster. When the cluster is saturated (commonly during an embedding ingest), the proxy starts shedding load by returning 502 responses. These come back in milliseconds and carry an HTML body, not an OpenSearch JSON response.

The difference matters for your retry logic:

• A 429 from OpenSearch means the write queue is full. Returns OpenSearch JSON. Slow to arrive (the request made it to the cluster).
• A 502 from the proxy means the proxy itself is shedding load. Returns HTML. Arrives in milliseconds (the request never reached the cluster).
• Both mean: try again later. Neither means the document is bad.

I discovered this the hard way: a retry strategy that treated 502 as permanent discarded an entire sub-batch of over 600 documents on the first occurrence. Because the proxy had already forwarded some of those requests before shedding, many of them had actually been indexed. The retry would have found nothing to do. Treating 502 as permanent meant those documents were silently dropped.

The fix is to treat {429, 500, 502, 503, 504} as retryable. Before marking anything as a permanent failure, refresh the index and check whether the documents are actually present. Bulk-applied-then-502 is a very common outcome during ML saturation.

Lesson 10: use equal jitter backoff, not full jitter, under sustained ML saturation

Full jitter backoff (random(0, cap)) is the standard recommendation for reducing thundering-herd behaviour. Under typical conditions it works well. Under sustained ML inference saturation it is the wrong choice, because the random range includes values near zero. Multiple workers will independently roll short delays and re-apply load before the cluster's predict queue has drained at all, converting a brief saturation spike into a sustained storm.

Equal jitter chooses a delay in the range [cap/2, cap], which provides a guaranteed minimum wait. Workers still spread out their retries (the jitter half of the range prevents a synchronised retry wave), but the minimum delay is long enough for the ML layer to drain between attempts.

import random

def backoff_delay(attempt: int, base: float = 2.0, cap: float = 60.0) -> float:
    """Equal jitter: guaranteed floor of cap/2, randomised ceiling of cap."""
    ceiling = min(cap, base * (2 ** attempt))
    return ceiling / 2 + random.random() * (ceiling / 2)

During the ingest run that identified this, a full-jitter strategy produced retry delays as short as 0.1 seconds against a cluster that had just rejected with 429. Several workers retried before anything had drained, and a brief queue spike became a sustained pattern of continuous rejections. Equal jitter with a 60-second cap resolved it.

Closing thought: debug with counts, not just results

Most of the lessons above share a diagnostic pattern: the query succeeds and returns plausible results, but the count of what it searched was radically smaller than you expected. The only way to find these problems is to check hit counts and candidate pool sizes, not just the quality of the top results.

A few habits I now run on every query change:

• Check hits.total.value on a size=0 query with an exists filter on a top-level field to count parent documents exactly. Do not use _count with match_all; it includes nested Lucene docs.
• Add include_os_request: true in your application config so every response includes the raw DSL that was actually sent. The query you think you built is often not the query the client sent.
• Build a recall regression suite. Mine asserts that a known company is returned when searching against twelve different kinds of text it contains. Running it after every query change catches filter regressions before they reach production.
• When you change ANN parameters, compare candidate pool sizes (the number of documents scoring above zero), not just the top-k results. A change that improves top-5 precision while halving the candidate pool is usually worse overall for a recommendation use case.

OpenSearch neural search is genuinely powerful once the mechanics are clear. The documentation is improving, but there is still a gap between the getting-started material and what you need to know to operate it at production scale with selective filters and managed infrastructure. I hope some of what is here closes that gap for someone else starting this work.