How to Set Up Hybrid Search for Local Markdown Notes with Ollama

You remember writing down the supplier reference, “NX-4821”, somewhere in your project notes. Searching for the reference should find the exact line immediately. But when you search for “why we changed the onboarding flow”, you also want notes that mention reducing setup friction, even if they never use the word “onboarding”. Keyword search handles the first request well. Embedding search helps with the second. Hybrid search combines both.

For a local Markdown collection, this can be a useful upgrade without sending your notes to a hosted service. It does add moving parts, though: a keyword index, an embedding model, a vector index and a method for combining results. The goal is not to make search technically impressive. It is to make your own notes easier to retrieve when you need them.

What hybrid search combines

A hybrid search system runs two different searches for the same query.

Search type	Best at finding	Weaknesses
Keyword search	Exact names, dates, acronyms, file titles, codes, quoted phrases	Misses related wording and synonyms
Embedding search	Similar concepts, paraphrased questions, loosely connected ideas	Can miss or dilute exact identifiers
Hybrid search	Both precise facts and conceptually related notes	Requires more setup and evaluation

Keyword search usually relies on a full-text index. Depending on your tools, this may use a method such as BM25, which considers terms that occur in both your query and a document, while giving less weight to very common words.

Embedding search converts note chunks and your query into lists of numbers called vectors. Similar meanings tend to produce vectors that sit closer together. A local model served through Ollama can create these embeddings on your own machine.

Neither approach is universally better. A note titled 2026-04-12-client-call.md may be essential when you search for a date, client name or contract code. An embedding model may not consider those tokens meaningful enough. Conversely, a keyword search for “prepare for difficult feedback conversation” may not find a note called “Talking through missed expectations”, despite that note being exactly what you need.

Decide whether the extra setup is worthwhile

Hybrid search is most useful when your notes contain a mixture of structured facts and free-form writing. This is common in project notes, meeting records, research, personal knowledge bases and technical documentation.

It is probably worth considering if you often search for:

• People, product names, ticket numbers, account codes or URLs
• Dates, version numbers, filenames and meeting titles
• Decisions described differently over time
• Broad questions such as “what did we decide about the pricing page?”
• Ideas that are spread across several notes rather than stated in one place

It may not be worth it yet if you have a small, consistently named collection and ordinary text search already works. Hybrid search cannot compensate for notes that are too vague to be useful. Before adding AI retrieval, make sure important notes have clear titles, dates where relevant and enough context to stand alone.

If you are still shaping your note structure, using Markdown frontmatter without overcomplicating your system can help you keep useful fields such as project, status and date consistent. Those fields can also support filtering later.

Prepare your Markdown notes for indexing

Start with a folder of Markdown files that represents the notes you actually want to search. Avoid indexing every exported archive, template and duplicate unless you have a clear reason.

For each file, capture at least:

• File path
• Note title, usually from the first heading or filename
• Modified date
• Frontmatter fields, if present
• Heading hierarchy
• The text of each searchable chunk

Chunking matters. An embedding model should not usually receive an entire long journal entry or lengthy project document as one item. Split notes at sensible boundaries, such as headings and paragraphs, while keeping enough surrounding context to preserve meaning.

A practical chunk might include the note title, its nearest heading and a few paragraphs of text. This means that a paragraph saying “we postponed it” still carries the project and topic that explain what “it” refers to.

Keep the original file path and heading with every chunk. Search results are only useful if you can open the source note and understand where the match came from.

Set up local embeddings with Ollama

Choose an embedding model that Ollama can run on your machine. Embedding models are designed for retrieval, not for conversational responses, so do not use a general chat model merely because it is already installed.

Your indexing process needs to do two jobs:

1. Send each note chunk to the local embedding model and store its vector.
2. Send each future search query to the same model before comparing it with stored vectors.

Use the same embedding model for both. Changing the model later means re-embedding your collection, because vectors from different models are not directly comparable.

Store vectors in a local vector index or database alongside the chunk text and metadata. The exact choice depends on your technical comfort and the application you use. What matters more than the database name is that you can rebuild the index from your Markdown folder. Your notes remain the source of truth. The index should be disposable derived data.

For a private collection, also check where your search application stores indexes, logs and backups. Running an embedding model locally does not automatically make every part of the workflow private if another service handles your files or queries. For a broader local-first approach, read how to use AI for personal notes without sending them to the cloud.

Build a keyword index alongside it

Your keyword index should cover the same chunks as the vector index. If one index searches whole files and the other searches short chunks, results can be confusing and hard to compare.

Include titles, headings and frontmatter values in the keyword index. These fields often contain the exact terms people search for:

• ACME
• Q3 planning
• 2026-10-04
• BUG-193
• invoice overdue

Treat punctuation and case thoughtfully. You may want bug-193 and BUG-193 to match, while still retaining the original spelling in displayed results. For codes and identifiers, exact phrase matching is especially valuable.

Do not strip numbers from your search pipeline. Dates, version numbers and reference codes are often the reason people lose trust in semantic search.

Combine results with a simple ranking method

Once you have both result lists, you need one ranked list. A good starting point is reciprocal rank fusion, often shortened to RRF.

Instead of trying to compare a keyword score directly with an embedding similarity score, RRF uses each result’s position in its own list. A chunk ranked highly by either search gets a boost. A chunk ranked well by both gets a larger boost.

This avoids a common problem: keyword and vector scores are calculated differently, so their raw numbers are not naturally compatible.

Start with equal weight for both systems. Then adjust only after testing real queries. If exact codes regularly fall too low, increase the contribution of keyword results or add a rule that promotes exact title and phrase matches. If broad questions return only literal matches, give the embedding results more influence.

A sensible result display should show why an item appeared. For example, show the note title, heading, date, a short excerpt and whether it was an exact match or a related match. This makes search easier to trust and easier to debug.

Test against real searches, not imagined ones

Create a small evaluation list before deciding that hybrid search works. Use queries you have genuinely typed, or were frustrated by in the past.

Include several categories:

Query category	Example
Exact identifier	NX-4821
Person or organisation	Mina Patel
Date or meeting	15 May retrospective
Conceptual question	why did we delay the launch?
Different wording	reducing customer setup effort
Mixed query	ACME contract decision

For each query, write down the note or section you expected to find. Then compare keyword-only, embedding-only and hybrid results. Check the top five results rather than just the first one.

Ask four practical questions:

1. Does the expected note appear near the top?
2. Are exact names, dates and codes still reliably found?
3. Are related but differently phrased notes now discoverable?
4. Do the extra results create more distraction than value?

Keep a short record of failures. If a decision is difficult to retrieve because it is buried in a meeting note, improving the note may be more valuable than tuning search weights. Turning meetings into clearly linked outcomes also helps, as described in how to turn meeting notes into follow-up tasks with AI without losing context.

Maintain the index without making it fragile

Re-index changed files when you edit them, and remove vectors for files you delete or move. Periodically run a full rebuild, especially after changing your chunking rules, metadata extraction or embedding model.

Keep the system modest. A local hybrid search setup is successful when it makes your Markdown notes more dependable, not when it becomes another project to maintain. Tools such as Fundus can be useful if you want local notes, tasks and AI assistance in one place, but the underlying principle remains the same: preserve your original notes, index them locally, and test search against the questions you actually ask.