Section 3: Knowledge, Reasoning, Tools & Memory — AI Foundations for Product Leaders

In Section 2, we established that foundation models are powerful but fundamentally limited: they hallucinate, forget everything between sessions, can't access real-time data, and can't interact with the outside world. This section covers the four upgrade layers that overcome those limitations. Every serious AI product you've used — ChatGPT, Perplexity, Notion AI, Microsoft Copilot, Amazon Q — is built on some combination of these.

┌───────────────────────────────────────────────────────────┐
│              YOUR AI PRODUCT                              │
├───────────────────────────────────────────────────────────┤
│                                                           │
│   🧠 REASONING        🔧 TOOLS                           │
│   Chain of Thought     APIs, Code Execution,              │
│   Tree of Thoughts     Web Search, File Ops               │
│   Self-Consistency                                        │
│                                                           │
│   📚 KNOWLEDGE (RAG)  💾 MEMORY                           │
│   Retrieval-Augmented  Conversation History,               │
│   Generation           User Profiles,                     │
│   Vector Search        Long-Term Store                    │
│                                                           │
├───────────────────────────────────────────────────────────┤
│              FOUNDATION MODEL                             │
│         (GPT-4, Claude, Gemini, Llama)                    │
└───────────────────────────────────────────────────────────┘

3.1 Knowledge Augmentation: Retrieval-Augmented Generation (RAG)

The Problem RAG Solves

Foundation models know what was in their training data — and nothing else. Your company's internal documents, today's pricing, last week's product update, the customer's order history — the model knows none of it.

RAG (Retrieval-Augmented Generation) is the most important pattern in production AI today. The idea is simple: before the model generates a response, first retrieve relevant information from an external knowledge source, then inject that information into the prompt as context.

Analogy: Imagine a brilliant consultant who has never worked in your industry. Before every meeting, an assistant hands them a folder containing exactly the documents they need. The consultant reads the folder, then gives you an expert answer grounded in your actual data. That assistant + folder system is RAG.

3.1.1 Naive RAG: The Basic Pattern

Naive RAG is the simplest implementation and where most teams start:

┌─────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  User Query  │────▶│  1. Embed Query   │────▶│  2. Search Vector │
│              │     │  (convert to      │     │     Database       │
│              │     │   vector)         │     │  (find similar     │
│              │     │                   │     │   chunks)          │
└─────────────┘     └──────────────────┘     └────────┬─────────┘
                                                       │
                                                       ▼
┌─────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  4. Return   │◀────│  3. Augment       │◀────│  Top-K relevant  │
│  Response    │     │  Prompt with      │     │  chunks           │
│  to User     │     │  retrieved chunks │     │                   │
│              │     │  + Generate       │     │                   │
└─────────────┘     └──────────────────┘     └──────────────────┘

Step by step:

Index phase (offline): Take your knowledge base (docs, FAQs, product pages, PDFs). Split each document into chunks (typically 200-500 tokens). Convert each chunk into a numerical vector (embedding) using an embedding model. Store these vectors in a vector database.
Query phase (real-time):
User asks a question: "What's the return policy for electronics?"
Embed the query into the same vector space
Search the vector database for the top-K most similar chunk vectors (typically K=3 to 10)
Insert those chunks into the LLM prompt: "Using the following context, answer the user's question: [retrieved chunks]. Question: What's the return policy for electronics?"
LLM generates a response grounded in the retrieved context

Naive RAG Limitations:

Problem	Description	Impact
Query mismatch	User's natural language question may not semantically match the best document chunk	Retrieves irrelevant or suboptimal context
Chunk boundary issues	Important information may span two chunks, and neither chunk alone is sufficient	Incomplete or inaccurate answers
No ranking intelligence	Returns top-K by vector similarity alone — doesn't account for recency, authority, or relevance refinement	May surface outdated or low-quality documents
Single retrieval pass	One shot to get the right documents — no ability to refine or follow up	Misses information that requires iterative search
Context window pollution	Irrelevant chunks waste context window space and can confuse the model	Lower quality responses, higher cost

3.1.2 Enhanced RAG: Smarter Retrieval

Enhanced RAG adds intelligence to each stage of the pipeline:

Query Rewriting / Expansion

The user's raw query is often a bad search query. Enhanced RAG rewrites it before searching.

Example: - User query: "Why is my order late?" - Rewritten queries: - "Shipping delay policy and estimated delivery timelines" - "Order tracking status delayed reasons" - "Late delivery compensation policy"

The system searches for all three rewrites and combines the results. This technique, called multi-query retrieval, dramatically improves recall.

Real-world example: Perplexity rewrites every user query into multiple search queries before hitting its search index. That's why it often finds better answers than if you searched Google yourself — it's running 3-5 searches per question.

Re-Ranking

After initial retrieval returns the top-K chunks, a re-ranker model (like Cohere Rerank, or a cross-encoder) re-evaluates each chunk in the context of the original query and reorders them by true relevance.

Why this matters: Embedding similarity (used in naive RAG) is a rough proxy for relevance. A re-ranker is a more expensive but much more accurate relevance scorer. Think of it as a first pass by a junior researcher (vector search) followed by a senior expert reviewing and reordering the results (re-ranker).

Hybrid Search

Combines semantic search (vector similarity — good at understanding meaning) with keyword search (BM25/TF-IDF — good at exact matches).

Search Type	Good At	Bad At
Semantic (vector)	Understanding intent, synonyms, paraphrasing	Exact matches, proper nouns, product SKUs, codes
Keyword (BM25)	Exact terms, identifiers, names, codes	Understanding meaning, handling paraphrasing
Hybrid	Both — uses Reciprocal Rank Fusion (RRF) to combine results	Slightly more complex to implement

Example: A user asks "AirPods Pro 2 noise cancellation specs." Vector search might retrieve general noise-cancelling headphone content. Keyword search finds exact matches for "AirPods Pro 2." Hybrid search returns the right answer.

3.1.3 Modular RAG: Composable Pipelines

Modular RAG treats each step (query processing, retrieval, re-ranking, generation) as a swappable component in a pipeline. This is how mature production systems are built.

┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
│  Query   │──▶│  Query   │──▶│ Retrieve │──▶│ Re-rank  │──▶│ Generate │
│  Input   │   │ Rewrite  │   │ (Multi-  │   │ & Filter │   │ Response │
│          │   │ & Route  │   │  source) │   │          │   │ + Cite   │
└──────────┘   └──────────┘   └──────────┘   └──────────┘   └──────────┘
                    │               │
                    ▼               ▼
              ┌──────────┐   ┌──────────────────────┐
              │ Classify │   │ Vector DB │ SQL DB   │
              │ & Route  │   │ Web Search│ Graph DB │
              │ to source│   │ API calls │ Cache    │
              └──────────┘   └──────────────────────┘

Key capabilities: - Query routing: Classify the query and send it to different retrieval backends. Product questions → product database. Policy questions → policy docs. Current events → web search. - Multi-source retrieval: Search vector DB, relational DB, knowledge graph, and web simultaneously. - Self-reflection / corrective RAG: After generation, a judge model reviews whether the answer actually addresses the query. If not, trigger a refined retrieval and re-generate. - Adaptive retrieval: Only trigger RAG when the model needs external knowledge. Simple greetings or general knowledge questions can skip retrieval entirely — saving latency and cost.

3.1.4 RAG Components Deep Dive

Embeddings: How Meaning Becomes Math

An embedding is a numerical representation of text — a vector of floating-point numbers (typically 768 to 3072 dimensions) that captures the meaning of the text.

Analogy: Think of GPS coordinates. "Paris," "the capital of France," and "the City of Light" are different text strings, but their embeddings would be nearby in vector space — just like how different descriptions of the same place would map to similar GPS coordinates.

How they work: 1. An embedding model (e.g., OpenAI text-embedding-3-large, Cohere embed-v3, or open-source BGE-large) processes text input 2. It outputs a fixed-length vector: [0.023, -0.841, 0.119, ..., 0.445] — typically 1536 or 3072 dimensions 3. Semantically similar texts produce vectors that are close together (measured by cosine similarity)

PM-relevant model comparison:

Embedding Model	Dimensions	Relative Quality	Cost (per 1M tokens)	Open Source?
OpenAI `text-embedding-3-large`	3072	⭐⭐⭐⭐⭐	~$0.13	No
OpenAI `text-embedding-3-small`	1536	⭐⭐⭐⭐	~$0.02	No
Cohere `embed-v3`	1024	⭐⭐⭐⭐⭐	~$0.10	No
Google `text-embedding-004`	768	⭐⭐⭐⭐	Free (limits apply)	No
`BGE-large-en-v1.5`	1024	⭐⭐⭐⭐	Free (self-hosted)	Yes
`E5-Mistral-7B-instruct`	4096	⭐⭐⭐⭐⭐	Free (self-hosted)	Yes

PM Decision Point: Embedding model choice affects retrieval quality. Better embeddings → better retrieval → better answers. But the difference between top models is often marginal. Don't over-optimize here before you've tuned your chunking strategy and prompt design.

Vector Databases: Where Embeddings Live

A vector database stores embedding vectors and enables fast similarity search over millions or billions of vectors.

Vector DB	Type	Best For	Scale	Notable Users
Pinecone	Fully managed cloud	Teams wanting zero ops overhead, fast time-to-market	Billions of vectors	Notion, Shopify
Weaviate	Open-source + cloud	Hybrid search (vector + keyword), multimodal	Billions of vectors	Red Hat, Stackla
Chroma	Open-source, lightweight	Prototyping, small-to-medium scale, local development	Millions of vectors	Popular in LangChain demos
pgvector	PostgreSQL extension	Teams already on Postgres, want vector search without a new DB	Millions of vectors	Supabase users
Qdrant	Open-source + cloud	High-performance filtering + vector search	Billions of vectors	Enterprise use cases
Milvus	Open-source + cloud (Zilliz)	Massive scale, cloud-native	Trillions of vectors	Many enterprise deployments

PM Decision Framework for Vector DB: - Prototype / MVP: Use Chroma (local) or Pinecone free tier - Production with small team: Pinecone (managed) or pgvector (if already on Postgres) - Production at scale with hybrid search needs: Weaviate or Qdrant - Massive scale / billion+ vectors: Milvus/Zilliz or Pinecone Enterprise

Chunking Strategies: How You Split Documents Matters More Than You Think

Chunking determines how source documents are broken into pieces before embedding. Bad chunking is the #1 cause of poor RAG quality.

Strategy	How It Works	Pros	Cons	Best For
Fixed-size	Split every N tokens (e.g., 256 tokens) with overlap	Simple, predictable	Cuts mid-sentence, mid-paragraph	Quick prototypes, uniform data
Sentence-based	Split on sentence boundaries	Preserves meaning units	Single sentences often lack context	Short FAQ-style content
Paragraph-based	Split on paragraph/section boundaries	Preserves logical units	Paragraphs vary wildly in size	Well-structured documents
Recursive character	Tries paragraph → sentence → character boundaries in order	Balances size and meaning	Needs tuning per content type	General purpose (LangChain default)
Semantic chunking	Uses embeddings to find natural topic boundaries	Best semantic coherence	Slower, more expensive to index	High-value knowledge bases
Document-structured	Respects document structure (headings, sections, tables)	Preserves document intent	Requires structured input (HTML/Markdown)	Product docs, legal documents

Rule of thumb: Start with recursive character splitting at 512 tokens with 50-token overlap. Iterate based on retrieval quality. Most RAG quality problems trace back to chunking — before you add re-ranking or fancy retrieval, check if your chunks make sense to a human reader.

Retrieval Strategies

Strategy	Description	When to Use
Top-K similarity	Return K most similar chunks	Simple queries, homogeneous knowledge base
Maximum Marginal Relevance (MMR)	Balance similarity with diversity — avoid returning K near-duplicate chunks	When top results tend to be repetitive
Filtered retrieval	Apply metadata filters before vector search (e.g., `category="electronics"`, `date > 2024-01-01`)	When you have structured metadata to narrow scope
Parent document retrieval	Index small chunks for precision, but retrieve the parent document/section for context	When full context matters for answer quality
Multi-query	Generate multiple query variants, retrieve for each, merge results	Complex or ambiguous queries
Self-query	LLM parses the query to extract filters + semantic component	"Show me electronics returns from last month" — needs both search and filter

3.1.5 RAG in the Wild: Real Product Examples

Product	What Gets Retrieved	RAG Approach	Why It Works
Notion AI	User's own workspace — notes, docs, databases	Enhanced RAG with workspace-scoped retrieval. Embeddings of all content within user's workspace. Permission-aware (only retrieves what the user can access).	Notion already has the user's content. RAG lets the AI "know" your notes without fine-tuning a model per user.
Glean	Enterprise data across 100+ SaaS apps (Slack, Drive, Jira, Confluence, etc.)	Modular RAG with connectors to each data source. Cross-application hybrid search. Enterprise ACL (access control) enforcement on every retrieval.	Solves the enterprise knowledge silo problem — one AI that searches everything, respecting permissions.
Microsoft Copilot for M365	Emails (Outlook), files (OneDrive/SharePoint), messages (Teams), calendar	Microsoft Graph-powered retrieval. Grounded in your tenant's M365 data.	The "killer app" of enterprise RAG — AI that knows your work context. Each user gets different answers based on their data access.
Amazon Q	AWS documentation, company data sources, Jira, ServiceNow, S3	Modular RAG with 40+ pre-built connectors. Document-structured chunking for technical docs.	Purpose-built for developer and enterprise use cases within the AWS ecosystem.
Perplexity	Live web search results for every query	Web-search RAG with real-time indexing, citation extraction, and re-ranking. Multi-query expansion for comprehensive coverage	Solves the knowledge cutoff problem entirely — every answer is grounded in current web sources.

3.1.6 RAG Comparison Matrix

Dimension	Naive RAG	Enhanced RAG	Modular RAG
Implementation time	Days	1-3 weeks	1-3 months
Quality	Good for simple use cases	Significantly better for complex queries	Best — production-grade
Cost	Low	Moderate (re-ranker adds cost)	Higher (more components)
Latency	~1-2s added	~2-4s added	~2-5s added (with caching: ~1-2s)
Best for	Internal tools, MVPs, simple Q&A	Customer-facing products, complex knowledge bases	Enterprise products, multi-source retrieval, high-stakes applications
Team needed	1 engineer	2-3 engineers	3-5+ engineers

When to Use What: RAG Decision Framework

Do you need knowledge beyond the model's training data?
├── No → Skip RAG entirely. Use the base model.
├── Yes → Is the knowledge source static or dynamic?
│   ├── Static (updates monthly or less)
│   │   └── Simple use case? → Naive RAG
│   │   └── Complex queries or high quality bar? → Enhanced RAG
│   ├── Dynamic (updates daily or real-time)
│   │   └── Single source? → Enhanced RAG with refresh pipeline
│   │   └── Multiple sources? → Modular RAG
│   └── Is this enterprise (permission-sensitive)?
│       └── Yes → Modular RAG with ACL enforcement (this is non-negotiable)

💡 PM Action Item: RAG Audit

For your product (or a product you admire), answer: 1. What knowledge does the AI need that isn't in the model's training data? 2. Where does that knowledge live today? (Docs? Database? API? User-generated content?) 3. How often does it change? 4. Who should have access to what? (Permission model) 5. What's the cost of a wrong retrieval? (Annoying? Costly? Dangerous?)

Your answers determine whether you need naive, enhanced, or modular RAG — or no RAG at all.

3.2 Reasoning: Making Models Think Better

The Problem

LLMs generate responses token-by-token in a single forward pass. By default, they don't "think through" a problem — they produce the first plausible completion. For simple questions ("What's the capital of France?"), this works fine. For complex, multi-step problems ("Should we enter the Japanese market and what should our pricing strategy be?"), the default mode produces shallow, pattern-matched answers.

Reasoning techniques force the model to show its work, consider alternatives, and produce higher-quality outputs. The tradeoff: they use more tokens (higher cost) and take more time (higher latency).

3.2.1 Type 1 vs. Type 2 Thinking (Kahneman's Framework Applied to AI)

Daniel Kahneman's Thinking, Fast and Slow describes two cognitive systems:

	Type 1 (Fast)	Type 2 (Slow)
In humans	Instinct, gut reaction, pattern matching	Deliberate, analytical, step-by-step reasoning
Example	"What's 2 + 2?" → Instant	"What's 347 × 28?" → Need to work through it
In LLMs (default)	Single forward pass → immediate answer	Not naturally available — must be engineered
In LLMs (enhanced)	Standard prompting	Chain of Thought, Tree of Thoughts, reasoning models

The key insight: By default, LLMs are Type 1 thinkers. All reasoning techniques are attempts to give LLMs Type 2 thinking. OpenAI's o1/o3 models are the most prominent attempt to bake Type 2 thinking directly into the model.

3.2.2 Chain of Thought (CoT)

What it is: Asking the model to think step-by-step before giving a final answer.

How it works: Instead of prompting "Answer this question: ...", you prompt "Think through this step by step, then give your final answer: ..." — or you provide few-shot examples that demonstrate step-by-step reasoning.

Example — without CoT:

Example — with CoT:

Why it works: By generating intermediate reasoning tokens, the model effectively creates "scratch space" in its own output. Each intermediate step becomes part of the context for the next step, allowing the model to maintain a chain of logical dependencies that would otherwise be lost in a single forward pass.

When to use: Multi-step math, logical reasoning, complex analysis, debugging, planning.

Cost/quality tradeoff: CoT uses 2-5x more output tokens. Response quality on reasoning tasks improves 15-40% (varies by model and task). At scale, this adds up. A customer service bot processing 1M queries/day that uses CoT may spend an additional $5,000-$20,000/day on output tokens.

3.2.3 Chain of Thought — Self-Consistency (CoT-SC)

What it is: Run CoT multiple times (e.g., 5-10 times) with sampling (temperature > 0), then take the majority vote on the final answer.

Analogy: Instead of asking one consultant for their analysis, you ask five consultants independently, then go with the answer most of them agree on.

Example:

When to use: High-stakes decisions, complex reasoning where you need higher confidence, tasks with verifiable correct answers (math, logic, classification).

Cost/quality tradeoff: 5-10x the cost of single CoT (you're running inference 5-10 times). Quality improves 5-15% over single CoT on reasoning-heavy benchmarks. Best reserved for high-value decisions where accuracy justifies cost.

3.2.4 Auto Chain of Thought

What it is: Instead of manually crafting CoT examples, the model automatically generates its own reasoning demonstrations. The system clusters similar questions, generates CoT examples for representative questions from each cluster, and uses those as few-shot examples.

Why it matters for PMs: Manual CoT requires human-written reasoning examples for each question type — which doesn't scale. Auto-CoT removes this bottleneck, making it practical to deploy CoT across diverse query types without maintaining a large library of hand-crafted examples.

When to use: Products serving diverse query types where manual CoT example curation is impractical — e.g., a general-purpose AI assistant handling thousands of distinct question categories.

3.2.5 Tree of Thoughts (ToT)

What it is: Instead of a single linear chain of reasoning, ToT explores multiple reasoning paths simultaneously, evaluates which paths are most promising, and can backtrack from dead ends.

                        [Problem]
                       /    |    \
                   Path A  Path B  Path C
                   /  \      |      /  \
                A1    A2    B1    C1    C2
                 ✗     |     ✗     |     ✗
                       ▼           ▼
                      A2.1        C1.1  ← Best solution

Analogy: Linear CoT is like hiking on a single trail. ToT is like sending scouts down three trails simultaneously, having them report back, then directing all resources to the most promising trail.

Example — Product Strategy Problem:

When to use: Strategic planning, creative problem-solving, game-playing, tasks where the solution space is large and exploration helps. Not worth the overhead for straightforward Q&A.

Cost/quality tradeoff: 5-20x the cost of single CoT. Massive latency increase (10-60 seconds). Best for offline analysis, not real-time user interactions.

3.2.6 Graph of Thoughts (GoT)

What it is: Extends Tree of Thoughts by allowing reasoning paths to merge, loop, and interconnect — forming a directed graph rather than a tree. Intermediate thoughts from different branches can be combined to form new insights.

      [Problem]
      /   |   \
    A     B     C
    |   / | \   |
    A1-B1  B2  C1     ← B1 incorporates insight from A
     \     |  /
      \    | /
       Combined        ← Merging multiple branches
          |
       [Solution]

When to use: Highly complex, interconnected problems where insights from one line of reasoning should inform another. Examples: architectural design reviews, complex policy analysis, multi-stakeholder tradeoff analysis.

Current state: GoT is largely a research technique as of early 2025. Few production systems implement it directly. However, the principle — letting reasoning paths interweave — is emerging in agentic orchestration frameworks.

3.2.7 Reasoning Strategy Comparison

Strategy	Quality Gain	Latency	Cost Multiplier	Best For	Real-Time Viable?
No reasoning (default)	Baseline	~1-3s	1x	Simple Q&A, classification	✅ Yes
Chain of Thought	+15-40% on reasoning tasks	~3-8s	2-5x	Multi-step problems, analysis	✅ Yes (with patience)
CoT Self-Consistency	+5-15% over CoT	~15-40s	5-10x	High-stakes decisions	⚠️ Marginal
Auto-CoT	Similar to manual CoT	~3-8s	2-5x	Diverse query types at scale	✅ Yes
Tree of Thoughts	+10-30% on complex tasks	~30-120s	10-50x	Strategic analysis, planning	❌ No (batch/offline)
Graph of Thoughts	Highest on interconnected problems	~60-300s	20-100x	Research, complex design	❌ No (research)
Reasoning models (o1/o3)	+20-50% on hard reasoning	~5-60s	3-15x	Math, code, science, logic	⚠️ Depends on task

Real-World Reasoning Products

OpenAI o1/o3: Purpose-built reasoning models that internally perform chain-of-thought before returning a response. The "thinking" tokens are generated but hidden from the user. o1-preview uses 5-50x more tokens internally than a standard GPT-4o call. o3 extends this with deeper reasoning chains. These models excel at competition mathematics, PhD-level science, and complex coding — but cost significantly more and are much slower.

Claude's extended thinking: Anthropic's approach to reasoning, where Claude explicitly shows its reasoning process. The extended thinking section can be tens of thousands of tokens, and you pay for all of them. Particularly strong for nuanced analysis, policy interpretation, and tasks where seeing the reasoning is itself valuable.

PM decision point: Use standard models (GPT-4o, Claude Sonnet) with CoT prompting for 90% of use cases. Reserve reasoning models (o1/o3) for genuinely hard problems — complex code generation, multi-step mathematical reasoning, scientific analysis. Don't use o1 for a customer service chatbot: you'll pay 10x more for marginal quality improvement on simple queries.

When to Use What: Reasoning Decision Framework

How complex is the reasoning required?
├── Simple (factual Q&A, classification, summarization)
│   └── Standard prompting. No reasoning overhead needed.
├── Moderate (multi-step analysis, comparison, structured output)
│   └── Chain of Thought. Add "think step by step" or few-shot CoT examples.
├── Hard (math, logic, code, constraint satisfaction)
│   └── Is accuracy critical?
│       ├── Yes → Reasoning model (o1/o3) or CoT-SC
│       └── No → CoT is probably sufficient
├── Very Hard (novel strategy, creative exploration, open-ended)
│   └── Is this real-time?
│       ├── Yes → CoT with reasoning model (accept higher latency)
│       └── No → Tree of Thoughts or agentic loop

💡 PM Action Item: Reasoning Cost Calculator

Pick a feature in your product that involves reasoning (product recommendations, customer issue diagnosis, content analysis). Estimate: - How many reasoning-heavy queries per day? - What's the cost of a wrong answer? (Customer churn? Revenue loss? Safety risk?) - What latency can users tolerate?

Use the comparison table above to calculate: Standard prompting cost vs. CoT cost vs. reasoning model cost. Then evaluate: does the quality improvement justify the cost increase?

3.3 Memory: Making AI Remember

The Problem

Every LLM call is stateless. The model has no memory of who you are, what you said five minutes ago, or what you discussed last week. Without memory engineering, your AI product is a brilliant amnesiac — astonishing in any single interaction, useless across interactions.

This is a product-defining limitation. The difference between a demo and a product is often memory.

3.3.1 Conversation Buffer Memory

What it is: Store the entire conversation history and inject it into every prompt.

How it works:

System: You are a helpful travel assistant.
[Previous messages]:
  User: I'm planning a trip to Japan.
  Assistant: Great! When are you planning to visit?
  User: March next year.
  Assistant: March is beautiful — cherry blossom season! ...
  User: What about hotels in Kyoto?    ← Current message

→ All prior messages are included in the prompt context.
→ The model "remembers" you're going to Japan in March.

Pros: Simple, preserves full context, no information loss. Cons: Cost grows linearly with conversation length. A 50-message conversation may consume 10,000+ tokens just for history — expensive and eventually hits the context window limit.

When to use: Short conversations (< 20 turns), customer support sessions, task-specific interactions with clear endpoints.

Real-world example: Most chatbot implementations (Intercom Fin, Drift) use buffer memory within a single support session. The full conversation is kept in context until the ticket is resolved.

3.3.2 Summary Memory

What it is: Instead of keeping every message, periodically summarize the conversation and replace the full history with the summary.

How it works:

After 10 messages, the system generates:
"Summary: User is planning a trip to Japan in March 2026.
 Interested in Kyoto. Budget is mid-range ($150-200/night).
 Prefers traditional ryokans over modern hotels.
 Has dietary restrictions (vegetarian)."

→ This summary replaces the full 10-message history.
→ New messages are appended to the summary.
→ Periodically, the summary is updated to incorporate new messages.

Pros: Bounded context usage — the summary stays roughly the same size regardless of conversation length. Enables very long conversations. Cons: Lossy — nuance and specific details may be dropped during summarization. The quality of the summary determines the quality of downstream interactions.

When to use: Long conversations (20+ turns), ongoing advisory interactions, situations where the gist matters more than exact quotes.

Real-world example: Character.ai uses a variant of summary memory to maintain personality-consistent conversations that span hundreds of messages — far beyond any model's context window.

3.3.3 Entity Memory

What it is: Track specific entities (people, projects, preferences, facts) mentioned in conversation and maintain a structured knowledge graph of known entities.

How it works:

Conversation:
  User: "My daughter Sophie starts college next fall."
  User: "My husband Mark is allergic to shellfish."
  User: "We're renovating our kitchen — budget is $40K."

Entity Store:
  Sophie → daughter, starting college fall 2026
  Mark → husband, shellfish allergy
  Kitchen renovation → in progress, $40K budget

→ Entities are injected into the prompt as structured context.
→ When the user later says "What recipes can you suggest for Mark?",
   the system injects Mark's shellfish allergy automatically.

Pros: Space-efficient. Captures the most important facts. Supports cross-conversation recall (entities persist). Highly structured — can be queried, updated, and deleted. Cons: Requires entity extraction (another LLM call or NLP pipeline). May miss implicit or contextual information.

When to use: Personalization-heavy products, CRM-style interactions, AI assistants that need to track facts about users, projects, or preferences across sessions.

Real-world example: ChatGPT's memory feature uses a variant of entity memory. When you tell ChatGPT "I prefer concise answers" or "I work at Google on the Ads team," it stores these as entity-level facts and injects them into future conversations.

3.3.4 Long-Term Memory

What it is: A persistent memory layer that stores information across sessions — days, weeks, months apart — enabling the AI to build a persistent understanding of the user.

Architecture:

┌─────────────────┐     ┌──────────────────┐
│  Conversation    │────▶│  Memory Writer    │
│  (real-time)     │     │  (extracts facts, │
│                  │     │   preferences,    │
│                  │     │   decisions)      │
└─────────────────┘     └────────┬─────────┘
                                 │
                                 ▼
                        ┌──────────────────┐
                        │  Long-Term Store  │
                        │  (vector DB +    │
                        │   metadata)      │
                        └────────┬─────────┘
                                 │
       ┌─────────────────┐       │
       │  New Conversation│◀─────┘
       │  (memory is      │  Memory Reader retrieves
       │   retrieved and  │  relevant past context
       │   injected)      │  based on current query
       └─────────────────┘

How it works: 1. After each conversation (or at key moments), a memory extraction process identifies facts, preferences, and decisions worth remembering 2. These are stored in a persistent store (vector database with metadata) 3. At the start of each new conversation, relevant memories are retrieved and injected into the system prompt 4. Memories can be updated, overridden, or deleted by the user

Pros: True personalization over time. The AI gets better as it knows more about the user. Enables relationship-like dynamics. Cons: Privacy implications (what do you store? for how long? can users delete?). Stale memories can cause problems (user changed jobs but AI still references old company). Requires careful memory management.

Real-world examples: - ChatGPT Memory (OpenAI): Stores user facts across conversations. Users can view, edit, and delete memories. Controlled by the user — they can say "remember this" or "forget that." - Google Gemini + Google account context: Uses your Google account data (with permission) to personalize responses — knowing your calendar, location, interests. - Rewind AI (now Limitless): Records everything you see and hear, creates a searchable persistent memory of your entire digital life.

3.3.5 Memory Decay

What it is: A technique where memories have a "freshness" score that decays over time. Recent memories are weighted more heavily than old ones. Memories that are neither accessed nor reinforced gradually fade — just like human memory.

How it works:

Memory: "User prefers window seats on flights."
 - Created: Jan 2025
 - Last accessed: Oct 2025
 - Decay score: 0.35 (fading)

Memory: "User is vegetarian."
 - Created: Jan 2025
 - Last accessed: Feb 2026 (yesterday)
 - Decay score: 0.95 (strong)

→ At retrieval, decay score weights relevance.
→ Vegetarian preference will be prioritized;
   window seat preference may not surface unless relevant.

Pros: Prevents memory bloat. Automatically handles outdated information. Models natural human memory patterns. Cons: May drop still-relevant but infrequently accessed information. Requires tuning decay rates per use case.

When to use: Products with long-term user relationships (AI companions, personal assistants, health coaching). Essential when memory stores grow to thousands of entries per user.

3.3.6 Memory Type Comparison

Memory Type	Persistence	Context Cost	Implementation Complexity	Best For
Buffer	Within session	High (grows linearly)	Low	Short sessions, support chats
Summary	Within session (can be persisted)	Bounded	Medium	Long sessions, advisory calls
Entity	Cross-session	Low (structured facts)	Medium-High	Personalization, CRM-like interactions
Long-Term	Persistent (days/months)	Medium (retrieved selectively)	High	AI companions, personal assistants
Memory Decay	Persistent with fading	Low-Medium	High	Long-term products with evolving user context

When to Use What: Memory Decision Framework

Does your product have multi-turn conversations?
├── No (single query/response) → No memory needed
├── Yes → How long are typical sessions?
│   ├── Short (< 10 turns) → Buffer Memory
│   ├── Long (10-50+ turns) → Summary Memory or Buffer with sliding window
│   └── Does the user return across sessions?
│       ├── No (one-off interactions) → Buffer or Summary are sufficient
│       └── Yes → What should persist?
│           ├── Specific facts/preferences → Entity Memory
│           ├── Broad user understanding → Long-Term Memory
│           └── Is context likely to change over time?
│               ├── Yes → Add Memory Decay
│               └── No → Static long-term store

💡 PM Action Item: Memory Design Exercise

Design the memory architecture for an AI fitness coach app: 1. What should the AI remember within a single workout session? (Buffer) 2. What should the AI remember between sessions? (Entity: weight, goals, injuries) 3. What should decay over time? (Old workout preferences, temporary injuries) 4. What should never be forgotten? (Allergies, chronic conditions) 5. What memory controls should the user have? (View, edit, delete)

Mind the privacy implications — memory inherently stores personal data. GDPR, CCPA, and other regulations apply. Build memory deletion ("right to be forgotten") from day one, not as an afterthought.

3.4 Tools: Letting Models Act in the Real World

The Problem

A base LLM can only read text and write text. It cannot check flight availability, run a database query, send an email, execute code, browse a webpage, or interact with any external system. But users expect AI products to do things, not just say things.

Tools bridge this gap. They give the model the ability to take actions in the real world — turning it from an oracle into an agent.

3.4.1 Types of Tools

Tool Category	Examples	What It Enables
APIs	Weather API, booking API, payment API, CRM API	Access real-time data, execute transactions, interact with third-party services
Databases	SQL query execution, NoSQL lookups, CRM record retrieval	Read/write structured data, look up user accounts, query product catalogs
Code Execution	Python interpreter, JavaScript runtime, Jupyter notebooks	Perform calculations, data analysis, visualization, run simulations
Web Browsing	URL fetching, web scraping, search engine queries	Access real-time web information, verify facts, research topics
File Operations	Read/write files, parse PDFs/CSVs/images, generate documents	Process uploaded documents, create reports, analyze spreadsheets
Communication	Email send, Slack message, calendar invite, SMS	Take action in the user's communication channels

3.4.2 How Tool Integration Works: Function Calling

Modern LLMs support function calling (also called "tool use") — a structured protocol where the model can request to call an external function with specific parameters, and the application layer actually executes it.

The Flow:

┌──────────┐    ┌───────────────┐    ┌──────────────┐    ┌──────────────┐
│  User:   │───▶│  LLM receives │───▶│  LLM decides │───▶│  App layer   │
│ "Book me │    │  prompt +     │    │  to call a   │    │  EXECUTES    │
│  a flight│    │  tool         │    │  tool:       │    │  the function│
│  to Tokyo│    │  descriptions │    │  search_     │    │  (API call   │
│  March 5"│    │               │    │  flights(    │    │   to airline) │
│          │    │               │    │  dest="TYO", │    │              │
│          │    │               │    │  date="3/5") │    │              │
└──────────┘    └───────────────┘    └──────┬───────┘    └──────┬───────┘
                                            │                    │
                                            ▼                    ▼
                                     ┌──────────────┐    ┌──────────────┐
                                     │  LLM formats │◀───│  Results     │
                                     │  response    │    │  returned to │
                                     │  using tool  │    │  LLM context │
                                     │  results     │    │              │
                                     └──────────────┘    └──────────────┘

Critical detail: The LLM never actually executes the function. It outputs a structured JSON request specifying which function to call and with what arguments. Your application code intercepts this, executes the real call, and feeds the results back to the LLM for response generation.

Example — function calling in practice:

// Tool description provided to the model:
{
  "name": "search_flights",
  "description": "Search for available flights between airports",
  "parameters": {
    "origin": "string — departure airport code",
    "destination": "string — arrival airport code",
    "date": "string — departure date (YYYY-MM-DD)",
    "passengers": "integer — number of passengers"
  }
}

// User says: "Find me flights from SFO to Tokyo on March 5th for 2 people"

// Model outputs (NOT a final response to user):
{
  "function_call": {
    "name": "search_flights",
    "arguments": {
      "origin": "SFO",
      "destination": "TYO",
      "date": "2026-03-05",
      "passengers": 2
    }
  }
}

// Your app executes the actual API call, gets results, feeds them back.
// Model then generates: "I found 3 flights from SFO to Tokyo on March 5th:
//   1. JAL 001 — Departs 11:30, arrives 15:30+1, $1,200/person
//   2. ANA 007 — Departs 13:00, arrives 17:00+1, $1,150/person
//   3. United 837 — Departs 10:00, arrives 14:00+1, $980/person"

Tool Description Quality Matters Enormously. The model decides which tool to use and how to fill parameters based entirely on the text descriptions you provide. Vague descriptions → wrong tool selections. Ambiguous parameter descriptions → incorrect arguments. Treat tool descriptions like API documentation for a very smart intern who has never used your system before.

3.4.3 Tool Orchestration Patterns

Pattern	Description	Example	When to Use
Sequential	Tools called one after another, each depending on the previous result	Search flights → Select flight → Book flight → Send confirmation email	Workflows with dependencies between steps
Parallel	Multiple tools called simultaneously, results combined	Check weather AND search hotels AND find restaurants for a trip	Independent data gathering from multiple sources
Conditional	Tool calls depend on previous results or conditions	If order status = "shipped" → call tracking API; if "processing" → call warehouse API	Branching workflows with different paths
Iterative	Tool called repeatedly until a condition is met	Query database → not enough results → broaden search → query again	Search/exploration tasks with uncertain scope
Nested	Tool results are fed into other tool calls as inputs	Search for company → extract CEO name → search for CEO's recent talks	Multi-hop information gathering

Real-world complexity: Production tool orchestration often combines multiple patterns. An AI travel agent might: 1. Parallel: Search flights, hotels, and car rentals simultaneously 2. Sequential: Take selected flight → check visa requirements for that route 3. Conditional: If visa required → provide application instructions; if not → proceed to booking 4. Iterative: If user doesn't like options → adjust parameters and search again

3.4.4 Tools in the Wild: Real Product Examples

Product	Tools Integrated	How It Works	Product Impact
ChatGPT Code Interpreter	Python execution environment, file upload/download	Model writes Python code, executes it in a sandboxed environment, returns results + visualizations	Solves math precisely, creates charts, analyzes data files — things the LLM alone can't do reliably
Claude Computer Use	Screen reading, mouse/keyboard control, application interaction	Claude can see a computer screen, click buttons, type text, navigate applications — like a human using a computer	Enables automation of any desktop workflow without custom API integrations
Gemini Extensions	Google Search, Maps, Hotels, Flights, YouTube, Workspace	Gemini calls Google's own services as tools, enabling real-time information and action within Google's ecosystem	Deep integration with Google's product suite gives Gemini unique action capabilities
Perplexity Web Search	Real-time web search, page fetching, citation extraction	Every query triggers web searches, fetches pages, extracts and cites relevant information	Turns an LLM into a research engine — every answer is grounded in current web sources
Expedia ChatGPT Plugin	Flight search, hotel search, activity search, trip planning APIs	ChatGPT queries Expedia's inventory through structured API calls	Natural language trip planning backed by real availability and pricing
Shopify Sidekick	Store analytics, product management, discount creation, report generation	AI assistant calls Shopify's internal APIs to read and modify store data	Merchants manage their store through conversation instead of navigating admin panels

3.4.5 Tool Design Principles for PMs

Principle of Least Privilege: Give the model access only to the tools it needs. A customer support bot shouldn't have access to your deployment pipeline. Each tool should have the narrowest possible permissions.
Human-in-the-Loop for Destructive Actions: For any tool that modifies data (database writes, emails, purchases), require user confirmation before execution. "I'll now book this $1,200 flight. Shall I proceed?" — not silent execution.
Graceful Failure: Tools will fail (APIs go down, rate limits hit, invalid parameters). Design for this: the model should explain the failure to the user and suggest alternatives, not crash silently.
Tool Latency Budget: Each tool call adds latency. If a tool call takes 3 seconds, and you chain 4 calls, that's 12 seconds of wait time before the user sees a response. Set latency budgets and design the UX accordingly — stream intermediate status updates.
Cost Accounting: Tool calls themselves may have costs (API fees, compute for code execution), plus each tool round-trip adds tokens to the prompt (tool descriptions, function call results). Account for both in your unit economics model.
Observability: Log every tool call, its parameters, its results, and the model's decision-making process. You need to debug why the model called the wrong tool, used wrong parameters, or missed a tool it should have called.

When to Use What: Tool Decision Framework

Does your AI need to interact with external systems?
├── No → No tools needed. Pure text generation.
├── Yes → What kind of interaction?
│   ├── Read-only data access → API calls / database queries
│   │   └── Real-time data? → Web search / live API
│   │   └── Your own data? → Database tools with read permissions
│   ├── Computation → Code execution (sandboxed)
│   ├── Actions (write/modify) → API calls with confirmation step
│   │   └── How critical? → Add human-in-the-loop for high-stakes actions
│   └── Multi-step workflows → Tool orchestration with sequential/parallel patterns
│       └── How complex? → Consider an agentic framework (LangChain, CrewAI)

💡 PM Action Item: Tool Inventory

For your AI product, create a tool inventory:

Tool	Category	Read/Write	Risk Level	Confirmation Required?	Latency Budget
Example: Check order status	Database	Read	Low	No	< 2s
Example: Process refund	API	Write	High	Yes — show amount, confirm	< 5s

Map every external action your AI needs to take. For each, define the risk level and whether human confirmation is required. This becomes a critical input into your security review and your UX design.

3.5 Putting It All Together: The Enhancement Stack

The four layers — Knowledge, Reasoning, Tools, Memory — are not independent. The best AI products combine them:

Example: A Premium AI Travel Agent

Component	Implementation
Knowledge (RAG)	Retrieves from: destination guides, visa requirements, hotel reviews, airline policies. Enhanced RAG with hybrid search and re-ranking.
Reasoning (CoT)	Uses chain-of-thought for trip planning: budget allocation → itinerary optimization → alternative suggestions.
Memory	Entity memory for traveler preferences (aisle seat, vegetarian, budget range). Long-term memory for past trips (don't suggest places they've been). Memory decay for time-sensitive preferences (visited Paris last year — low decay; prefers Marriott — high persistence).
Tools	Flight search API, hotel booking API, weather API, visa checker API, email for sending itineraries, code execution for budget calculations. Sequential + parallel orchestration.

User: "Plan me a week in Japan in March, similar budget to my Italy trip but
       with more cultural experiences."

1. MEMORY retrieves: Italy trip was $4,500. Budget ~= $4,500-5,000.
   User prefers ryokans. User is vegetarian. User has valid passport
   (expires 2028).

2. RAG retrieves: Japan travel guide chunks about March (cherry blossom
   season), cultural experiences (tea ceremonies, temple stays, kaiseki
   dining), vegetarian dining in Kyoto/Tokyo.

3. TOOLS execute (parallel): Search flights SFO→TYO March dates.
   Search ryokans in Kyoto. Check Japan visa requirements for US citizens.
   Get March weather forecast.

4. REASONING (CoT): Given $5K budget and 7 days, allocate: flights ~$1,200,
   accommodation ~$1,800 (7 nights × $250/night ryokan), activities ~$800,
   food ~$700, transport ~$500. Remaining buffer: ~$400.
   Optimize itinerary: Tokyo (3 days) → Kyoto (3 days) → Osaka (1 day).

5. Response: Detailed, personalized itinerary with booking links, budget
   breakdown, and cultural experience recommendations — all grounded in
   real availability and the user's actual preferences.

3.6 PM Action Items & Exercises

Exercise 1: Enhancement Layer Mapping

Pick a product you use daily (ChatGPT, Notion AI, Google Gemini, Perplexity). For each enhancement layer, identify:

Layer	Is it used?	How do you know?	How good is the implementation?
Knowledge (RAG)
Reasoning
Memory
Tools

Exercise 2: Design a RAG Pipeline

You're the PM for a customer support AI at a large e-commerce company. Design the RAG pipeline: 1. What knowledge sources need to be indexed? (Help articles, order data, return policies, product specs...) 2. What chunking strategy would you use for each source? 3. Would you use naive, enhanced, or modular RAG? Why? 4. How would you handle permission-sensitive data (orders belong to specific customers)? 5. How would you measure RAG quality?

Exercise 3: Reasoning Cost-Benefit Analysis

Your team wants to add CoT reasoning to your AI product's complex query handling. Currently, 20% of queries are "complex." Calculate: - How many complex queries per day? - What's the current failure rate on complex queries? - What improvement would CoT provide (use the comparison table)? - What's the additional cost in tokens and dollars? - What's the business value of that quality improvement?

Should you use CoT? CoT-SC? A reasoning model? Or is the cost not justified?

Exercise 4: Memory Architecture Design

Design the memory system for one of these products: - (A) An AI-powered personal finance advisor - (B) A language learning app with an AI tutor - (C) An AI customer support agent for a SaaS product

For your chosen product, specify: 1. What types of memory do you need? (Buffer/Summary/Entity/Long-term/Decay) 2. What specific information should be stored? 3. What should decay vs. persist permanently? 4. What user controls are needed? (View, edit, delete, export) 5. What are the privacy/compliance implications?

Exercise 5: Tool Safety Audit

Review the tools that an AI agent for an online banking app might need: - Check account balance (read) - View transaction history (read) - Transfer money between accounts (write) - Pay bills (write) - Open a new account (write) - Close an account (write — destructive)

For each tool: 1. What's the risk level? 2. What confirmation UX is needed? 3. What are the maximum parameter values (e.g., transfer limit)? 4. What fraud detection should wrap the tool call? 5. What audit trail is required?

3.7 Discussion Questions

The RAG Quality Ceiling: Your team built a RAG-powered customer support bot. It answers 70% of queries correctly. The remaining 30% get wrong retrievals. You've already implemented enhanced RAG with re-ranking. What's your next move — better chunking? More data? Fine-tuning the embedding model? Or is 70% the ceiling for this architecture?
The Reasoning Cost Dilemma: Your AI-powered code review tool uses o1-level reasoning and catches 40% more bugs than GPT-4o — but costs 12x more per review. The engineering team generates 500 code reviews/day. At what point does the quality improvement justify the cost? How would you design a tiered approach?
Memory and Privacy Tension: Users of your AI health coach love that it remembers their conditions, medications, and fitness history. But you're launching in the EU (GDPR) and in healthcare (HIPAA). How do you design a memory system that is both deeply personalized and fully compliant? What happens when a user exercises their "right to be forgotten"?
Tool Risk Management: Your AI travel agent can now book flights and hotels on behalf of users — executing real financial transactions. A bug causes the model to misinterpret "cancel my booking" as "book a new trip." How would you prevent this class of error? What's the right balance between automation and confirmation friction?
Build Order for a New Product: You're building an AI-powered research analyst for investment firms. You can't build all four enhancement layers at once. In what order would you build Knowledge (RAG), Reasoning, Memory, and Tools? Justify your sequencing based on user value, implementation complexity, and risk.
Competitive Moat: If everyone uses the same foundation models (GPT-4, Claude) and the same RAG frameworks (LangChain, LlamaIndex), where does competitive advantage come from? Is it data? UX? Orchestration logic? Memory design? Tool integrations? Which enhancement layer is hardest for competitors to replicate?

3.8 Key Takeaways

RAG is the single most important production AI pattern. It solves hallucination and knowledge cutoff simultaneously by grounding model responses in retrieved, verified information. Start with naive RAG, graduate to enhanced and modular as your quality requirements increase. The choice of chunking strategy alone can make or break your RAG system.
Reasoning is a compute-for-quality tradeoff. Chain of Thought, Tree of Thoughts, and reasoning models (o1/o3) dramatically improve output quality on complex tasks — at 2-50x the cost. Use reasoning selectively: triage queries by complexity and apply reasoning only where it changes the outcome. Don't use an o1 model for tasks a Haiku can handle.
Memory transforms a demo into a product. Without memory, every interaction starts from zero. Buffer memory handles sessions; entity and long-term memory handle personalization over time; memory decay prevents staleness. Design memory with privacy as a first-class concern — you're storing personal data, and regulations apply.
Tools turn language models into actors. Function calling lets models interact with APIs, databases, code interpreters, and real-world services. The model decides what to do; your application executes it. Apply least privilege, require confirmation for destructive actions, and budget for tool latency in your UX.
The four layers compound. The best AI products combine RAG, reasoning, tools, and memory into integrated systems. A travel agent that remembers your preferences (memory), searches real inventory (tools), reasons about budget allocation (reasoning), and grounds recommendations in travel guides (RAG) delivers a profoundly better experience than any single layer alone.
Your competitive moat is in the enhancement layers, not the model. Everyone has access to GPT-4 and Claude. Your differentiation comes from your proprietary knowledge base (RAG), your domain-specific reasoning chains, your accumulated user memory, and your unique tool integrations. Design these layers as your core product IP.
Start with the highest-impact, lowest-risk layer. For most products: RAG first (grounds answers in real data), then tools (lets the AI take action), then memory (personalizes over time), then advanced reasoning (improves quality on hard tasks). This sequence maximizes early user value while managing implementation risk.

🛠️ Section 3: Knowledge, Reasoning, Tools & Memory

3.1 Knowledge Augmentation: Retrieval-Augmented Generation (RAG)

The Problem RAG Solves

3.1.1 Naive RAG: The Basic Pattern

3.1.2 Enhanced RAG: Smarter Retrieval

Query Rewriting / Expansion

Re-Ranking

Hybrid Search

3.1.3 Modular RAG: Composable Pipelines

3.1.4 RAG Components Deep Dive

Embeddings: How Meaning Becomes Math

Vector Databases: Where Embeddings Live

Chunking Strategies: How You Split Documents Matters More Than You Think

Retrieval Strategies

3.1.5 RAG in the Wild: Real Product Examples

3.1.6 RAG Comparison Matrix

When to Use What: RAG Decision Framework

💡 PM Action Item: RAG Audit

3.2 Reasoning: Making Models Think Better

The Problem

3.2.1 Type 1 vs. Type 2 Thinking (Kahneman's Framework Applied to AI)

3.2.2 Chain of Thought (CoT)

3.2.3 Chain of Thought — Self-Consistency (CoT-SC)

3.2.4 Auto Chain of Thought

3.2.5 Tree of Thoughts (ToT)

3.2.6 Graph of Thoughts (GoT)

3.2.7 Reasoning Strategy Comparison

Real-World Reasoning Products

When to Use What: Reasoning Decision Framework

💡 PM Action Item: Reasoning Cost Calculator

3.3 Memory: Making AI Remember

The Problem

3.3.1 Conversation Buffer Memory

3.3.2 Summary Memory

3.3.3 Entity Memory

3.3.4 Long-Term Memory

3.3.5 Memory Decay

3.3.6 Memory Type Comparison

When to Use What: Memory Decision Framework

💡 PM Action Item: Memory Design Exercise

3.4 Tools: Letting Models Act in the Real World

The Problem

3.4.1 Types of Tools

3.4.2 How Tool Integration Works: Function Calling

3.4.3 Tool Orchestration Patterns

3.4.4 Tools in the Wild: Real Product Examples

3.4.5 Tool Design Principles for PMs

When to Use What: Tool Decision Framework

💡 PM Action Item: Tool Inventory

3.5 Putting It All Together: The Enhancement Stack

3.6 PM Action Items & Exercises

Exercise 1: Enhancement Layer Mapping

Exercise 2: Design a RAG Pipeline

Exercise 3: Reasoning Cost-Benefit Analysis

Exercise 4: Memory Architecture Design

Exercise 5: Tool Safety Audit

3.7 Discussion Questions

3.8 Key Takeaways