Train an agent to use google search as a tool with RL

Want to train an AI agent that can use external tools like search engines, calculators, or APIs? This tutorial shows you how to build tool-calling agents using reinforcement learning — agents that learn when and how to request external information to solve complex problems.

We use VERL (a production-ready RL training framework) to train agents with search tool integration and SkyPilot to run and scale training on any AI infrastructure, including Kubernetes and clouds.

What You'll Build #

By the end of this tutorial, you'll have:

1. Set up a search-augmented RL pipeline — Bring up a retrieval service, prepare tool-aware data, and configure VERL for tool-calling rollouts.
2. Trained and evaluated the agent — Compare the trained agent's answers against a baseline (no tools) on questions that require external knowledge.
3. Swapped retrieval backends — Run the same agent with Wikipedia and Google Search to compare behavior and quality.
4. Scaled the system — Separate the retrieval service from RL training across nodes to improve throughput and reliability.

All of this is orchestrated from a single SkyPilot YAML file, with optional multi-node setup that cleanly separates retrieval and training for scale.

Why Tool-Calling Agents Matter #

Without tool usage, the LLM has only knowledge up to when it was trained and cannot verify any results

Traditional language models are limited by their training data cutoff. They can’t:

• Access real-time information
• Query specialized databases
• Perform calculations beyond their learned patterns
• Retrieve specific documents on demand

Tool-calling agents overcome these limitations by learning to:

• Recognize when they need external information
• Formulate appropriate queries to tools (search, APIs, calculators)
• Integrate retrieved results into their reasoning
• Optimize for task success using both internal knowledge and external tools

How Tool-Calling Works During Training #

During RL training, there is a trainer and multiple rollout workers that coordinate tool calls:

1. Model requests a tool call (e.g., <search> protein in egg yolks </search>)
2. Rollout worker processes the request asynchronously by calling the retrieval service
3. Model receives the response and continues generation with the retrieved information
4. Trainer calculates the rewards based on the final answer and updates the model

This creates a training loop where the agent learns optimal tool-calling strategies through reinforcement—when to search, how to formulate queries, and how to synthesize retrieved information.

Agent-tool interaction: The agent decides when to search, retrieves documents, and synthesizes answers

Example: Training a Search-Augmented Q&A Agent #

Let’s build an agent that uses Wikipedia search to answer questions it couldn’t otherwise answer correctly.

Step 1: Setting Up the Retrieval Service

Before training, we need a retrieval service that the agent can query. VERL’s example uses a local Wikipedia search service based on:

• E5-base-v2 embeddings (dense retrieval)
• FAISS indexing for fast similarity search
• 132GB Wikipedia corpus (full text + indices)
• FastAPI server exposing HTTP endpoints

The retrieval service runs independently from the training process, communicating via HTTP. This decoupled design means:

• Retrieval can scale separately from training
• Multiple training jobs can share one retrieval service
• Easy to swap different retrieval backends

Step 2: Preparing Tool-Calling Data

The data format extends VERL’s standard format with tool-calling annotations. Here’s how a training example looks:

Example Question:

Who won the Nobel Prize in Physics in 2023?

Expected Behavior:

1. Agent recognizes it doesn’t know (post-training-cutoff information)
2. Agent calls search tool: search("Nobel Prize Physics 2023")
3. Retrieval service returns Wikipedia excerpt
4. Agent synthesizes answer from retrieved text

Data Format:

Key differences from standard RL data:

• tool_config specifies available tools and endpoints
• requires_search hint (optional, for analysis)
• Multi-turn expected (agent may search multiple times)

The preprocessing script (preprocess_search_r1_dataset.py) handles:

• Downloading question-answer pairs
• Formatting for multi-turn conversations
• Validating retrieval service accessibility
• Creating train/validation splits

Step 3: Training an Agent with Tool Calling

The training uses SGLang backend for rollout to support tool calling. Key configuration differences:

Complete SkyPilot YAML:

Click to go to verl-search-tool.yaml in skypilot repo

Launch training:

What SkyPilot does under the hood?

SkyPilot orchestrating the retrieval service and distributed training

How effective is the RL training for tool use? #

Click to go to verl-search-interaction-infer.yaml in skypilot repo

Comparison 1: Trained Agent vs Base Model

Let’s first compare the trained agent against the base model on questions requiring search:

Test Question: “How much protein in four boiled egg yolks?”

Key takeaway: The trained agent recognizes when it needs external information and uses tools effectively, while the base model hallucinates with false confidence.

Comparison 2: Wikipedia (Stale Corpus) vs Google Search (Current Data)

The trained agent can work with different retrieval backends. This comparison highlights a key advantage: Google Search provides up-to-date information, while a Wikipedia corpus from 2023 may be outdated.

Test Question: “Who are the latest Nobel Prize winners in Physics?”

Step 4: Swapping the Retrieval Backend with Real Google Search

Once trained, agents can swap retrieval backends (Wikipedia → Google → custom APIs) without retraining, as they’ve learned the general skill of tool use. This lets you test the same agent with different knowledge sources and compare their effectiveness.

Running inference with different retrieval backends:

For Google Search backend:

Click to go to verl-search-interaction-google-infer.yaml in skypilot repo

The agent will interactively answer questions using the configured retrieval backend, demonstrating the search behaviors shown in the comparisons above.

Key takeaway: Once trained on the general skill of tool use, agents can swap retrieval backends without retraining. This flexibility lets you optimize for different use cases—Wikipedia for encyclopedic knowledge, Google for current information, or custom APIs for domain-specific data.

Scale up the RL post-training: Separating Rollout and Training #

Distributed RL training architecture with tool calling: Rollout workers handle tool invocations asynchronously (from VERL Agentic RL documentation)

For large-scale training, you can distribute the retrieval service and training across different machines to optimize resource utilization and performance.

Why separate retrieval and training:

1. Resource isolation: Retrieval service (CPU/memory intensive) doesn’t compete with training (GPU intensive)
2. Independent scaling: Scale retrieval throughput and training capacity separately
3. Better utilization: Dedicate high-memory nodes to retrieval, GPU nodes to training
4. Fault tolerance: Retrieval service can persist across training restarts

Architecture:

Multi-node scaling with SkyPilot:

The YAML can be extended to run retrieval and training on separate node types. We can scale the retrieval service easily using sky serve!

Click to go to verl-search-interaction-retrieval.yaml in skypilot repo

Benefits of this architecture:

• Use every spare GPU: Rollout workers are just HTTP clients, so you can run them on any available GPU (on-prem, k8s, or cloud) and point them at the same trainer + retrieval endpoints.
• Decouple sampling from training: Scale rollout workers for more trajectories, or trainer GPUs for faster updates — each can scale independently.

Screensshots:

RL trainer node

Retrieval node

Why SkyPilot for Tool-Calling Agents #

Infrastructure Challenges:

1. Need to run retrieval service + training simultaneously
2. Retrieval service requires persistent state (loaded indices)
3. Multi-node coordination with tool access
4. Checkpointing both model and tool usage stats

SkyPilot Solutions:

• Multi-process orchestration: Manages retrieval service + training
• Persistent storage: Mounts Wikipedia corpus efficiently
• Auto-recovery: Restarts all services on spot preemption
• Cross-cloud: Same YAML works on any infrastructure

Conclusion #

Tool-calling agents learn to access external knowledge beyond their training data. This tutorial showed how to train them with VERL and scale them with SkyPilot.

Key takeaway: In RL training, the rollout phase (where agents generate responses and call tools) and training phase (where models are updated) have different resource needs. Rollout needs fast retrieval access but minimal GPU, while training needs heavy GPU compute. SkyPilot lets you separate these cleanly—run the retrieval service on cheap CPU/GPU nodes that the rollout workers query via HTTP, while reserving expensive GPU nodes purely for the training loop. This architecture means you can maintain a persistent retrieval service shared across experiments while training nodes scale up or down as needed, optimizing both cost and performance.

Everything from single-node prototypes to multi-node production runs orchestrates from a single YAML file, on any infrastructure.

Next Steps #

• Complete YAML files: GitHub repo
• SkyPilot documentation: https://skypilot.readthedocs.io
• SkyPilot Slack: https://slack.skypilot.co

Ready to build tool-calling agents? Launch the example and join the community for support!