AI Agents ACT NOW OPPORTUNITY

Nvidia's Nemotron Lightning and Switchyard: free tools for companies running AI agents

ShiniFrontiers · Source: NVIDIA

Section 1 of 6 FREE PREVIEW

What problem does it solve?

An AI agent is software that uses an AI model to finish a task on its own, step by step, such as a coding assistant that reviews code or a support bot that answers billing questions. Nvidia has released two free, open tools for companies running one: Nemotron 3.5 Lightning, a small AI model that does the agent's routine steps quickly, and NeMo Switchyard, a library that decides which AI model handles each step.

A. Nemotron 3.5 Lightning — for developers building AI agents

Problem: An AI agent makes many small calls: run a tool, check a result, format an answer. Sending each to a top-tier paid model adds cost, delay and outside data sharing.

What it does: A small open model that handles those routine calls fast, on a hosted endpoint or the business's own graphics card.

Note: Hard reasoning and long multi-step work still need a larger model.

B. NeMo Switchyard — for teams paying for several AI models

Problem: Sending everything to the best model overspends; sending everything to a cheap one loses quality on hard tasks.

What it does: Sends each request to a model from the business's own list (any mix of Nvidia, open or paid models), moving up to a stronger one when the work gets hard.

Note: Early software (version 0.3); settings can change between releases.

See the difference

Situation	Today	With Lightning and Switchyard
Nemotron 3.5 LightningA coding agent runs hundreds of routine steps a day	Every step goes to a top-tier model, paid per use	Routine steps run on Lightning on the business's own graphics card, no per-use fee
Nemotron 3.5 LightningCustomer data must not leave the business	Pick a weak local model, or send the data out	Runs in-house on one H100 (Nvidia's data-centre graphics card) or one DGX Spark (its desktop AI computer)
NeMo SwitchyardA support conversation goes smoothly, then hits a hard problem	One fixed model for the whole conversation: too costly, or too weak	Starts on the cheaper model; Switchyard moves it to the top-tier model when it struggles
NeMo SwitchyardThe business adds a new model or switches provider	Developers rewrite each connection by hand	The business updates its model list; the routing setup stays the same

Illustrative situations based on the sources; measured partner results appear further down.

What it saves: Top-tier AI fees on an agent's routine steps, which move to a cheaper model.
What it costs: Both are free to download. The business pays for graphics cards or hosted use, plus developer time to set up and test.
How ready: Working in the real world

The model is released for commercial use, with a free trial endpoint. The router is early, pre-1.0 software; its standalone server is demo-only.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

Join free to read it all →

Section 2 of 6 MEMBERS

How does it work?

Join free to read this section

Nvidia's Nemotron Lightning and Switchyard: free tools for companies running AI agents — how it fits together.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

The big picture

Picture a law firm: a senior partner plans the case, and junior associates handle routine paperwork at a fraction of the cost. Lightning is the fast junior. Switchyard is the office manager who decides which work goes to whom, and passes a file up to the partner when a junior gets stuck.

How it fits into the business's AI setup

A request from the business's AI agent travels down the stack. Colours show who owns each part; the dashed line is where data leaves the company.

Agent. The business's AI agent: a support bot or coding agent that customers and staff use. Each task is many small model calls. Connects once to Switchyard instead of to each model; adding or swapping a model later needs no rewiring.

Routing. NeMo Switchyard: free, open source, early (v0.3). 74% lower cost in LangChain's test (Escalation setting), at about 6 points lower accuracy.

Models. Nemotron 3.5 Lightning: on the business's own graphics card. Data stays in; no per-call fee. Only the hard 7 in 100 calls are paid at top-tier prices. If Lightning is hosted, all calls leave the company.

Hardware. One H100 or one DGX Spark: Nvidia's data-centre graphics card or desktop AI computer; ordinary desktop cards work with a shorter context window. Cost moves from per-call fees to hardware the business owns.

Switchyard: technical view

Why not just use one top-tier model for everything?

One top-tier model is the simplest setup and the most accurate, but every routine call pays the top price. Model routing trades a small, measurable accuracy drop for a lower bill.

	One top-tier model for every step	Switchyard routing between models
Cost	❌ Top price on every call, routine or not	✅ Routine calls go to a cheaper model
Accuracy on hard tasks	✅ The highest available	⚠️ Slightly lower: about 6 points in LangChain's test
Where data goes	⚠️ To an outside AI provider	✅ Routine steps can stay in-house with a local model like Lightning
Changing models later	❌ Rewire the agent	✅ Edit the model list; the agent stays the same
Setup effort	✅ One connection	❌ A routing setup to configure and test

How good is Lightning?

Lightning is a mixture-of-experts model: 30 billion parts in total, but only 3 billion work on each word, so it runs at the cost of a small model. On NVIDIA's tests it gets 83 in 100 everyday agent tasks right and fixes about half of real code bugs, but only 23 in 100 hard command-line tasks: route those to a larger model.

Test	Lightning's score	What that means
PinchBench: everyday assistant-style agent tasks	83 in 100	Strong on the routine work it is built for
SWE-bench Verified: fixing real bugs in open-source code	53 in 100	Fixes about half the bugs it is given; changes still need review
Terminal-Bench 2.1: hard multi-step command-line tasks	23 in 100	Weak here: send this work to a larger model
τ³-bench (Banking): an agent test in a banking setting	9 in 100	Very weak: do not rely on it alone for this kind of work
Speed	Up to 4× the output of similar-sized models	Finished 10,000 tasks 30% faster than Qwen3.6 35B at similar accuracy

Part A: how Lightning fits in

Who	What happens	In everyday terms
1. The developer	Chooses where it runs: a hosted endpoint, the business's server, or a desktop	Deciding whether to hire through an agency or in-house
2. The agent	Sends its routine calls to Lightning	The partner hands paperwork to a junior
3. Lightning	Answers fast, using speculative decoding: drafting several words at once and checking them	A fast typist who guesses the next few words and corrects as needed
4. The developer (optional)	Trains it further on the business's own data with NVIDIA's NeMo tools	Training the junior on the firm's own past cases

Part B: how a request moves through Switchyard

Who	What happens	In everyday terms
1. The agent	Sends its next step, with recent tool results	A junior brings the office manager the next task
2. Switchyard	Picks a model from the business's list; routine work goes to the efficient one	The manager gives routine paperwork to a junior
3. Switchyard	Moves the session to the top-tier model after repeated errors or loops	A stuck file goes up to the partner
4. The chosen model	Returns the answer; the agent carries on	The finished work comes back

How Switchyard decides which model gets a call

Switchyard is not tied to Nvidia: the business lists any mix of models, such as Lightning, Claude or other open and paid models, and Switchyard picks one for each call. The business chooses how it decides, and Auto is the default.

Setting	How it decides	In everyday terms
Auto (default)	Watches the agent's recent work: starts on the low-cost model; repeated errors, going in circles or long searching move it up; steady edits and passing tests keep it low	A manager who steps in when a junior keeps hitting the same wall
Task	A judge model reads each request and decides up front whether the low-cost model can handle it	Sorting the post before handing it out
Escalation	Starts low; a judge model checks progress each turn and moves the whole session up when it sees sustained difficulty	A help desk passing a stuck case to a senior
Composite	The judge sets the starting level; the agent's recent work decides after that	Sorting first, then stepping in if needed

Lightning's weights, training data and recipes are published under the OpenMDW-1.1 licence, and the model card marks it ready for commercial use.
Switchyard is open source under Apache 2.0; the business chooses the models, and its own gateway or application makes the calls.

Join free → I have an account

Section 3 of 6 MEMBERS

How to set it up, and what it costs

Join free to read this section

Nvidia's Nemotron Lightning and Switchyard: free tools for companies running AI agents — steps and cost.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

Before you start: is it a fit?

	Lightning	Switchyard
Good fit if	Agents make many routine calls now billed at top-tier prices	The business already pays for two or more AI models
	Data must stay on the business's own hardware	Agents run long, multi-step tasks
	The business has an H100, a DGX Spark, or uses a hosted endpoint	A developer can test routing against the current single-model setup
Who acts on it	A developer or AI engineering lead	The AI platform lead, with whoever owns the AI bill
Not yet if	The work is mostly hard reasoning or long command-line tasks	The business needs a supported, production-grade router today

Try it now — no install

Try the model in the browser (anyone) Open https://build.nvidia.com/nvidia/nemotron-3.5-lightning-30b-a3b, which offers a free endpoint for prototyping. A free version is also on OpenRouter: https://openrouter.ai/nvidia/nemotron-3.5-lightning:free. Expect: answers to the business's own kind of task, to judge speed and quality.

Before you begin

You need	Why
A developer	To connect the model and the router to the agent
A graphics card, or a hosted endpoint	One H100 (Nvidia's data-centre graphics card) or one DGX Spark (its desktop AI computer) runs the model; ordinary desktop graphics cards run it with a shorter context window
A baseline	The agent's current cost and success rate on real tasks, to compare against
For Switchyard: a gateway plug-in or a developer	Either a plug-in for the LiteLLM or NeMo Relay gateway, or the library added to the business's own software (it is written in Rust, a programming language; a Python example is provided)

Part 1: Run Lightning on the business's own machine

Run it on a desktop with Ollama (the developer) Expect: a chat with tool-calling and reasoning on. Ollama sets the context window by graphics memory: 4K tokens under 24GB, 32K for 24-48GB, 256K above.
Or serve it with llama.cpp (the developer) Expect: an OpenAI-compatible API on port 8000.
On a data-centre graphics card, use the model card's recipe (the developer) The Hugging Face model card gives tuned vLLM, SGLang and TensorRT-LLM commands for DGX Spark, H100, GB200 and A100. Copy the one for the business's hardware.
Point the agent at it (the developer) Use the model name the server reports at GET /v1/models; the name below is the vLLM default. For coding agents, the model card says to add force_nonempty_content to each call.

Part 2: Add routing with Switchyard

Choose the model pool (the AI platform lead) List the models the agent may use, for example Lightning for routine steps and the current top-tier model for hard ones.
Choose how to connect it (the developer) Use the LiteLLM or NeMo Relay plug-in, or embed the library in the business's own code. The standalone server is for demos only, not production.
Start with the Auto routing choice (the developer) In Switchyard's configuration file. Auto starts on the efficient model and moves up based on recent tool activity.
Compare against the current setup (the AI platform lead, with the owner of the AI bill) Run the same real tasks both ways and compare cost per finished task, not per call: Switchyard's guide warns a cheaper call does not guarantee a cheaper finished task. Expect: a cost and success-rate comparison to decide on.
Pin the version (the developer) Switchyard is pre-1.0, so behaviour can change between releases.

What it costs

Item	Cost
The model	Free to download (OpenMDW-1.1 licence; marked ready for commercial use)
The router	Free (Apache 2.0 open-source licence)
Hosted trial	A free prototyping endpoint on build.nvidia.com; a free version on OpenRouter
NVIDIA's packaged server in production	Needs NVIDIA AI Enterprise; a 90-day evaluation is offered. The sources give no price.
Hardware	One H100 or one DGX Spark for the model; desktop graphics cards with a shorter context window
People	Developer time to connect and test; platform-lead time to compare results

Join free → I have an account

	What changes	What it can mean for the business
AI bill	Routine agent calls move off top-tier models	A smaller bill for the same agent work
Speed	Routine steps run on a fast small model	Agents finish long tasks sooner
Data	Routine steps can run on the business's own hardware	Less customer data sent to outside providers
Flexibility	Models sit behind one routing setup	Switching or adding a provider does not mean rewiring agents

Partner	What they reported	What it means
LangChain	74% lower cost over 145 multi-turn tasks; 7% of calls went to the top-tier model; about 6 points less accurate	Large savings, if a small accuracy drop is acceptable
Ramp	58% lower cost and 33% shorter runtime, matching a top-tier model on its own coding test	Savings with no reported accuracy loss on that test
Cognition	28% lower average cost, within 2.8 points of Opus 5, routing Opus 5 and Kimi K2.7	Near top-tier coding results for less
Classmethod	27% lower cost in initial testing, quality maintained	An early, internal result
Boomi	59% of traffic went to a 5× faster fine-tuned model; later turns 21% faster	Quicker replies in long conversations

Section 5 of 6 MEMBERS

What are the limits?

Join free to read this section

Nvidia's Nemotron Lightning and Switchyard: free tools for companies running AI agents — where it falls short.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

What these tools can't do yet

Limit	Why it matters	What to do
The router is pre-1.0	Its interface, settings and routing behaviour can change; the standalone server is demo-only.	Pin a version; in production use a gateway plug-in or the library, not the demo server.
Weak on hard command-line work	Lightning scores 23 in 100 on Terminal-Bench 2.1.	Route such tasks to a larger model.
Short context on desktops	Ollama gives 4K tokens under 24GB of graphics memory; the llama.cpp example uses about 40K.	Use a data-centre graphics card for long documents.
NVIDIA's own scores	Measured on NVIDIA's harness; the blog and model card differ on PinchBench (86 against 83.4).	Test on the business's own tasks before relying on it.
Routing can cost accuracy	LangChain's 74% saving came with about 6 points lower accuracy.	Set a quality bar first, and measure against it.

Limit	Why it matters	What to do
The LiteLLM plug-in is limited	Experimental, install-from-source only, and supports only Execution and Random routing.	Use NeMo Relay or the library for Task or escalation routing.
A NeMo Relay error bug	Relay 0.8.x and 0.9.0 can lose upstream error details with the plug-in on, even for other models.	Read Switchyard's Relay compatibility note before enabling it.
No published RTX 5090 recipe	The model card's hardware table marks the RTX 5090 setup "confirm", with no recipe.	Test it before planning an RTX 5090 rollout.

What it does not do

Switchyard does not host models or run a service; the business's gateway or application makes the calls.
Lightning does not replace a top-tier model for planning and hard reasoning.
Neither tool guarantees a saving; the business has to measure it on its own work.
Lightning reads and writes text only.

Join free → I have an account

Nvidia's Nemotron Lightning and Switchyard: free tools for companies running AI agents

What problem does it solve?

How does it work?

The big picture

How it fits into the business's AI setup

Why not just use one top-tier model for everything?

How good is Lightning?

Part A: how Lightning fits in

Part B: how a request moves through Switchyard

How Switchyard decides which model gets a call

How to set it up, and what it costs

Before you start: is it a fit?

Try it now — no install

Before you begin

Part 1: Run Lightning on the business's own machine

Part 2: Add routing with Switchyard

What it costs

What do you gain?

What partners measured with Switchyard

What are the limits?

What these tools can't do yet

What it does not do

Sources