Deploy and Host Mastra Evals with Memory on Railway

Mastra is the open-source TypeScript framework for building AI agents and workflows. This template runs three working recipes for evaluating memory-enabled agents with Mastra evals: a global runEvals call with shared memory, a per-item loop that gives each dataset row its own thread, and a Dataset.startExperiment run that forwards memory through an inline task. It uses a deterministic mock model, so runs are reproducible and need no API keys.

About Hosting Mastra Evals with Memory

This is a run-once demo service. On deploy, it executes all three eval recipes and prints scored results to the deploy logs, then exits. Redeploy to run the evals again. Swap the mock model for a real provider and point the dataset at your own agent to turn it into a real eval harness for CI or scheduled quality checks (add a cron schedule in service settings).

Common Use Cases

Learn how to wire Mastra evals to agents that use conversation memory
Bootstrap an eval harness for your own agents with reproducible mock runs
Run scheduled agent-quality checks by adding a cron schedule to the service

Dependencies for Mastra Evals with Memory Hosting

None. The demo uses a mock model and temporary databases.

Deployment Dependencies

Mastra evals docs

Implementation Details

View results in the deployment logs:

railway logs

Why Deploy Mastra Evals with Memory on Railway?

Railway is a singular platform to deploy your infrastructure stack. Railway will host your infrastructure so you don't have to deal with configuration, while allowing you to vertically and horizontally scale it.

By deploying Mastra Evals with Memory on Railway, you are one step closer to supporting a complete full-stack application with minimal burden. Host your servers, databases, AI agents, and more on Railway.