robots.txt

robots.ts: Crawl Rules as Code

Similar to sitemap.ts, creating a file named robots.ts inside app/ and exporting a default function returning a rules object is all it takes — Next.js serves the result as a properly formatted robots.txt at the site's root.

This is most commonly used to disallow crawling of routes that shouldn't be indexed — admin dashboards, API routes, or internal tooling — while explicitly allowing everything else, and to point crawlers toward your sitemap.

// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: "/",
      disallow: ["/admin/", "/api/"],
    },
    sitemap: "https://example.com/sitemap.xml",
  };
}

This allows crawling of the entire site except /admin/ and /api/, and points crawlers to the sitemap.

Robots Rule Shape

{
  rules: {
    userAgent: string | string[]; // "*" matches all crawlers
    allow?: string | string[];
    disallow?: string | string[];
  };
  sitemap?: string;
  host?: string;
}

• userAgent: "*" applies the rule to every crawler; specific crawler names can be targeted individually.
• allow and disallow accept a single path or an array of paths.
• sitemap should point to the fully-qualified sitemap URL generated by sitemap.ts.
• Multiple rule objects can be provided (as an array) to give different crawlers different rules.

robots.txt Cheat Sheet

Key facts about the robots.ts convention.

Different Rules for Different Crawlers

You can provide an array of rule objects to apply different allow/disallow rules to different user agents — for example, giving a specific rule to a particular bot while keeping a general rule for everyone else.

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", allow: "/" },
      { userAgent: "SomeAggressiveBot", disallow: "/" },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

robots.txt vs. Per-Page noindex

robots.txt controls crawling (whether a bot visits a URL at all), while a page's metadata.robots field (part of the Metadata API) controls indexing (whether a page that was crawled should appear in search results). A page can be crawlable but marked noindex, or vice versa — they solve related but distinct problems.

Common Mistakes

• Confusing robots.txt (crawling) with per-page noindex metadata (indexing) — they solve different problems.
• Accidentally disallowing your entire site (disallow: "/") with an overly broad rule.
• Forgetting to reference the sitemap, missing an easy way to help crawlers discover content.
• Assuming robots.txt enforces security — it's a voluntary directive, not an access control mechanism.

Key Takeaways

• app/robots.ts generates /robots.txt automatically from an exported function.
• allow/disallow rules control which paths crawlers may visit.
• sitemap in the returned object points crawlers to your generated sitemap.
• robots.txt controls crawling; per-page metadata controls indexing — they're complementary, not interchangeable.
• robots.txt is a voluntary directive, not a security boundary — never rely on it to hide sensitive content.