llms.txt Generator

Generate a standardized llms.txt file from your website's sitemap or URL list. Help AI crawlers understand your content structure and prioritize important pages.

Choose Your Input Method

Use Sitemap ^{(recommended)}

Parse your sitemap.xml to automatically discover and categorize URLs

Enter URLs Manually

Provide a custom list of URLs (max 100)

Enter Sitemap URL

Enter URLs

Advanced: Custom Category Patterns

Customize how URLs are categorized using regex patterns. These patterns apply to both sitemap parsing and manual URL analysis.

Core Pattern (regex)

Product Pattern (regex)

Documentation Pattern (regex)

Resources Pattern (regex)

Support Pattern (regex)

Blog Pattern (regex)

Company Pattern (regex)

Legal Pattern (regex)

Sitemap Parsed Successfully

Total URLs Found

URLs Included

Sitemaps Processed

Automatic prioritization applied: Found more than 100 URLs. The tool automatically prioritized core pages, product pages, documentation, and blog posts.

Analyzed:

Failed:

pages selected for generation

Auto-Detected Site Metadata

Title Suffix:
Site Name:
Root URL:
Description:

These values have been auto-populated below. You can edit them if needed.

Site Metadata

Title Suffix (regex pattern or plain text - will be removed from titles in preview and output)

Examples: | Example Company (plain text) or \| Example Company( Blog| Solutions)? (regex)

Site Name

Site URL

Site Description

Your llms.txt File

Total Pages

About llms.txt

The llms.txt file is an emerging standard for providing LLM-friendly documentation of your website's content. Similar to how robots.txt tells search engine crawlers what to index, llms.txt helps AI language models understand your site structure and content priorities.

Why Use llms.txt?

Better AI Understanding: Provides context about your pages that helps LLMs give more accurate answers about your business
Content Prioritization: Highlight your most important pages for AI crawlers
Structured Information: Organize content by category (core, product, docs, blog) for better comprehension
SEO for AI: As AI-powered search becomes more prevalent, llms.txt helps your content appear in AI-generated responses

How It Works

Generate: Use this tool to create your llms.txt file from a sitemap or URL list
Customize: Review and edit the generated file to match your needs
Deploy: Place the file at https://yoursite.com/llms.txt
Update: Regenerate periodically as your content evolves

Best Practices

Focus on evergreen content rather than time-sensitive posts
Include clear, descriptive titles and summaries for each page
Organize content logically by category
Keep the file under 100 entries for optimal performance
Update quarterly or when major content changes occur

Related Tools

Check out our SEO Analyzer for comprehensive on-page SEO analysis and optimization recommendations.

Frequently Asked Questions

What is llms.txt?

llms.txt is a standardized markdown file that provides LLM-friendly documentation of your website's content. It lists important pages with titles and descriptions, organized by category (core pages, product pages, documentation, resources, support, blog posts, company info, legal pages). This helps AI crawlers and language models understand your site structure and prioritize content when answering questions about your business.

Where should I place the llms.txt file?

Place the generated llms.txt file in your website's root directory (e.g., https://example.com/llms.txt). This makes it easily discoverable by AI crawlers following the llms.txt standard.

How does automatic categorization work?

When you provide a sitemap, the tool automatically categorizes URLs based on common patterns:

Core Pages: /about, /contact, /pricing, /features, /services
Product Pages: /product/, /solutions/, /plans/
Documentation: /docs/, /guides/, /api/, /reference/, /tutorial/
Resources: /resources/, /case-studies/, /whitepapers/, /ebooks/, /templates/, /downloads/
Support: /help/, /faq/, /support/, /knowledge-base/, /troubleshooting/
Blog Posts: /blog/, /posts/, /articles/, /news/
Company: /company/, /careers/, /jobs/, /team/, /press/, /investors/
Legal: /privacy/, /terms/, /cookies/, /gdpr/, /compliance/

You can provide custom regex patterns for your specific URL structure.

What if my sitemap has more than 100 URLs?

The tool automatically prioritizes URLs when your sitemap exceeds 100 entries. Priority order: Core → Product → Documentation → Resources → Support → Blog → Company → Legal → Other. This ensures your most important pages are included. You can review and adjust the selection before generating.

How do I handle multiple title suffix variations?

The Title Suffix field supports both plain text and regular expressions. If your site uses different suffixes (like " | FreshJuice DEV Docs", " | FreshJuice Tools", " | FreshJuice"), you can use a regex pattern like \\| FreshJuice( DEV Docs| Tools)? to match all variations. The tool automatically detects if your pattern contains regex characters and applies it accordingly.

Can I edit the generated file?

Yes! The llms.txt file is just formatted markdown text. After downloading, you can edit it in any text editor to add custom sections, reorder pages, or update descriptions. The tool provides a starting point that you can customize to your needs.

How is this different from robots.txt or sitemap.xml?

While robots.txt tells bots what not to crawl and sitemap.xml lists all URLs for search engines, llms.txt provides human-readable context about your content specifically for LLMs. It includes descriptions and categorization that help AI understand what each page is about, making it easier to provide accurate answers about your business.