Executive Summary
This report delivers a targeted comparative analysis and optimization strategy for Empathia.ai's robots.txt and llms.txt files. Our goal is to maximize both web discoverability and AI/LLM alignment while protecting sensitive or non-public areas. We preserve all original data and present actionable suggestions, including per-URL directives and industry best practices, in full BCG consulting report format.
Key Finding
Empathia.ai’s robots.txt currently grants universal access to all referenced search and AI bots; the site is fully open, but can be optimized further by restricting private/test directories and structuring LLM metadata for maximal AI discoverability.
1. robots.txt Analysis & Optimization
Existing robots.txt (Summarized)
User-agent: *
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Gemini
Allow: /
User-agent: Googlebot-News
Allow: /
User-agent: Google-CloudVertexBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: OAI-SearchBot/1.0; +https://openai.com/searchbot
Allow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Allow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: Claude-Web
Allow: /
Sitemap: https://www.empathia.ai/sitemap.xml
- Universal allowance to all referenced bots
- Sitemap linked
- No Disallow directives or restrictions
Optimization Recommendations
- Maintain universal access for public content
- Explicitly disallow sensitive/private directories (e.g., /admin/, /checkout/, /private/, /cart/, /search, /test/) if such folders exist
- Retain all key bots in allow lists or unify using User-agent: * if policies align
- Keep sitemap linked and updated
| URL | Current Status | Optimization Suggestion |
|---|---|---|
| https://www.empathia.ai/ | Allowed | Keep allowed |
| https://www.empathia.ai/product | Allowed | Keep allowed |
| https://www.empathia.ai/billing-code | Allowed | Keep allowed |
| https://www.empathia.ai/specialties/(...) | Allowed | Keep allowed |
| https://community.empathia.ai/onboarding-handbook | Allowed | Keep allowed |
| https://support.empathia.ai/support/solutions | Allowed | Keep allowed |
| https://love.empathia.ai | Allowed | Keep allowed |
| https://www.empathia.ai/blog | Allowed | Keep allowed |
Example Optimized robots.txt
# robots.txt for https://www.empathia.ai/
User-agent: *
Allow: /
# Disallow private/test areas (add as present)
Disallow: /admin/
Disallow: /checkout/
Disallow: /cart/
Disallow: /private/
Disallow: /search
Disallow: /test/
Sitemap: https://www.empathia.ai/sitemap.xml
Note: Only add Disallow lines if the directories exist.
Current vs. Optimized Coverage Table
| Area | Current | Optimized |
|---|---|---|
| Main site & key pages | Allowed | Allowed |
| Sitemap | Linked | Linked |
| Low-value/private URLs | Not blocked | Blocked if present |
Expert SEO Tips
Routinely review your robots.txt for sensitive folder exposure. Check Google Search Console for crawl anomalies, and ensure your sitemap remains accurate.
2. llms.txt Analysis & Optimization
Existing llms.txt (Frevana-based Template)
# Frevana
> Frevana is Your AI team for Generative Engine Optimization (GEO) and beyond
Frevana enables users to Launch an AI team in minutes to get their brand mentioned in AI results, which includes
### Metadata
title: Frevana | Your AI team for Generative Engine Optimization
description: Launch an AI team in minutes to get your brand mentioned in AI results
domain: www.frevana.com
language: en
category: ...
keywords: ...
### Core Pages
- [Homepage](...)
### Accessibility
alt_text_present: true
structured_data: true
mobile_friendly: true
### SEO
robots_txt: /robots.txt
Optimized llms.txt for www.empathia.ai
# Empathia AI
> Empathia AI is an all-in-one AI-powered healthcare assistant automating intake, notes, charting, and billing for clinicians across specialties.
### Metadata
title: Empathia AI | Streamlined Documentation and Billing for Healthcare
description: Empathia AI provides real-time medical charting, intake automation, coding, and billing with seamless EMR integration. Designed for physicians and healthcare teams.
domain: www.empathia.ai
language: en
category: Healthcare AI, Medical Automation, Billing, Documentation, Clinical Workflow
keywords: Empathia AI, healthcare automation, AI scribe, charting, medical billing, physician workflow, SureCode AI
### Core Pages
- [Homepage](https://www.empathia.ai)
- [Product](https://www.empathia.ai/product)
- [Billing and Coding](https://www.empathia.ai/billing-code)
- [Specialties](https://www.empathia.ai/specialties/)
- [Onboarding Handbook](https://community.empathia.ai/onboarding-handbook)
- [Support](https://support.empathia.ai/support/solutions)
- [Testimonials](https://love.empathia.ai)
- [Blog](https://www.empathia.ai/blog)
### Page-specific Semantics
- Homepage: Overview, features, integrations, app links.
- Product: Modules, automation benefits, collaboration tools.
- Billing and Coding: SureCode AI for ICD/CPT coding, claims, compliance.
- Specialties: Supported medical fields, specialty-specific templates.
- Onboarding Handbook: Self-serve guides, integration tutorials.
- Support: FAQs, troubleshooting, user manuals, contact.
- Testimonials: Verified reviews, success stories.
- Blog: Community, best-practice, healthcare AI updates.
### Accessibility
alt_text_present: true
structured_data: true
mobile_friendly: true
multi_language_support: true
### SEO
robots_txt: /robots.txt
sitemap_present: true
### Compliance
HIPAA: true
PHIPA: true
PIPEDA: true
Per-URL Optimization Comments
| URL | Optimized Structure | Optimization Notes |
|---|---|---|
| / | Clear overview, feature list, keywords. | Use structured summary and highlights for LLM/SEO. |
| /product | Bulleted modules/features/keywords. | Ensure value proposition is explicit and scannable. |
| /billing-code | Emphasize automation, accuracy, compliance. | Highlight integration and standards adherence. |
| /specialties/(...) | List all specialties and tailored features. | Use per-specialty templates for clarity. |
| /onboarding-handbook | Outline guides, checklists, video resources. | Use section/meta tags for easy parsing. |
| /support/support/solutions | List resource types/FAQ, clear help section. | Emphasize troubleshooting, link to manuals. |
| /love | Use verified reviews and specific benefit cases. | Structured testimonials, show specialties helped. |
| /blog | List article types (insights/guides/news). | Tag author, date, keywords for each post. |
3. Final Summary Comparison
| Area | Existing | Optimized |
|---|---|---|
| robots.txt Accessibility | Everything allowed | Everything allowed; sensitive folders blocked if present |
| Bots Included | All major bots allowed | All major bots allowed; can use User-agent: * |
| Sitemap | Linked | Linked |
| Security | No protections | Block backend if relevant/needed |
| llms.txt Metadata | Frevana template | Rich, Empathia-specific, enumerates all main URLs |
| Semantics & Keywords | Generic | Targeted LLM/SEO metadata, accessibility, compliance |
| Compliance | Not noted | Explicit: HIPAA, PHIPA, PIPEDA |
4. Actionable Recommendations
-
robots.txt:
- Audit for sensitive or low-value folders (e.g., admin, checkout, test) as new features launch.
- Keep sitemap link active and up to date for search and AI bots.
- Monitor indexing health in Google Search Console and equivalent tools.
-
llms.txt:
- Document and maintain structured, bullet-pointed descriptions for every main/public page.
- Prominently list all AI-related and compliance features (HIPAA, PHIPA, PIPEDA).
- Update accessibility, metadata, and multi-language availability as features are added.
- Tag all specialty and dynamic pages in future updates for LLM discoverability.
Next Steps
For detailed page-level llms.txt samples (e.g., for each specialty or sub-product), or to extend policy with granular robots.txt controls for new microsites, request a follow-up technical brief.