1. Content Analysis — Reference 1 (robots.txt)
- Finding:
robots.txt is currently missing (404 Not Found).
- Importance: A missing robots.txt means no explicit crawl guidance for search engines or AI bots (including LLM-based agents).
- Best Practice: Place the
robots.txt at the root: https://shoplyai.ai/robots.txt
- Reference Suggestions: Disallow sensitive/user/account/cart/orders/admin-related paths, while allowing CSS/JS/images and providing a sitemap.
Optimized robots.txt (based on provided example and reference best practice):
User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /register/
Disallow: /cart/
Disallow: /checkout/
Disallow: /user/
Disallow: /account/
Disallow: /orders/
Disallow: /private/
Allow: *.css
Allow: *.js
Allow: /images/
Allow: /
Sitemap: https://shoplyai.ai/sitemap.xml
# LLM Bots and AI Agents—Explicit Allow
User-agent: Googlebot
Allow: /
User-agent: Gemini
Allow: /
User-agent: Googlebot-News
Allow: /
User-agent: Google-CloudVertexBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: OAI-SearchBot/1.0; +https://openai.com/searchbot
Allow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Allow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: Claude-Web
Allow: /
- How this improves:
- Protects sensitive/user data from indexing.
- Ensures search, LLM, and specialized AI bots get full content access where appropriate.
- Enables CSS/JS/image asset indexing for enhanced rendering.
- Explicitly enables modern LLM bots (OpenAI, Google, Anthropic/Claude, Gemini).
- Points to sitemap for improved crawling.
- Action: Upload at https://shoplyai.ai/robots.txt.
2. Content Analysis — Reference 2 (llms.txt)
- Finding:
llms.txt is also missing (404 Not Found).
- Importance: Modern AI agents and LLM bots increasingly look for
llms.txt to understand site-level AI/Ops metadata, API info, and allow/deny signals.
- Best Practice: Place
llms.txt at the domain root: https://shoplyai.ai/llms.txt
- Reference Suggestions: Include structured metadata, supported LLMs, description, API endpoints, and support.
Optimized llms.txt (based on provided example, modified for ShoplyAI):
# ShoplyAI llms.txt — AI and LLM Operations Metadata
title: ShoplyAI | Intelligent E-commerce Personalization
domain: shoplyai.ai
language: en
category: AI, E-commerce, Personalization, LLM, Recommendations
keywords: ShoplyAI, AI Shopping, Personalization, Recommendation Engine, Chatbot, Ecommerce AI, LLM, Smart Retail, API
status: active
ai_friendly: true
llms_supported: GPT-4, Claude 3, Gemini 1.5
description: ShoplyAI delivers state-of-the-art personalized product recommendations and customer assistance for online stores, powered by large language models.
apis_available: true
api_docs_url: https://shoplyai.ai/developer/api-docs
models_updated: 2024-06-01
contact: [email protected]
# Accessibility
alt_text_present: true
structured_data: true
mobile_friendly: true
# SEO
robots_txt: /robots.txt
- How this improves:
- Enables LLMs and AI agents to instantly detect and parse important metadata.
- Matches/aligns with best practices for clarity, up-to-date models, and support.
- Supports SEO and accessibility claims.
- States API and developer resources.
- Action: Upload at https://shoplyai.ai/llms.txt.
Final Recommendations
- Create and deploy both files at the root of your domain:
/robots.txt for search/AI bot indexing control/llms.txt for LLM/AI access and metadata sharing
- Keep both files up-to-date as your features, API, models, or policies evolve.
- Monitor bot activity and LLM agent behavior for any crawl, indexing, or consumption issues.
- (Optional) Add human-readable help links for developers or AI bot operators, if needed.
These steps will make your platform maximally accessible, indexable, and friendly to both human users and the new generation of AI/LLM-powered agents!
