# ============================================================================= # robots.txt for landkit.pro # ============================================================================= # LandKit is an AI/GEO discoverability product, so we explicitly opt INTO # AI training, AI inference, and search indexing across all major crawlers. # # Content Signals follow the IETF draft (draft-romm-aipref-contentsignals) # and the contentsignals.org spec: # - search : may use content for traditional search-result snippets # - ai-input : may fetch content at query time to ground an AI answer # - ai-train : may use content as training data for generative models # # Spec refs: # https://contentsignals.org/ # https://datatracker.ietf.org/doc/draft-romm-aipref-contentsignals/ # # Each User-agent group is independent — RFC 9309 says a crawler matches the # MOST specific User-agent line and ignores other groups, so disallow rules # are repeated per named bot rather than inherited from "*". # ============================================================================= # ----------------------------------------------------------------------------- # Default policy — applies to any crawler not explicitly listed below # ----------------------------------------------------------------------------- User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /dashboard/ Disallow: /new Disallow: /project/ Disallow: /update-password Disallow: /login Disallow: /signup Disallow: /admin Disallow: /admin/ Disallow: /checkout Disallow: /shared/ Disallow: /api/ Disallow: /auth/ Disallow: /*?token= Disallow: /*?share= Disallow: /*?session= # ============================================================================= # Traditional search engines # ============================================================================= User-agent: Googlebot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /dashboard/ Disallow: /new Disallow: /project/ Disallow: /update-password Disallow: /login Disallow: /signup Disallow: /admin Disallow: /checkout Disallow: /shared/ Disallow: /api/ Disallow: /auth/ User-agent: Googlebot-Image Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /shared/ User-agent: Googlebot-News Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /shared/ User-agent: Bingbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /dashboard/ Disallow: /new Disallow: /project/ Disallow: /update-password Disallow: /login Disallow: /signup Disallow: /admin Disallow: /checkout Disallow: /shared/ Disallow: /api/ Disallow: /auth/ User-agent: Slurp Content-Signal: search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ User-agent: DuckDuckBot Content-Signal: search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ User-agent: Baiduspider Content-Signal: search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ User-agent: YandexBot Content-Signal: search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # ============================================================================= # AI training crawlers — LandKit benefits from being in foundation models, # so all of these are allowed full access to public marketing content. # ============================================================================= # OpenAI (ChatGPT model training) User-agent: GPTBot Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ Disallow: /auth/ # Anthropic (Claude model training) User-agent: ClaudeBot Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ Disallow: /auth/ User-agent: anthropic-ai Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ User-agent: claude-web Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Google AI / Gemini training (separate from search Googlebot) User-agent: Google-Extended Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Perplexity User-agent: PerplexityBot Content-Signal: ai-train=yes, ai-input=yes, search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Common Crawl — feeds many open-source LLMs User-agent: CCBot Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # ByteDance (Doubao / TikTok AI) User-agent: Bytespider Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Amazon (Alexa / Rufus / Bedrock) User-agent: Amazonbot Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Apple Intelligence training User-agent: Applebot-Extended Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ User-agent: Applebot Content-Signal: search=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Meta AI (Llama training, Meta AI assistant) User-agent: Meta-ExternalAgent Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ User-agent: Meta-ExternalFetcher Content-Signal: ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ User-agent: FacebookBot Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Mistral User-agent: MistralAI-User Content-Signal: ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Cohere User-agent: cohere-ai Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ User-agent: cohere-training-data-crawler Content-Signal: ai-train=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # You.com User-agent: YouBot Content-Signal: ai-train=yes, ai-input=yes, search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Diffbot (knowledge-graph extractor used by many AI products) User-agent: Diffbot Content-Signal: ai-train=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Webz.io / omgili — feeds LLM training datasets User-agent: omgili Content-Signal: ai-train=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ User-agent: omgilibot Content-Signal: ai-train=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # ImageSift (image-AI training) User-agent: ImagesiftBot Content-Signal: ai-train=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ # Huawei Petal Search AI User-agent: PetalBot Content-Signal: ai-train=yes, ai-input=yes, search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Timpi search index User-agent: Timpibot Content-Signal: search=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ # Kagi User-agent: KagiBot Content-Signal: search=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ # Velen / DataForSEO crawlers (note: LandKit USES DataForSEO too) User-agent: VelenPublicWebCrawler Content-Signal: search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ User-agent: DataForSeoBot Content-Signal: search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ # ============================================================================= # AI agent / live-fetch crawlers — these run at query time when an AI assistant # answers a user. Allowing them means LandKit can be cited in answers. # ============================================================================= # OpenAI ChatGPT browse mode User-agent: ChatGPT-User Content-Signal: ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # OpenAI search index User-agent: OAI-SearchBot Content-Signal: ai-input=yes, search=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Perplexity user-triggered fetch User-agent: Perplexity-User Content-Signal: ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # DuckDuckGo Assist User-agent: DuckAssistBot Content-Signal: ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # Brave Search / Leo AI User-agent: BraveBot Content-Signal: search=yes, ai-input=yes Allow: / Disallow: /dashboard/ Disallow: /admin Disallow: /login Disallow: /signup Disallow: /checkout Disallow: /shared/ Disallow: /api/ # ============================================================================= # Sitemaps & AI manifests # ============================================================================= # Sitemap is proxied via Cloudflare Pages from the Supabase Edge Function. Sitemap: https://landkit.pro/sitemap.xml # llms.txt — structured site summary for AI agents (see llmstxt.org) # https://landkit.pro/llms.txt