# MED-CLINIC - robots.txt # https://med-clinic.be # Last Updated: 2026-06-19 # IndexNow API Key for instant indexing # Key Location: https://med-clinic.be/med-clinic-indexnow-key-2025.txt # ============================================ # SEARCH ENGINES - Major Crawlers # ============================================ User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-News Allow: / User-agent: Googlebot-Video Allow: / User-agent: Storebot-Google Allow: / User-agent: Google-InspectionTool Allow: / User-agent: Bingbot Allow: / User-agent: BingPreview Allow: / User-agent: msnbot Allow: / User-agent: Yandex Allow: / User-agent: YandexBot Allow: / User-agent: YandexImages Allow: / User-agent: YandexNews Allow: / User-agent: Baiduspider Allow: / User-agent: Baiduspider-image Allow: / User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: NaverBot Allow: / User-agent: Yeti Allow: / User-agent: Seznam Allow: / User-agent: SeznamBot Allow: / User-agent: Sogou Allow: / User-agent: Qwantify Allow: / User-agent: Applebot Allow: / User-agent: PetalBot Allow: / User-agent: Ecosia Allow: / # ============================================ # SOCIAL MEDIA BOTS # ============================================ User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: Facebot Allow: / User-agent: LinkedInBot Allow: / User-agent: Pinterest Allow: / User-agent: Pinterestbot Allow: / User-agent: Slackbot Allow: / User-agent: Slackbot-LinkExpanding Allow: / User-agent: WhatsApp Allow: / User-agent: TelegramBot Allow: / User-agent: Discordbot Allow: / User-agent: Redditbot Allow: / User-agent: Snapchat Allow: / User-agent: vkShare Allow: / # ============================================ # AI/LLM BOTS - 30+ AI Crawlers Allowed # ============================================ # OpenAI User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / # Anthropic (Claude) User-agent: anthropic-ai Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / # Google AI (Gemini) User-agent: Google-Extended Allow: / User-agent: Gemini Allow: / User-agent: GoogleOther Allow: / # Perplexity User-agent: PerplexityBot Allow: / User-agent: Perplexity Allow: / # Cohere User-agent: cohere-ai Allow: / User-agent: CohereBot Allow: / # Mistral AI User-agent: MistralBot Allow: / User-agent: Mistral Allow: / # Meta AI (LLaMA) User-agent: Meta-ExternalAgent Allow: / User-agent: FacebookBot Allow: / User-agent: meta-externalfetcher Allow: / # Amazon AI User-agent: Amazonbot Allow: / User-agent: AmazonProductBot Allow: / # Microsoft AI (Copilot) User-agent: Copilot Allow: / User-agent: bingbot/Copilot Allow: / # Apple AI User-agent: Applebot-Extended Allow: / # ByteDance AI User-agent: Bytespider Allow: / # Baidu AI User-agent: Baiduspider-render Allow: / # You.com AI User-agent: YouBot Allow: / # Brave AI User-agent: BraveBot Allow: / # Neeva AI User-agent: NeevaBot Allow: / # AI Research Crawlers User-agent: CCBot Allow: / User-agent: DataForSeoBot Allow: / User-agent: Diffbot Allow: / User-agent: ImagesiftBot Allow: / User-agent: Omgilibot Allow: / User-agent: Timpibot Allow: / User-agent: VelenPublicWebCrawler Allow: / User-agent: Webzio Allow: / User-agent: img2dataset Allow: / # AI Assistants User-agent: AI2Bot Allow: / User-agent: Ai2Bot-Dolma Allow: / User-agent: iaskspider Allow: / User-agent: Kangaroo Allow: / User-agent: peer39_crawler Allow: / User-agent: AwarioBot Allow: / # xAI (Grok) User-agent: xAI Allow: / User-agent: Grok Allow: / User-agent: GrokBot Allow: / # Newer answer engines (2025–2026) User-agent: PhindBot Allow: / User-agent: KomoBot Allow: / User-agent: AndiBot Allow: / User-agent: LinerBot Allow: / User-agent: ProRataAI Allow: / User-agent: ChatGLM-Spider Allow: / User-agent: DeepSeekBot Allow: / User-agent: TikTokSpider Allow: / User-agent: ZhipuBot Allow: / User-agent: QwenBot Allow: / User-agent: ArkBot Allow: / # ============================================ # SEO & MONITORING TOOLS - Allowed # ============================================ User-agent: AhrefsBot Allow: / User-agent: SemrushBot Allow: / User-agent: MJ12bot Allow: / User-agent: DotBot Allow: / User-agent: rogerbot Allow: / User-agent: MojeekBot Allow: / User-agent: YepBot Allow: / # ============================================ # NOTE: Legacy WordPress / WooCommerce / archive Disallow rules # are consolidated under the `User-agent: *` block below to avoid # duplication and orphan directives. # ============================================ # ============================================ # BLOCKED - Tech Detection & Scraping Bots # ============================================ User-agent: WhatCMS Disallow: / User-agent: BuiltWith Disallow: / User-agent: Wappalyzer Disallow: / User-agent: W3Techs Disallow: / User-agent: SimilarTech Disallow: / User-agent: Netcraft Disallow: / User-agent: StackShare Disallow: / User-agent: Datanyze Disallow: / User-agent: Hype Stat Disallow: / User-agent: SiteAnalyzer Disallow: / User-agent: WhatRuns Disallow: / User-agent: CMS Detector Disallow: / User-agent: NerdyData Disallow: / User-agent: PublicWWW Disallow: / User-agent: Webalizer Disallow: / User-agent: WebsiteOutlook Disallow: / User-agent: Spyse Disallow: / User-agent: SpiderLabs Disallow: / User-agent: SurveyBot Disallow: / User-agent: Scrapy Disallow: / User-agent: node-fetch Disallow: / User-agent: curl Disallow: / User-agent: wget Disallow: / User-agent: HTTrack Disallow: / User-agent: WebCopier Disallow: / User-agent: WebZIP Disallow: / User-agent: siteexplorer Disallow: / User-agent: Urlbox Disallow: / # ============================================ # DEFAULT RULES - Other Bots # ============================================ User-agent: * Allow: / Allow: /fr/ Allow: /en/ Allow: /.well-known/ Allow: /site.webmanifest Disallow: /api/ Disallow: /admin/ Disallow: /_ Disallow: /*.json$ Disallow: /*.map$ Disallow: /node_modules/ Disallow: /src/ Disallow: /functions/ # Legacy WordPress paths - No longer active (410 Gone) Disallow: /de/ Disallow: /nl/ Disallow: /tag/ Disallow: /fr/tag/ Disallow: /feed/ Disallow: /feed Disallow: /*/feed/ Disallow: /*?feed=* Disallow: /rss/ Disallow: /atom/ Disallow: /trackback/ Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /xmlrpc.php Disallow: /author/ Disallow: /auteur/ Disallow: /attachment/ Disallow: /embed/ Disallow: /*?p=* Disallow: /*?page_id=* Disallow: /*?preview=* Disallow: /*?replytocom=* Disallow: /*?wordfence* Disallow: /*wordfence_lh* Disallow: /2019/ Disallow: /2020/ Disallow: /2021/ Disallow: /2022/ Disallow: /2023/ Disallow: /2024/ # NOTE: /2025/ and /2026/ intentionally NOT blocked — current/active content lives there. Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /winkelwagen/ Disallow: /mijn-account/ Disallow: /winkelmand/ Disallow: /product/ Disallow: /product-category/ Disallow: /shop/ Disallow: /winkel/ Disallow: /webshop Disallow: /webshop-de Disallow: /online-shop-en # /behandelingen/ is LIVE content - do NOT disallow Disallow: /gezichtsbehandelingen-antwerpen/ Disallow: /lichaamsbehandelingen-antwerpen/ Disallow: /by-circadia-antwerpen/ Disallow: /by-mesoestetic-antwerpen/ Disallow: /headspa-antwerpen/ Disallow: /headspa-keravive/ Disallow: /headspa/ Disallow: /behandelingen__trashed/ Disallow: /ongecategoriseerd/ Disallow: /nicht-kategorisiert/ Disallow: /uncategorized/ # ============================================ # SITEMAPS & RSS FEEDS # ============================================ # Sitemap Index (bundelt alle sitemaps) Sitemap: https://med-clinic.be/sitemap-index.xml Sitemap: https://med-clinic.be/sitemap.xml Sitemap: https://med-clinic.be/sitemap-treatments.xml Sitemap: https://med-clinic.be/sitemap-cities.xml Sitemap: https://med-clinic.be/sitemap-conditions.xml Sitemap: https://med-clinic.be/sitemap-fr-locaties.xml Sitemap: https://med-clinic.be/sitemap-blog.xml Sitemap: https://med-clinic.be/sitemap-images.xml # RSS Feeds for content discovery # RSS Feed NL: https://med-clinic.be/rss.xml # RSS Feed FR: https://med-clinic.be/fr/rss.xml # ============================================ # CRAWL DIRECTIVES # ============================================ # Crawl-delay reduced from 10 to 1 (was throttling Bing/Yandex over 2300+ URLs) Crawl-delay: 1 # ============================================ # 2026 SEO DIRECTIVES # ============================================ # Host directive (legacy but still used by some crawlers) Host: https://med-clinic.be # LLMs.txt for AI optimization (short index) # https://med-clinic.be/llms.txt # Security policy # https://med-clinic.be/.well-known/security.txt