# Master Allowing User Agent User-agent: * Allow: / # Training: Inclusion in AI foundational models # OpenAI’s GPT crawler User-agent: GPTBot Allow: / # Perplexity AI search assistant crawler User-agent: PerplexityBot Allow: / # Anthropic Claude web crawler User-agent: ClaudeBot Allow: / # Google Gemini AI crawler User-agent: Gemini-Ai Allow: / # Cohere AI training crawler User-agent: Cohere-Ai Allow: / # xAI (Elon Musk) crawler User-agent: xAI-Bot Allow: / # You.com AI search bot User-agent: YouBot Allow: / # Allen Institute for AI crawler User-agent: AI2Bot Allow: / # Together AI crawler User-agent: Together-Bot Allow: / # DuckDuckGo AI assistant crawler User-agent: DuckAssistBot Allow: / # Webzio extended AI crawler User-agent: Webzio-Extended Allow: / # Emerging AI bot User-agent: KangarooBot Allow: / # Chinese AI bot User-agent: PanguBot Allow: / # Image/vision-related AI bot User-agent: ImagesiftBot Allow: / # Structured data extraction bot User-agent: Diffbot Allow: / # Anthropic alternative crawler signature User-agent: anthropic-ai Allow: / # Indexing: Real-time visibility in ChatGPT search # OpenAI indexing/search bot User-agent: OAI-SearchBot Allow: / # Browsing: On-demand retrieval via ChatGPT user requests # ChatGPT user-triggered browsing bot User-agent: ChatGPT-User Allow: / Disallow: /*?property_type Disallow: /tel:+917283008300 Disallow: /tel:+91-7283008300 Disallow: /javascript:void(0) Disallow: /property_amenity/gym Disallow: /property_amenity Disallow: /comments/feed Disallow: /property_location Disallow: /property_location/binnyfields-binnypet-bengaluru Disallow: /property_location/noida Disallow: /property_location/pokhran-road-2-thane Disallow: /godrej-golf-links Disallow: /submit-a-property Disallow: /gallery Disallow: /category/webinar Disallow: /category/news Disallow: /category Disallow: /home-map Disallow: /profile Disallow: /sample-page Disallow: /property_type/commercial Disallow: /property_type/residential Disallow: /property_type Disallow: /tag/roundup #Disallow: /tag Disallow: /agents Disallow: /wp-login.php?action=lostpassword Disallow: /wp-login.php Disallow: /ho Disallow: /2020/05 #Disallow: /? Disallow: /wp-json Disallow: /wp-content Disallow: /wp-content/ Disallow: /blogComment/ajax Disallow: /modal-enquiry Disallow: /property-query Disallow: /web Disallow: /web/ Disallow: /web/public Disallow: /web/public/ Disallow: /index2.html Disallow: /carrer-list Disallow: /news-letter Disallow: /refer-data Disallow: /career-query Disallow: /banner_form Disallow: /subscribe Disallow: /flats-for-sale-in-noida/https://realtyassistant.in/ #Disallow: /*?utm_source= #Disallow: /*&submit=Search # Block malformed queries with %20 or other Disallow: /*%20 Disallow: /*class= Disallow: /*