Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Why would Bing require a robots.txt when all the other major search engines are fine with a sitemap.xml? Or are these separate somehow?


Because this is the lowest-hanging fruit imaginable, why not take the 5 minutes to add a robots.txt and then use https://www.bing.com/webmasters/robotstxttester to confirm that Bing can fetch/parse it?


You're certainly getting a lot of free SEO help today. I'm not the one who has problems getting my site indexed, but it seems like they are different things.

  # Allowing all web crawlers access to all content
  User-agent: * 
  Disallow: 
https://moz.com/learn/seo/robotstxt

At least you'll be able to look in your logs and see if BingBot is grabbing the file or not.


A day later, I see you added the robots.txt and it shows up first now.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: