Why Isn'y My Website in the Search Engines?

Reasons Why A Website Isn't Indexed

• No Inbound Links To The Site - You need at least one good quality link from another website to be included (ie. "indexed") in the major search engines. This is the main reason why new websites are not indexed. Start by submitting your site to a good quality directory like JoeAnt, or one of the others I recommend in my Building Links SEO Tips article. Ask other websites whose main topic is related to yours to link to your site. A good quality link comes from a prominent page on a high quality website. Forum signatures and blog comments are not likely to help your ranking much, but they can help get your site indexed.


• Site Blocked By The robots.txt File - The robots.txt file is a plain text file that controls how search engines access your site. Many website designers block the search engines from crawling a site while it's under construction, and then forget to remove that blocking instruction when the design in completed. So check your robots.txt file in Google's Webmaster Tools, where a tool is provided that will test your robots.txt file. You'll also find information there on how to use your robots.txt file properly.


• Site Blocked By robots <META> Tag - Similarly, some web designers will insert a robots <META> tag set to "noindex" on a website's main page while it's in the testing stage. The "noindex" robots <meta> tag will prevent a page from being indexed, or will remove it if it has previously been indexed. Remove the tag and you should recover in a short time.


• Server And DNS Problems - It's rare, but not unheard of, for server problems to be severe enough to prevent the search engines from accessing a site. Sometimes hosting services have automatic controls that see search engine crawlers (a.k.a. "robots", a.k.a. "spiders") as malicious and block them. Use the "Fetch As Googlebot" tool in Google's Webmaster Tools console to make sure that Googlebot (Google's crawler) can access your site. Extremely slow response times can also prevent indexing. An improperly configured Domain Name Server (DNS) can sometimes cause problems as well. Check your domain's DNS at intoDNS, and if their system reports problems, consult your hosting service about it immediately. If you don't get a satisfactory response, you should move to a new hosting service.


• Frame-Forwarded Domain - Some free and low-cost services have webmasters create a <frameset> page as the main page of a domain name package, but the website's actual content resides on a completely different domain. The major search engines see such a website as having no content of its own and will not index it. In this situation, the search engines (epsecially Google) will often index the content at the domain where it truly resides. The solution is to buy a conventional hosting service, or at least use a reliable free host, so that your content is seen to be on your own domain.


• Invalid/Poorly Constructed Webpages - The search engines' crawlers are very tolerant of errors in the HTML mark-up of webpages, but it is possible for mark-up and syntax errors to prevent indexing in extreme cases. Valid HTML mark-up will not only help insure that your site is indexed properly, it will also insure that your users see what you wanted them to see when they visit your site. Check your site's HTML with the W3C's HTML Validator and correct any errors it reports.


• Previously-Owned Domain Banned/Penalized - If your domain name was previously owned and was banned for violating the Guidelines in the past, the search engines might continue to exclude it from the index. You can check the history of the domain name by using the Wayback Machine operated by archive.org. Look at the old pages shown in the Wayback Machine for signs of a poor quality website. If the website used to be about pornography, drugs, or other often-abused topic, or if the site looks like it was created by copying the content of other websites, there's a good chance it was banned. The solution is to file a Reconsideration Request in Google's Webmaster Tools console, or use the Email Support form in Bing's Webmaster Center. Tell them that you are the new owner of a domain you believe has been penalized, and that you intend to adhere to their guidelines.

As I say, the search engines are anxious to include your website so they can show it in the search results. You just have to make sure that they see signals of quality (like an inbound link!), and that your site behaves properly. Of course, being indexed is not the same as ranking well, but it's a very necessary first step in your Search Engine Optimization and Marketing program.

This page was last updated on September 25, 2020