Google Neglects Its Own

In September 2009, I created my family web page on Google. Now, more than 18 months later, it still does not appear on Google's own search engine even though Google itself hosts the site.

In stark contrast, I posted the same content on my family web page on Comcast a full year later in September 2010, and yet Google's search engine displayed my Comcast site within a few short weeks.

This anomaly compelled me to research ways to create visibility for my Google-hosted web page on Google's own search engine. The following saga will outline the sequence of events that Google led me to take and that you presumably must take as well.

The first necessary step involved logging on to Google Webmaster Tools using a Gmail account. After signing in, execute the following steps:

• In the "Verification home" window, click the button "Add a site..."
• In the next pop-up window, enter the applicable web site url in the field provided (in my case, http://sites.google.com/site/gumby32/)
• In the "Verify ownership" window, click on the second "Alternate methods" tab
• Select the first option of the three: "Add a meta tag to your site's home page." Indeed, Google's own help entry states: "This is the best option if you don't have access to your server." That is certainly true for those of us who are merely clients of Google sites. Meta tags will look something like this, only all on one single line:

    <meta name="google-site-verification"
    content="b0l0X5ZOJiT9kRSABjuXlE-
    w3kkjL8zPTUJyMMOO6x0" />

• Copy the entire meta tag that appears in the window and save it somewhere convenient. You will soon need to paste this meta tag into a particular field in your web page header information.
• Click the "Verify" button to complete the verification process.

Once your site has been verified, you may exit Webmaster Tools and then sign in to your Google web site. After you log on, follow these steps:

• Click "More actions | Manage site"
• From the menu at the left, click "Site setting - General"
• In the window to the right, paste your saved entry from Google Webmasters above into the field "Google Webmaster Tools verification"
• Click the "Save changes" button at the bottom to save your work

As added insurance, I availed myself of a service Google offers to add your URL to Google. Entering the top-level url of your web site into the designated field and then submitting it supposedly adds your site to Google's index each time Google's web crawler scans the web.

By going through all of these contortions, I thought I had covered all of my bases. Boy, was I mistaken: I had no idea of what still lay ahead.

I had completed my web site verification process and submitted my url to Google on 9 April 2011. I had read that Google's web crawler operates early each month, so I was prepared to wait a few weeks for my site to appear in a Google search sometime in early May.

I waited patiently until 1 May, when I began to check almost daily. To my increasing chagrin, my site still failed to appear in any Google search over the next two full weeks. In frustration, I re-submitted my url to Google once again on 14 May, but another full week passed with no results.

In the course of researching these requirements, I was dimly aware of an additional process whereby users could also submit a site map of their web page to Google. Finally, out of sheer desperation, I undertook the following steps:

• Create a plain text file with all of the links you want Google to crawl. Log on to your web site, navigate your way to the sitemap location, and click the "List" button. Then:
• highlight all links in the list
• right-click the selected links
• in the resulting menu, select "Copy Selected Links"
• paste all links to a file "sitemap.txt"
• Upload the newly-created file "sitemap.txt" anywhere onto your web site. "Web Site Overview" is always appropriate, but it really doesn't matter because the file is only a temporary device that you can eventually delete.
• Log on to Google Webmaster Tools once again
• In the Home window, click the site at issue (in my case, "sites.google.com/site/gumby32/")
• In the window at left, click on "Site Configuration"
• In the expanded menu, click "Sitemaps"
• Back in the window at the right, click the button "Submit a Sitemap"
• Copy the relevant portion of the link from the plain text file that you uploaded earlier and then paste that portion into the blank field. In my case, the correct entry was "home/sitemap.txt?attredirects=0&d=1" even though the complete link of the attached file was "https://sites.google.com/site/gumby32/home/sitemap.txt?attredirects=0&d=1"
• Click the "Submit Sitemap" button
• Once the URLs are accepted and acknowledged, you may delete the "sites.txt" file that you uploaded (and which will appear as an attachment) because it is no longer needed. You probably don't want an unsightly text file with no ostensible purpose littering your web site.

This technique appeared to work because the status column in the sitemap listing showed a green check mark instead of the red "X" displayed in my first two failed attempts. I was further encouraged because the field above the sitemap listing displayed the correct count of 33 submitted URLs along with the entry "Index count pending."

I was now at the end of my rope. I knew of no other measures to take. I had submitted 33 specific URLs that were all accepted and acknowledged. If that failed to work, then I was fully prepared to give up.

I completed this process on 24 May and initially expected to wait until the monthly web crawl in early June. However, after all this time and effort, I was beginning to feel anxious. I could not wait any longer, so I launched my inquiry on 26 May.

Eureka!

• My first Google search ("gumby32 site:sites.google.com") suddenly yielded all 33 of my web site links where it previously displayed only the message "Your search ... did not match any documents."
• A search for "gumby32 greiner" only showed seven items, but numbers six and seven were my résumé and main web page, respectively.
• A search for my full name "Robert Brian Greiner" produced 633,000 results, but my main web page was listed fourth on the first page!

Not every search came up roses, however. My search for gumby32 alone produced 3,700 results, but after wading through the first six pages, I found no mention of my site. Even so, I have at least some reason to believe that my web site is now on the radar scope. Clearly, adding the site map links was the turning point.

While I am pleased to finally have my web page finally listed in Google searches, I am not at all happy about the obstacles I had to overcome. I am both perplexed and somewhat dismayed that Google erects so many barriers in the path of its own customers while requiring no special action by the customers of its competitors (in my case, Comcast) to be included in a Google search.

Is it just me, or is there something wrong with this picture?