Google Fellow/software engineer Amit Singhal writes about the challenges inherent to searching for images in an essay for Google's Think Quarterly corporate webzine:

At Google, when we talk about organizing the world’s information, we don’t mean only text; images and videos contain a wealth of information. In the early days, this type of content simply didn’t exist online. Now, through efforts like Google Earth and Street View, we can provide something incredibly valuable: images of your physical world.

However, in many ways, getting visual information online is the easy part. What’s hard is understanding that information. Unlike text, we cannot simply read an image or video. We have to look inside them, dig out the pixels and translate them into something meaningful. For a long time, we considered this a pipe dream, but by combining search methodology and technological breakthroughs in computer vision, today we can match pictures at a visual level. Search for ‘Mount Rushmore’ on Google and our algorithms will analyze many factors, such as the shape and texture that produces a good image of Mount Rushmore, then return those images to you in striking full-color.