So Where's Your Site Search Engine, Fractal?

I'll let Jared explain why I think a site like FyrenIyce is especially ill-suited to those trendy, online search engines popping up everywhere. You cannot afford to miss information about this illness, and I really don't want something on the site that might be responsible for you missing something critical. Read on and hear what Jared has to say about his research into the efficacy of onsite search engines.

From: Jared M. Spool
Organization: User Interface Engineering
Subject: On-Site Searching Results

On-Site Searching

In a recent study, we sent users off on scavenger hunts of information on various web sites. While all of the information that they sought could be found just by using the links provided, users would often use the search functionality provided by the site. (We saw users use search on 75% of the sites in our study.)

This makes sense. Theoretically, if you know the keywords, you should just be able to type them in and instantly find the page you're looking for. However, in practice, it doesn't work that well.

Search engines didn't make finding information easier, they made it harder. When users found information without a search engine, they did 50% better than when they tried to use a search engine.

Overall, having a search engine on a site doesn't seem to make the site more usable. Of the different sites we tested, the best search engine helped users find the target information only 50% of the time. Some were as low as 25%.

We think there are four major reasons for this:

1) Users don't know how to narrow searches



Few users did anything other than simple keyword searches. (Almost no one used booleans or other features of the search engines.) They would often type in very broad terms, such as "Videos" when looking for how a video about the Wild West to get a friend. This is the equivalent of walking up to a librarian and just saying 'Travel' and expecting they'll instantly find the book on Hawaii that you are interested in.

When users got back a result set that was too large to be practical, they often tried completely different keywords instead of adding to the set they had. They didn't seem to know any strategies that would have allowed them to take the stuff already found and narrow it down further (for instance, "regular tire rotation").

None of the sites that we tested provided any useful information on how to narrow a search. In fact, most seemed to assume that users knew how to search effectively and didn't provide any clues at all.

2) Full-text Search engines are not indexed



Users didn't seem to understand what a full-text search is. The dynamics of full-text searches are different than looking something up in an index, but users didn't seem to grasp this.

For instance, they were surprised when they typed in the word 'Tire' on the Car Talk site to find results that contained the word 'entire' or the phrase "I'm tired." Although the site did present the option to search for entire words or partial words, users didn't change the setting (the default was partial).

We also saw that users didn't understand that plurals and singular words would produce different results and were surprised. Users didn't know that typing errors would produce poor results and couldn't tell that it was a typo, instead of a lack of content, that got them the "nothing found" message. (For example, one user mistyped 'Videos' as 'Vidoes' and got zero hits -- and then assumed that there weren't any videos on the site.)

Full-text searching produces a lot of irrelevant information for users. For instance, one of the tasks we had for the Smithsonian Magazine site was, "Your six-year-old son has to do a report on dinosaurs for school. You remember an excellent article in an old issue of the Smithsonian Magazine that you think would be a fine reference. Go find it."

For this task, users naturally typed in the keyword 'dinosaur.' The first article returned was on the American steel industry -- one of the great American industrial dinosaurs. (Go try this yourself, it's great!)

Indexing is a craft that takes a lot of skill. No professional indexer with any self-respect would ever put an article about the steel industry under the topic of dinosaurs. As sites get bigger, this problem will only become more of an issue. Full-text searches will get more noisy and irrelevant as more words are introduced without any sense of what makes them important to the content.

Successful searching is essentially an indexing task. To help user search more effectively, this intelligence must be designed into the site. We think that professional indexers and others who have these skills will become more valuable in the years to come.

3) Multiple search areas are not clear



Many of the sites provided the ability to search different types of content or different areas of the site. This capability differed from site to site. For instance, the Smithsonian Magazine site lets users search either Feature stories, Columns, or Back Issues '89-'94. Users didn't know which one to pick.

Car Talk has four search areas that do not overlap. The designers have scattered the search screens in several different locations (usually with the content it searches). Users not finding information in one area didn't know that they should search other areas.

Disney lets users narrow the areas to search only after they've done an initial search. There is no explanation to users as to how the results from the other areas will be different from the search they just conducted.

4) Search engines are short cuts



As a short cut, the search engine is intended to get users quickly to their content. But like other types of short cuts, users first need to know how to get there "the long way."

The short cut requires that users understand how the search engine works, how the content has been segmented and indexed, and how content has been labeled (page titles). This is a lot to ask of users and most are not up to the task.

Our data shows that the site uses a search engine, there is a strong correlation to users failing to find the right information (r-squared > .7). It also shows that users will gravitate to a search engine when the links are not clear. On sites where no search option was provided, users complained, but did 50% better than the best site that had a search engine. (This is more proof that users don't necessarily know what's good for them.) And with all the times that users failed using search engines, they never seemed to associate the failure with the search engine -- it never occured to them to try a different tack. Instead, they gave up.

At this point, based on this information, our recommendation is for site designers to focus on making the long way -- the links of the site -- work effectively for users.

Over the next few weeks, we'll be publishing other findings from our most recent study of web usability in UIETips, which you can subscribe to by sending the word SUBSCRIBE in the body of a message to UIETips-Request@uie.com.

User Interface Engineering mailto:jspool@uie.com 800 Turnpike Street, Suite 101 North Andover, MA 01845 USA (978) 975-4343 fax: (978) 975-5353

DISCLAIMER