Semantic text processing essentially understands linguists. Think of a statement; say, I am optimizing a paragraph for search engine. At least three to four words (I, am, a, for) in the statement are excesses, in the sense that they don't contribute actively towards the meaning of the sentence. They simply add value to the sentence grammatically. In this way, natural language contains numerous redundant and unnecessary words, from the point of view of search engines or semantic meanings. Functional words, conjunctions, prepositions, auxiliary verbs, and several other forms of words just add meaning to a sentence but do not add much content. Ironically, these are the most frequently used words in English.

In the very the first step in LSI, these words are picked and thrown off. The document is then left with words that may have some semantic meaning. We can discard:

• Articles, prepositions, and conjunctions
• Common verbs and pronouns
• Common adjectives (big, late, high)
• Frilly words (therefore, thus, however, albeit, etc.)
• Any word that appear uniquely in every document or in a particular document

Obviously, a grid may contain a cross or a blank. There is no midway and this way we can have an analysis of our document on keyword search. Note that we have left out any word or may have included it under any other column head if the form of the word varies, say it is 'topologies' that appear somewhere in the document and not 'topology'. If instead of looking for the presence of each keyword in a document we go for how many times a word appears in a given document, the grid may show like this:

These figures give certain mathematical meaning. We can calculate the mean, median, and mode of the occurrence of certain words in the document and the correlation between them. This gives us a detailed analysis on our document collection. In case of LSI, we do exactly this. After removing unnecessary words from the documents, we generate the term-document matrix. A graphical representation of this matrix would give you the term-space and will have as many dimension as the number of content-wise meaningful words. This is because, to graphically represent the matrix, you will need as many axes to the graph as there are content words.

Going by this application of the theory, if we try to analyse a real-life document collection and note down the occurrence of each content word, we will get numerous relevant content words. If these are recorded in the matrix, as above, and plotted on a graph, the result in the term space will also have numerous dimensions. This is true for each document in our collection. Each document is considered as a vector with the content words as their component. The documents with several common words will have vectors that are near to each other and hence, will be concluded to be semantically close. Documents with fewer common words will have vectors that are far apart and hence, are semantically distant.

It is mathematically possible to describe this space, although it is difficult to visualize such a space. However, if you try to visualize this multi-dimensional space, you can gain another interesting insight into LSI. Try looking at a branch of a tree full of green leaves. Since, there are leaves propping out at every possible direction, you will always fail to see all the leaves. That is, from whichever angle you try to look at the branch, few leaves will be hidden behind few others so that you can never see all the leaves at one go.

This idea can be contemplated as ‘loss in information' and is a similar idea that you can use to visualize your n-dimensional term space. From whichever angle you look from, some vectors in your n-dimensional term space always overlaps others and the boundaries blur or collapse. In other words, similar keywords or content words loses their distinct identity and get squeezed together. Hence, the difference between singular and plurals, or synonyms or similar meaning words tend to attain a null value

This idea can be contemplated as ‘loss in information' and is a similar idea that you can use to visualize your n-dimensional term space. From whichever angle you look from, some vectors in your n-dimensional term space always overlaps others and the boundaries blur or collapse. In other words, similar keywords or content words loses their distinct identity and get squeezed together. Hence, the difference between singular and plurals, or synonyms or similar meaning words tend to attain a null value.

One thing to note here is that, although loss of information is deemed as a bad idea, it is

converted into a blessing when it comes to LSI. This technique of using or exploiting the feature of natural language, namely, similar-meaning words occur together, cuts off noise or unnecessary information. In the final lap, we can remove the hash from the hay.

Everyday, Google is taking a step to convert its whole search mechanism into an LSI-enabled one. Although, LSI is not adapted uniformly and in entirety, and not all searches will return a semantic word set now, the transition is visible in the search results. Conducting a search for 'phone' will show results in which the keyword 'phone' is contained and highlighted. However, if you add the tilde (~) before your keyword and search, ('~phone') your result will show the Web site for Nokia and the word ‘Nokia' is now highlighted. From its new method of indexing, Google has determined that Nokia is relevant to phone.

About The Author

Get in Touch

Call an expert on: +91 11 41623530/31
E-mail: vikas[at]mosaic-service.com
Request a Free Quote

Using This Article

If you want to publish this article on your own web site, please use the whole article up to and including About the Author.