Programming Collective Intelligence: Building Smart Web 2.0 Applications

Front Cover
"O'Reilly Media, Inc.", Aug 16, 2007 - Computers - 362 pages
26 Reviews

Want to tap the power behind search rankings, product recommendations, social bookmarking, and online matchmaking? This fascinating book demonstrates how you can build Web 2.0 applications to mine the enormous amount of data created by people on the Internet. With the sophisticated algorithms in this book, you can write smart programs to access interesting datasets from other web sites, collect data from users of your own applications, and analyze and understand the data once you've found it.

Programming Collective Intelligence takes you into the world of machine learning and statistics, and explains how to draw conclusions about user experience, marketing, personal tastes, and human behavior in general -- all from information that you and others collect every day. Each algorithm is described clearly and concisely with code that can immediately be used on your web site, blog, Wiki, or specialized application. This book explains:

  • Collaborative filtering techniques that enable online retailers to recommend products or media
  • Methods of clustering to detect groups of similar items in a large dataset
  • Search engine features -- crawlers, indexers, query engines, and the PageRank algorithm
  • Optimization algorithms that search millions of possible solutions to a problem and choose the best one
  • Bayesian filtering, used in spam filters for classifying documents based on word types and other features
  • Using decision trees not only to make predictions, but to model the way decisions are made
  • Predicting numerical values rather than classifications to build price models
  • Support vector machines to match people in online dating sites
  • Non-negative matrix factorization to find the independent features in a dataset
  • Evolving intelligence for problem solving -- how a computer develops its skill by improving its own code the more it plays a game
Each chapter includes exercises for extending the algorithms to make them more powerful. Go beyond simple database-backed applications and put the wealth of Internet data to work for you.

"Bravo! I cannot think of a better way for a developer to first learn these algorithms and methods, nor can I think of a better way for me (an old AI dog) to reinvigorate my knowledge of the details."
-- Dan Russell, Google

"Toby's book does a great job of breaking down the complex subject matter of machine-learning algorithms into practical, easy-to-understand examples that can be directly applied to analysis of social interaction across the Web today. If I had this book two years ago, it would have saved precious time going down some fruitless paths."
-- Tim Wolters, CTO, Collective Intellect

What people are saying - Write a review

User ratings

5 stars
4 stars
3 stars
2 stars
1 star

User Review - Flag as inappropriate

I was drawn to this book because I want to understand how to model data gushing from social networks and large data sets. Although I wasn't sure of how the book would approach the topics of data mining, I hypothesized that this book would give me some basic skills to leverage large data sets, at least giving me some examples of how data can be analyzed algorithmically.
I have discovered after reading the book that one can learn information pertaining to several skill sets:
1. Mining data from the internet and filtering that data. Since beginning to read the book, I have borrowed several techniques for gathering and saving data.
2. Introduction to about a dozen mathematical concepts and algorithms that use said data to filter, categorize, predict properties of, and determine relationships betweeen the data. I have realized that much entrepreneurial activity involved in enhancing human interaction is achieved through manipulation of data and these algorithms.
As far as the mathematics and algorithms themselves, much of it was more advanced than my current understanding, and will service in the long run as a springboard from which I must pursue these concepts. The book has given me a lot of starting points from further exploration.
3. An introduction to the python programming languages and some useful tools / packages written it it. Most of the python you will glean from this book is not laid out explicitly, but gathered from reading examples of data mining and analysis with python. Some basic syntax and script structure, as well as data types like lists and dictionaries, and functions for operating on data structures and performating mathematics are utilized.
Tools in python introduced in the book include FeedParser, Python Imaging Library, Beautiful Soup for html/xml parsing, pysqlite for database creation, NumPy for linear algebra and matrix mathematics, and matplotlib for 2D graphics.
Reading this book has piqued my interest in algorithms and mathematical analysis of datasets. From here I will pusue the study of these fields and a more in-depth understanding of some of the methods and algorithms presented. It was also a nice mental exercise to learn to read python code from real-world examples.

User Review - Flag as inappropriate

This looks like the 21st century successor to the AI programming books of the previous century (mine, and Charniak, Riesbeck, & McDermott). Lots of interesting applications; you can learn from them without much background required.

All 5 reviews »


Making Recommendations
Discovering Groups
Searching and Ranking
Document Filtering
Modeling with Decision Trees
Building Price Models
Kernel Methods and SVMs
Finding Independent Features
Evolving Intelligence
Algorithm Summary

Other editions - View all

Common terms and phrases

Popular passages

Page 9 - Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on Win32 Type "help", "copyright", "credits" or "license
Page 14 - This function will return a value between -1 and 1. A value of 1 means that the two people have exactly the same ratings for every item.

References to this book

About the author (2007)

Toby Segaran is the author of Programming Collective Intelligence, a very popular O'Reilly title. He was the founder of Incellico, a biotech software company later acquired by Genstruct. He currently holds the title of Data Magnate at Metaweb Technologies and is a frequent speaker at technology conferences.

Bibliographic information