Tool Enables Scientists to Uncover Patterns in Vast Data Sets

With support from the National Science Foundation, researchers from the Broad Institute and Harvard University recently developed a tool that can uncover patterns in large data sets in a way that no other software program can.

Called Maximal Information Coefficient or MIC, the tool can can tease out multiple, recurring events or sets of data hidden in health information from around the globe, or in the changing bacterial landscape of the gut or even in statistics amassed from a season of competitive sports--and much more. The researchers report their findings in the Dec. 16th issue of the journal Science.

Part of a suite of statistical tools called MINE for Maximal Information-based Nonparametric Exploration, MIC has the ability to sort through today's mass of research variables--from attempts to track hurricanes, efforts to model earthquakes, endeavors to identify the Higgs Boson and efforts to glean insights from affecting the world economy and social networking interaction.

Researchers currently use advanced technology to gather big, complex, data sets, which may be incredibly useful in enhancing system understanding, if, in fact, vast amounts of data can be organized so that telling information may be extracted. Sophisticated computer programs research these data sets with great speed, but fall short in even-handedly detecting different kinds of patterns in large data collections, essential for more sophisticated analysis.

One of the greatest strengths of this newly discovered tool within MINE is its ability to detect and analyze a broad spectrum of patterns and characterize them according to a number of different parameters a researcher might be interested in. Other statistical tools work well for searching for a specific pattern in a large data set, but cannot score and compare different kinds of possible relationships. Researchers can also use MINE to generate new ideas and connections.

Learn more about MINE, MIC and patterns identified in biological and health data, as well as statistics from the 2008 baseball season by visiting the Broad Institute website. A video about this work also is available on the website.

-NSF-