MANHASSET NY -- IBM researchers successfully scanned 10 billion files on a single system in 43 minutes, shattering the previous record of one billion files in three hours by a factor of 37.
The breakthrough was achieved by tweaking its General Parallel File System (GPFS) to run on a cluster of 10 eight-core systems and solid state storage from Violin Memory.
The previous record was set by IBM researchers at the Supercomputing 2007 conference where they demonstrated the ability to scan one billion files in three hours.
GPFS was unveiled in 1998 for applications requiring high-speed access to large volumes of data such as data mining, seismic data processing, risk management and financial analysis, weather modeling and scientific research. GPFS’s advanced algorithm makes possible the full use of all processor cores on all of these machines in all phases of the task (data read, sorting and rules evaluation).
In a technical white paper researchers at IBM Advanced Storage Laboratory, IBM Almaden Research Center said this only the first step in the constant drum call for the information processing power consumed by leading business, government and scientific organizations whose data storage needs continues to grow at a 90 percent CAGR.
IBM’s white paper is available here.IBM, Researchers, Data IBM researchers perform data scan record
No comments:
Post a Comment