Dealing with Very Massive Data Moves

Dealing with big data moves is on the mind of almost any company you can think of these days. Even the tiny “mom and pop” internet companies can find themselves have to occasionally get a file over 1 GB in size to some other remote location. (And in all seriousness, 1 GB is small by comparision to what some companies have to move.)

So, what are some of these big files? How about a file that lets you analyze Web clickstreams, call data records, financial trading data, log files, or other forms of machine-generated information. Any time you are into this kind of file storage and moving, you are having to deal with very big files.

There are two new products on the market that have been specifically designed for this purpose. Enter IBM Netezza and Infobright to the big file data movement game.

IBM began making public announcemnts about its new product, IBM Netezza, on Wednesday of this week. It is designed to move files in the petabytes range. It is designed to help people that, for a host of reasons, need to keep archived files that are very large. The new hardware and software system is designed to hold up to 20 500 terabyte racks, for a total of 10 petabytes of user accessable data. It is making use of improved technology to the standard offered TwinFin hard drive sytem. Now the speed is not the absolute best in the world ( a standard query can run 2.5 times faster on the TwinFin system than the new Netzza system), but that is usually not an issue for companies wanting to search large databases for , say, regulatory queries. On the flip side, the newer hard drives use less energy than ever before, so that has an appeal also.
Infobright, though, has come up with a new software application that is designed to meet the needs of smaller companies. It’s new piece of software is designed to simply and effeciently move files of 40 terabytes or less. It will still allow many companies all the data storage they need, without sacrificing a lot of the speed of queries.

So, has you notion of big data files changed any? We’ll talk next time.

Johnny Smoes