Managing data at RAM speed

Two of IT’s most consistent mandates are lower costs and more speed. To meet this demand for cheap speed, standard disk-based database management systems need help. Increasingly, this help is coming in the form of memory-centric data management technology. Two of IT’s most consistent mandates are lower costs and more speed. To meet this demand for cheap speed, standard disk-based database management systems need help. Increasingly, this help is coming in the form of memory-centric data management technology. Text

Conventional DBMSs are designed to get data on and off of disks as safely, quickly and flexibly as possible. Much of their optimization is focused on one key bottleneck — the length of time it takes to find a random byte of data on disk, which is 1 million times as long as it might take to find the same byte in RAM.

But the optimizations and access methods designed to address this bottleneck don’t work so well once the data is safely in main memory. Memory-centric data management tools, using access methods that would be ridiculous in a disk-centric setup, can perform vastly better.

If you want to query a used-book database more than 1 million times per minute, that’s hard to do in a standard relational DBMS. But Progress Software’s ObjectStore gets it done for Amazon.com. If you want to recalculate a set of OLAP cubes in real time, don’t look to a disk-based system of any kind. But Applix’s TM1 can do just that.

And if you want to stick DBMS instances on 99 nodes of a telecommunications network, all persisting data to a 100th node, a disk–centric system isn’t your best choice — but Solid Information Technology has a product that works just fine. At their core, each of those products relies on the same technical approach: vast amounts of pointer traversal. Access that random is pretty impractical on disk, where it can take over a millisecond to get from one point to the next. But it works great in 100- to 1,000-MHz RAM.

There’s actually a broad variety of memory-centric products, most of them specialized for some particular kind of processing, whether OLAP or OLTP or event stream. They can be hard to find, being positioned as DBMS, quasi-DBMS, business intelligence features or some utterly new kind of middleware. They may come from top-tier software vendors or from the rawest of start-ups. But they are out there.

While memory-centric analytic technology has been around for a while, you may easily have missed it. It’s been held back by the addressability limits of 32-bit processors and even more by the scalability limits on most parallel hardware architectures. But that was before massively parallel (a.k.a. blade/grid) architectures made it practical to link huge numbers of CPUs together. You want 100GB of RAM on your server? No problem. One terabyte is achievable too, if you can afford that much silicon. And it’s just going to get easier, as the amount of RAM per board keeps doubling, based on the increasing performance of 64-bit chips.

Perhaps the simplest memory-centric approach to analytic data management is used in SAP’s superfast BI Accelerator (nee HPA, nee Euclid). Basically, it takes an InfoCube (a prebuilt SAP star-schema warehouse), sucks it into RAM with suitable sparsity compression and executes all analytic queries as full-table scans. Competing with that is tough when you’re bound by disk access speeds.

The most unique approach is probably that of Applix’s TM1, a memory-centric MOLAP tool that allows fully flexible evaluation rules without on-disk precalculation (and hence without the resulting data explosion). Thus, it simultaneously offers most of the benefits of analytic DBMS and most of the benefits of ordinary spreadsheets, and it deserves to be the basis of most planning applications unless and until something better comes along. Memory-centric OLTP, not as dependent on total RAM available, has probably made more of a market impact to date, although mainly in a few niches. I suspect that the ‘niche’ label is about to come off.Text

Memory-centric OLTP, not as dependent on total RAM available, has probably made more of a market impact to date, although mainly in a few niches. ObjectStore provides object-oriented memory-centric solutions for some of the most demanding OLTP apps in the world — not just the Amazon bookstore, but some airline reservations and scheduling systems as well. Products from Progress Software, TimesTen and start-up StreamBase Systems are used for highly real-time financial trading systems. TimesTen and Solid Information Technology support real-time management of billing and other functions on huge telecommunications networks.

I suspect that the “niche” label is about to come off. Oracle acquired TimesTen, and it probably didn’t do so just for a few particular apps. Did you buy a lot of Real Application Clusters recently? Well, the sequel is a generic caching-performance story, and it’s coming soon from an Oracle salesman near you.

For backup detail to this column, please see the DBMS2 blog at this Web site.

– Curt A. Monash is a consultant in Acton, Mass. You can reach him at [email protected].

Don’t miss related articles, white papers, product reviews, useful links and more! Visit our Spotlight on Data Management

Would you recommend this article?

Share

Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.


Jim Love, Chief Content Officer, IT World Canada

Featured Download

Featured Articles

Cybersecurity in 2024: Priorities and challenges for Canadian organizations 

By Derek Manky As predictions for 2024 point to the continued expansion...

Survey shows generative AI is a top priority for Canadian corporate leaders.

Leaders are devoting significant budget to generative AI for 2024 Canadian corporate...

Related Tech News

Tech Jobs

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.

Tech Companies Hiring Right Now