BEST OF THE WEB

Eight tips for resilient big data apps

One of the problems with big data applications is they have to handle big data — we’re talking huge data sets.

As Supreet Oberoi, vice-president of Concurrent Inc. , a maker of the Cascading application development framework, points out in a column for GigaOM, if they aren’t tough enough they may fail in production.

The solution is to build resilient, well-tested applications before they go out the door. “This is a matter of philosophy and architecture as much as technology,” he says, in putting forward eight tips for building big data apps that can hold up to demanding environments:

–Define a blueprint for resilient applications, with a systemic enterprise architecture and methodology for your company  approaches big data applications.

This means answering a number of questions, including where your current architecture is failing;

–Size shouldn’t matter.  Apps have to be tested with small-scale datasets, then fail or take too long with larger ones. They have to handle all sizes of data;

–Have a transparent process for finding problems, so developers and operations staff can diagnose and respond to problems when they happen;

–Abstraction and simplicity work. “Resilient applications tend to be future-proof because they employ abstractions that simplify development, improve productivity and allow substitution of implementation technology,” he writes. Developers should be able to build apps without being mired in the implementation details. Then data scientists should b able to use the app and access any type of data source;

–Build in security, auditing and compliance;

–Test-driven development should provide the ability to step through the code, establish invariants, and utilize other defensive programming techniques;

–Be portable. Applications should be designed to run on a variety of platforms and products;

–No black arts. Code should be shared, reviewed and commonly owned by multiple developers, not dependent on one person.

“If companies follow these eight rules, they will create resilient, scalable applications that allow them to tap into the full power of big data,” Oberoi writes.

How many of these rules does your developer team follow — or break?

Howard Solomon
Howard Solomon
Currently a freelance writer, I'm the former editor of ITWorldCanada.com and Computing Canada. An IT journalist since 1997, I've written for several of ITWC's sister publications including ITBusiness.ca and Computer Dealer News. Before that I was a staff reporter at the Calgary Herald and the Brampton (Ont.) Daily Times. I can be reached at hsolomon [@] soloreporter.com

Would you recommend this article?

Share

Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.


Jim Love, Chief Content Officer, IT World Canada

Featured Download

ITW in your inbox

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.

More Best of The Web