Hadoop gets a Real-Time Processing by Open Source vets

Nodeable solves real time Big Data issues

Big Data is certainly on a lot of people's lips these days. There is no doubt that we are certainly generating lots of data. Analyzing that data and making it useful is fueling literally millions of dollars of investment in companies around Hadoop, NoSQL, etc. One area where Big Data has some challenges is real-time analysis. With all of that data, analyzing in real time to get actionable intelligence into the hands of users is a big challenge. That is the the challenge that Nodeable is seeking to tackle.

Nodeable is led by a couple of open source veterans. Dave Rosenberg, formerly of Mule Source among a few other open source projects, is the CEO of Nodeable. With him are several folks who have worked with him in his previous open source companies. Additionally, Matt Assay, another veteran open source company builder, is on board at Nodeable as well.

I had a chance to sit down with Dave and talk about what he and his team are doing with Nodeable. You can listen in on our 15 or so minute conversation below. Let me warn you, the audio is a bit uneven at some points, but it isn't too bad and I think the quality of the conversation is well worth the problem with the quality of the audio.

The Nodeable team is using an open source program called Storm, which was originally developed by some folks at Twitter. Nodeable is seeking to commercialize this and build on top of it. This is a model that Dave has followed in the past and has lots of experience with.

Nodeable has been kicking around for a while now, but only recently really went public with this model. It is not competitive with Hadoop or other Big Data solutions, rather it brings another needed facet of Big Data to analytics.

So have a listen to Dave and check out a new and different Big Data solution coming to market.

VMware launches open source toolkit to run Hadoop on virtual machines

VMware is ramping up its big data push with Serengeti, a new open source toolkit that lets enterprises run Apache Hadoop on virtual machines.

In a statement Thursday, the virtualisation juggernaut said the toolkit will allow enterprises to deploy a Hadoop cluster in minutes on VMware’s vSphere virtualisation platform, plus common Hadoop components such as Apache Pig and Apache Hive.

VMware is also working with the Apache Hadoop community to contribute extensions that will make key components “virtualisation-aware” to support elastic scaling and improve Hadoop’s performance in virtual environments.

Apache Hadoop is an open source platform commonly used by large enterprises in the growing area of big data processing, where complex data sets are broken down into smaller chunks for analysis by clusters of computers to derive key business insights. It is based on MapReduce, a programming model conceived by Google to overcome the problem of creating web search indexes.

According to VMware, deployment and operational complexity, the need for dedicated hardware, and concerns about security and service level assurance have prevented many enterprises from taking advantage of Hadoop.

“By decoupling Apache Hadoop nodes from the underlying physical infrastructure, VMware can bring the benefits of cloud infrastructure – rapid deployment, high-availability, optimal resource utilization, elasticity, and secure multi-tenancy – to Hadoop,” it said.

Tony Baer, principal analyst at technology consultancy Ovum said: “Hadoop must become friendly with the technologies and practices of enterprise IT if it is to become a first-class citizen within enterprise IT infrastructure. The resource-intensive nature of large Big Data clusters make virtualisation an important piece that Hadoop must accommodate”.

“VMware’s involvement with the Apache Hadoop project and its new Serengeti Apache project are critical moves that could provide enterprises the flexibility that they will need when it comes to prototyping and deploying Hadoop,” Baer added.

Earlier this month, VMware partnered with HortonWorks to develop a high availability architecture that allows companies to run HortonWorks’ Hadoop clusters on vSphere. In April, it also acquired big data start-up Cetas that provides analytics applications on top of Hadoop.

The R Programming Language

Take the open road to statistical analysis

Statistical analysis has been around since mainframes were introduced to academia and corporations back in the 1960s.

But the great diversity of telemetry collected by systems today, the need to sift through it for insight and the growing popularity of open-source alternatives is transforming the R programming language for statistical analysis and visualisation. Its new nickname is Red Hat for stats.

Everybody loves R, particularly those selling big-data products such as data warehouses and Hadoop data munchers.

Part of the reason is that R is an open source package that solicits input from a large and clever community of statisticians and quantitative analysts who are able to steer its development.

Alphabet soup
This was not the case for proprietary tools created by SAS Institute and SPSS at the dawn of the mainframe era, and their follow-ons in the distributed computing era.

Just as Linux can be thought of as an open-source analog to Unix, the R programming language borrows heavily from the S language.

This was created by John Chambers at Bell Labs in 1976, as a reaction to the pricey but well respected SPSS and SAS tools that came out nearly a decade earlier.

S is very much a child of the VAX and Unix minicomputer era, while R is a product of the PC and Linux era.

The R language was created in 1996 by Ross Ihaka and Robert Gentleman, two stats professors from the University of Auckland in New Zealand who are still core members of the R development team. (Incidentally, so is Chambers, the creator of S, and it is no accident that some data crunching routines for S will run unchanged in the R environment.)

R can be thought of as a modern implementation of S. So can S-PLUS, created by a company called Insightful, which licensed S from Lucent Technologies in 2004 and was eaten by Tibco Software in 2008.

Come the revolution
Unlike S and to a certain extent S-PLUS, R is not just some code created in an ivory tower.

It is the product of a community of statisticians and coders which has created more than 2,500 plug-ins for chewing on various data sets and doing statistical analysis tuned specifically for particular data types or industries.

R is used by more than two million quantitative analysts worldwide, according to estimates made by Revolution Analytics, which was founded in 2007 to create a parallel implementation of R.

Since then, the company has taken an open-core approach to R, offering commercial support for the open-source package, while at the same time extending the R environment to run better on clusters of machines and in conjunction with Hadoop clusters.

To date, no one has commercialised the PSPP open-source alternative to SPSS (acquired by IBM in July 2009), but it would not be surprising to see this happen at some point, if PSSP matures.

Revolution Analytics has not exactly made the R community happy by peddling proprietary extensions to R in its R Enterprise distribution, after getting some seed money from Intel Capital in 2008 and $9m in venture money in 2009.

Since then, Revolution Analytics has parallelised the underlying R statistical engine so it runs better on multicore/multithreaded processors and across server clusters; added a NoSQL-like format called XDF to help parallelise data sets; and added support for native SAS file formats and conversion to XDF

Most recently it has tweaked its R implementation so each node in a Hadoop cluster can run R analytics locally on the Hadoop cluster on data stored in the Hadoop Distributed File System and then aggregate the results of those calculations, much like MapReduce operations on unstructured data.

Revolution Analytics has soaked up a lot of the oxygen in the R room for the past few years. But other companies are doing interesting things, integrating R tools with their own products and making life easier for analysts seeking answers in mountains of data.

Parallel universe
Seeking some kind of advantage over its rivals in the data warehousing space, Netezza opened up the Netezza software stack in February 2010.

Netezza is a maker of data warehousing appliances based on a heavily customised and parallelised version of the PostgreSQL database, which uses field programmable gate arrays (FPGAs) to boost its performance running on x86 clusters.

Netezza opened up its software development environment with a set of APIs that allow SAS and R algorithms to run in parallel on its warehouse appliances.

It also similarly offered hooks for Java, C++, Fortran, or Python applications to reach into the data warehouse and use the FPGAs to extract data stored in the warehouse rather than using the SQL database query language.

Seven months later, as it became clearer that big data was going to be big business, IBM snapped up privately held Netezza for a cool $1.7bn.

In October 2010, data warehouse maker Teradata added its own in-database analytics to its eponymous data warehouses with a package called TeradataR.

This turns the Teradata Warehouse Miner tool into a plug-in for the R console, allowing for 44 different analytical functions in Teradata databases, as well as any stored procedures in data warehouses to be exposed to R and called from R programs. There are another 20 functions that let R work in the Teradata environment.

The idea is to stay within the R console and run the analytics in parallel on the database, instead of trying to suck information down into a workstation and running R locally.

Oracle joins in
Even Oracle is getting in on the R act. In February, the company launched Advanced Analytics, a bridge between Oracle databases and the R analytical engine.

Advanced Analytics is Oracle's Data Mining add-on for its 11g R2 database. When R programmers want to run a statistical routine, they call the equivalent SQL function in the Data Mining toolbox and run that against the database.

If there is no such SQL function, then an embedded R engine spread across database nodes (if it is a cluster) runs the R routines, collects up summary data and presents it back to the R console as an answer.

Oracle also ships something called the R Connector for Hadoop for its Big Data Appliance, a version of the Cloudera CDH3 Hadoop environment running on Oracle's Exa x86 cluster iron.

This connector lets an R console talk to Hadoop Distributed File System and NoSQL databases running on the Big Data Appliance.

Hadoop wil get big growth by revenues: IDC

The market for software related to the Hadoop and MapReduce programming frameworks for large-scale data analysis will jump from $77 million in 2011 to $812.8 million in 2016, a compound annual growth rate of 60.2%, according to a new report released Monday by analyst firm IDC.

Hadoop is an open-source implementation of the MapReduce framework. It is hosted at the Apache Software Foundation along with a number of supporting software projects, including the Hadoop Distributed File System (HDFS) and Pig programming language.

MapReduce and Hadoop are based on the principle of splitting up large amounts of data and then processing the chunks in parallel across large numbers of nodes. It's closely associated with the industry buzzword "big data," which refers to the ever-larger volumes of information, particularly of unstructured form, being generated by websites, social media, sensors and other sources.

Overall, Hadoop has enjoyed a steady stream of interest from commercial analytics and database vendors in recent years, who have begun offering commercial products and services for it.

While "fantastic and largely unsupportable claims have been made" regarding Hadoop and MapReduce's use cases and benefits, "there can be no doubt that it does provide a relatively low-cost means of deriving considerable value from very large collections of unorganized data," IDC analysts Carl Olofson and Dan Vesset wrote in the report.

Therefore, the conditions are right for significant growth in the Hadoop-MapReduce "ecosystem," according to IDC.

This year, "Leading adopters in the mainstream IT world will move from 'proof of concept' to real value," the report states.

However, lack of qualified talent will limit the technology's rise during the next two to three years, it adds.

The coming years will also see a "battle between open source purists, who believe that the core of Hadoop deployment must be based purely on the Apache project code," according to IDC. However, most IT organizations will use a mix of commercial and open-source components in their Hadoop environments, the report adds.

Still, "competition between open source vendors and their closed source counterparts may force lower license fees from the latter group, resulting in somewhat slower software revenue growth than would be the case if open source projects did not represent so large a component of this market space."

IDC is a subsidiary of IDG News Service's parent company, International Data Group.