Apache Spark adds a much needed spark to Big Data processing

Apache Spark has been the new kid on the block that is now being touted as the next big thing in Big Data. It is the largest open source project in data processing and comes equipped with features that make it fast, easy to use and make it a unified engine. From the point of inception, Spark has taken followers in big companies such as Yahoo, Amazon, eBay, Groupon etc. on a massive scale. It has in a short span of time become the largest open source community in Big Data, with over 750 contributors from 200+ organizations.

Spark is a framework that enables parallel, distributed data processing. It offers a simple programming abstraction that provides powerful cache and persistence capabilities. Its framework can be deployed through Apache Mesos, Apache Hadoop via Yarn, or Spark’s own cluster manager. It also serves as a foundation for additional data processing frameworks such as Shark, which provides SQL functionality for Hadoop.