diff options
Diffstat (limited to 'content/blog/Storm-vs-Spark.markdown')
| -rw-r--r-- | content/blog/Storm-vs-Spark.markdown | 461 |
1 files changed, 0 insertions, 461 deletions
diff --git a/content/blog/Storm-vs-Spark.markdown b/content/blog/Storm-vs-Spark.markdown deleted file mode 100644 index e5f7776..0000000 --- a/content/blog/Storm-vs-Spark.markdown +++ /dev/null @@ -1,461 +0,0 @@ ---- -title: "Apache Storm and Apache Spark Streaming" -description: "Comparison of Apache Storm and Apache Spark Streaming" -tags: - - "Apache Storm" - - "Apache Spark" - - "Apache" - - "Real-time Streaming" - - "ZData Inc." -date: "2014-09-08" -categories: - - "Apache" - - "Development" - - "Real-time Systems" -slug: "apache-storm-and-apache-spark" ---- - -This is the last post in the series on real-time systems. In the [first -post][3] we discussed [Apache Storm][1] and [Apache Kafka][5]. In the [second -post][4] we discussed [Apache Spark (Streaming)][3]. In both posts we examined -a small Twitter Sentiment Analysis program. Today, we will be reviewing both -systems: how they compare and how they contrast. - -The intention is not to cast judgment over one project or the other, but rather -to exposit the differences and similarities. Any judgments made, subtle or not, -are mistakes in exposition and/ or organization and are not actual endorsements -of either project. - -## Apache Storm ## - -"Storm is a distributed real-time computation system" [[1][1]]. Apache Storm is -a [task parallel][7] continuous computational engine. It defines its workflows -in Directed Acyclic Graphs (DAG's) called "topologies". These topologies run -until shutdown by the user or encountering an unrecoverable failure. - -Storm does not natively run on top of typical Hadoop clusters, it uses -[Apache ZooKeeper][8] and its own master/ minion worker processes to -coordinate topologies, master and worker state, and the message guarantee -semantics. That said, both [Yahoo!][9] and [Hortonworks][10] are working on -providing libraries for running Storm topologies on top of Hadoop 2.x YARN -clusters. Furthermore, Storm can run on top of the [Mesos][11] scheduler as -well, [natively][12] and with help from the [Marathon][13] framework. - -Regardless though, Storm can certainly still consume files from HDFS and/ or -write files to HDFS. - -## Apache Spark (Streaming) ## - -"Apache Spark is a fast and general purpose engine for large-scale data -processing" [[2][2]]. [Apache Spark][2] is a [data parallel][8] general purpose -batch processing engine. Workflows are defined in a similar and reminiscent -style of MapReduce, however, is much more capable than traditional Hadoop -MapReduce. Apache Spark has its Streaming API project that allows for -continuous processing via short interval batches. Similar to Storm, Spark -Streaming jobs run until shutdown by the user or encounter an unrecoverable -failure. - -Apache Spark does not itself require Hadoop to operate. However, its data -parallel paradigm requires a shared filesystem for optimal use of stable data. -The stable source can range from [S3][14], [NFS][15], or, more typically, -[HDFS][16]. - -Executing Spark applications does not _require_ Hadoop YARN. Spark has its own -standalone master/ server processes. However, it is common to run Spark -applications using YARN containers. Furthermore, Spark can also run on Mesos -clusters. - -## Development ## - -As of this writing, Apache Spark is a full, top level Apache project. Whereas -Apache Storm is currently undergoing incubation. Moreover, the latest stable -version of Apache Storm is `0.9.2` and the latest stable version of Apache -Spark is `1.0.2` (with `1.1.0` to be released in the coming weeks). Of course, -as the Apache Incubation reminder states, this does not strictly reflect -stability or completeness of either project. It is, however, a reflection to -the state of the communities. Apache Spark operations and its process are -endorsed by the [Apache Software Foundation][27]. Apache Storm is working on -stabilizing its community and development process. - -Spark's `1.x` version does state that the API has stabilized and will not be -doing major changes undermining backward compatibility. Implicitly, Storm has -no guaranteed stability in its API, however, it is [running in production for -many different companies][34]. - -### Implementation Language ### - -Both Apache Spark and Apache Storm are implemented in JVM based languages: -[Scala][19] and [Clojure][20], respectively. - -Scala is a functional meets object-oriented language. In other words, the -language carries ideas from both the functional world and the object-oriented -world. This yields an interesting mix of code reusability, extensibility, and -higher-order functions. - -Clojure is a dialect of [Lisp][21] targeting the JVM providing the Lisp -philosophy: code-as-data and providing the rich macro system typical of Lisp -languages. Clojure is predominately functional in nature, however, if state or -side-effects are required, they are facilitated with a transactional memory -model, aiding in making multi-threaded based applications consistent and safe. - -#### Message Passing Layer #### - -Until version `0.9.x`, Storm was using the Java library [JZMQ][22] for -[ZeroMQ][23] messages. However, Storm has since moved the default messaging -layer to [Netty][24] with efforts from [Yahoo!][25]. Although Netty is now -being used by default, users can still use ZeroMQ, if desired, since the -migration to Netty was intended to also make the message layer pluggable. - -Spark, on the other hand, uses a combination of [Netty][24] and [Akka][26] for -distributing messages throughout the executors. - -### Commit Velocity ### - -As a reminder, these data are included not to cast judgment on one project or -the other, but rather to exposit the fluidness of each project. The continuum -of the dynamics of both projects can be used as an argument for or against, -depending on application requirements. If rigid stability is a strong -requirement, arguing for a slower commit velocity may be appropriate. - -Source of the following statistics were taken from the graphs at -[GitHub](https://github.com/) and computed from [this script][38]. - -#### Spark Commit Velocity #### - -Examining the graphs from -[GitHub](https://github.com/apache/spark/graphs/commit-activity), over the last -month (as of this writing), there have been over 330 commits. The previous -month had about 340. - -#### Storm Commit Velocity #### - -Again examining the commit graphs from -[GitHub](https://github.com/apache/incubator-storm/graphs/commit-activity), -over the last month (as of this writing), there have been over 70 commits. The -month prior had over 130. - -### Issue Velocity ### - -Sourcing the summary charts from JIRA, we can see that clearly Spark has a huge -volume of issues reported and closed in the last 30 days. Storm, roughly, an -order of magnitude less. - -Spark Open and Closed JIRA Issues (last 30 days): - -{{< figure src="/media/spark_issues_chart.png" - link="https://issues.apache.org/jira/browse/SPARK/" - alt="Apache Spark JIRA issues" >}} - -Storm Open and Closed JIRA Issues (last 30 days): - -{{< figure src="/media/storm_issues_chart.png" - link="https://issues.apache.org/jira/browse/STORM/" - alt="Apache Storm JIRA issues" >}} - -### Contributor/ Community Size ### - -#### Storm Contributor Size #### - -Sourcing the reports from -[GitHub](https://github.com/apache/incubator-storm/graphs/contributors), Storm -has over a 100 contributors. This number, though, is just the unique number of -people who have committed at least one patch. - -Over the last 60 days, Storm has seen 34 unique contributors and 16 over the -last 30. - -#### Spark Contributor Size #### - -Similarly sourcing the reports from [GitHub](https://github.com/apache/spark), -Spark has roughly 280 contributors. A similar note as before must be made about -this number: this is the number of at least one patch contributors to the -project. - -Apache Spark has had over 140 contributors over the last 60 days and 94 over -the last 30 days. - -## Development Friendliness ## - -### Developing for Storm ### - -* Describing the process structure with DAG's feels natural to the - [processing model][7]. Each node in the graph will transform the data in a - certain way, and the process continues, possibly disjointly. - -* Storm tuples, the data passed between nodes in the DAG, have a very natural - interface. However, this comes at a cost to compile-time type safety. - -### Developing for Spark ### - -* Spark's monadic expression of transformations over the data similarly feels - natural in this [processing model][6]; this falls in line with the idea - that RDD's are lazy and maintain transformation lineages, rather than - actuallized results. - -* Spark's use of Scala Tuples can feel awkward in Java, and this awkwardness - is only exacerbated with the nesting of generic types. However, this - awkwardness does come with the benefit of compile-time type checks. - - - Furthermore, until Java 1.8, anonymous functions are inherently - awkward. - - - This is probably a non-issue if using Scala. - -## Installation / Administration ## - -Installation of both Apache Spark and Apache Storm are relatively straight -forward. Spark may be simpler in some regards, however, since it technically -does not _need_ to be installed to function on YARN or Mesos clusters. The -Spark application will just require the Spark assembly be present in the -`CLASSPATH`. - -Storm, on the other hand, requires ZooKeeper to be properly installed and -running on top of the regular Storm binaries that must be installed. -Furthermore, like ZooKeeper, Storm should run under [supervision][35]; -installation of a supervisor service, e.g., [supervisord][28], is recommended. - -With respect to installation, supporting projects like Apache Kafka are out of -scope and have no impact on the installation of either Storm or Spark. - -## Processing Models ## - -Comparing Apache Storm and Apache Spark's Streaming, turns out to be a bit -challenging. One is a true stream processing engine that can do micro-batching, -the other is a batch processing engine which micro-batches, but cannot perform -streaming in the strictest sense. Furthermore, the comparison between streaming -and batching isn't exactly a subtle difference, these are fundamentally -different computing ideas. - -### Batch Processing ### - -[Batch processing][31] is the familiar concept of processing data en masse. The -batch size could be small or very large. This is the processing model of the -core Spark library. - -Batch processing excels at processing _large_ amounts of stable, existing data. -However, it generally incurs a high-latency and is completely unsuitable for -incoming data. - -### Event-Stream Processing ### - -[Stream processing][32] is a _one-at-a-time_ processing model; a datum is -processed as it arrives. The core Storm library follows this processing model. - -Stream processing excels at computing transformations as data are ingested with -sub-second latencies. However, with stream processing, it is incredibly -difficult to process stable data efficiently. - -### Micro-Batching ### - -Micro-batching is a special case of batch processing where the batch size is -orders smaller. Spark Streaming operates in this manner as does the Storm -[Trident API][33]. - -Micro-batching seems to be a nice mix between batching and streaming. However, -micro-batching incurs a cost of latency. If sub-second latency is paramount, -micro-batching will typically not suffice. On the other hand, micro-batching -trivially gives stateful computation, making [windowing][37] an easy task. - -## Fault-Tolerance / Message Guarantees ## - -As a result of each project's fundamentally different processing models, the -fault-tolerance and message guarantees are handled differently. - -### Delivery Semantics ### - -Before diving into each project's fault-tolerance and message guarantees, here -are the common delivery semantics: - -* At most once: messages may be lost but never redelivered. - -* At least once: messages will never be lost but may be redelivered. - -* Exactly once: messages are never lost and never redelivered, perfect - message delivery. - -### Apache Storm ### - -To provide fault-tolerant messaging, Storm has to keep track of each and every -record. By default, this is done with at least once delivery semantics. -Storm can be configured to provide at most once and exactly once. The delivery -semantics offered by Storm can incur latency costs; if data loss in the stream -is acceptable, at most once delivery will improve performance. - -### Apache Spark Streaming ### - -The resiliency built into Spark RDD's and the micro-batching yields a trivial -mechanism for providing fault-tolerance and message delivery guarantees. That -is, since Spark Streaming is just small-scale batching, exactly once delivery -is a trivial result for each batch; this is the _only_ delivery semantic -available to Spark. However some failure scenarios of Spark Streaming degrade -to at least once delivery. - -## Applicability ## - -### Apache Storm ### - -Some areas where Storm excels include: near real-time analytics, natural -language processing, data normalization and [ETL][36] transformations. It also -stands apart from traditional MapReduce and other course-grained technologies -yielding fine-grained transformations allowing very flexible processing -topologies. - -### Apache Spark Streaming ### - -Spark has an excellent model for performing iterative machine learning and -interactive analytics. But Spark also excels in some similar areas of Storm -including near real-time analytics, ingestion. - -## Final Thoughts ## - -Generally, the requirements will dictate the choice. However, here are some -major points to consider when choosing the right tool: - -* Latency: Is the performance of the streaming application paramount? Storm - can give sub-second latency much more easily and with less restrictions - than Spark Streaming. - -* Development Cost: Is it desired to have similar code bases for batch - processing _and_ stream processing? With Spark, batching and streaming are - _very_ similar. Storm, however, departs dramatically from the MapReduce - paradigm. - -* Message Delivery Guarantees: Is there high importance on processing _every_ - single record, or is some nominal amount of data loss acceptable? - Disregarding everything else, Spark trivially yields perfect, exactly once - message delivery. Storm can provide all three delivery semantics, but - getting perfect exactly once message delivery requires more effort to - properyly achieve. - -* Process Fault Tolerance: Is high-availability of primary concern? Both - systems actually handle fault-tolerance of this kind really well and in - relatively similar ways. - - - Production Storm clusters will run Storm processes under - [supervision][35]; if a process fails, the supervisor process will - restart it automatically. State management is handled through - ZooKeeper. Processes restarting will reread the state from ZooKeeper on - an attempt to rejoin the cluster. - - - Spark handles restarting workers via the resource manager: YARN, Mesos, - or its standalone manager. Spark's standalone resource manager handles - master node failure with standby-masters and ZooKeeper. Or, this can be - handled more primatively with just local filesystem state - checkpointing, not typically recommended for production environments. - -Both Apache Spark Streaming and Apache Storm are great solutions that solve the -streaming ingestion and transformation problem. Either system can be a great -choice for part of an analytics stack. Choosing the right one is simply a -matter of answering the above questions. - -## References ## - -[spark_jira_issues]: https://kennyballou.com/media/spark_issues_chart.png - -[storm_jira_issues]: https://kennyballou.com/media/storm_issues_chart.png - -[1]: http://storm.incubator.apache.org/documentation/Home.html - -* [Apache Storm Home Page][1] - -[2]: http://spark.apache.org - -* [Apache Spark][2] - -[3]: http://www.zdatainc.com/2014/07/real-time-streaming-apache-storm-apache-kafka/ - -* [Real Time Streaming with Apache Storm and Apache Kafka][3] - -[4]: http://www.zdatainc.com/2014/08/real-time-streaming-apache-spark-streaming/ - -* [Real Time Streaming with Apache Spark (Streaming)][4] - -[5]: http://kafka.apache.org/ - -* [Apache Kafka][5] - -[6]: http://en.wikipedia.org/wiki/Data_parallelism - -* [Wikipedia: Data Parallelism][6] - -[7]: http://en.wikipedia.org/wiki/Task_parallelism - -* [Wikipedia: Task Parallelism][7] - -[8]: http://zookeeper.apache.org - -* [Apache ZooKeeper][8] - -[9]: https://github.com/yahoo/storm-yarn - -* [Yahoo! Storm-YARN][9] - -[10]: http://hortonworks.com/kb/storm-on-yarn-install-on-hdp2-beta-cluster/ - -* [Hortonworks: Storm on YARN][10] - -[11]: http://mesos.apache.org - -* [Apache Mesos][11] - -[12]: https://mesosphere.io/learn/run-storm-on-mesos/ - -* [Run Storm on Mesos][12] - -[13]: https://github.com/mesosphere/marathon - -* [Marathon][13] - -[14]: http://aws.amazon.com/s3/ - -[15]: http://en.wikipedia.org/wiki/Network_File_System - -[16]: http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html - -[17]: https://issues.apache.org/jira/browse/STORM/ - -[18]: https://issues.apache.org/jira/browse/SPARK/ - -[19]: http://www.scala-lang.org/ - -[20]: http://clojure.org/ - -[21]: http://en.wikipedia.org/wiki/Lisp_(programming_language) - -[22]: https://github.com/zeromq/jzmq - -[23]: http://zeromq.org/ - -[24]: http://netty.io/ - -[25]: http://yahooeng.tumblr.com/post/64758709722/making-storm-fly-with-netty - -[26]: http://akka.io - -[27]: http://www.apache.org/ - -[28]: http://supervisord.org - -[29]: http://xinhstechblog.blogspot.com/2014/06/storm-vs-spark-streaming-side-by-side.html - -* [Storm vs Spark Streaming: Side by Side][29] - -[30]: http://www.slideshare.net/ptgoetz/apache-storm-vs-spark-streaming - -* [Storm vs Spark Streaming (Slideshare)][30] - -[31]: http://en.wikipedia.org/wiki/Batch_processing - -[32]: http://en.wikipedia.org/wiki/Event_stream_processing - -[33]: https://storm.incubator.apache.org/documentation/Trident-API-Overview.html - -[34]: http://storm.incubator.apache.org/documentation/Powered-By.html - -[35]: http://en.wikipedia.org/wiki/Process_supervision - -[36]: http://en.wikipedia.org/wiki/Extract,_transform,_load - -[37]: http://en.wikipedia.org/wiki/Window_function_(SQL)#Window_function - -[38]: https://gist.github.com/kennyballou/c6ff37e5eef6710794a6 |