In this week, I research about why the spark was needed if already there was Hadoop, what kind of problem does it help me to solve or why were map reduce was not enough or capable solution to process big data and what were the challenges with map reduce. Firstly, I am going to compare the framework of Apache Spark and Apache Hadoop. Then, moving ahead we will compare both the Big Data frameworks on different parameters to analyze their strengths and weaknesses.
| Apache Hadoop | Apache Spark |
| Hadoop is a framework that allows you to first store Big Data in a distributed environment so that you can process it parallelly. There are basically two components in Hadoop | Apache Spark is a framework for real-time data analytics in a distributed computing environment. It executes in-memory computations to increase the speed of data processing. |
| 1. HDFS
HDFS creates an abstraction of resources, Similar as virtualization, you can see HDFS logically as a single unit for storing Big Data, but actually you are storing your data across multiple nodes in a distributed fashion. Here, you have master-slave architecture. In HDFS, Namenode is a master node and Datanodes are slaves.
|
|
| NameNode: It is the master daemon that maintains and manages the DataNodes (slave nodes). It records the metadata of all the files stored in the cluster, e.g. location of blocks stored, the size of the files, permissions, hierarchy, etc. It records each and every change that takes place to the file system metadata.
For example, if a file is deleted in HDFS, the NameNode will immediately record this in the EditLog. It regularly receives a Heartbeat and a block report from all the DataNodes in the cluster to ensure that the DataNodes are live. It keeps a record of all the blocks in HDFS and in which nodes these blocks are stored.
|
2. Spark Streaming – Spark Streaming is the component of Spark which is used to process real-time streaming data. Thus, it is a useful addition to the core Spark API. It enables high-throughput and fault-tolerant stream processing of live data streams
|
| DataNode
These are slave daemons which run on each slave machine. The actual data is stored on DataNodes. They are responsible for serving read and write requests from the clients. They are also responsible for creating blocks, deleting blocks and replicating the same based on the decisions taken by the NameNode.
|
3. Spark SQL: Spark SQL is a new module in Spark which integrates relational processing with Spark’s functional programming API. It supports querying data either via SQL or via the Hive Query Language. For those of you familiar with RDBMS, Spark SQL will be an easy transition from your earlier tools where you can extend the boundaries of traditional relational data processing |
| 2. Map Reduce Engine: This engine is primarily responsible for managing resources as well as controlling and monitoring computing processes
· Job Tracker: Complex computing processes (jobs) are separated into individual parts and then distributed by a so-called Job Tracker. · Task Trackers ensure that the subprocesses are processed in a parallelized manner. |
4. GraphX: GraphX is the Spark API for graphs and graph-parallel computation. Thus, it extends the Spark RDD with a Resilient Distributed Property Graph. At a high-level, GraphX extends the Spark RDD abstraction by introducing the Resilient Distributed Property Graph: a directed multigraph with properties attached to each vertex and edge.
|
| 3. YARN
YARN performs all your processing activities by allocating resources and scheduling tasks. It has two major daemons, i.e. ResourceManager and NodeManager. ResourceManager It is a cluster level (one for each cluster) component and runs on the master machine. It manages resources and schedule applications running on top of YARN. NodeManager It is a node level component (one on each node) and runs on each slave machine. It is responsible for managing containers and monitoring resource utilization in each container. It also keeps track of node health and log management. It continuously communicates with ResourceManager to remain up-to-date. So, you can perform parallel processing on HDFS using MapReduce.
|
5. MLlib (Machine Learning): MLlib stands for Machine Learning Library. Spark MLlib is used to perform machine learning in Apache Spark.
|
DeepakKumarPRJ702 