In this week, I research some articles on site apache. spark.org where I find some useful articles on Apache Spark and Apache Hadoop. Moreover, it is a really good site. I watch many videos on YouTube regarding Apache Spark and Apache Hadoop especially the Edureka site of youtube. In this week, I compare both the Big Data frameworks on different parameters to analyze their strengths and weaknesses.
Performance
Spark works very fast because it is working on in-memory processing. It can also use disk for data that doesn’t all fit into memory. Spark’s in-memory processing delivers near real-time analytics. Real-time analytics makes the Spark suitable for credit card processing system, machine learning, security analytics and Internet of Things sensors.
Hadoop was originally set up to continuously gather data from multiple sources without worrying about the type of data and storing it across a distributed environment. MapReduce uses batch processing. MapReduce was never built for real-time processing, the main idea behind YARN is parallel processing over distributed dataset.
The problem with comparing the two is that they perform processing differently.
Ease of Use
Spark is written in with user-friendly APIs for Scala, Java, Python, and Spark SQL. Spark SQL is very similar to SQL, so it becomes easier for SQL developers to learn it. Spark also provides an interactive shell for developers to query & perform other actions, & have immediate feedback.
You can ingest data in Hadoop easily either by using shell or integrating it with multiple tools like Sqoop, Flume, etc. YARN is just a processing framework and it can be integrated with multiple tools like Hive and Pig.
Data Processing
Batch Processing vs Stream Processing
In Hadoop, YARN is a basically a batch-processing framework. When we submit a job to YARN, it reads data from the cluster, performs operation & writes the results back to the cluster. Then it again reads the updated data, performs the next operation & write the results back to the cluster and so on.
Spark performs similar operations, but it uses in-memory processing and optimizes the steps. GraphX allows users to view the same data as graphs and as collections. Users can also transform and join graphs with Resilient Distributed Datasets (RDDs).
Fault Tolerance
HDFS and YARN both, master daemons (i.e. NameNode & ResourceManager respectively) checks heartbeat of slave daemons (i.e. DataNode & NodeManager respectively). As Hadoop uses commodity hardware, another way in which HDFS ensures fault tolerance is by replicating data.
RDDs can persist a dataset in memory across operations, which makes future actions 10 times much faster. If a RDD is lost, it will automatically be recomputed by using the original transformations. This is how Spark provides fault-tolerance.
Security
Hadoop supports Kerberos for authentication but the handling of it is difficult. Hadoop also supports like LDAP (Lightweight Directory Access Protocol) for authentication.
Spark currently supports authentication via a shared secret. Spark can integrate with HDFS and it can use HDFS ACLs and file-level permissions. Spark can also run on YARN leveraging the capability of Kerberos.
DeepakKumarPRJ702 