In this week, I have researched on Apache Spark and Apache Hadoop. I researched on what are these tools and if Apache Hadoop was there then why there was a need to develop Apache Spark. Apache Spark and Apache Hadoop are two different tools for querying big data. Basically, these are two different tools for processing and transforming big data and mining out meaningful information from a huge amount of data. Previously, Hadoop was a big thing. There were some problem issues with Hadoop and that is why apache spark was developed at the University of California, Berkeley AmpLab to overcome the issues with Hadoop. Next week, I will research on comparative research on the framework of both tools.
Apache Hadoop is an open source, Scalable, and Fault-tolerant framework written in Java. It efficiently processes large volumes of data on a cluster of commodity hardware. Hadoop is not only a storage system but is a platform for large data storage as well as processing.
Apache Spark is also an open source cluster computing framework. Spark runs applications in Hadoop clusters up to 100x faster in memory and 10x faster on disk. By reducing the number of read/write cycle to disk and storing intermediate data in-memory spark makes it possible.
Comparison
| Apache Spark | Apache Hadoop |
| Apache Spark is used for real-time data processing and as well as batch processing. | Apache Hadoop is used for only batch processing. |
| Easy to program and does not require abstractions. | Difficult to program requires abstractions |
| Programmers can perform streaming, batch processing, and machine learning, all in the same cluster. | It is used for generating reports that help find answers to historical queries. |
| Has in-built interactive mode. | Non- built interactive mode except tools like Pig and Hive |
| Executes jobs 10 to 100 times faster than Hadoop Map Reduce. | Hadoop MapReduce does not leverage the memory of the Hadoop cluster to the maximum. |
| Programmers can modify the data in real-time through Spark Streaming. | Allows you to just process a batch of stored data. |
Use Cases of Apache Hadoop:
There are many real-life use cases of big data Hadoop.
Use of Big Data in Retail Industry: Transaction data in its raw form helps a company understand its sales pattern. Savvy retailers can use big data combining data from web browsing patterns, social media, industry forecasts, existing customer records, and many other data to predict trends, prepare for demand, pinpoint customers, optimize pricing and promotions, and monitor real-time analytics and results.
Use Cases of Spark
1.Credit Card Fraud Detection: As millions of people are using credit card nowadays, it has become very necessary to protect people from frauds. It has become a challenge for Credit card companies to identify whether the requested transaction is fraudulent or not.
2. In the game industry, processing and discovering patterns from the potential firehose of real-time in-game events and being able to respond to them immediately is a capability that could yield a lucrative business, for purposes such as player retention, targeted advertising, auto-adjustment of complexity level, and so on.
https://link.springer.com/article/10.1007/s41060-016-0027-9
https://ieeexplore.ieee.org/abstract/document/7021746
DeepakKumarPRJ702 