In this week, I already finish my report and I also send my report to my supervisor for checking. I face a lot of difficulties in the whole project. Initially, the report writing was difficult for me on Apache Spark vs Apache Hadoop because it has a lot of technical terms in my project which was difficult for me to understand. I watch a lot of YouTube videos and YouTube Channels that greatly helps me to understand all the difficult terms and theoretical parts of my project. But I complete the report with the help of the useful guidance of my supervisor. He helps me by directing me in a good direction. Initially, I was worried because I was facing a lot of difficulties on how to write a good logical report because it was not easy for me to understand all the technical terms. I give my 100% efforts and put all my efforts and hard work to complete my project and I finally achieve it. It was a really good experience with my supervisor. He helps me a lot and really helps me at every stage of my project by providing good guidance.
Week 11
In this, I complete my report. But there are some parts that are left in the report. I write the APA references in ascending order. But I still need some time to organize my research in a logical manner. I think there might be some grammatical mistakes so I have to check it. Moreover, I also have to check the Turnitin report and need to check my report twice. I also give the APA references of video that I use as resources for my report. Some topics still need to cover in the report so I need to cover those also.
Week 10
In this week, I research on the companies who are using Apache Spark and Apache Hadoop. Moreover, I research some of the major commands to carry out different workloads across the Hadoop platform. Some of the major commands of Hadoop are
- Checksum
- Movefromlocal
- Hadoop infrastructure.
- Appendtofile
- Copy to local
- Chgrep
Moreover, I am writing my report. Some part of my report is completed but some part is left. I spend most of my time in writing the report. I face difficulty in organizing my research for the report. I also research on the current market situation of both tools.
Week 9
In this week, I start to write the report but before writing the report, my project supervisor Lars Dam told me to write the references of all the research papers and findings before writing the report so that it becomes easy for to write the report. I prepare the report template so that I can show it to my supervisor and get the feedback about it. Moreover, I complete some part of the report. I organize my research and make a plan to present research in a logical way.
Week 8
In this week, I research on the use cases of Apache Spark and Apache Hadoop. Moreover, I start writing the report but I face difficulty in writing the report. I will ask from my supervisor what is the exact format and structure of the report and how the table of content should look like.
Use Cases of Apache Hadoop
- Healthcare
- Wildlife
- Retail Industry
- Income Tax to scrutinize bank Account
- Fraud Detection
- Sentimental Analysis
- Networking Security
- Education
Use Cases of Apache Spark
- Streaming Data
- Streaming ETL
- Data Enrichment
- Trigger Event Detection
- Complex Session Analysis
- Machine learning
- Interactive Analysis
- Fog Computing
- Weather Analysis
Week 7
In this week, I research some articles on site apache. spark.org where I find some useful articles on Apache Spark and Apache Hadoop. Moreover, it is a really good site. I watch many videos on YouTube regarding Apache Spark and Apache Hadoop especially the Edureka site of youtube. In this week, I compare both the Big Data frameworks on different parameters to analyze their strengths and weaknesses.
Performance
Spark works very fast because it is working on in-memory processing. It can also use disk for data that doesn’t all fit into memory. Spark’s in-memory processing delivers near real-time analytics. Real-time analytics makes the Spark suitable for credit card processing system, machine learning, security analytics and Internet of Things sensors.
Hadoop was originally set up to continuously gather data from multiple sources without worrying about the type of data and storing it across a distributed environment. MapReduce uses batch processing. MapReduce was never built for real-time processing, the main idea behind YARN is parallel processing over distributed dataset.
The problem with comparing the two is that they perform processing differently.
Ease of Use
Spark is written in with user-friendly APIs for Scala, Java, Python, and Spark SQL. Spark SQL is very similar to SQL, so it becomes easier for SQL developers to learn it. Spark also provides an interactive shell for developers to query & perform other actions, & have immediate feedback.
You can ingest data in Hadoop easily either by using shell or integrating it with multiple tools like Sqoop, Flume, etc. YARN is just a processing framework and it can be integrated with multiple tools like Hive and Pig.
Data Processing
Batch Processing vs Stream Processing
In Hadoop, YARN is a basically a batch-processing framework. When we submit a job to YARN, it reads data from the cluster, performs operation & writes the results back to the cluster. Then it again reads the updated data, performs the next operation & write the results back to the cluster and so on.
Spark performs similar operations, but it uses in-memory processing and optimizes the steps. GraphX allows users to view the same data as graphs and as collections. Users can also transform and join graphs with Resilient Distributed Datasets (RDDs).
Fault Tolerance
HDFS and YARN both, master daemons (i.e. NameNode & ResourceManager respectively) checks heartbeat of slave daemons (i.e. DataNode & NodeManager respectively). As Hadoop uses commodity hardware, another way in which HDFS ensures fault tolerance is by replicating data.
RDDs can persist a dataset in memory across operations, which makes future actions 10 times much faster. If a RDD is lost, it will automatically be recomputed by using the original transformations. This is how Spark provides fault-tolerance.
Security
Hadoop supports Kerberos for authentication but the handling of it is difficult. Hadoop also supports like LDAP (Lightweight Directory Access Protocol) for authentication.
Spark currently supports authentication via a shared secret. Spark can integrate with HDFS and it can use HDFS ACLs and file-level permissions. Spark can also run on YARN leveraging the capability of Kerberos.
Week 6
In this week I find some research papers of big data analytics of apache spark vs Apache Hadoop. I do a thorough study of those papers and get some useful information about both tools. Moreover, I research on the companies which are using especially apache spark and how apache spark has helped those companies to boost their businesses. Following are the important links of the papers and other sources that I have found in last week.
https://www.computerresearch.org/index.php/computer/article/view/1137/1124
Raj, A., & D’Souza, R. (2019). A Review on Hadoop Eco System for Big Data.
Week 5
In this week, I research about why the spark was needed if already there was Hadoop, what kind of problem does it help me to solve or why were map reduce was not enough or capable solution to process big data and what were the challenges with map reduce. Firstly, I am going to compare the framework of Apache Spark and Apache Hadoop. Then, moving ahead we will compare both the Big Data frameworks on different parameters to analyze their strengths and weaknesses.
| Apache Hadoop | Apache Spark |
| Hadoop is a framework that allows you to first store Big Data in a distributed environment so that you can process it parallelly. There are basically two components in Hadoop | Apache Spark is a framework for real-time data analytics in a distributed computing environment. It executes in-memory computations to increase the speed of data processing. |
| 1. HDFS
HDFS creates an abstraction of resources, Similar as virtualization, you can see HDFS logically as a single unit for storing Big Data, but actually you are storing your data across multiple nodes in a distributed fashion. Here, you have master-slave architecture. In HDFS, Namenode is a master node and Datanodes are slaves.
|
|
| NameNode: It is the master daemon that maintains and manages the DataNodes (slave nodes). It records the metadata of all the files stored in the cluster, e.g. location of blocks stored, the size of the files, permissions, hierarchy, etc. It records each and every change that takes place to the file system metadata.
For example, if a file is deleted in HDFS, the NameNode will immediately record this in the EditLog. It regularly receives a Heartbeat and a block report from all the DataNodes in the cluster to ensure that the DataNodes are live. It keeps a record of all the blocks in HDFS and in which nodes these blocks are stored.
|
2. Spark Streaming – Spark Streaming is the component of Spark which is used to process real-time streaming data. Thus, it is a useful addition to the core Spark API. It enables high-throughput and fault-tolerant stream processing of live data streams
|
| DataNode
These are slave daemons which run on each slave machine. The actual data is stored on DataNodes. They are responsible for serving read and write requests from the clients. They are also responsible for creating blocks, deleting blocks and replicating the same based on the decisions taken by the NameNode.
|
3. Spark SQL: Spark SQL is a new module in Spark which integrates relational processing with Spark’s functional programming API. It supports querying data either via SQL or via the Hive Query Language. For those of you familiar with RDBMS, Spark SQL will be an easy transition from your earlier tools where you can extend the boundaries of traditional relational data processing |
| 2. Map Reduce Engine: This engine is primarily responsible for managing resources as well as controlling and monitoring computing processes
· Job Tracker: Complex computing processes (jobs) are separated into individual parts and then distributed by a so-called Job Tracker. · Task Trackers ensure that the subprocesses are processed in a parallelized manner. |
4. GraphX: GraphX is the Spark API for graphs and graph-parallel computation. Thus, it extends the Spark RDD with a Resilient Distributed Property Graph. At a high-level, GraphX extends the Spark RDD abstraction by introducing the Resilient Distributed Property Graph: a directed multigraph with properties attached to each vertex and edge.
|
| 3. YARN
YARN performs all your processing activities by allocating resources and scheduling tasks. It has two major daemons, i.e. ResourceManager and NodeManager. ResourceManager It is a cluster level (one for each cluster) component and runs on the master machine. It manages resources and schedule applications running on top of YARN. NodeManager It is a node level component (one on each node) and runs on each slave machine. It is responsible for managing containers and monitoring resource utilization in each container. It also keeps track of node health and log management. It continuously communicates with ResourceManager to remain up-to-date. So, you can perform parallel processing on HDFS using MapReduce.
|
5. MLlib (Machine Learning): MLlib stands for Machine Learning Library. Spark MLlib is used to perform machine learning in Apache Spark.
|
Week 4
In this week, I have researched on Apache Spark and Apache Hadoop. I researched on what are these tools and if Apache Hadoop was there then why there was a need to develop Apache Spark. Apache Spark and Apache Hadoop are two different tools for querying big data. Basically, these are two different tools for processing and transforming big data and mining out meaningful information from a huge amount of data. Previously, Hadoop was a big thing. There were some problem issues with Hadoop and that is why apache spark was developed at the University of California, Berkeley AmpLab to overcome the issues with Hadoop. Next week, I will research on comparative research on the framework of both tools.
Apache Hadoop is an open source, Scalable, and Fault-tolerant framework written in Java. It efficiently processes large volumes of data on a cluster of commodity hardware. Hadoop is not only a storage system but is a platform for large data storage as well as processing.
Apache Spark is also an open source cluster computing framework. Spark runs applications in Hadoop clusters up to 100x faster in memory and 10x faster on disk. By reducing the number of read/write cycle to disk and storing intermediate data in-memory spark makes it possible.
Comparison
| Apache Spark | Apache Hadoop |
| Apache Spark is used for real-time data processing and as well as batch processing. | Apache Hadoop is used for only batch processing. |
| Easy to program and does not require abstractions. | Difficult to program requires abstractions |
| Programmers can perform streaming, batch processing, and machine learning, all in the same cluster. | It is used for generating reports that help find answers to historical queries. |
| Has in-built interactive mode. | Non- built interactive mode except tools like Pig and Hive |
| Executes jobs 10 to 100 times faster than Hadoop Map Reduce. | Hadoop MapReduce does not leverage the memory of the Hadoop cluster to the maximum. |
| Programmers can modify the data in real-time through Spark Streaming. | Allows you to just process a batch of stored data. |
Use Cases of Apache Hadoop:
There are many real-life use cases of big data Hadoop.
Use of Big Data in Retail Industry: Transaction data in its raw form helps a company understand its sales pattern. Savvy retailers can use big data combining data from web browsing patterns, social media, industry forecasts, existing customer records, and many other data to predict trends, prepare for demand, pinpoint customers, optimize pricing and promotions, and monitor real-time analytics and results.
Use Cases of Spark
1.Credit Card Fraud Detection: As millions of people are using credit card nowadays, it has become very necessary to protect people from frauds. It has become a challenge for Credit card companies to identify whether the requested transaction is fraudulent or not.
2. In the game industry, processing and discovering patterns from the potential firehose of real-time in-game events and being able to respond to them immediately is a capability that could yield a lucrative business, for purposes such as player retention, targeted advertising, auto-adjustment of complexity level, and so on.
https://link.springer.com/article/10.1007/s41060-016-0027-9
https://ieeexplore.ieee.org/abstract/document/7021746
Week 3
In the last week, I understand what is big data and do some research about Apache Spark and Apache Hadoop.
Big data
The amount of data on the internet is in the order of exabytes and it becomes (10^18 bytes) and such huge data sets are what you call big data. For instance, Google, Youtube, Facebook, and many other sites contain lot contents being uploaded by a lot many users, lots are shared, comments given on post and much such activity occur every minute from all over the world. We call ‘Big Data ‘, which always keep increasing at exponential rates. Big Data Hadoop is called a huge amount Of Data. It is also structured and Unstructured Data. Big Data Hadoop Is defined to Five V’s Of Big Data Hadoop. Volume, Velocity, Variety, Value, Veracity.
Volume: The meaning of volume is measured by information produced every second. Simply think about every one of the messages, twitter messages, photographs, video cuts, sensor information and so on we deliver and share each second.
Velocity: Speed can really compare and it is just the speed at which data is generated.
Variety: The third V is called a variety. Any company can get information from a wide range of sources: from in-house gadgets to cell phone GPS innovation or what individuals are stating on interpersonal organizations. The significance of these wellsprings of data shifts relying upon the idea of the business.
Veracity: The fourth V is veracity, which is equal to quality. We have every one of the information, however, would we be able to miss something? Is the information clean and precise?
Value: The extraction of useful data from huge amount of data is called value.
Some Use cases of Big Data
- Amazon is the biggest Use case of Big Data. Each and every click you do on their website is recorded, analyzed and the next time you visit you will come up with the best deals and prices for the items one checked and looked. You may also get emails with such offerings. It records your IP your location your social media accounts etc. This is done on a real-time basis.
- Analytics for Fraud Detection: Banks can find an unusual card behavior in real time (if somebody else, not the owner, is using it) and block suspicious activities or banks can at least postpone them to inform the owner.
DeepakKumarPRJ702 