Big Data Analytics

1.3.1. Volume

1.3.2. Velocity

1.3.3. Variety

1. Structured
   
   • Data with a fixed format.
   • For examples, spreadsheets and databases are structured data.
   • DBMS operations are very fast, and this is very scalable.
2. Unstructured
   
   • Data with no fixed format.
   • For example, images, videos, text.
   • 80% of the data is unstructured.
   • To deal with unstructured data, the various ways are
     • Data mining
     • NLP
     • Text Analytics
     • Noisy Text Analytics
3. Semi-Structured
   
   • It might have a fixed format, but might contain unstructured data.
   • XML is a great example - it has a format (tags), but the content can be unstructured. A simpler example would be an email (from, to, CC, BCC form a structure, but the content can be anything).
   • It’s often either data with labels, or data in the form of key-value pairs.

1.3.4. Newer V’s introduced in recent times

1. Veracity
   
   • Trustworthiness/credibility of data.
2. Value
   
   • The ability to extract meaningful data.
3. Validity
   
   • Is the data correct or appropriate for its intended use?
4. Visualization
   
   • Putting data in a human-interpretable form.
5. Variability
   
   • Changes in the nature of the data.
6. Vulnerability
   
   • Compromises in system security.
7. Viscosity
   
   • How easy/difficult it is to combine/transfer data.
8. Volatility
   
   • How long data is valid for, and how long it should be stored.
   • This is the characteristics of data that deals with its retention.
9. Virality
   
   • How far the existing data spreads around.

8.7.1. 3Cs

9.1.1. Consistency

9.1.2. Availability

9.1.3. Partition Tolerance

9.3.1. CP

9.3.2. AP

9.3.3. CA

12.2.1. Hadoop 1.x

1. HDFS
   
   • This is for reliable storage.
   • HDFS stands for Hadoop Distributed File System.

   • Given any file, you split the file into blocks

     File (1GB)
     ↓
     Block1 (128MB)
     Block2 (128MB)
     Block3 (128MB)
     ...
     

     Then you distribute the blocks across different nodes:

     Node1 → Block1
     Node2 → Block2
     Node3 → Block3
     Node4 → Block4
     

   1. Blocks
      
      • This is the minimum amount of data that can be read or written. (64 MB or 128 MB).
      • By default there are 3 replicas of data, and they will be placed across at least 2 racks.
        • The first replica is placed on the local node
        • The second and third replica are palaced on a remote rack during write pipeline.
      • The client reads from the nearest replica.
   2. Rack
      
      • It’s a collection of 40-50 data nodes using the same network switch.

      Rack1	Rack2	Rack3
      Block1	_	_
      _	Block1	_
      _	_	_
      _	Block1	_

   3. Rack Awareness
      
      • Choosing data node racks that are closest to each other is called rack awareness.
      • Improves cluster performance by reducing network traffic.
      • This is how you move the computation to where the data is stored.
   4. Master/Slave Architecture - Node
      
      1. NameNode
         
         • Master of the system
         • Maintains and manages the blocks present on the Data Node.
         • It doesn’t store the files themselves. It only stores metadata, locates the files and tracks changes.
         • These are up and running all the time.
         • There exists a secondary NameNode, which takes over if the primary NameNode is not there.

         • There are two components of a NameNode:

           FSImage (File System Image)	Edit Logs
           Point-in-time snapshot of the file system metadata	Journals recording every change (write/delete) made to the namespace since the last snapshot

         1. Secondary NameNode
            
            • NameNode only stores metadata (FSImage) and logs. So incase it restarts, it has to replay the entire log to manage the cluster.
            • The Secondary NameNode pulls the current FSImage and Edit Logs from the Active NameNode.
            • It merges them, creating a new, up-to-date FSImage. This means the current snapshot is modified to accommodate the new changes presented in Edit logs.
            • The new FSImage is sent back to the Primary NameNode, and the old edit log is cleared
      2. DataNode
         
         • These are the machines that actually store data and serve it to the client.
         • They send heartbeat signals to the NameNode about every 3 seconds to indicate health.
         • Responsible for serving read/write requests to the client.
2. MapReduce
   
   • MapReduce is a programming model for cluster resource management and data processing.
   • It’s also known as Hadoop’s processing unit.
   • Map is for loading, parsing, transforming and filtering key-value pairs.
   • Reduce is for grouping and aggregating data produced by map.
   • There are various steps for MapReduce:
   1. Record Reader
      
      • You read documents, where the key is the name of the file and the value is the contents of the file.
      • For example, a <key, value> pair could be <input.txt, "the quick brown fox the">.
   2. Mapper
      
      • Mapper splits each word from the value, and now each word will be the key and the count of that word will be the value.

      • So the output of Mapper would be:

        <"the", 1>
        <"quick", 1>
        <"brown", 1>
        <"fox", 1>
        <"the", 1>
        

        This is hardcoded, so if a word appears twice, there will be duplicate keys, and each key will have value 1.

   3. Combiner
      
      • This reads each key-value pair and for common keys, it combines the values as a collection.

      • So the output of Combiner would be:

        <"the", [1, 1]>
        <"quick", [1]>
        <"brown", [1]>
        <"fox", [1]>
        

   4. Reducer
      
      • This reduces a set of intermediate values (the values are now a collection), into a final value.

      • So the output of Reducer would be:

        <"the", 2>
        <"quick", 1>
        <"brown", 1>
        <"fox", 1>
        

   5. Some Map-Reduce Terminology
      

      Job
       ├── Task (Map)
       │    ├── Task Attempt 1
       │    ├── Task Attempt 2
       │    └── Task Attempt 3
       ├── Task (Map)
       │    └── Task Attempt 1
       └── Task (Reduce)
            ├── Task Attempt 1
            └── Task Attempt 2
      

      1. Job
         
         • The problem statement i.e. what you want to run Map-Reduce on.
      2. Task
         
         • Smaller piece of a job
         • There’s the map task and the reduce task
      3. Task Attempt
         
         • It’s an execution attempt of a task.
         • For one task, there may be multiple task attempts, because of node failures or task crashes.
   6. Some applications of MapReduce
      
      1. Reverse Web-Link Graph
         
         • You have a bunch of tuples (a,b) where there’s a link from page a to b.

         • The output is that for each page we have to display the list of pages that link to it.

           For example, given (a,b), (c,b) and (d,b), the output will be

           <b, [a,c,d]>
           

         • Map will take (a,b) as input and output (b,a) (it simply swaps, so that we can group by b)

           map(a, b):
               emit(b, a)
           

         • Hadoop automatically does something called shuffle and sort phase, where all entries with the same key are grouped.

         • The reduce phase would simply output the list.

           reduce(b, list_of_a):
               emit(b, list_of_a)
           

   7. Some things you CAN’T use Map-Reduce for
      
      • Real-time analytics because of high latency. MapReduce was designed for massive batch jobs.
      • Iterative algorithms because re-reading and writing data to HDFS is expensive.
      • A large number of small files, because
        • They can overwhelm the NameNode and create metadata overhead.
        • The shuffle and sort phase moves data all over the nodes consuming bandwidth.
      • Tasks which need Caching, because in MapReduce, the same dataset is re-read from the disk for every subsequent job.
   8. Daemons in Hadoop 1.x
      
      • NameNode (Master Node in HDFS)
      • Secondary NameNode (Helper)
      • DataNode (Slave Node in HDFS)
      • JobTracker (Master Node in MapReduce)
      • TaskTracker (Slave Node in MapReduce)

12.2.2. Hadoop 2.x

1. HDFS
   
   • Reliable Storage.
   • HDFS is still the file system of Hadoop 2.x.
   1. Standby NameNode
      
      • Along with primary and secondary NameNode, there also exists a Standby NameNode.
      • It acts as a passive backup to the active NameNode.
   2. Data Pipelining
      
      • It’s a multistep processing of moving large volumes of data across a distributed computing cluster.
2. MapReduce
   
   • Data processing.
   • There is no longer any concept of job tracker or task tracker
3. Yarn
   
   • Cluster Resource Managment
   • Yarn stands for Yet Another Resource Negotiator.
   • Yarn is called the operating system of Hadoop 2.x.
   • Yarn defines two resources:
     • v-cores: virtual cores (CPU power allocated to a container on a node)
     • memory: memory allocated to a container on a node
4. Daemons in Hadoop 2.x
   
   • NameNode (Master node in HDFS)
   • Secondary NameNode (helper)
   • DataNode (slave node in HDFS)
   • Resource Manager (master node in YARN)
     • This keeps a running total of the cluster’s available resources.
   • NodeManager (slave node in YARN)
   • ApplicationMaster
     • This isn’t a daemon, but it’s a framework-specific library to manage the entire lifecycle of a job.
   • Container
     • These are the resources allocated to a single node on a cluster.
5. How Yarn works
   
   • First the application talks to the ResourceManager for the cluster
   • In response, the ResourceManager first allocates one single container.
   • The ApplicationMaster starts running within that container.
   • The ApplicationMaster requests subsequent containers from ResourceManager.
   • Once all tasks are finished, ApplicationMaster exists and the container is de-allocated.
6. Key Features
   
   1. Scalability
      
   2. Flexibility
      
      • Supports way more processing models than just MapReduce, including interactive SQL, real-time streaming.
   3. Improved Utilization