> For the complete documentation index, see [llms.txt](https://gitbook.nicacton.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://gitbook.nicacton.com/cloud-computing/aws/analytics/elastic-mapreduce-emr.md).

# Elastic MapReduce (EMR)

EC2 instances for big data processing based off Hadoop framework

## Essentials

* Simplifies running big data frameworks (Hadoop, Spark) on AWS
* Processing of data in large batches
* A managed service for running Hadoop clusters on EC2
* EMR is used to analyze and process vast amounts of data

### Features and Advantages

* S3 for storage
  * Load into HDFS or keep in S3 (EMRFS)
* Transient clusters
  * Can also be persistent
  * Steps or tasks to be completed are defined, then termination
  * Saves a lot of money
* Spot Instances
* Bootstrapping
  * Customized configuration while launching
* Preconfigured application frameworks
* Scalability (add and remove core and task nodes)

### Master Node

* Node that manages the cluster by running software components which coordinate the distribution of data and tasks among other (slave) nodes for processing
* The master node trackks the status of tasks and monitors the health of the cluster
* SSH to the master node is permitted to run interactive jobs

### Slave Nodes

Two Types

* Core Node
  * A slave node has software components which run tasks AND store data in the Hadoop Distributed File System (HDFS) on your cluster
  * The core nodes do the "heavy lifting" with the data
* Task Node
  * A slave node that has software components which only run tasks
  * Task nodes are optional

## Mapping Job

### Map Phase

* Mapping is a function that defines the processing of data for each split
* The block size for HDFS is 128MB, which is the optimum split size.
* The larger the instance size used in our EMR cluster, the more chunks you can map and process at the same time.
* If there are more chunks than nodes/mappers, the chunks will queue for processing

### Reduce Phase

* Reducing is a function that aggregates the data into a single output file
* Reduced data needs to be stored (maybe in S3) as data processed by the EMR cluster is not persistent.
*