Hadoop and Spark
Context: FIT1043_MOC, FIT3003_MOC · the two big open-source distributed-processing platforms · Hadoop = batch, Spark = in-memory/real-time · FIT3003 W12: Spark is the compute layer of a Data Lakehouse
Quick Revision
- 🎯 Objective: choose a big-data engine ➔ Hadoop for cheap batch/offline Map-Reduce; Spark for fast in-memory/real-time.
- ⚡ Key Constraint: Hadoop writes to disk between steps (not suited to streaming); Spark keeps data in memory, so it’s much faster and handles real-time.
📝 How It Works
1. Hadoop
- What ➔ open-source Java implementation of Map-Reduce (Doug Cutting @ Yahoo!).
- Architecture ➔ Common (Java libraries/utilities) + MapReduce (the core paradigm); a huge tool ecosystem.
- Three components (FIT3003 W12) ➔ HDFS (storage) + MapReduce (processing) + YARN (resource management).
- Fit ➔ inexpensive platform for parallelising processing; batch/offline only — not suited to streaming; past its hype peak.
2. Spark
- What ➔ an Apache top-level project (AMPLab, UC Berkeley); builds on Hadoop infrastructure.
- Interfaces ➔ Java, Scala, Python, R.
- Edge ➔ in-memory analytics ➔ real-time processing, much faster than Hadoop; works with parts of the Hadoop ecosystem; includes Map-Reduce capabilities.
- Deployment (FIT3003 W12) ➔ the in-memory successor to MapReduce; runs standalone, on YARN or on Kubernetes — Hadoop infrastructure is optional; reads HDFS, S3/ADLS object storage, or databases via JDBC; supports SQL, batch and streaming.
3. What makes distributed processing hard (FIT3003 W12)
- Four costs ➔ network communication and data shuffling · uneven work across nodes (data skew) · fault recovery and coordination · infrastructure and processing cost ➔ the price paid for splitting a workload across machines (Distributed Processing and Map-Reduce).
⚖️ Core Decision Matrix
| Aspect | Hadoop | Spark |
|---|---|---|
| Core | HDFS + Map-Reduce + YARN (Java) | in-memory engine — standalone, YARN or Kubernetes |
| Speed | disk-based, slower | in-memory, much faster |
| Processing | batch / offline | real-time / streaming + batch |
| Interfaces | Java (MapReduce) | Java, Scala, Python, R |
When It Flips: the difference is where data lives between steps — Hadoop spills to disk (cheap, batch), Spark keeps it in RAM (fast, real-time). For real-time processing, choose Spark.
⚠️ Common Mistakes
- 💡 Hadoop can’t stream ➔ its disk-based Map-Reduce suits offline batch; use Spark (or a streaming platform like Kafka) for real-time.
- 💡 Spark ⊃ Map-Reduce ➔ Spark isn’t a rejection of Map-Reduce; it includes those capabilities and adds in-memory speed.
🧠 Active Recall
Differentiate Hadoop and Spark, and say which suits real-time processing.
Answer
- Short answer: Hadoop is a disk-based Java Map-Reduce platform for cheap batch/offline processing; Spark is an in-memory engine built on Hadoop infra that is much faster and supports real-time — so Spark for real-time.
- Why: In-memory vs disk ➔ Spark avoids Hadoop’s between-step disk writes, enabling streaming/interactive speed.