Spark 2 Workbook Answers Apr 2026

def fetch_batch(it): session = requests.Session() for url in it: yield session.get(url).text session.close()

1. **Ingestion** – `spark.read.json` or `textFile`. 2. **Parsing** – `withColumn` + `from_unixtime`, `regexp_extract`. 3. **Cleaning** – filter out malformed rows, `na.drop`. 4. **Enrichment** – join with a static lookup table (broadcast). 5. **Aggregation** – `groupBy(date, status).agg(count("*").as("cnt"))`. 6. **Output** – write to Parquet partitioned by `date` **or** stream to console for debugging.

sc = SparkContext(appName="DistinctWordCount")

# 4️⃣ Action – trigger the computation and collect the count unique_word_count = distinct_words.count() spark 2 workbook answers

Add a short paragraph for each stage, explaining why you chose that API.

- [ ] All code compiles/run on Spark 2.x (no 3.x‑only APIs). - [ ] Comments are present for every non‑obvious line. - [ ] You’ve referenced at least **one** Spark concept (lazy eval, shuffle, broadcast, etc.). - [ ] Edge cases are discussed. - [ ] The answer is written **in your own words** (no copy‑pasting from the internet).

| Operation | PySpark | Scala | |-----------|---------|-------| | **Read CSV** | `spark.read.option("header","true").csv(path)` | `spark.read.option("header","true").csv(path)` | | **Write Parquet** | `df.write.parquet("out.parquet")` | `df.write.parquet("out.parquet")` | | **Cache** | `df.cache()` | `df.cache()` | | **Repartition** | `df.repartition(10)` | `df.repartition(10)` | | **Window** | `from pyspark.sql.window import Window` | `import org.apache.spark.sql.expressions.Window` | | **UDF** | `spark.udf.register("toUpper", lambda s: s.upper(), StringType())` | `udf((s: String) => s.toUpperCase, StringType)` | | **Streaming read** | `spark.readStream.format("socket")...` | `spark.readStream.format("socket")...` | | **Stop Spark** | `spark.stop()` | `spark.stop()` | def fetch_batch(it): session = requests

import requests

```python from pyspark import SparkContext

val spark = SparkSession.builder() .appName("DeptSalary") .getOrCreate() "true").csv(path)` | `spark.read.option("header"

sc = SparkContext(appName="WordCount") lines = sc.textFile("hdfs:///data/myfile.txt")

### 🎯 Your Next Step

## 7. Putting It All Together – A Mini‑Project Blueprint