UnionAll Operation in PySpark DataFrames: A Comprehensive Guide

PySpark’s DataFrame API is a powerful tool for big data processing, and the unionAll operation is a key method for combining multiple DataFrames by stacking their rows vertically. Whether you’re merging datasets from different sources, appending new records, or consolidating data for analysis, unionAll provides a straightforward way to unify DataFrames with matching schemas. Built on Spark’s Spark SQL engine and optimized by Catalyst, it ensures scalability and efficiency across distributed systems. This guide covers what unionAll does, the various ways to apply it, and its practical uses, with clear examples to illustrate each approach.

Ready to master unionAll? Explore PySpark Fundamentals and let’s get started!

What is the UnionAll Operation in PySpark?

The unionAll method in PySpark DataFrames combines two or more DataFrames by stacking their rows vertically, returning a new DataFrame with all rows from the input DataFrames. It’s a transformation operation, meaning it’s lazy; Spark plans the union but waits for an action like show to execute it. Requiring DataFrames to have identical schemas (same column names and types), unionAll preserves duplicates and is widely used for appending datasets, merging similar data, or building comprehensive collections for processing. It’s identical to union in PySpark, serving as an alias with the same functionality but named to align with SQL conventions.

Here’s a basic example:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("UnionAllIntro").getOrCreate()
data1 = [("Alice", 25), ("Bob", 30)]
data2 = [("Cathy", 22), ("David", 28)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
unionall_df = df1.unionAll(df2)
unionall_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# |Cathy| 22|
# |David| 28|
# +-----+---+
spark.stop()

A SparkSession initializes the environment, and two DataFrames (df1 and df2) are created with matching schemas (name, age). The unionAll(df2) call stacks df2’s rows below df1’s, and show() displays the combined result with all four rows. For more on DataFrames, see DataFrames in PySpark. For setup details, visit Installing PySpark.

Various Ways to Use UnionAll in PySpark

The unionAll operation offers multiple ways to combine DataFrames, each tailored to specific needs. Below are the key approaches with detailed explanations and examples.

1. Basic UnionAll of Two DataFrames

The simplest use of unionAll combines two DataFrames with identical schemas by stacking their rows vertically, preserving all data including duplicates. This is ideal when you need to append one dataset to another, such as adding new records to an existing table, without altering the original content.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("BasicUnionAll").getOrCreate()
data1 = [("Alice", 25), ("Bob", 30)]
data2 = [("Cathy", 22), ("David", 28)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
basic_unionall_df = df1.unionAll(df2)
basic_unionall_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# |Cathy| 22|
# |David| 28|
# +-----+---+
spark.stop()

The DataFrame df1 has two rows, and df2 has two rows with the same schema. The unionAll(df2) call stacks df2 below df1, resulting in 4 rows. The show() output displays all rows in the order they were combined. This method is straightforward for basic row concatenation.

2. UnionAll of Multiple DataFrames

The unionAll operation can combine more than two DataFrames by chaining multiple unionAll calls, stacking all rows vertically into a single DataFrame. This is useful when you need to consolidate data from several sources, such as monthly datasets or logs from different systems, into one unified collection.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MultiUnionAll").getOrCreate()
data1 = [("Alice", 25)]
data2 = [("Bob", 30)]
data3 = [("Cathy", 22)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
df3 = spark.createDataFrame(data3, ["name", "age"])
multi_unionall_df = df1.unionAll(df2).unionAll(df3)
multi_unionall_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# |Cathy| 22|
# +-----+---+
spark.stop()

Three DataFrames (df1, df2, df3) each have one row with the same schema. The chained unionAll(df2).unionAll(df3) call stacks all rows, resulting in 3 rows. The show() output shows the combined data. This method scales to append multiple DataFrames efficiently.

3. UnionAll with Duplicates Preserved

The unionAll operation preserves duplicates when combining DataFrames, including identical rows within or across the datasets. This is valuable when you need to retain all occurrences, such as logging events or tracking repeated entries, without deduplication.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DuplicateUnionAll").getOrCreate()
data1 = [("Alice", 25), ("Bob", 30)]
data2 = [("Bob", 30), ("Cathy", 22)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
dup_unionall_df = df1.unionAll(df2)
dup_unionall_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# |  Bob| 30|
# |Cathy| 22|
# +-----+---+
spark.stop()

The DataFrame df1 has "Bob, 30," and df2 repeats it. The unionAll(df2) call stacks all rows, preserving the duplicate "Bob, 30" entries. The show() output shows 4 rows, including both instances. This method ensures all data, even duplicates, is retained.

4. UnionAll with Column Renaming for Schema Alignment

The unionAll operation requires matching schemas, and withColumnRenamed can align column names before combining DataFrames with differing but compatible structures. This is essential when merging datasets with similar data but mismatched column names, allowing seamless stacking.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("RenameUnionAll").getOrCreate()
data1 = [("Alice", 25), ("Bob", 30)]
data2 = [("Cathy", 22), ("David", 28)]
df1 = spark.createDataFrame(data1, ["full_name", "age"])
df2 = spark.createDataFrame(data2, ["name", "years"])
renamed_df2 = df2.withColumnRenamed("name", "full_name").withColumnRenamed("years", "age")
rename_unionall_df = df1.unionAll(renamed_df2)
rename_unionall_df.show()
# Output:
# +---------+---+
# |full_name|age|
# +---------+---+
# |    Alice| 25|
# |      Bob| 30|
# |    Cathy| 22|
# |    David| 28|
# +---------+---+
spark.stop()

The DataFrame df1 uses "full_name" and "age," while df2 uses "name" and "years." The withColumnRenamed calls align df2’s schema to df1’s, and unionAll(renamed_df2) stacks the rows. The show() output shows a unified DataFrame. This method resolves schema mismatches.

5. UnionAll with Filtering or Transformation

The unionAll operation can be combined with filter or other transformations post-union to refine the combined DataFrame, such as removing unwanted rows or adjusting data. This is useful when you need to process the unified dataset further, like excluding nulls or standardizing values, after stacking.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("FilteredUnionAll").getOrCreate()
data1 = [("Alice", 25), ("Bob", None)]
data2 = [("Cathy", 22), ("David", 28)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
filtered_unionall_df = df1.unionAll(df2).filter(col("age").isNotNull())
filtered_unionall_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |Cathy| 22|
# |David| 28|
# +-----+---+
spark.stop()

The unionAll(df2) call combines df1 and df2, including a null age for Bob. The filter(col("age").isNotNull()) removes rows with null ages, and show() displays the filtered result. This method refines the unioned data efficiently.

Common Use Cases of the UnionAll Operation

The unionAll operation serves various practical purposes in data integration.

1. Combining Data from Multiple Sources

The unionAll operation merges datasets from different sources, such as logs or files.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MultiSourceUnionAll").getOrCreate()
data1 = [("Alice", 25), ("Bob", 30)]
data2 = [("Cathy", 22), ("David", 28)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
source_unionall_df = df1.unionAll(df2)
source_unionall_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# |Cathy| 22|
# |David| 28|
# +-----+---+
spark.stop()

Data from two sources is combined.

2. Appending New Records

The unionAll operation adds new records to an existing dataset, such as updates.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("AppendRecords").getOrCreate()
data1 = [("Alice", 25), ("Bob", 30)]
data2 = [("Cathy", 22)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
append_df = df1.unionAll(df2)
append_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# |Cathy| 22|
# +-----+---+
spark.stop()

New record "Cathy" is appended.

3. Consolidating Partitioned Data

The unionAll operation consolidates data split across partitions or files.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ConsolidatePartitions").getOrCreate()
data1 = [("Alice", "HR", 25)]
data2 = [("Bob", "IT", 30)]
df1 = spark.createDataFrame(data1, ["name", "dept", "age"]).repartition(1)
df2 = spark.createDataFrame(data2, ["name", "dept", "age"]).repartition(1)
consolidated_df = df1.unionAll(df2)
consolidated_df.show()
# Output:
# +-----+----+---+
# | name|dept|age|
# +-----+----+---+
# |Alice|  HR| 25|
# |  Bob|  IT| 30|
# +-----+----+---+
spark.stop()

Partitioned data is unified.

4. Building Comprehensive Datasets with Duplicates

The unionAll operation builds datasets, retaining duplicates for completeness.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("BuildWithDuplicates").getOrCreate()
data1 = [("Alice", 25), ("Bob", 30)]
data2 = [("Bob", 30), ("Cathy", 22)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
build_df = df1.unionAll(df2)
build_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# |  Bob| 30|
# |Cathy| 22|
# +-----+---+
spark.stop()

A dataset includes duplicate "Bob" entries.

FAQ: Answers to Common UnionAll Questions

Below are answers to frequently asked questions about the unionAll operation in PySpark.

Q: How does unionAll differ from union?

A: They are identical in PySpark; union is an alias.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FAQVsUnion").getOrCreate()
data1 = [("Alice", 25)]
data2 = [("Bob", 30)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
unionall_df = df1.unionAll(df2)
union_df = df1.union(df2)
unionall_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# +-----+---+
union_df.show()  # Same output
spark.stop()

Both produce the same result.

Q: Does unionAll remove duplicates?

A: No, unionAll preserves all rows, including duplicates.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FAQDuplicates").getOrCreate()
data1 = [("Alice", 25)]
data2 = [("Alice", 25)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
dup_df = df1.unionAll(df2)
dup_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |Alice| 25|
# +-----+---+
spark.stop()

Duplicates are retained.

Q: How does unionAll handle null values?

A: Nulls are preserved in the combined DataFrame.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FAQNulls").getOrCreate()
data1 = [("Alice", 25)]
data2 = [("Bob", None)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
null_df = df1.unionAll(df2)
null_df.show()
# Output:
# +-----+----+
# | name| age|
# +-----+----+
# |Alice|  25|
# |  Bob| null|
# +-----+----+
spark.stop()

"Bob"’s null age is included.

Q: Does unionAll affect performance?

A: It’s efficient for small unions; large datasets increase memory use.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FAQPerformance").getOrCreate()
data1 = [("Alice", 25)]
data2 = [("Bob", 30)]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "age"])
perf_df = df1.unionAll(df2)
perf_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# +-----+---+
spark.stop()

Small unions are lightweight.

Q: Can I unionAll DataFrames with different schemas?

A: No, schemas must match; use renaming or casting first.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FAQSchema").getOrCreate()
data1 = [("Alice", 25)]
data2 = [("Bob", "30")]
df1 = spark.createDataFrame(data1, ["name", "age"])
df2 = spark.createDataFrame(data2, ["name", "years"])
# This would fail: df1.unionAll(df2)
df2_aligned = df2.withColumnRenamed("years", "age")
schema_df = df1.unionAll(df2_aligned)
schema_df.show()
# Output:
# +-----+---+
# | name|age|
# +-----+---+
# |Alice| 25|
# |  Bob| 30|
# +-----+---+
spark.stop()

Renaming aligns schemas for unionAll.

UnionAll vs Other DataFrame Operations

The unionAll operation stacks DataFrames vertically, unlike join (merges horizontally), groupBy (aggregates groups), or filter (row conditions). It differs from withColumn (adds/modifies columns) by combining datasets and leverages Spark’s optimizations over RDD operations.

More details at DataFrame Operations.

Conclusion

The unionAll operation in PySpark is a versatile way to combine DataFrame data vertically. Master it with PySpark Fundamentals to enhance your data integration skills!