Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Databricks Certified Associate Developer for Apache Spark 3.5 – Python Exam Practice Test

RDD actions like reduce() aggregate values across all partitions and return the result to the driver.

To compute the maximum processing time, reduce() is ideal because it combines results from all tasks efficiently.

Example:

max_time = rdd_times.reduce(lambda x, y: max(x, y))

This aggregates maximum values from all executors into a single result on the driver.

Why the other options are incorrect:

A: Broadcast variables distribute read-only data; they cannot aggregate results.

B: Spark UI provides visualization, not programmatic collection.

D: Accumulators support additive operations only (e.g., counters, sums), not non-associative ones like max.

In Apache Spark’s execution hierarchy, the relationships are structured as follows:

Job: Created when an action (e.g., count(), collect(), save()) is triggered on an RDD or DataFrame.

Stage: Each job is divided into one or more stages, separated by shuffle boundaries (e.g., after a reduceByKey or join).

Task: Each stage consists of multiple tasks, one per partition, executed in parallel on executors.

Execution Hierarchy:

Job → Stage(s) → Task(s)

So, a job contains multiple stages, and each stage contains multiple tasks.

Why the other options are incorrect:

A: A job does not directly contain tasks without stages.

B: A stage cannot contain multiple jobs; it belongs to a single job.

C: Tasks do not contain jobs.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

Spark Architecture Overview — Execution Hierarchy: Jobs, Stages, and Tasks.

Databricks Exam Guide (June 2025): Section “Apache Spark Architecture and Components” — describes execution hierarchy and lazy evaluation.

===========

To enable a Structured Streaming query to recover from failures or intentional shutdowns, it is essential to specify the checkpointLocation option during the writeStream operation. This checkpoint location stores the progress information of the streaming query, allowing it to resume from where it left off.

According to the Databricks documentation:

"You must specify the checkpointLocation option before you run a streaming query, as in the following example:

option("checkpointLocation", "/path/to/checkpoint/dir")

toTable("catalog.schema.table")

— Databricks Documentation: Structured Streaming checkpoints

By setting the checkpointLocation during writeStream, Spark can maintain state information and ensure exactly-once processing semantics, which are crucial for reliable streaming applications.

Structured Streaming supports three output modes:

Append: Writes only new rows since the last trigger.

Update: Writes only updated rows.

Complete: Writes the entire result table after every trigger execution.

For aggregations like groupBy().count(), only complete mode outputs the entire table each time.

Example:

aggDF.writeStream \

outputMode("complete") \

format("console") \

start()

Why the other options are incorrect:

A: “AGGREGATE” is not a valid output mode.

C: “REPLACE” does not exist.

D: “APPEND” writes only new rows, not the full table.

The correct way to overwrite an existing file using the DataFrameWriter is:

df.write.mode("overwrite").json("path/to/file")

Option D is also technically valid, but Option A is the most concise and idiomatic PySpark syntax.

According to Spark’s watermarking rules:

“Records that are older than the watermark (event time < current watermark) are considered too late and are dropped.”

So, if a record’s event_time is earlier than (max event_time seen so far - 10 minutes), it is discarded.

In Apache Spark's cluster mode:

"The driver program runs on the cluster’s worker node instead of the client’s local machine. This allows the driver to be close to the data and other executors, reducing network overhead and improving fault tolerance for production jobs."

(Source: Apache Spark documentation – Cluster Mode Overview)

This deployment is ideal for production environments where the job is submitted from a gateway node, and Spark manages the driver lifecycle on the cluster itself.

Option A is partially true but less specific than D.

Option B is incorrect: the driver never executes all tasks; executors handle distributed tasks.

Option C describes client mode, not cluster mode.

To sort a PySpark DataFrame by multiple columns with mixed sort directions, the correct usage is:

python

CopyEdit

df.orderBy("age", "salary", ascending=[True, False])

age will be sorted in ascending order

salary will be sorted in descending order

The orderBy() and sort() methods in PySpark accept a list of booleans to specify the sort direction for each column.

Documentation Reference: PySpark API - DataFrame.orderBy

Partitioning a dataset divides data into separate directories based on partition column values. When queries filter on partitioned columns, Spark can prune irrelevant partitions — meaning it only reads files that match the filter criteria.

Advantage:

Reduces I/O and improves performance by scanning only relevant subsets of data.

Example:

/data/sales/year=2023/month=10/...

/data/sales/year=2024/month=01/...

A query filtering WHERE year = 2024 reads only the relevant partition.

Why the other options are incorrect:

A: Compression is independent of partitioning.

B: Spark does not automatically clean partitions unless managed manually.

C: Partitioning does not cause Spark to load entire data into memory.

The Spark UI is the standard and most effective way to inspect executor logs, task time, input size, and shuffles.

From the Databricks documentation:

“You can monitor job execution via the Spark Web UI. It includes detailed logs and metrics, including task-level execution time, shuffle reads/writes, and executor memory usage.”

(Source: Databricks Spark Monitoring Guide)

Option A is incorrect: logs are not guaranteed to be in /tmp, especially in cloud environments.

B. —verbose helps during job submission but doesn't give detailed executor logs.

D. spark-sql is a CLI tool for running queries, not for inspecting logs.

Hence, the correct method is using the Spark UI → Stages tab → Executor logs.

na.drop(how='any') drops any row that has at least one null value.

This is exactly what's needed when the goal is to retain only fully complete records.

Usage:CopyEdit

filtered_df = users_raw_df.na.drop(how='any')

Explanation of incorrect options:

A: thresh=0 is invalid — thresh must be ≥ 1.

B: how='all' drops only rows where all columns are null (too lenient).

D: spark.na.drop doesn't support mixing how and thresh in that way; it's incorrect syntax.

Apache Spark is a distributed data processing engine designed for large-scale, cluster-based computation.

Advantages:

Horizontal Scalability: Spark can distribute tasks across many machines, handling datasets larger than the memory of a single node.

Fault Tolerance: Spark automatically recovers from node or task failures using the lineage graph (RDD recovery mechanism) and retry logic.

These two features allow Spark to process huge datasets efficiently and reliably, unlike standard Python scripts that are limited to one machine and fail on single-node errors.

Why the other options are incorrect:

B: Spark runs on commodity hardware; no specialized machines required.

C: Spark emphasizes in-memory processing, not disk-only operations.

D: Spark still requires user code in Python, Scala, SQL, or Java.

Option B is the correct and idiomatic approach in PySpark to split a string column (like email) based on a delimiter such as "@".

The split(col("email"), "@") function returns an array with two elements: username and domain.

getItem(0) retrieves the first part (username).

getItem(1) retrieves the second part (domain).

withColumn() is used to create new columns from the extracted values.

Example from official Databricks Spark documentation on splitting columns:

from pyspark.sql.functions import split, col

df.withColumn("username", split(col("email"), "@").getItem(0)) \

withColumn("domain", split(col("email"), "@").getItem(1))

⚠️ Why other options are incorrect:

A uses fixed substring indices (substr(0, 5)), which won’t correctly extract usernames and domains of varying lengths.

C uses substring_index, which is available but less idiomatic for splitting emails and is slightly less readable.

D removes "@" from the email entirely, losing the separation between username and domain, and ends up duplicating values in both fields.

Therefore, Option B is the most accurate and reliable solution according to Apache Spark 3.5 best practices.

The approx_percentile function in Spark is a performance-optimized alternative to percentile. It takes an optional accuracy parameter:

approx_percentile(column, percentage, accuracy)

Higher accuracy values → more precise results, but increased memory/computation.

Lower values → faster but less accurate.

From the documentation:

“Increasing the accuracy improves precision but increases memory usage.”

Final Answer: D

In Spark’s client mode, the driver runs on the local machine that submitted the job. If that machine is resource-constrained (e.g., low memory), performance degrades.

From the Spark documentation:

"In cluster mode, the driver runs inside the cluster, benefiting from cluster resources and scalability."

Option A is incorrect — executors do not help the driver directly.

Option B might help short-term but does not scale.

Option C is correct — switching to cluster mode moves the driver to the cluster.

Option D (local mode) is for development/testing, not production.

Final Answer: C

When queries frequently filter on certain columns, partitioning by those columns ensures partition pruning, allowing Spark to scan only relevant directories instead of the entire dataset.

Correct code:

final.write.partitionBy("event_year", "event_month").parquet("events.liveLatest")

This improves read performance dramatically for filters like:

SELECT * FROM events.liveLatest WHERE event_year = 2024 AND event_month = 5;

bucketBy() helps in clustering and joins, not in partition pruning for file-based tables.

Why the other options are incorrect:

B: Bucket count changes parallelism, not query pruning.

C: sortBy organizes data within files, not across partitions.

D: Partitioning by only one column limits pruning benefits.

The number of tasks is controlled by the number of partitions. By default, spark.sql.shuffle.partitions is 200. If stages are showing very few tasks (less than total cores), you may not be leveraging full parallelism.

From the Spark tuning guide:

"To improve performance, especially for large clusters, increase spark.sql.shuffle.partitions to create more tasks and parallelism."

Thus:

A is correct: increasing shuffle partitions increases parallelism

B is wrong: it further reduces parallelism

C is invalid: increasing dataset size doesn’t guarantee more partitions

D is irrelevant to task count per stage

Final Answer: A

Apache Spark employs a lazy evaluation model for transformations. This means that when transformations (e.g., map(), filter()) are applied to a DataFrame, Spark does not execute them immediately. Instead, it builds a logical plan (lineage) of transformations to be applied.

Execution is deferred until an action (e.g., collect(), count(), save()) is called. At that point, Spark's Catalyst optimizer analyzes the logical plan, optimizes it, and then executes the physical plan to produce the result.

This lazy evaluation strategy allows Spark to optimize the execution plan, minimize data shuffling, and improve overall performance by reducing unnecessary computations.

In Pandas API on Spark (previously Koalas), the method .to_spark() converts a pyspark.pandas.DataFrame into a PySpark DataFrame.

Correct usage:

spark_df = pdf.to_spark()

This enables interoperability between the Pandas API on Spark and the PySpark SQL API, allowing developers to switch seamlessly between both for transformations or performance optimization.

Why the other options are incorrect:

A (to_pandas): Converts to a local Pandas DataFrame, not a PySpark DataFrame.

C (to_dataframe): Not a valid API method.

D (spark): Not an existing DataFrame method.

Temp views like createOrReplaceTempView are session-scoped.

They disappear once the Spark session ends.

To retain data across sessions, it must be persisted:

df.write.saveAsTable("users_vw")

Thus, the view needs to be persisted as a table to survive session termination.

Spark's best practice is to estimate partition count based on data volume and a reasonable partition size — typically 128 MB to 256 MB per partition.

With 1 TB of data: 1 TB / 128 MB ≈ ~8000 partitions

This ensures that tasks are distributed across available CPUs for parallelism and that each task processes an optimal volume of data.

Option A (equal to cores) may result in partitions that are too large.

Option B (fixed 200) is arbitrary and may underutilize the cluster.

Option C (nodes) gives too few partitions (10), limiting parallelism.

The message "GC overhead limit exceeded" typically indicates that the JVM is spending too much time in garbage collection with little memory recovery. This suggests that the driver or executor is under-provisioned in memory.

The most effective remedy is to increase the driver memory using:

--driver-memory 4g

This is confirmed in Spark's official troubleshooting documentation:

“If you see a lot of GC overhead limit exceeded errors in the driver logs, it’s a sign that the driver is running out of memory.”

— Spark Tuning Guide

Why others are incorrect:

A may help but does not directly address the driver memory shortage.

B is not a valid action; GC cannot be disabled.

D increases memory usage, worsening the problem.

Pandas UDFs (vectorized UDFs) process entire Pandas Series objects, not scalar values. Each invocation operates on a column (Series) rather than a single value.

Correct syntax:

@pandas_udf(StringType())

def shake_256(raw: pd.Series) -> pd.Series:

return raw.apply(lambda x: hashlib.shake_256(x.encode()).hexdigest(20))

This allows Spark to apply the function in a vectorized way, improving performance significantly over traditional Python UDFs.

Why the other options are incorrect:

A/D: These define scalar functions — not compatible with Pandas UDFs.

B: Uses an invalid type hint [pd.Series] (not a valid Python type annotation).

Spark automatically uses a broadcast hash join when one side of the join is small enough to fit within the broadcast threshold.

Default threshold:

spark.sql.autoBroadcastJoinThreshold = 10MB (as of Spark 3.5)

Since df2 is 8 MB, Spark automatically broadcasts it to all executors. This avoids a shuffle on the large dataset (df1) and speeds up the join.

Why the other options are incorrect:

A: 8 MB < 10 MB threshold → broadcast join is efficient.

B: AQE is not required for broadcast joins; it’s a static optimization.

C: Broadcast hints are optional — Spark infers automatically.

When writing a DataFrame using .partitionBy() in Spark, the data is physically organized into directory structures corresponding to unique combinations of the partition columns.

Example:

/path/to/output/color=Red/taste=Sweet/part-0001.parquet

/path/to/output/color=Green/taste=Sour/part-0002.parquet

This structure improves query performance by pruning partitions when filtering on these columns.

Why the other options are incorrect:

A: Appending requires .mode("append"), which isn’t used here.

B: Null values in partition columns are handled; they don’t raise errors.

D: Partitioning prevents storing all data in a single file.

The .mode("overwrite") ensures that existing files at the path will be replaced.

partitionBy("country") optimizes queries by writing data into partitioned folders.

Correct syntax:

df.write.mode("overwrite").partitionBy("country").parquet("/data/output")

— Source: Spark SQL, DataFrames and Datasets Guide

Underutilization with timeout warnings often indicates insufficient parallelism — meaning there aren’t enough executors to process all tasks concurrently.

Solution:

Increase the number of executors to allow more parallel task execution and better resource utilization.

Example configuration:

--conf spark.executor.instances=8

This distributes the workload more effectively across cluster nodes and reduces idle time for pending tasks.

Why the other options are incorrect:

A: Extending timeouts hides the symptom, not the root cause (lack of executors).

B: More memory per executor won’t fix scheduling bottlenecks.

C: Reducing partition size may increase overhead and does not fix resource imbalance.

The groupBy() operation causes a shuffle because it requires all values for a specific key to be brought together, which may involve moving data across partitions.

In contrast:

filter() and select() are narrow transformations and do not cause shuffles.

coalesce() tries to reduce the number of partitions and avoids shuffling by moving data to fewer partitions without a full shuffle (unlike repartition()).

When using the MEMORY_AND_DISK storage level, Spark attempts to cache as much of the DataFrame in memory as possible. If the DataFrame does not fit entirely in memory, Spark will store the remaining partitions on disk. This allows processing to continue, albeit with a performance overhead due to disk I/O.

As per the Spark documentation:

"MEMORY_AND_DISK: It stores partitions that do not fit in memory on disk and keeps the rest in memory. This can be useful when working with datasets that are larger than the available memory."

— Perficient Blogs: Spark - StorageLevel

This behavior ensures that Spark can handle datasets larger than the available memory by spilling excess data to disk, thus preventing job failures due to memory constraints.

Watermarking in Structured Streaming defines how late a record can arrive based on event time before Spark discards it.

Behavior:

withWatermark("event_time", "10 minutes")

This means Spark will keep state for 10 minutes beyond the maximum event time seen so far.

Any data arriving later than 10 minutes after the current watermark is ignored — it will not be included in the aggregation or output.

Why the other options are incorrect:

B: Late data beyond the watermark threshold is not included.

C: Late data is not moved to a new window; it’s simply dropped.

D: True for late data within the watermark threshold, not after it.

To process data in fixed micro-batch intervals, use the .trigger(processingTime="interval") option in Structured Streaming.

Correct usage:

query = df.writeStream \

outputMode("append") \

trigger(processingTime="5 seconds") \

start()

This instructs Spark to process available data every 5 seconds.

Why the other options are incorrect:

B: continuous triggers are for continuous processing mode (different execution model).

C: once=True runs the stream a single time (batch mode).

D: Default trigger runs as fast as possible, not fixed intervals.

Adaptive Query Execution (AQE) in Spark 3.x automatically optimizes query plans at runtime based on the actual data characteristics observed during job execution.

Key features of AQE include:

Dynamic switching of join strategies: Changes between sort-merge join and broadcast join based on actual shuffle sizes.

Coalescing shuffle partitions: Reduces small tasks and improves parallelism efficiency.

Handling skew joins: Dynamically splits large partitions to avoid data skew.

Thus, the most accurate answer describing AQE’s function is “dynamically switching join strategies.”

Why the other options are incorrect:

B: Table statistics are collected manually or by the metastore, not by AQE.

C: AQE benefits multi-stage jobs involving shuffles, not single-stage jobs.

D: Delta file optimization is handled by Databricks utilities, not AQE.

By default, the spark.read.text() method reads a text file one line per record. This means that each line in a text file becomes one row in the resulting DataFrame.

To read each file as a single record, Apache Spark provides the option wholetext, which, when set to True, causes Spark to treat the entire file contents as one single string per row.

Correct usage:

df = spark.read.option("wholetext", True).text(txt_path)

This way, each record in the DataFrame will contain the full content of one file instead of one line per record.

To also include the file path, the function input_file_name() can be used to create an additional column that stores the complete path of the file being read:

from pyspark.sql.functions import input_file_name

df = spark.read.option("wholetext", True).text(txt_path) \

withColumn("file_path", input_file_name())

This approach satisfies both requirements from the question:

Each record holds the entire contents of a file.

Each record also contains the file path from which the text was read.

Why the other options are incorrect:

B or D (lineSep) – The lineSep option only defines the delimiter between lines. It does not combine the entire file content into a single record.

C (wholetext=False) – This is the default behavior, which still reads one record per line rather than per file.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

PySpark API Reference: DataFrameReader.text — describes the wholetext option.

PySpark Functions: input_file_name() — adds a column with the source file path.

Databricks Certified Associate Developer for Apache Spark Exam Guide (June 2025): Section “Using Spark DataFrame APIs” — covers reading files and handling DataFrames.

To handle JSON files with evolving or differing schemas, Spark SQL supports the option mergeSchema 'true', which merges all fields across files into a unified schema.

Correct syntax:

CREATE EXTERNAL TABLE users

USING json

OPTIONS (path '/data/input.json', mergeSchema 'true');

This creates an external table directly on the JSON data, inferring schema automatically and merging variations.

Why the other options are incorrect:

B: Missing schema merge configuration — fails with inconsistent files.

C: inferSchema applies to CSV/other file types, not JSON.

D: mergeAll is not a valid Spark SQL option.

Pandas API on Spark provides a distributed implementation of the Pandas DataFrame API on top of Apache Spark.

Advantages:

Executes transformations in parallel across all nodes and cores in the cluster.

Maintains Pandas-like syntax, making it easy for Python users to transition.

Enables scaling of existing Pandas code to handle large datasets without memory limits.

Therefore, it combines Pandas usability with Spark’s distributed power, offering both speed and scalability.

Why the other options are incorrect:

B: While it uses Python, that’s not its main advantage.

C: It runs distributed across the cluster, not on a single node.

D: Pandas API on Spark uses lazy evaluation, not eager computation.

In Spark SQL and DataFrame operations, the configuration parameter spark.sql.shuffle.partitions defines the number of partitions created during shuffle operations such as join, groupBy, and distinct.

The default value (in Spark 3.5) is 200.

If this number is too low, Spark creates fewer tasks, leading to idle executors and poor cluster utilization.

Increasing this value allows Spark to create more tasks that can run in parallel across executors, effectively using more cluster resources.

Correct approach:

spark.conf.set("spark.sql.shuffle.partitions", 400)

This increases the parallelism level of shuffle stages and improves overall resource utilization.

Why the other options are incorrect:

B: Reducing partitions further would decrease parallelism and worsen the underutilization issue.

C: Dynamic resource allocation scales executors up or down based on workload, but it doesn’t fix low task parallelism caused by insufficient shuffle partitions.

D: Increasing dataset size is not a tuning solution and doesn’t address task-level under-parallelization.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

Spark SQL Configuration: spark.sql.shuffle.partitions — controls the number of shuffle partitions.

Databricks Exam Guide (June 2025): Section “Troubleshooting and Tuning Apache Spark DataFrame API Applications” — tuning strategies, partitioning, and optimizing cluster utilization.

===========

To compare a column value with a Python literal constant in a DataFrame expression, use F.lit() to convert it into a Spark literal.

Correct refactor:

from pyspark.sql import functions as F

min_price = 110.50

result_df = prices_df.filter(F.col("price") > F.lit(min_price)).agg(F.count("*"))

This avoids type mismatches and ensures Spark executes the filter expression on the cluster.

Why the other options are incorrect:

B: where() syntax is valid, but F.lit("price") is incorrect — wraps string literal, not a column.

C: withColumn adds a column, not needed for this aggregation.

D: Comparison logic reversed.

To write a structured streaming DataFrame to Parquet files, the correct way to specify the format and output directory is:

writeStream

format("parquet")

option("path", "path/to/destination/dir")

According to Spark documentation:

“When writing to file-based sinks (like Parquet), you must specify the path using the .option("path", ...) method. Unlike batch writes, .save() is not supported.”

Option A incorrectly uses .option("location", ...) (invalid for Parquet sink).

Option B incorrectly sets the format via .option("format", ...), which is not the correct method.

Option C repeats the same issue.

Option D is correct: .format("parquet") + .option("path", ...) is the required syntax.

Final Answer: D

Adaptive Query Execution (AQE) dynamically selects join strategies based on actual data sizes at runtime. If the size of post-shuffle partitions is below the threshold set by:

spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold

then Spark prefers to use a shuffled hash join.

From the Spark documentation:

“AQE selects a shuffled hash join when the size of post-shuffle data is small enough to fit within the configured threshold, avoiding more expensive sort-merge joins.”

Therefore:

A is wrong — Cartesian joins are only used with no join condition.

B is correct — this is the optimized join for small partitioned shuffle data under AQE.

C and D are used under other scenarios but not for this case.

Final Answer: B