Skip to content

Using the interactive Spark Cluster Client

  • This example demonstrates how to connect to the Practicus AI Spark cluster we created, and execute simple Spark operations.
  • Please run this example on the Spark Coordinator (master).
import practicuscore as prt 

# Let's get a Spark session
spark = prt.distributed.get_client()

Behind the scenes

  • After the above code, new Spark executors will start running.
  • This is specific to auto-scaled Spark only and not the dfault behavior.
# And execute some code
data = [("Alice", 29), ("Bob", 34), ("Cathy", 23)]
columns = ["Name", "Age"]

# Create a DataFrame
df = spark.createDataFrame(data, columns)

# Perform a transformation
df_filtered = df.filter(df.Age > 30)

# Show results
df_filtered.show()

# Explicitly delete spark session
prt.engines.delete_spark_session()
# Unlike the standard Spark cluster, the below won't work for auto-scaled.
# spark.stop()

Terminating the cluster

  • You can go back to the other worker where you created the cluster to run:
coordinator_worker.terminate()
  • Or, terminate "self" and children workers with the below:
prt.get_local_worker().terminate()

Previous: Start Cluster | Next: Batch > Batch Job