Distributed Deep Learning Made Easy with Spark 3.4

Summary

Distributed deep learning has become a crucial aspect of modern data processing pipelines. With the release of Apache Spark 3.4, users now have access to built-in APIs for both distributed model training and model inference at scale. This article explores how Spark 3.4 simplifies distributed deep learning, making it easier to integrate deep learning models into large-scale data processing workflows.

Simplifying Distributed Deep Learning with Spark 3.4

Distributed deep learning is a critical component of many modern data processing pipelines. From sales predictions to content recommendations, sentiment analysis, and fraud detection, deep learning models are increasingly being used to analyze large datasets. However, combining deep learning training and inference with large-scale data has historically been a challenge for Spark users.

The Challenge of Distributed Deep Learning

Most deep learning frameworks were designed for single-node environments, and their distributed training and inference APIs were often added as an afterthought. This disconnect between single-node deep learning environments and large-scale distributed environments has led to the development of third-party solutions such as Horovod-on-Spark, TensorFlowOnSpark, and SparkTorch. However, these solutions were not natively built into Spark, requiring users to evaluate each platform against their own needs.

Spark 3.4: A Game-Changer for Distributed Deep Learning

With the release of Spark 3.4, users now have access to built-in APIs for both distributed model training and model inference at scale. This simplifies the process of integrating deep learning models into large-scale data processing workflows.

Distributed Training

For distributed training, Spark 3.4 introduces a new TorchDistributor API for PyTorch, which follows the spark-tensorflow-distributor API for TensorFlow. These APIs simplify the migration of distributed deep learning model training code to Spark by taking advantage of Spark’s barrier execution mode to spawn the distributed deep learning cluster nodes on top of the Spark executors.

Once the deep learning cluster has been started by Spark, control is essentially handed off to the deep learning frameworks through the main_fn that was passed to the TorchDistributor API. This requires only minimal code changes to run standard distributed deep learning training on Spark with this new API.

Distributed Inference

For distributed inference, Spark 3.4 introduces a new predict_batch_udf API, which builds on the Spark Pandas UDF to provide a simpler interface for deep learning model inference. Pandas UDFs provide several advantages over row-based UDFs, including faster serialization of data through Apache Arrow and faster vectorized operations through Pandas.

However, while the Pandas UDF API may be a great solution for ETL use cases, it is still not ideal for deep learning inference use cases. The predict_batch_udf API addresses this by providing a data-parallel architecture, where each executor loads the model and predicts on their portions of the dataset, so the model must fit in the executor memory.

End-to-End Example for Spark Deep Learning

To try these new APIs, users can check out the Spark DL Training and Inference Notebook for an end-to-end example. Based on the Distributed Training E2E on Databricks Notebook from Databricks, the example notebook demonstrates:

• How to train a MNIST model from single-node to distributed, using the new TorchDistributor API.
• How to use the new predict_batch_udf API for distributed inference.
• How to load training data from a distributed file store, like S3, using NVTabular.

Benefits of Spark 3.4 for Distributed Deep Learning

The release of Spark 3.4 brings several benefits to distributed deep learning:

• Simplified Integration: With built-in APIs for distributed model training and inference, users can easily integrate deep learning models into large-scale data processing workflows.
• Improved Performance: The predict_batch_udf API provides faster serialization of data through Apache Arrow and faster vectorized operations through Pandas.
• Scalability: Spark 3.4 allows users to scale their deep learning workloads to meet the demands of large datasets.

Table: Comparison of Distributed Deep Learning Solutions

Table: Benefits of Spark 3.4 for Distributed Deep Learning

Conclusion

Distributed deep learning is a critical component of modern data processing pipelines. With the release of Spark 3.4, users now have access to built-in APIs for both distributed model training and model inference at scale. This simplifies the process of integrating deep learning models into large-scale data processing workflows, providing improved performance and scalability. By leveraging these new APIs, users can unlock the full potential of distributed deep learning and drive innovation in their organizations.