Prefect Brings Dataflow Automation to Databricks

Prefect Technologies Inc., the company behind the open-source data workflow automation platform Prefect, has partnered with Databricks, a data and AI company, to provide data engineers, data scientists and analytics engineers the ability to automate dataflows for the most data-intensive organizations and applications.

Organizations will have greater control and confidence in their workflows along with the ability to connect various resources across the modern data stack on the lake house. The Databricks Lakehouse Platform combines elements of data lakes and data warehouses, enabling users to unify their data, analytics and AI, build on open-source technology, and maintain a consistent platform across clouds. Combined with Prefect’s data workflow orchestration platform, data teams will have greater performance, reliability and visibility into their data.

“As the Databricks Lakehouse Platform becomes the hub of data-driven organizations, workflow orchestration ensures that accurate, reliable data is ready for critical use cases – with minimal human intervention,” said Roger Murff, VP of Technology Partners, at Databricks. “Prefect saves data engineers time, offering end-to-end visibility and increasing confidence in the accuracy and availability of data.”

For users migrating to Databricks, Prefect allows them to work with tools across the data stack and operate across multiple infrastructures simultaneously. Additionally, data engineers can access all of the tools they currently use while developing in Python.”

The Prefect and Databricks partnership is driven by the companies’ shared vision to provide engineers with flexibility and open-source support.

“As data engineers encounter a growing number of tools across the stack, we’re excited to add Databricks as a partner to help companies continue to coordinate their dataflows,” said Jeremiah Lowin, CEO of Prefect. “Databricks’ validation of Prefect’s technology also helps further our growth.”

More information on Prefect and Databricks can be found at https://www.prefect.io/partner/databricks/.