Job Queueing for Notebook in Microsoft Fabric | Spark Compute

About this episode

Text from the YouTube description.

Microsoft Fabric supports job queueing for Spark compute when capacity limits are reached, ensuring jobs are retried until they either execute or expire. Admins allocate resources within Azure-based capacities, affecting how jobs are queued. The system supports queueing for pipeline-triggered Notebook jobs and Spark job definitions but not for interactive or public API-triggered Notebook jobs. Jobs are processed in a First-In-First-Out manner.

https://learn.microsoft.com/en-us/fabric/data-engineering/job-queueing-for-fabric-spark

🎙 Meet the Speakers:

👤 Guest from Product Group: Santhosh Kumar Ravindran, Senior Product Manager

Santhosh Ravindran currently leads Spark Compute and Settings for Microsoft Fabric Spark. He focuses on building capabilities that meet the data engineering needs like Spark Pools, Queueing and Scheduling, and Job orchestration for big data workloads in large enterprises using Spark. Prior to this, Santhosh was a Product Manager and Engineering Lead building Metadata scanners, access policy orchestration, lineage and data catalog systems for enterprise data estates as part of the Microsoft Governance Platform (Microsoft Purview).

Linkedin: https://www.linkedin.com/in/thisissanthoshkumar/

Twitter: https://www.twitter.com/iamsanthoshkr

👤 Host: Estera Kot, Principal Product Manager at Microsoft.

LinkedIn: https://www.linkedin.com/in/esterakot/
Twitter: https://twitter.com/estera_kot

👍 Liked this video? Don't forget to hit the 'Like' button and share it with your peers!