Microsoft Fabric Optimistic Job Admission for Apache Spark - maximum compute utilization by default
Episode 63 of Fabric Espresso, hosted by Dr. Estera Kot.
About this episode
Text from the YouTube description.
Welcome to our latest video where we dive into the powerful capabilities of Microsoft Fabric's Data Engineering and Data Science experiences! 🚀
Discover how this fully managed Apache Spark compute platform is revolutionizing data analytics with unparalleled speed and efficiency. With starter pools, you can initialize Apache Spark sessions in just 5 to 10 seconds—no manual setup required! Plus, enjoy the flexibility to customize Spark pools to meet your unique data engineering and data science needs.
We'll explore how ApacheSpark for Fabric uses the innovative optimistic job admission technique to ensure your Spark jobs run smoothly. Whether you're working with interactive or batch jobs from notebooks, lakehouses, or Spark job definitions, this platform has got you covered.
Learn how jobs start with minimum node settings and can scale up dynamically, optimizing performance based on job stages. We'll also discuss how the job admission and throttling layer on Fabric Spark allows for seamless scaling within the maximum burst cores assigned, ensuring your analytics experience is always top-notch.
Wanna try a lab, go to https://aka.ms/fabconlake
Read more here:
➡️ https://blog.fabric.microsoft.com/en-us/blog/introducing-optimistic-job-admission-for-fabric-spark
➡️ https://learn.microsoft.com/en-us/fabric/data-engineering/job-admission-management
🎙 Meet the Speakers:
👤 Guest Expert: Santhosh Kumar Ravindran, Senior Product Manager
Santhosh Ravindran currently leads Spark Compute and Settings for Microsoft Fabric Spark. He focuses on building capabilities that meet the data engineering needs like Spark Pools, Queueing and Scheduling, and Job orchestration for big data workloads in large enterprises using Spark. Prior to this, Santhosh was a Product Manager and Engineering Lead building Metadata scanners, access policy orchestration, lineage and data catalog systems for enterprise data estates as part of the Microsoft Governance Platform (Microsoft Purview).
Linkedin: https://www.linkedin.com/in/thisissanthoshkumar/
Twitter: https://www.twitter.com/iamsanthoshkr
👤 Host: Estera Kot, PhD, Principal Product Manager at Microsoft.
LinkedIn: https://www.linkedin.com/in/esterakot/
Twitter: https://twitter.com/estera_kot
👍 Liked this video? Don't forget to hit the 'Like' button and share it with your peers!