Dr. Estera Kot

Microsoft Fabric Spark Utilities - mssparkutils

Episode 36 of Fabric Espresso, hosted by Dr. Estera Kot.

Video thumbnail for Microsoft Fabric Spark Utilities - mssparkutils. Opens the video on YouTube.

Length 21:16. Published Nov 7, 2023 on the Azure Synapse Analytics YouTube channel. Topic: Spark and Data Engineering.

Watch on YouTube

About this episode

Text from the YouTube description.

Welcome back to another episode of Fabric Espresso DE&DS series. In this episode, we introduce and explore the nuances of Fabric Notebooks and a library named MSSparkUtils. Microsoft Spark Utilities (MSSparkUtils) is a built-in package to help you easily perform common tasks. You can use MSSparkUtils to work with file systems, to get environment variables, to chain notebooks together, and to work with secrets. MSSparkUtils are available in PySpark (Python), Scala, .NET Spark (C#), and R (Preview) notebooks and Synapse pipelines.

🎙 Meet the Speakers:

👤 Guest from Microsoft Fabric Product Group: Jene Zhang, Senior Program Manager of Fabric Notebooks

Jene is a Senior Program Manager at Microsoft with a passion for developing advanced developer tools and AI technology. Currently, she is spearheading the creation of a specialized Spark notebook aimed at enabling data engineers and scientists to excel in their domains.

LinkedIn: https://www.linkedin.com/in/edelweissno1/

Twitter: https://twitter.com/edelweissno1

👤 Host: Estera Kot, Senior Product Manager at Microsoft and a member of the Fabric Product Group. She holds the role of Product Owner for Apache Spark-based runtimes in Microsoft Fabric and Synapse Analytics. Estera is a Data & AI Architect and is passionate about computer science.
LinkedIn: https://www.linkedin.com/in/esterakot/
Twitter: https://twitter.com/estera_kot

👍 Like this video? Don't forget to hit the 'Like' button and share it with your network!

🔔 Stay Updated: For more insights into Microsoft Fabric Data Engineering and Data Science, and all things tech, make sure to subscribe to our channel and hit the notification bell so you never miss an episode!

#Microsoft #MicrosoftFabric #FabricNotebooks #DataScience #BigData #DataAnalytics #ApacheSpark #JupyterNotebook #SynapseNotebook #IPywidgets #TechInnovation #mssparkutils #MicrosoftSparkUtilities

Chapters

  1. 0:00 Introduction to MS Spark
  2. 0:55 What is MS Spark Utils
  3. 2:05 Key Utilities and Use Cases
  4. 4:21 File System API Demo
  5. 6:03 Mounting Storage Demo
  6. 8:36 Working with Mount Paths
  7. 9:36 Notebook Resources Management
  8. 10:54 Fast Copy Functionality
  9. 13:45 Reference Run API Demo
  10. 17:45 Unique Features and Summary

All Fabric Espresso episodes