Field Notes
Technical notes by Dr. Estera Kot
Notes from both sides of the data platform: the Apache Spark runtime Dr. Kot owned at Microsoft, and the client platforms she designs as CTO at Clouds on Mars. Topics are Apache Spark, Microsoft Fabric, lakehouse architecture, and platform migration.
On this site
- Three doors out of Fabric: FabCon Europe 2026 in 18 demos
What FabCon Europe 2026 changed in Microsoft Fabric, read through 18 recorded demos: which door your data now leaves by, what is GA, and what to check first.
- Fabric does not live in your VNet: workspace identity and the enterprise Landing Zone
Fabric meets an Azure Landing Zone at four crossings: identity, secrets, outbound, and inbound. Where workspace identity works, where the gateway stops it, and the Conditional Access trap.
- Microsoft Fabric implementation: a practitioner's guide
Nine implementation decisions in order, from F SKU capacity and OneLake design to Synapse migration and the Native Execution Engine. Common mistakes and an FAQ close it.
- The Microsoft Fabric Maturity Model
Five levels from Foundation to Excellence, each with objectives, success criteria, and a self-assessment.
FabCon Europe 2026 in 18 demos
- “Fabric data in Microsoft Copilot” is on for everyone
A Microsoft 365 admin center setting decides who can ask Copilot about Power BI data, and it ships set to All users. Four gates, three owners, and the Monday check.
- One AI toggle on every semantic model
Each semantic model has a switch that opens it to Copilot, data agents, and Microsoft MCP tools for read-only users. It starts on, and models built on it do not inherit it.
- Fabric IQ MCP server: query a semantic model from your coding agent
Fabric IQ MCP is GA: six read-only tools let a coding agent find a semantic model and run DAX as you. Your agent writes the DAX, and users can grant consent themselves by default.
- Fabric Core MCP server: 43 tools, 25 of them write
Fabric Core MCP is GA, and Learn's tools reference lists 43 tools, 25 of them write, including delete_workspace and OneLake role replacement. Where the safeguards sit, and Learn's warning quoted.
- OneLake Table Read API: REST reads that respect row security
The OneLake Table Read API returns Fabric table rows as Apache Arrow over REST, with OneLake RLS and CLS applied on the server. Preview. The request path, six caller outcomes, and the cost catch.
- Unity Catalog and Iceberg REST on OneLake: one table, no copy
OneLake serves the same Delta tables through the Iceberg REST Catalog and a Unity Catalog compatible API, with no copy. Why OneLake security blocks instead of filtering there, and which door to use.
- OneLake catalog search API: find any item in the tenant from code
One POST to /v1/catalog/search finds Fabric items across every workspace you can access, and Fabric Core MCP runs the same search. Preview. Two permission gates, and what search exposes.
- Project Osmos: a data engineering agent that plans, runs, and checks
The Fabric data engineering agent turns one outcome into notebooks and a validated Delta table on SparkCore, as you. The eight-setting card, the permission wall, and the catches.
- Operations agent: root cause for a failed pipeline in Monitor hub
Select Investigate on a failed pipeline run and the operations agent explains the root cause, read-only and as you. It needs workspace monitoring on before the failure.
- Ontology in Fabric IQ: business rules an agent can read, not run
The ontology agent drafts entities, relationships, and metrics from a semantic model and writes only in Act mode. Rules are context, not enforcement. Old ontology retires Jan 31, 2027.
- Copilot for Real-Time Intelligence: a dashboard from one sentence
Copilot turns one sentence into a twelve-tile Real-Time Dashboard on an eventhouse. Preview. What it generates, and the four things to review before you share it.
- Fabric data agent: plain-language questions over a lakehouse
The Fabric data agent turns English questions into read-only SQL that runs as you. Grounding, the English-only limit, and how it reaches Microsoft 365 Copilot and Copilot Studio.
- OneLake security: define RLS and CLS once, enforce them everywhere
OneLake security roles define RLS and CLS once, and every Fabric engine enforces them, shortcuts included. Who they restrict, and why DefaultReader must change first.
- Outbound access protection: stop data leaving through shortcuts and agents
Workspace outbound access protection blocks every outbound connection from a Fabric workspace. What it covers (GA for shortcuts, preview for data agents), what it misses, and what it breaks.
- Policies in Fabric: an allow list with no Deny
Fabric policies, in preview, allow an action when a rule matches and block everything else, at tenant or capacity scope. How rules evaluate, and why the first rule changes the default.
- OneLake catalog Govern: one screen for labels, DLP, and settings
The OneLake catalog Govern tab shows label coverage, DLP results, and recommended actions, refreshed once a day. What it covers, what it misses, and how to turn its cards into a backlog.
- AI functions in Spark: translate, classify, and score sentiment in one line
ai.translate, ai.analyze_sentiment, and ai.classify on a Fabric DataFrame column through a built-in endpoint with no API key: runtimes, cost per token, regions, and when to bring your own model.
- Materialized lake views: a medallion in SQL with drop-on-mismatch rules
Silver and gold as CREATE MATERIALIZED LAKE VIEW statements with CHECK rules: DROP versus FAIL, lineage, incremental refresh, and a demo where three bad rows never reach gold.
One post per demo recording, with the status of each feature checked on Microsoft Learn. Start with the series overview.
From Events to Answers
- From Events to Answers: Real-Time Intelligence and Fabric IQ from zero
Real-Time Intelligence and Fabric IQ from zero, on one invented oud house. The demo writes the winner rule twice, in Activator and in the ontology, and Fabric still has no link between them.
- Real-Time Intelligence in plain words: the path of one event
Five parts, one job each. Follow one tester's vote through Eventstream, Eventhouse, a dashboard, and an Activator rule, and see why the alert waits for 20 testers before 60 counts.
- What an ontology is, and why a data agent needs one
An ontology in Microsoft Fabric IQ is the dictionary of the business. One worked example: a rule that a winner has first choice above 60 changes the data agent's answer, with no change to the data.
- The agent writes the sentence, the table holds the number
A Fabric data agent should write the sentence and the query, never the number. One demo answer taken apart, the guards that keep figures in tables, four build failures, and capacity facts.
- From demo to production: six gaps and three ways to build
A Fabric IQ demo proves the mechanics, not quality at scale. Six gaps to close, three ways to build, preview risk, and a production checklist for data agents.
- Fabric for 50 people in one evening
A hands-on Fabric session fails at sign-in, not at the demo. Accounts, MFA, a CU budget, and teardown for 50 people, from one evening at Microsoft Qatar.
- Teaching Microsoft Fabric from zero in two hours
Beginners leave at the first word nobody explained. How a two-hour Fabric session was built: five level rules, a minute clock, a glossary in pictures, a paper walkthrough, and a recovery line per demo step.
Seven parts from a two-hour community session on Real-Time Intelligence and Fabric IQ, from zero. Start with Part 1.
On the Microsoft Fabric Updates Blog
- Fabric Runtime 1.3 is Generally Available!
Fabric Runtime 1.3, with Apache Spark 3.5 and Delta Lake 3.2, moves from public preview to general availability.
- Public Preview of Native Execution Engine for Apache Spark on Fabric Data Engineering and Data Science
The Native Execution Engine enters public preview on Fabric Runtime 1.2: columnar data and vectorized processing for Spark queries.
More posts from her Microsoft years are on her author page.