Spark

In a previous post I showed how to trigger a semantic model refresh from a Fabric notebook using sempy.fabric. That approach works great for small models, but in production you’ll quickly run into problems when the model has many tables or large fact tables. The problem: out of memory The sempy default for max_parallelism is 10. For a model with 27 tables including several large fact tables (tens of millions of rows), processing 10 tables concurrently can easily exhaust the available memory on the capacity and the refresh fails with a generic “out of memory” error — or worse, it succeeds but takes an unreasonably long time because of internal retries. ...

The sempy Python library in Fabric is quite powerful and offers a range of capabilities that make working with Fabric environments more efficient and flexible. One of the most useful features I’ve implemented in my projects is the ability to trigger semantic model refreshes directly from a Spark notebook. This approach allows to trigger the refresh right after the data load is finished. import sempy.fabric as fabric import time workspace = fabric.resolve_workspace_name() dataset = "My_Amazing_Semantic_Model" refresh_request_id = fabric.refresh_dataset(workspace=workspace, dataset=dataset, refresh_type="full") # refresh api is async, so we need to poll until it completes while True: time.sleep(60) res = fabric.get_refresh_execution_details( dataset=dataset, workspace=workspace, refresh_request_id=refresh_request_id ) if res.status != "Unknown": break if res.status == "Completed": print(f"Refresh Dataset completed with success {res.extended_status}") else: errors = res.messages["Message"].str.cat(sep="\n") raise Exception(f"Refresh Dataset failed: \n\n{errors}") This script starts by resolving the current workspace and identifying the target dataset. It then initiates a full refresh of that dataset and waits for the process to complete, checking periodically for updates. ...