A #dataOp ramblings

Fabric Semantic Model Refresh: One Table at a Time

Wed, 07 Oct 2026 00:00:00 +0000

In a previous post I showed how to trigger a semantic model refresh from a Fabric notebook using sempy.fabric. That approach works great for small models, but in production you’ll quickly run into problems when the model has many tables or large fact tables.

The problem: out of memory

The sempy default for max_parallelism is 10. For a model with 27 tables including several large fact tables (tens of millions of rows), processing 10 tables concurrently can easily exhaust the available memory on the capacity and the refresh fails with a generic “out of memory” error — or worse, it succeeds but takes an unreasonably long time because of internal retries.

Microsoft Fabric External Data Sharing Hidden Limitation

Thu, 09 Apr 2026 19:36:00 +0100

Microsoft Fabric external data sharing is a powerful feature that allows us to share data with external users. This is particularly interesting because it enables data to be shared in-place from the sharer’s tenant without any data copy. We simply create shortcuts.

The use case for the customer in question involves data from Microsoft Dynamics 365 (D365) residing on two different Azure tenants. I needed a solution to share data from one tenant to the other.

Refreshing Power BI Datasets with Notebooks

Wed, 12 Nov 2025 00:00:00 +0000

The sempy Python library in Fabric is quite powerful and offers a range of capabilities that make working with Fabric environments more efficient and flexible. One of the most useful features I’ve implemented in my projects is the ability to trigger semantic model refreshes directly from a Spark notebook.

This approach allows to trigger the refresh right after the data load is finished.

import sempy.fabric as fabric
import time

workspace = fabric.resolve_workspace_name()
dataset = "My_Amazing_Semantic_Model"

refresh_request_id = fabric.refresh_dataset(workspace=workspace, dataset=dataset, refresh_type="full")

# refresh api is async, so we need to poll until it completes
while True:
 time.sleep(60)

 res = fabric.get_refresh_execution_details(
 dataset=dataset, workspace=workspace, refresh_request_id=refresh_request_id
 )

 if res.status != "Unknown":
 break

if res.status == "Completed":
 print(f"Refresh Dataset completed with success {res.extended_status}")
else:
 errors = res.messages["Message"].str.cat(sep="\n")
 raise Exception(f"Refresh Dataset failed: \n\n{errors}")

This script starts by resolving the current workspace and identifying the target dataset. It then initiates a full refresh of that dataset and waits for the process to complete, checking periodically for updates.

Terraform/OpenTofu Notes

Wed, 05 Nov 2025 00:00:00 +0000

Using Workspaces for Environment Separation

Terraform workspaces are a powerful feature that allows you to manage multiple environments (such as dev, staging, and prod) within the same configuration. For example, to switch to the production workspace:

tofu workspace select prod

Applying Configuration Changes

When applying changes to your Terraform configuration, you can specify a variable file for different environments and approve the changes automatically:

tofu apply -var-file ./dev.tfvars -auto-approve

Importing Existing Resources

To import an existing resource into Terraform’s state, use the import command with the appropriate variable file:

The credentials provided for the UsageMetricsDataConnector source are invalid.

Mon, 14 Jul 2025 00:00:00 +0100

Last weeks I had several meetings with Microsoft support to try this wrong credentials issue. The problem was that after right clicking to “View Usage Metrics Report” the report would come up empty and after 24h you would get an email with the following message:

<ccon>The credentials provided for the UsageMetricsDataConnector
source are invalid. (Source at UsageMetricsDataConnector.)</ccon>. The
exception was raised by the IDbCommand interface

We tried several approaches until we found this piece of documentation: https://learn.microsoft.com/en-us/power-bi/collaborate-share/service-modern-usage-metrics#considerations-and-limitations

About me

Wed, 09 Jul 2025 00:00:00 +0000

I’m a Portuguese Solution Data Architect based in Braga, with more than 15 years of experience building Business Intelligence and data solutions — always aiming to help teams find meaning in their data.

My path has taken me from software development through BI consulting and a spell leading the BI practice at Salsa — where I worked with QlikView and Qlik Sense — to a period at DevScope on projects involving Snowflake, to where I am today: working freelance as a Solution Data Architect with NoseDat, a German BI company. My focus is now the modern Azure stack — Microsoft Fabric, Databricks, Power BI and DAX — alongside infrastructure-as-code with Terraform. I’m currently a Microsoft Certified: Fabric Data Engineer Associate.

Issues with v4l2loopback and OBS

Wed, 09 Jul 2025 00:00:00 +0100

I started my Linux journey a long time ago with Red Hat in the middle of the nineties and moved after to Slackware. Still to this day one of my favourite distros. I tried Ubuntu for a shortwhile but found a new home on Arch. It was my main distro for the better part of the 2010’s but then tried Void Linux and it’s now my home since at least 3 years.

Tabular Editor Scripts: Simplifying Your Power BI Modeling Experience

Sun, 27 Apr 2025 10:23:00 +0100

As a data enthusiast, I’m always on the lookout for ways to streamline my workflow and make my life easier. In this post, I’ll be sharing two Tabular Editor scripts that have saved me countless hours of manual formatting in my Power BI projects.

There are two versions of this software. To use the scripts below you only need version two. This version is free and open source: https://tabulareditor.github.io/TabularEditor/

If you have the money for the license you can support the author (Daniel Otykier) by buying the license for version 3: https://tabulareditor.com

Extracting tables from SQL queries by using Sqlfluff

Sun, 03 Sep 2023 22:16:00 +0100

I got an assignment the other day to produce documentation to send to a customer. The extraction of the table names required to execute a certain Databricks notebook was part of the task. The plan was to build an object dependency tree.

The query spanned 279 lines. How can you extract only the table names from a file without having to manually look for them? Can we make use of this technique again in the future?

Azure Active Directory extraction with Databricks

Mon, 08 May 2023 16:51:00 +0100

During data engineering projects I tend to try and minimize the tools being used. I think it’s a good practice. Having too many tools causes sometimes for errors going unnoticed by the teams members.

One of the advantages of having a tool like Databricks is that it allows us to use all the power of python and avoid, like I did in the past, to have something like Azure Functions to compensate for the limitations of some platform.

List of errors from Databricks API

Fri, 17 Feb 2023 11:01:00 +0000

I’m currently working on a project where I’m adapting a code base of Databricks notebooks for a new client. There are a few errors to hunt but the Web UI is not really friendly for this purpose.

Just wanted a quick and easy way to not have to click around to find the issues.

Here’s a quick script to just do that:

import os, json
import configparser
from databricks_cli.sdk.api_client import ApiClient
from databricks_cli.runs.api import RunsApi


def print_error(nb_path, nb_params, nb_run_url, nb_error="Unknown"):
 error = nb_error.partition("\n")[0]
 params = json.loads(nb_params) if nb_params != "" else {}
 print(
 f"""
Path: {nb_path}
Params: {json.dumps(params,indent=2)}
RunUrl: {nb_run_url}
Error: {error}
"""
 )


databricks_cfg = "~/.databrickscfg"

conf = configparser.ConfigParser()
conf.read(os.path.expanduser(databricks_cfg))

api_client = ApiClient(
 host=conf["DEFAULT"]["host"],
 token=conf["DEFAULT"]["password"]
)

runs_api = RunsApi(api_client)

for x in range(1, 101, 25):
 x = runs_api.list_runs(
 job_id=None,
 active_only=None,
 completed_only=None,
 offset=x,
 limit=25,
 version="2.1",
 )
 if len(x["runs"]) > 0:
 for y in x["runs"]:
 if y["state"]["result_state"] == "FAILED":
 z = runs_api.get_run_output(run_id=y["run_id"])

 if "error" in z:
 print_error(
 z["metadata"]["task"]["notebook_task"]["notebook_path"],
 z["metadata"]["task"]["notebook_task"]["base_parameters"][
 "Param1Value"
 ],
 z["metadata"]["run_page_url"],
 z["error"],
 )
 else:
 print_error(
 z["metadata"]["task"]["notebook_task"]["notebook_path"],
 z["metadata"]["task"]["notebook_task"]["base_parameters"][
 "Param1Value"
 ],
 z["metadata"]["run_page_url"],
 )

Follow this documentation to install the requirements. There’s a lot more you can do with databricks-cli to make your life easier. It’s a great tool to add to your toolbox.

Calculating workdays in Databricks

Thu, 05 Jan 2023 10:23:00 +0000

There’s not an official function to calculate workdays in Databricks. Here are some solutions.

Having a DimCalendar with holidays and at least Databricks Runtime 12

If you have a DimCalendar in the system you can now do LEFT LATERAL JOIN without the correlated subquery errors when using non-equality predicates. Check SPARK-36114 for more details.

Calculating working days is then as simple as run a query like:

SELECT mt.StartDate,
 mt.EndDate,
 dc.Workdays
 FROM myTable mt
 LEFT JOIN LATERAL
 (SELECT COUNT(DateKey) - 1 AS Workdays
 FROM dimCalendar dc
 WHERE mt.StartDate <= dc.DateKey
 AND mt.EndDate >= dc.Datekey
 AND dc.IsDayWeekDay = TRUE
 AND dc.IsDayHoliday = FALSE
 )

If this query is slow, please check if the data types for the dates columns match. All are DATE or all are TIMESTAMP. If the sluginess remains check if any of the fields are not part of the statistics of the table.

Mass changing power query connection strings in Excel using PowerShell

Wed, 07 Dec 2022 10:33:00 +0000

A while ago I’ve received an email from Jason Alvarez asking me if I knew a way to change Power Query connections inside an Excel file. The problem is similar to my previous post on Mass changing pivot table connection strings in Excel using PowerShell. Turns out you can and he was able to find this solution:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


#root path to the collection of excel docs:
$rootPath = "C:\Temp\PowerQueryDemo\"

#Current query file path:
$currentPath = "C:\Temp\PowerQueryDemo\"
#Updated query file path:
$newPath = "C:\Temp\PowerQueryDemo\PowerQueryNewSource"

Get-ChildItem -Path $rootPath -Recurse -Include *.xlsx | ForEach-Object{
 $docPath = $_.FullName

 ## Create an instance of excel:
 $ExcelAppInstance = New-Object -ComObject "Excel.Application"
 $ExcelAppInstance.Visible = $false

 ## Open the workbook object within our Excel instance:
 $workbook = $ExcelAppInstance.workbooks.Open($docPath)

 ##iterate the list of queries embedded in the workbook,
 ##updating the file path to our new one:
 $workbook.Queries | ForEach-Object{
 Write-Output $_.Formula
 Write-output $_.Formula.replace($xPath,$tPath)
 $_.Formula = $_.Formula.replace($xPath,$tPath)
 }

 $workbook.Save()
 $workbook.Close()
 $ExcelAppInstance.Quit()
}

The key here is the Queries property inside the Workbook object.

PBIPT #41 - Solving Wordle with Power BI

Wed, 29 Jun 2022 09:32:00 +0100

Yesterday I had the pleasure to be back to in-person to give a small fun presentation about bending Power BI to help us solve Wordle puzzles.

The content is in Portuguese but I will upload an English version during this week.

Here’s the link on GitHub where you can download the file:

https://github.com/fpvmorais/presentations

If you have any questions please reach me.

Have fun!

JavaException: Must have Java 8 or newer installed.

Sun, 24 Apr 2022 19:47:00 +0100

While creating a new machine to be the Integration Runtime for Purview and after I have installed the mandatory JRE that allows the connection to Snowflake I kept getting this error:

Error: (3913) JavaException: Must have Java 8 or newer installed.

This puzzled me because I had installed version 17. Started troubleshooting and followed this guide to check if the installation was correct. Everything seemed alright but the error was still there.

Python Azure Functions Tips

Sat, 16 Apr 2022 11:03:00 +0100

Here are some tips of the things I’ve learned while creating them.

App Plan

If you already have an Windows ‘App Service Plan’ you’ll need a different one for running Python.

You can’t host Linux and Windows apps in the same resource group. If you have an existing resource group named AzureFunctionsQuickstart-rg with a Windows function app or web app, you must use a different resource group.

Python version

If you need to use ‘azure-identity’ you’ll need to use python 3.8 instead of the 3.9 otherwise you’ll get:

Accessing local settings while unit testing Azure Functions

Sat, 09 Apr 2022 16:45:00 +0100

There’s a lot to chew while unit testing Azure Functions. I going to be quite liberal with the terminology because technically some of this will be in fact integration testing and not unit testing per se.

Either way, Azure Functions load the local.settings.json on startup, creating several environment variables that then we can use in our code. In C# we would access them like so:

tenantId = configRoot["TenantId"];
appId = configRoot["AppId"];

or in Python:

Add multiple scans using the Purview API

Sun, 06 Feb 2022 19:02:00 +0000

As I explained in a previous post about Purview with the official Snowflake connector you need to create a scan per database/schema pair. In the customer where I was working there are a few hundred databases plus their schemas.

Doing this by using the UI would mean that we will be wasting a lot of time clicking away and be prone to forget some database/schema pair.

I’ve explored the documentation about the Scans - REST API and the rest of this post is just some PowerShell code to automate the creation of scans and their schedules in Purview.

Add user(s) to multiple Power BI workspaces quickly

Fri, 14 Jan 2022 17:28:00 +0000

A few days ago I had to add a new team member to more that 50 workspaces. I started dying inside just thinking of going to each workspace and add it using the portal ui.

Here’s a little snippet that saved me a few hundred clicks:

Login-PowerBI

$workspaces = Get-PowerBIWorkSpace

$user = "mySpecial.User@MyDomain.com"

foreach($workspace in $workspaces)
{
 Write-Host "Adding User to : " $workspace.Name
 try
 {
 Add-PowerBIWorkspaceUser -Id $workspace.Id -UserPrincipalName $user -AccessRight Member
 }
 catch
 {
 Write-Host "Message: [$($_.Exception.Message)"] -ForegroundColor Red
 }
}

Adapt it to your needs.

Snowflake Connector for Purview

Mon, 29 Nov 2021 09:58:13 +0000

In the last few weeks I’ve been working with Purview doing some code to integrate Snowflake entities because there wasn’t an official connector. You can imagine my surprise when I’ve noticed last friday that there’s a new unannounced preview connector that does the work for us.

PyApacheAtlas

If you already read or watched something about the recommended way to create custom entities on Purview you have heard something about this library.

Deprecating Google Spreadsheets Custom Connector

Mon, 22 Nov 2021 11:30:31 +0000

There’s now an official connector to Google Spreadsheets on Power BI

With the November 2021 Power BI release, came the official connector support for Google Spreadsheets from Microsoft. As such there’s no more need for my custom connector. The repository on GitHub is now archived.

No more `Relationship tag contains incorrect attribute. Line 2, position 86`

During the last week I’ve tested it so see if it had the same limitations as mine had and I’m happy to report the XML errors when trying to import data from a Google Sheet with a pivot table don’t occur in the new one. No more having to create a proxy file and use the IMPORTRANGE function.

Advent of Code 2020 - Day 16

Sat, 17 Apr 2021 17:25:00 +0100

Today’s puzzle is called ‘Ticket Translation’. We have a bunch of fields in a language we do not understand and we need to figure out which is which.

There’s a set of rules we collected in several ways to a document that will serve as the puzzle input.

If you want to follow along please download AdventOfCode2020.pbit and check Day 1 post for instructions.

Part 1

Our input file is divided into three sections separated by a blank line:

Advent of Code 2020 - Day 15

Fri, 02 Apr 2021 19:07:00 +0100

Rambunctious Recitation is number game that follows the rules of the Van Eck sequence. Here’s a great video from the Numberphile channel on YouTube about it. And here start our troubles because it seems no one has yet found an efficient algorithm for generating this sequence. We will really need to brute force it.

How can we resolve this predicament with the tools available to us?

If you want to follow along please download AdventOfCode2020.pbit and check Day 1 post for instructions.

Advent of Code 2020 - Day 14

Sun, 21 Mar 2021 22:09:00 +0000

The troubles are far from over, this time it’s the docking program on the ferry that’s not compatible with the port computer system. We need to emulate the port’s software by using a bitmasking system to figure out the correct instructions. Let’s see if we are able to implement this logic with Power BI.

If you want to follow along please download AdventOfCode2020.pbit and check Day 1 post for instructions.

Part 1

As usual there’s a file with a set of instructions. Some lines refer to memory addresses and others to bitmasks. We need to apply the mask to the values stored in the memory addresses beneath it until we reach the next mask repeating the process to the end of the file.

Advent of Code 2020 - Day 13

Sun, 28 Feb 2021 18:00:00 +0000

We arrived to the island after the ferry trip and we see ourselves in need of transportation to the nearest airport. Fortunately there’s a shuttle bus service between the sea port and the airport that could take us there.

Day 13 is a wonderful mathematical puzzle, following the theme of modular arithmetic of the previous one. Let’s dive into it as there some surprises with the numeric limits on Power BI.

Advent of Code 2020 - Day 12

Sun, 21 Feb 2021 18:22:00 +0000

After the waiting room we are at a ferry navigating to the island. There seems to be a problem with the navigation system, and we volunteered to help. We have a set of instructions and we need to make some sense of them to help the captain circumvent the storm.

The solution to the problem will be given by calculating the Manhattan distance (sum of the absolute values of its east/west position and its north/south position).

Advent of Code 2020 - Day 11

Sat, 20 Feb 2021 17:28:00 +0000

Today’s puzzle title is “Seating System” but it’s actually a variation of a well known zero-player game called Conway’s Game of Life. Many of us may recall this game as a popular choice for work assignments in computer science courses.

Dr. John Conway, the English mathematician who created this game was last year a victim of COVID-19. If you want to know a little more about him you can read this article that was featured this week on Hacker News.

Advent of Code 2020 - Day 10

Sun, 14 Feb 2021 18:20:00 +0000

Day 10 puzzle brings us an array of adapters. In the first part we will need to find the number of differences of 1 and 3 jolts. Part two asks us to find the total number of distinct ways in which we can arranje the adapters to connect the device to the charging outlet.

If you want to follow along please download AdventOfCode2020.pbit and check Day 1 post for instructions.

Part 1

After reading that long text, we can see in the example given that all we have to do is:

Advent of Code 2020 - Day 9

Sun, 07 Feb 2021 18:52:00 +0000

This puzzle looks a lot like day 1. We have a list of numbers and need to do some operations that will return some big numbers.

Let’s try to DAX out them.

If you want to follow along please download AdventOfCode2020.pbit and check Day 1 post for instructions.

Part 1

We have a list of numbers of each the first 25 are a preamble. After this, any number can be calculated as a sum of any pair of numbers in that interval.

Advent of Code 2020 - Day 8

Sun, 07 Feb 2021 12:29:00 +0000

In day 8 we find a some weird infinite loop on a handheld device. To fix this problem we are asked to create a program that goes trough a stack of instructions.

nop +0
acc +1
jmp +4
acc +3
jmp -3
acc -99
acc +1
jmp -4
acc +6

And after we find where the infinite loop is located we replace the bad command with the correct instruction.

Advent of Code 2020 - Day 7

Sun, 31 Jan 2021 12:47:00 +0000

Day 7: Handy Haversacks is the hardest puzzle to solve using only Power BI until now. It took me a lot of time to find a viable and fast solution.

I have first tried to solve this using DAX PATH but had issues with the different levels of recursion. It was possible in the real data input for the same bag to be child of different bags in different levels. Only after a deep dive on using the List.Generate a light began to appear on the end of the tunnel.

Advent of Code 2020 - Day 6

Sun, 24 Jan 2021 15:46:00 +0000

Time to check for some custom declarations forms. In the first part we will check the distinct number of questions anyone on each group answered “yes”, and on the second part the questions where everyone gave a positive response.

If you want to follow along please download AdventOfCode2020.pbit and check Day 1 post for instructions.

Part 1

The input for this problem is very similar to the passports we have seen on day 4. We have several lines that represent the answers of the persons in a group and the groups are separated by a blank line.

Advent of Code 2020 - Day 5

Sat, 23 Jan 2021 15:59:00 +0000

This problem seems a lot of work at first but in reality it’s just binary number conversion to decimal.

Picking up the provided example:

FBFBBFF RLR

If we replace the lower half [F, L] by 0, and the upper half [B, R] by 1 we get:

0101100 101

Doing the conversions:

Advent of Code 2020 - Day 4

Fri, 22 Jan 2021 18:58:00 +0000

Day 4 is about passport fields validation. In part 1 we need to check if all the fields exist and in part 2 check if they are valid.

This seems like a job where the heavy lifting should be made by Power Query.

https://adventofcode.com/2020/day/4

If you want to follow along please download AdventOfCode2020.pbit and check Day 1 post for instructions.

Part 1

Following this strategy we load the data to our file and we find our first challenge. “Each passport is represented as a sequence of key:value pairs separated by spaces or newlines. Passports are separated by blank lines.”

Advent of Code 2020 - Day 3

Wed, 20 Jan 2021 20:31:00 +0000

Now we need to check for trees in paths. The interesting bit is that the path can grow ad infinitum to the right.

Check the details at https://adventofcode.com/2020/day/3

If you want to follow along please download AdventOfCode2020.pbit and check Day 1 post for instructions.

Part 1

How do we avoid repeating the pattern? Looking at the provided sample we have:

..##.......
#...#...#..
.#....#..#.
..#.#...#.#
.#...##..#.
..#.##.....
.#.#.#....#
.#........#
#.##...#...
#...##....#
.#..#...#.#

The interesting part begins after line 3, where we are going to need to get positions that are over the length of the row. The value that we wish for row 4 is 1, which is the value of the remainder of dividing 12 by the lenth of the track which is 11. Adding a new column with the remainder we can crosscheck with our data and everything seems to work:

Advent of Code 2020 - Day 2

Sun, 17 Jan 2021 21:34:00 +0000

We are at an toboggan rental shop and we need to help the shopkeeper to validate a bunch of corrupted passwords.

(If you want to follow along please download AdventOfCode2020.pbit and check Day 1 post for instructions)

Part 1

In part one we need to check if a given character occurs between a certain lower and upper bound.

This kind of operation seems to be more suited to be done in Power Query as we are going row by row checking for the validity of the rule. My idea here is to create a column with value of 1 if the password is valid and 0 otherwise.

Advent of Code 2020 - Day 1

Sat, 16 Jan 2021 17:35:00 +0000

This will be the first post of a series where I hope to be able to solve the problems from the Advent of Code using only Power BI.

This event, sadly, only came to my attention this year. It started in 2015 and it’s a lot of fun. In these posts I will talk about my thinking and strategy to try and solve these problems.

Without further ado let’s get started:

Google Spreadsheets Custom Connector

Wed, 26 Feb 2020 09:36:30 +0000

⚠ WARNING: The content of this post has been made deprecated due to the availability of the official Google Sheets connector and you should use that feature to import the data. ⚠
Check the official announcement on the link below:
https://powerbi.microsoft.com/en-us/blog/power-bi-november-2021-feature-summary/#post-17912-_Toc87473946
This post is just for future reference

In the beggining

I’ve been tasked to find a way of exporting data from Google Spreadsheets for a client. I’ve started researching for an already working solution and got to the following links:

Import data from another Power BI dataset

Mon, 24 Feb 2020 09:14:52 +0000

⚠ WARNING: The content of this post has been made deprecated due to the availability of composite models and you should use that awesome feature to import the data. ⚠
Check the official announcement on the link below:
https://powerbi.microsoft.com/en-us/blog/directquery-for-power-bi-datasets-and-azure-analysis-services-preview/
This post is just for future reference

While I was working with a client they were trying to import data from another Power BI dataset to their file and they got this error:

Emacs Major Modes for Power BI

Wed, 22 Jan 2020 13:53:10 +0000

Emacs Major Modes for Power BI

After many years of using Vim I finally understood the Emacs way. What I was trying to do all along was trying to make Vim behave exactly like Emacs.

To celebrate this I made some bad Elisp code and shared it on GitHub. I based my code on the Simple Emacs Major Mode for Syntax Coloring by Xah Lee.

Power Query Mode

This mode gives you syntax coloring and code completion inside Emacs. It has all the functions listed by '#shared' and color codes them. For more details on listing all Power Query functions you can check the post from RADACAD.

Diffing Power BI files

Sun, 02 Jun 2019 14:39:06 +0100

Do you really can diff Power BI files?

While working with some customers a few times this question came up: “How can I diff my current file with a older one? I really want to compare my files, to know what changed from one to another.”

Many many people working with Power BI end up with a directory full of files looking like: “myPowerBIFileV1.pbix”, “myPowerbifilev2.pbix”… “myPowerbifilevN.pbix”. The scenario wull get worse if we have two or more people working in the same file, and until recently my answer was always the same: “You can’t”.

EzApi - Define flat file columns from connection string

Thu, 25 Apr 2019 16:08:33 +0100

So what’s new?

Thanks to the contribution via GitHub from Hadi Fadl there’s a new method available for the flat file connection manager that adds the existing columns in the source file. This is very handy because in the past you would have to define each and every column that existed in the flat file.

I’ve updated the nuget to version 0.8.93.

Please give me your feedback of anything you might think should be improved.

EzApi - Adding Event Handlers

Tue, 19 Jun 2018 09:16:56 +0100

How to add event handlers to generated packages

I had a request to give an example of adding event handlers to a generated SSIS package. The code is quite convoluted. There’s no method to initialize a DtsEventHandler, you need to cast it after adding to the package with a String describing the type of event. Here’s a list of the available events:

Here’s a sample of how to do insert a ExecuteSQLTask to a event handler using a small console application:

Openssh Windows 10 Issues

Mon, 07 May 2018 12:57:12 +0100

Warning: Unprotected Private Key File

As an avid user of OpenSSH on Windows after last update I’ve started seen this on my PowerShell:


Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for 'C:\\Users\\My User/.ssh/id_rsa' are too open.

File System Security PowerShell Module

Because file and folder permissions are not easy with PowerShell I’m using the File System Security PowerShell Module. You can install from the gallery using the command:

EzApi 0.8.92

Fri, 04 May 2018 12:57:41 +0100

New Release 0.9.82

My small contribution for the BI world has been this SSIS generation framework resurrection. Amazingly it has almost 1500 downloads @NuGet. We are using this @DevScope for creating our SSIS packages based on metadata and it’s on production on some of our clients.

We can see the rest of the details at the project repository.

The Future

I finally managed to move to a new house and I will have (hopefully) more time to write on the blog and to continue to work on this side project.

Mass changing pivot table connection strings in Excel using PowerShell

Sat, 20 Jan 2018 18:22:21 +0000

The Problem

As Business Intelligence consultants after a migration to a brand new server inevitably we get asked by the users: “Do why have to redo all my pivot tables in Excel again? All my workbooks stopped working…”

Well, they can but they don’t need to. Where’s a simple code to replace the part that matters. This will iterate all the Excel files in the folder where the Powershell Script is located and and apply the substitution to all the OLEDBConnections that exist in each file:

Using Powershell to read and modify SSIS packages

Tue, 28 Nov 2017 20:10:47 +0000

Introduction

In these days developing in SSIS, you should be using a generation framework (BIML or EzApi. The benefits era a bit obvious:

Your packages will have consistent layout and logic
It’s easy and fast to apply the same change to all packages
No more mouse programming (almost :))
It’s the right way to do metadata driven ETL

The problem

In a client I had a project with 97 packages. Each one of them started as copy of the first one. The request was to change the value of a single variable on all of them.

How to access SFTP with SSIS using SSH.NET

Wed, 06 Sep 2017 16:34:25 +0100

Tutorial to download from SFTP in Integration Services

Hello World

Mon, 04 Sep 2017 15:58:01 +0100

Inspired by this post I’ve decided to start this blog.

Thank you Andy for all the times you’ve helped me without even knowing it :)

Contact

Fri, 01 Sep 2017 00:00:00 +0000

How to contact me

A #dataOp ramblings

Fabric Semantic Model Refresh: One Table at a Time

The problem: out of memory

Microsoft Fabric External Data Sharing Hidden Limitation

Refreshing Power BI Datasets with Notebooks

Terraform/OpenTofu Notes

Using Workspaces for Environment Separation

Applying Configuration Changes

Importing Existing Resources

The credentials provided for the UsageMetricsDataConnector source are invalid.

About me

Issues with v4l2loopback and OBS

Tabular Editor Scripts: Simplifying Your Power BI Modeling Experience

Extracting tables from SQL queries by using Sqlfluff

Azure Active Directory extraction with Databricks

List of errors from Databricks API

Calculating workdays in Databricks

Having a DimCalendar with holidays and at least Databricks Runtime 12

Mass changing power query connection strings in Excel using PowerShell

PBIPT #41 - Solving Wordle with Power BI

JavaException: Must have Java 8 or newer installed.

Python Azure Functions Tips

App Plan

Python version

Accessing local settings while unit testing Azure Functions

Add multiple scans using the Purview API

Add user(s) to multiple Power BI workspaces quickly

Snowflake Connector for Purview

PyApacheAtlas

Deprecating Google Spreadsheets Custom Connector

There’s now an official connector to Google Spreadsheets on Power BI

No more Relationship tag contains incorrect attribute. Line 2, position 86

Advent of Code 2020 - Day 16

Part 1

Advent of Code 2020 - Day 15

Advent of Code 2020 - Day 14

Part 1

Advent of Code 2020 - Day 13

Advent of Code 2020 - Day 12

Advent of Code 2020 - Day 11

Advent of Code 2020 - Day 10

Part 1

Advent of Code 2020 - Day 9

Part 1

Advent of Code 2020 - Day 8

Advent of Code 2020 - Day 7

Advent of Code 2020 - Day 6

Part 1

Advent of Code 2020 - Day 5

Advent of Code 2020 - Day 4

Part 1

Advent of Code 2020 - Day 3

Part 1

Advent of Code 2020 - Day 2

Part 1

Advent of Code 2020 - Day 1

Google Spreadsheets Custom Connector

In the beggining

Import data from another Power BI dataset

Emacs Major Modes for Power BI

Emacs Major Modes for Power BI

Power Query Mode

Diffing Power BI files

Do you really can diff Power BI files?

EzApi - Define flat file columns from connection string

So what’s new?

EzApi - Adding Event Handlers

How to add event handlers to generated packages

Openssh Windows 10 Issues

Warning: Unprotected Private Key File

File System Security PowerShell Module

EzApi 0.8.92

New Release 0.9.82

The Future

Mass changing pivot table connection strings in Excel using PowerShell

The Problem

Using Powershell to read and modify SSIS packages

Introduction

The problem

How to access SFTP with SSIS using SSH.NET

No more `Relationship tag contains incorrect attribute. Line 2, position 86`