Skip to main content
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Calling all Data Engineers! Fabric Data Engineer (Exam DP-700) live sessions are back! Starting October 16th. Sign up.

Reply
ati_puri
New Contributor III

Handling version controlled Metadata files in Fabric

Hi,

 

Currently, in our project we are handling configuration files for the metadata-driven pipelines/notebooks in the file section of fabric lakehouse, however when we sync our lakehosue with GIT, only lakehouse metadata goes into GIT and not datafiles. This leads to an issue when we want to have those config files in our other environments like Test, Prod via GIT.

 

One solution is to have those metadata files in GIT as .josn files, is there any other solution which is currently being implemented as a version controlled metadata files inside or outside fabric.

1 ACCEPTED SOLUTION
gaya3krishnan86
New Contributor II

We use metadata driven configuration json files and we save them in git.  To deploy the files to lakehouse we use fabric cli library as part of our CI CD process. 

View solution in original post

3 REPLIES 3
v-sshirivolu
Contributor III

Hi @ati_puri ,
Thanks for reaching out to Microsoft Fabric Community.

When syncing a Lakehouse with Git, only artifact metadata is pushed, not the data layer or configuration files. To manage JSON configs, store them as .json files in Git for version control and CI/CD support.Additionally, many teams use one or more of the following methods, depending on the configuration type:

Azure Key Vault / App Config: For managing secrets, credentials, or secure settings that vary by environment, accessible at runtime by pipelines and notebooks.

Pipeline/global parameters: For environment-specific values like paths or schema names, which can be dynamically passed into pipelines and notebooks.

Dedicated Config Lakehouse with deployment step: For configurations that must remain as files, .json files are maintained in Git and copied to a Config Lakehouse folder during deployment.

Typically, a hybrid model is adopted, using Git for structural or lookup metadata, Key Vault or pipeline parameters for sensitive and environment-specific settings, and an optional Config Lakehouse for file-based configurations within Fabric. This approach ensures both version control and flexibility when promoting solutions across Dev, Test, and Prod environments.

ati_puri
New Contributor III

Has anyone tried Variable library for the above?

gaya3krishnan86
New Contributor II

We use metadata driven configuration json files and we save them in git.  To deploy the files to lakehouse we use fabric cli library as part of our CI CD process. 

Helpful resources

Announcements
Users online (7,586)