Dataflows vs Data Pipelines vs Deployment Pipelines

[Anonymous]

I have 4 Gen2 dataflows in a data pipeline, in a Dev workspace.

Initially the dataflows are the standard Gen2 dataflows, which were in a data pipeline, but won't work in a Deployment Pipeline.

So I began creating the new CI/CD Gen2 dataflows, which I've read do work in a deployment pipeline, but I can't add them to a data pipeline (when I add the dataflow step the CI/CD dataflow isn't there, but the other ones are).

What can I do to have a series of dataflows, that are in a pipeline, and work with deployment pipelines?

[nilendraFabric · Accepted solution]

Hello @Anonymous 

I understand this is not ideal

CI/CD-enabled Dataflows Gen2 cannot be directly added to a Data Pipeline for orchestration ( As you mentioned). Only standard Dataflows Gen2 are supported in Data Pipelines.

One option is to split or maintain 2 copied - ( Not Ideal) 

• Use standard Dataflows Gen2 within your Data Pipeline for orchestration.
• Separately manage CI/CD-enabled Dataflows for deployment purposes until full integration support is available.

Once CI/CD-enabled Dataflows are available in Data pipelines , delete the Non CICD one 

Hope this helps 

Thanks

[v-menakakota]

Hi @Anonymous ,

Thank you for helpful respose  @nilendraFabric ,Standard Gen2 Dataflows  can be added to a data pipeline for orchestration but won't work in a deployment pipeline.I request you to follow the suggestions provided by Nilendra and let us know if it was helpful.If you still need any assistance feel free to reachout.

Regards,
Menaka.

[v-menakakota]

Hi @Anonymous ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.