- Issue: Facing lock condition errors when writing multiple records to an audit log table simultaneously.
- Setup:
- One master pipeline invoking multiple child pipelines.
- Audit log notebook placed in both master and child pipelines.
- During a child pipeline failure, the notebook activity updates the audit log table in parallel for both child and master pipeline rows.
- Problem: Despite updates being made as separate rows, lock conditions occur, causing pipeline errors.
- Attempts to Resolve:
- Applied partition by.
- Used update statements.
- These methods did not resolve the issue in Fabric.
Framed Question:
What are some effective strategies or best practices to avoid lock condition errors when writing multiple records to an audit log table simultaneously in a pipeline setup, especially in a Fabric environment?
