Hi @ex_kjetilh,
Below is a practical way to unit-test PySpark transformations from Fabric notebooks and run them in CI/CD. It boils down to: put your transform logic in plain Python modules, test them with pytest + a local Spark session (or Spark’s own testing helpers), and optionally add Fabric-side integration tests for end-to-end coverage.
- Refactor your notebook code into testable functions.
Put all transformation logic in /src/your_pkg/transforms.py (imported by your notebook), so tests don’t depend on a notebook runtime. See Databricks’ pattern (same idea) for testing notebook code by moving logic into modules: Unit testing for notebooks. - Write pytest unit tests that spin up a local Spark.
Create /tests/conftest.py and /tests/test_transforms.py. Use Spark’s built-in test helpers (Spark 4.x) like pyspark.testing.assertDataFrameEqual.
# tests/conftest.py
import pytest
from pyspark.sql import SparkSession
@pytest.fixture(scope="session")
def spark():
return (SparkSession.builder
.master("local[*]")
.appName("unit-tests")
.getOrCreate())
# tests/test_transforms.py
from pyspark.sql import Row
from your_pkg.transforms import clean_and_join
from pyspark.testing import assertDataFrameEqual # Spark ≥ 4.x
def test_clean_and_join(spark):
left = spark.createDataFrame([Row(id=1, v="a "), Row(id=2, v="b")])
right = spark.createDataFrame([Row(id=1, w=10), Row(id=2, w=20)])
actual = clean_and_join(left, right) # your transform
expected = spark.createDataFrame([Row(id=1, v="a", w=10), Row(id=2, v="b", w=20)])
assertDataFrameEqual(actual, expected, checkRowOrder=False)
- Run tests in CI/CD (GitHub Actions or Azure DevOps).
Pin your local pyspark to the Fabric runtime version to avoid surprises (check your Fabric Spark runtime, then set the same pyspark==x.y.* in tests). Example GitHub Actions job:
name: unit-tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: '3.10' }
- name: Install Java for Spark
uses: actions/setup-java@v4
with: { distribution: 'temurin', java-version: '11' }
- name: Install deps
run: |
pip install "pyspark==<match_Fabric_runtime>" pytest chispa
- name: Run pytest
run: pytest -q --maxfail=1 --disable-warnings
Fabric CI/CD background: Deployment pipelines overview,
Git integration overview. Good write-ups with examples: Unit tests on Microsoft Fabric items (pytest), Optimizing for CI/CD in Microsoft Fabric.
- Add Fabric integration tests.
Keep unit tests fast/local. For end-to-end checks inside Fabric (e.g., against a Lakehouse table), you can:- Trigger a Notebook job in a test workspace that seeds tiny test data, calls your transform, and asserts results (either via Spark asserts or by writing a small “result” table and checking row counts/values).
- Or orchestrate with deployment pipelines/fabric-cicd and run a smoke-test notebook after deploy. Example concept posts: Automate testing Fabric pipelines with YAML, fabric-cicd library initial tests.
- In notebooks, import your package; don’t re-implement.
Your Fabric notebook should import your_pkg.transforms so the code under test and the code you run in Fabric are the same. General notebook authoring doc: Develop, execute, and manage Fabric notebooks.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
