What this project is about
Retail extracts can contain stale duplicates, missing reference records and values that cannot be trusted in a dashboard. RetailPulse demonstrates how customers, products and orders can be transformed into reviewable reporting tables while preserving rejected orders for investigation. The local workflow makes quality rules visible before introducing a distributed data platform.
The repository contains a runnable standard-library Python pipeline, a separate PySpark job and an Azure-oriented architecture. Bronze, silver and gold refer to raw inputs, cleaned detail and reporting-ready outputs respectively. The Azure services in the architecture describe a deployment path; the repository does not demonstrate an operating Data Factory schedule, provisioned cloud environment or production Power BI service.
How it works
- ADLS bronze extracts
- Validated silver detail
- Gold reporting tables
Apply visible rules to the local batch
The local runner reads three CSV extracts and keeps the last received record for each business key. It checks order dates, positive integer quantities, non-negative prices and customer/product references. Accepted orders are enriched with descriptive attributes, and an empty customer region becomes Unknown. Rejected orders retain their input fields with one or more reason labels rather than silently becoming successful reporting rows.
Separate detail from reporting aggregates
Accepted rows become a detailed sales export. Further outputs aggregate revenue by month and region and rank product popularity by units. A metrics file records source orders, accepted and rejected counts, removed duplicates and repeat customers. These measures answer different questions: duplicate records across all three extracts must not be mistaken for duplicate orders when reconciling order totals. Decimal arithmetic supports explicit two-place revenue values.
Provide a distributed implementation starting point
The PySpark job selects records by their updated_at timestamp, casts order fields, joins customer and product data and writes Delta outputs for sales detail and monthly regional revenue. Delta is the table storage format proposed for the distributed path. The script is narrower than the local runner: it filters invalid rows and uses inner joins without exporting rejected rows, and it does not reproduce every local output.
The important design choices
Make local review inexpensive
Supply a small Python reference pipeline alongside the Spark job.
The trade-offRules can be exercised without a cluster, but two implementations can drift. Deduplication order, rejection behavior and generated tables need explicit parity tests before treating either path as interchangeable.
Keep transformations outside the report
Publish prepared tables and version-controlled DAX measures with Power BI construction notes.
The trade-offThe business logic remains inspectable, but a report still needs to be assembled, validated and deployed. The repository's assets do not establish that a finished PBIX report or managed refresh exists.
A batch with duplicates and invalid orders
A local sample batch contains a repeated order key, an unknown customer and an order with a negative quantity.
- The local runner keeps the final row for the repeated business key according to file order.
- The invalid orders go to the rejected-order export with reasons; valid orders are joined to customer and product attributes.
- The accepted detail feeds revenue and product summaries, while the metrics make the sample's accepted, rejected and duplicate accounting reviewable.
What this showsThe local result distinguishes usable reporting data from records needing correction. The same rejection behavior must still be implemented and tested in Spark.
Illustrative example, not a production incident.
How it was tested—and what it shows
The checks
The local tests verify last-record deduplication and run the supplied sample batch into temporary outputs. They check order reconciliation with the known sample duplicate, removed-record counts, non-negative revenue and expected regional coverage. The fixtures deliberately include bad and duplicated records. These tests concern the local implementation; they do not validate Spark execution, cloud permissions or equivalence between the two transformation paths.
What to take from the results
The project combines explicit data-quality rules, rejected-row evidence and usable reporting outputs. The distributed Spark version still needs the local runner’s full quality checks: a design for Azure is not yet a verified cloud deployment with rejected-row handling and reconciled totals.
What to keep in mind
- Cloud-shaped code and configuration do not by themselves establish production operation.
- Data checks cover the implemented rules, not every possible business error.
Technical notes & source
Built with
Python · PySpark · Azure Data Lake · Power BI
Terms used in this project
- Quarantine
- Keep questionable records separate for review instead of silently using or discarding them.