Real-Time Data Pipeline for Car Parking Management

The client operated a car parking management system that generated substantial operational data, but lacked a unified way to process and visualize it. Data arrived from multiple sources—some in real-time through message queues, others in batches via file storage. Without a proper pipeline, the team had no visibility into parking operations, occupancy trends, or system performance. They needed a solution that could handle real-time data streams while remaining flexible enough to accommodate future changes. Importantly, they wanted full control over the infrastructure—no reliance on expensive SaaS tools or vendor lock-in.

We started by mapping out the data sources and understanding the operational questions the team needed answered. The key requirement was flexibility—the ability to modify the pipeline easily as business needs evolved. We chose Mage AI as the orchestration layer. As an open-source tool that can be self-hosted on any server, it gave the client complete control over their data pipeline. Mage's block-based architecture meant that adding, updating, or removing transformations could be done directly in the pipeline without complex redeployments.

We built a data pipeline using Mage AI that ingests data from two primary sources: Real-time data flows in through a message queue, capturing live events from the parking management system. This includes transactions, occupancy changes, and operational events as they happen. Batch data arrives via S3, handling larger historical datasets and periodic data dumps. The pipeline processes both streams, applies the necessary transformations, and stores everything in a denormalized data structure. The denormalized approach was a deliberate design decision. By flattening the data at ingestion time, we eliminated complex joins at query time. This means dashboards load quickly and analysts can write simple queries without needing deep knowledge of the underlying data relationships. On top of this data layer, we built custom Grafana dashboards for observability and monitoring. Following established dashboard design principles, we created views that surface the most critical metrics while allowing drill-down into operational details.

The entire infrastructure is self-hosted, giving the client full ownership and control. Mage AI runs on their servers, orchestrating the data flow from source to dashboard. The pipeline architecture makes changes straightforward. Need to add a new metric? Update a transformation block in Mage. Want to change how data is aggregated? Modify the relevant pipeline step. Need to debug performance issues? Mage provides built-in monitoring to identify bottlenecks. This flexibility was critical for the client. Parking operations evolve, and the data needs change with them. Having a pipeline that can adapt without major re-engineering work means the analytics infrastructure grows with the business rather than becoming a constraint.

The client now has complete visibility into their parking operations through real-time dashboards. Occupancy data, transaction metrics, and operational KPIs are all available at a glance. More importantly, the system is built to last. When new requirements emerge—a new data source, a different aggregation, an additional dashboard—the team can implement changes directly in Mage without rebuilding the pipeline. The denormalized data structure means queries remain fast regardless of the complexity of the questions being asked. The self-hosted approach eliminated ongoing SaaS costs and gave the client full control over their data. No vendor dependencies, no per-seat pricing, no concerns about data leaving their infrastructure.