From Factory Sensors to Failure Warnings: An IoT Pipeline on AWS and Databricks

Last updated: October 2026

Many plants already collect a great deal of sensor data. It sits in local historian servers, and nobody analyses it, so a failing bearing on a rolling mill is noticed when the line stops.

This article describes the pipeline we built for a steel manufacturer, covered in our manufacturing IoT case study. The plant had 847 sensors across 12 production lines, generating roughly 2 TB of data a day, and each unplanned stoppage cost an average of $180,000 in lost production, scrap and emergency repairs. Maintenance was about 90% reactive. Project details come from that engagement. The rule SQL and code are simplified illustrations written for this article.

Start with the network, not the code

The hardest problem was not technical. The operational technology (OT) team controlled the production network and did not trust IT, or us, to touch it, and the IT team had no access to that network. Sensor data was only approved to leave the production network once we had set up an isolated DMZ with a unidirectional data diode.

On a manufacturing project, we recommend planning three to four weeks of security review and network architecture work before writing any pipeline code.

Normalise payloads in IoT Core rules

All 847 sensors connect to AWS IoT Core over MQTT and publish every five seconds: vibration amplitude, temperature, pressure, current draw and RPM. The payloads were not standardised. Six sensor manufacturers each had their own format and timestamp precision; some sent epoch seconds, others ISO 8601 strings with time zone offsets, and two sent local time with no time zone at all.

We built a normalisation layer in IoT Core rules that turns every payload into one common schema before it reaches Kinesis. A rule is a SQL statement over an MQTT topic filter, with + and # as wildcards, and Kinesis actions can take the partition key from a substitution template. One rule per manufacturer keeps each mapping small and testable.

-- Topic layout: plant/sensors/<line_id>/<sensor_id>/vendor_a
SELECT
    topic(3)          AS line_id,
    topic(4)          AS sensor_id,
    'vendor_a'        AS vendor,
    vib_amp           AS vibration_amplitude,
    temp_c            AS temperature_c,
    pressure_kpa      AS pressure_kpa,
    ts * 1000         AS event_epoch_ms,      -- vendor A sends epoch seconds
    timestamp()       AS received_epoch_ms
FROM 'plant/sensors/+/+/vendor_a'

-- Kinesis action on this rule:
--   streamName:   sensor-readings
--   partitionKey: ${topic(3)}-${topic(4)}

Time zones need the most care. IoT Core's time_to_epoch() function treats a timestamp string with no time zone as UTC, so a local time parsed in the rule would be silently shifted by the plant's UTC offset. For devices that send local time without a zone, we recommend passing the original string through unchanged, along with the plant's time zone, and converting it in Databricks, where the conversion can be tested. A rule's error action receives any message whose action fails, with the original payload, which is worth routing to a separate stream for inspection.

Partition Kinesis so no shard runs hot

From IoT Core, messages go to Kinesis Data Streams. In provisioned mode each shard accepts up to 1 MB per second or 1,000 records per second of writes, and a single partition key always maps to one shard. That second fact decides the key. Partitioning by production line alone caps each line at one shard's throughput; adding the sensor ID to the key, as in the rule above, spreads one line across shards when it needs more.

In the project, Kinesis was partitioned by production line, and each of the 12 lines had its own group of shards, which avoided hot partitions. On-demand mode is an alternative that scales automatically, but AWS notes that it does not isolate hot hash keys, so provisioned mode remains the better fit when keys are very uneven.

Parquet on S3, then Delta with Auto Loader

A Firehose delivery stream reads from Kinesis and writes compressed Parquet files to S3, partitioned by line, date and hour. Firehose converts JSON to Parquet using a schema held in the AWS Glue Data Catalog. On top of S3 we built a Databricks lakehouse on Delta Lake, whose ACID transactions matter when dashboards read the same tables that streaming ingestion is writing. Delta uses optimistic concurrency without locks, and two concurrent INSERTs cannot conflict under its default isolation level.

Auto Loader is the simplest way to pick up new Parquet files as they land. Its default mode lists the directory; Databricks now recommends file notification mode with file events for most workloads, which needs Databricks Runtime 14.3 LTS or later on S3.

(spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "parquet")
    .option("cloudFiles.useManagedFileEvents", "true")
    .option("cloudFiles.schemaLocation", "/Volumes/iot/raw/schemas/sensor_readings")
    .load("s3://<bucket>/sensors/")
  .writeStream
    .option("checkpointLocation", "/Volumes/iot/raw/checkpoints/sensor_readings")
    .trigger(processingTime="1 minute")
    .toTable("iot.bronze.sensor_readings"))

Auto Loader records which files it has processed in the checkpoint, so each file is loaded exactly once even if the stream restarts.

Converting local timestamps in Databricks

For readings that arrive in local time, the silver layer converts the local value to UTC using the plant's zone. If you also want to keep the original local value as a typed column, Databricks' TIMESTAMP_NTZ type, available from Databricks Runtime 13.3 LTS, stores a timestamp without any time zone, which matches a value that came without one.

SELECT
    line_id,
    sensor_id,
    vendor,
    CASE
        WHEN event_epoch_ms IS NOT NULL
            THEN timestamp_millis(event_epoch_ms)
        ELSE to_utc_timestamp(local_ts_string, plant_time_zone)   -- e.g. 'America/Chicago'
    END AS event_ts_utc,
    vibration_amplitude,
    temperature_c,
    pressure_kpa
FROM iot.bronze.sensor_readings;

Local times without a zone are ambiguous for one hour each year where daylight saving time applies, when clocks go back and the same local hour occurs twice. If the plant observes daylight saving, flag readings from that hour and use the receipt time from the IoT rule to place them.

From history to failure warnings

The value came from the models. We trained anomaly detection models in Databricks on 18 months of historical sensor data, correlated with maintenance logs, and they flag three failure precursors:

Alerts go to a dashboard on large monitors on the shop floor and, at the same time, to the maintenance team's pagers through Amazon SNS. Reactive maintenance fell from about 90% of the workload to 35%, and the plant now saves $3.1 million a year in unplanned downtime. The whiteboard in the break room is still there, but it now counts the days since the last unplanned stoppage.