Big Data and Data Analytics

Big Data and Data Analytics

Turning raw records into useful decisions

Big data refers to datasets whose volume, arrival rate, variety or variability calls for a scalable architecture to store and process them effectively. Data analytics is the work of examining data to identify relationships, problems and possible decisions. One describes the data and processing challenge; the other describes how useful information is extracted.

Factory measurements, online-store orders and records from IoT devices can all support analytical work. Artificial intelligence and machine learning may be involved, but not every analysis requires AI, and not every AI application requires big data.

Editor’s note: there is no universal terabyte threshold that makes a dataset “big.” What matters is whether it can be processed within the required time and cost. Extracting a useful finding from a small measurement table is still data analytics; collecting large numbers of records is not, by itself, analysis.

Why volume, velocity and variety matter

Volume is the amount of data to retain and query. Velocity concerns how quickly data arrives and needs to be processed; not every application requires an immediate result. Variety covers tables alongside text, images, audio and machine logs. Semi-structured data, such as JSON event records, also belongs in this picture.

Data trustworthiness and business value are also common evaluation dimensions. Recording a sensor’s incorrect reading millions of times does not make it accurate. The usefulness of the data depends on the question it can answer and the decision it can improve.

Distinguishing four analytical approaches

Descriptive analytics — What happened? Summarises past activity. Examples include downtime by shift, daily energy consumption and the proportion of rejected products. Charts and summary tables make the situation visible.

Diagnostic analytics — Why did it happen? Investigates which products, operating conditions or events are associated with increased downtime. Two variables moving together do not, by themselves, prove that one causes the other.

Predictive analytics — What might happen? Uses historical and current data to estimate future demand, quality problems or failure risk. The result is an estimate based on data and assumptions, rather than certain knowledge of the future.

Prescriptive analytics — What should be done? Combines predictions with constraints such as maintenance time, production requirements and cost to suggest an action. “Failure risk is increasing” is a prediction; recommending a particular maintenance window is a proposed decision.

Exploratory data analysis examines distributions, unusual values and unexpected relationships. It does not replace these four approaches with a separate mandatory classification; it can support different analytical tasks throughout the investigation.

The journey of industrial data

In production, data exchanged between PLCs and SCADA, SCADA historical records, maintenance forms and quality measurements can be examined together. IIoT connections add further sources. Values alone are insufficient: retain timestamps, engineering units, device identifiers and operating states as well.

Data is collected, checked, stored appropriately and prepared for analysis. Batch processing handles a bounded dataset, such as a completed production shift. Stream processing handles events that continue to arrive. Systems such as Apache Flink can process both forms; distributed infrastructure and cloud deployment are not compulsory for every project.

A simple volume calculation: recording 1,000 measurement points once per second produces 86.4 million records per day. Assuming just 16 bytes per record gives approximately 1.38 GB of raw data per day. This example excludes indexes, additional fields, backups and replication. Sampling intervals and retention periods directly affect the amount of data stored.

A motor example: rising current does not necessarily mean failure

An increase in motor current may reflect a heavier load, a speed change or mechanical friction. Current becomes more informative when examined alongside speed, temperature, vibration and maintenance history. This illustrates data analytics in production. The same data may feed a digital twin, although adding charts to a dashboard does not create a digital twin by itself.

Testing a predictive model on the same records used to develop it can exaggerate its apparent success. Assess it using a separate test period, a simple baseline, false alarms and missed events. Success is measured by how usefully the model performs in the field.

Security, privacy and anonymisation

Industrial cybersecurity should cover the entire data pipeline. Consider access permissions, encryption, access logs, backups and retention limits together. If connected-device security is neglected, an otherwise functioning analytics platform may receive incorrect or manipulated data.

Adding employee names, locations or shift identifiers to machine data can introduce personal privacy risks. Pseudonymisation can reduce risk by keeping identifying information separate; it is different from anonymisation. Removing a name is insufficient if other information can identify the person again. Assessing anonymisation also requires considering whether records can be linked back to individuals through other datasets.

Separating surveillance concerns from unsupported claims

The potential use of data collection for surveillance, profiling or unfair classification is a real issue. Dismissing security and privacy concerns as ignorance is therefore unhelpful. Claims that a system has a hidden intention or knows everything still require evidence. Examine what is collected, who can access it, how long it is retained and how decisions can be reviewed.

Define the question first

Clarify whether the aim is to reduce downtime, explain quality losses or track energy consumption per product. An early step in digital transformation is defining the problem and the measurements it requires. Success criteria should describe a concrete improvement, such as less downtime, lower scrap or reliable early detection. Big data is a tool serving these objectives within Industry 4.0 projects.