Define the prediction problem first
Supervised machine learning learns a predictive rule from input features and known targets. For example, synthetic sensor readings can be inputs for predicting device state. Features must be available at prediction time; the target is the state to predict. Including a maintenance conclusion available only afterward changes the problem.
State whether deployment concerns new times on known devices or entirely unseen devices. This changes the split unit, metrics and the scope of generalization claims. No percentage split can replace that problem definition.
What each dataset is for
Training data fits the model. Validation data supports development choices. Test data is reserved for final evaluation. Repeatedly changing the model in response to test scores makes test information part of selection, so it is no longer an entirely independent final check.
Duplicates or closely related samples from one source may cross split boundaries. Preprocessing that estimates parameters from data should fit them only on training data and apply the same transformation elsewhere. Plan splitting and preprocessing together.
Further reading: Google ML Crash Course: Dividing datasets
Align group splitting with the question
For unseen-device evaluation, avoid placing data from the same device in both training and test sets. For future-time prediction, chronology also matters. Group splitting is not universally the only correct method; it is a design matched to a specific generalization question.
The six rows below demonstrate overlap checking only. They train no model and are insufficient to estimate real-world performance. Actual research still needs adequate sample size, representativeness, label quality and uncertainty in its metrics.
Example: three devices with two rows each
Rows are A1, A2, B1, B2, C1 and C2. Putting alternating rows into train and test places all three devices on both sides.
That split may concern new records from known devices, but does not directly answer the unseen-device question. Assigning A to train, B to validation and C to test demonstrates group separation; one test device still does not establish broad generalization.
Conclusion: Match the split to the question before discussing scores.
Offline exercise: detect group overlap
Use Python 3 and six synthetic IDs defined in the program. No data downloads, personal data or model training are involved.
- Read the code and predict naive_overlap and grouped_overlap.
- Run python3 split_demo.py and check both output lines.
- State the additional split condition needed if deployment changes to future times on the same device.
rows = [(device, sample) for device in "ABC" for sample in (1, 2)]
naive_train = {d for d, _ in rows[::2]}
naive_test = {d for d, _ in rows[1::2]}
train, validation, test = {"A"}, {"B"}, {"C"}
print("naive_overlap=" + repr(sorted(naive_train & naive_test)))
print("grouped_overlap=" + repr(sorted((train & validation) | (train & test) | (validation & test))))Expected result
naive_overlap=['A', 'B', 'C']; grouped_overlap=[]. The second line establishes only that the example groups do not overlap.
Check your work
Why does no overlap not prove model accuracy? Identify the step this exercise never performed.
Finish and clean up
The program creates no data files. Keep the output and your explanation.