What this project is about
SplitCheck is a small command-line check for a common evaluation error: the same text or source group appearing in both training and test data. A model can appear to handle new examples while recognizing repeated material or patterns from a shared campaign. The tool checks these overlaps before the test results are treated as evidence of performance on independent data.
Its scope is narrower than detecting every kind of data leakage, where information from the test set improperly influences training. It compares standardized text and optional group identifiers. It does not detect paraphrases, translations, time-related leakage or every edited copy. Choosing a meaningful group—sender, campaign, patient or organisation—requires knowledge of the dataset and intended use.
How it works
- Train / validation / test splits
- Text + source-group overlap checks
- Inspectable findings
Check the input files
The loader accepts configurable ID and text columns, supports UTF-8 CSV files including those with an encoding marker, and rejects empty files or missing required columns. Each file must have non-empty, unique IDs. Reported overlaps can therefore be traced to specific rows instead of ambiguous duplicate identifiers.
Standardize text for comparison
Normalization removes capitalization differences, keeps sequences of ASCII letters and numbers, and joins them with single spaces. Case and punctuation changes therefore do not hide otherwise identical messages. A lookup of standardized training text allows each matching test record to be reported with a corresponding training-record ID. CSV rows are now streamed rather than retaining both complete files. Training-text lookups and identifier/group sets still grow with the input, so streaming does not make memory use constant.
Check shared source groups
An optional group column must exist in both files. The tool finds non-empty group identifiers shared between them and returns a sorted list alongside text matches. Its JSON report includes row counts, overlap evidence and a pass flag. Either kind of overlap produces a failure exit status for automated checks.
The important design choices
Keep matching rules explicit
Use fixed text-processing rules rather than a model that estimates similarity in meaning.
The trade-offResults are easy to reproduce, but restricting tokens to ASCII letters and numbers needs care for multilingual or non-Latin text. Removing punctuation can also erase meaningful differences.
Report, do not silently repair
Identify overlapping records and groups without deleting or moving rows.
The trade-offThe user must decide how to rebuild the split, but the tool does not silently change the dataset or introduce a new evaluation policy.
Find a repeated campaign
A training CSV and test CSV contain differently capitalized copies of a message and share a campaign identifier.
- Run the audit with the dataset's text, ID and group column names.
- Inspect the returned record pair and shared campaign rather than only the pass flag.
- Rebuild the split using groups appropriate to the intended use, then rerun the check.
What this showsAn automated integration check can stop an evaluation with known overlap, while leaving the data-repair decision explicit.
Illustrative example, not an additional measured result.
How it was tested—and what it shows
The checks
Tests included in the repository cover ignoring case and punctuation, finding both text and campaign overlap, and rejecting duplicate IDs. The README example deliberately contains one repeated message and one shared campaign, so it should exit with status 1. These examples check intended behaviour; they do not establish a measured detection rate for real-world data leakage.
What to take from the results
The output is a repeatable audit, not proof that a dataset has no leakage. Passing means only that these standardized-text and optional group checks found no overlap. The processing rules and meaning of each group still need review. Missing group values, similar messages, time order and preprocessing learned from test data all require separate checks.
What to keep in mind
- Does not detect paraphrases, translations or substantially edited copies.
- Memory still grows with distinct indexed texts, identifiers, groups and reported matches.
- Text normalisation and the choice of source group must suit the dataset.
Technical notes & source
Built with
Python · data validation · leakage checks
Terms used in this project
- Data leakage
- Information from the test data influencing training or evaluation in a way that makes the test misleading.
- Group split
- Keep related records, such as messages from the same campaign, on one side of the training/test boundary.