What this project is about
PhishGuard tests phishing-message classification using TF-IDF, which converts text patterns into numerical features weighted by how common they are, and logistic regression, a statistical classifier. The central question is why excellent training scores can fail on differently worded examples. The project compares word and character patterns, chooses a decision threshold without using the final challenge and keeps individual mistakes visible.
The evidence uses 240 generated training messages and 20 AI-assisted synthetic challenge messages. No external collection of real emails has been evaluated, so this is not an operational filter. Text alone omits sender identity, attachment behaviour, link-destination reputation and delivery headers—information that may be essential for determining whether an actual message is malicious.
How it works
- Keep related messages together
- Compare words and characters
- Check incoming datasets
Keep related messages together
Four-way cross-validation repeatedly trains on three groups and checks the fourth, keeping messages from the same body template together. Text features are learned afresh from each training portion. Predictions for messages excluded from that training choose the threshold; the final model then uses all training data and is tested once on the fixed challenge.
Compare words and characters
The experiment compares individual words and word pairs with sequences of three to five characters, using the same evaluation structure. It saves predictions, threshold comparisons, error lists and data for plotting performance at different thresholds. These outputs can be regenerated locally instead of relying on an isolated dashboard score.
Check incoming datasets
The runner accepts separate training and challenge CSVs with mappings for column names and labels. It rejects identical text after normalization and shared declared groups. A future real-email evaluation would still need checks for reworded duplicates, information from future messages leaking into training, and the accuracy of each source's labels.
The important design choices
Inspect unfamiliar wording
Keep a fixed challenge and review each error after selecting the threshold using training data only.
The trade-offA 20-message synthetic challenge is easy to inspect but too small and artificial to establish real deployment performance.
Keep the model inspectable
Use a classifier whose numerical weights are associated with explicit text patterns.
The trade-offThe pipeline is lightweight, but those weights alone do not prove why one prediction occurred or supply missing evidence about the sender and linked website.
Review a false alarm
A synthetic legitimate message mentions account security while directing users to their normal portal.
- Inspect the model's score and decision at the selected threshold.
- Compare the complete message with training vocabulary and available context.
- Record the error and test controlled wording changes before asserting a cause.
What this showsThe review moves beyond an overall score to examine what the model sees, how the message was labelled and which context is missing.
Illustrative example, not an additional measured result.
How it was tested—and what it shows
The checks
Both representations achieve training cross-validation F1 of 1.00 and choose threshold 0.5. F1 balances precision—how many flagged messages are labelled phishing—with recall—how many phishing-labelled messages are caught. The perfect score is a warning: subject templates and label-associated wording still recur. An earlier perfect random-split result did not establish performance on new messages. Output scores have not been calibrated to represent reliable probabilities.
What to take from the results
On the fixed challenge, word features achieve F1 .632 and ROC AUC .70; character features achieve F1 .267 and ROC AUC .41. ROC AUC measures how well scores rank phishing-labelled messages above legitimate ones across thresholds. The word model wrongly flags three legitimate messages and misses four phishing-labelled messages. Security vocabulary overlap and plausible malicious pretexts offer hypotheses to investigate, not verified explanations of these errors.
What to keep in mind
- Only 20 synthetic challenge messages were evaluated.
- No external email corpus has been tested, so real-world filtering performance is unknown.
Technical notes & source
Built with
Python · scikit-learn · TF-IDF · Streamlit
Terms used in this project
- False positive
- A normal message incorrectly flagged as phishing.
- F1 score
- A combined measure of how many flagged messages are correct and how many threats are found.