What this project is about
Combining image features with information from a patient's record can improve a prediction while creating a dependency on fields that may be missing. This pilot asks how much structured information helps six-class skin-lesion classification, and whether training with missing fields makes the combined model more robust.
The study uses PAD-UFES-20 and a pretrained ImageNet ResNet-18 image model whose parameters remain fixed. A metadata-only experiment covers 2,298 images from 1,373 patients; a separate 452-image pilot compares image and combined models on 90 held-out images. The contribution is the feature extraction, patient-grouped evaluation and missing-data experiment around established resources. This is a research pilot, not a clinical diagnostic system.
How it works
- Image + recorded patient fields
- Combine model inputs
- Test with missing fields
Keep each patient's images in one data group
Patient identifiers determine which examples can enter training or testing, so images from the same patient cannot appear on both sides. The full metadata experiment holds out one of five groups formed to keep patients together while preserving the class mix, producing 1,838 training and 460 test rows. The image pilot applies the same grouping procedure to its smaller population. Identifiers and the biopsy indicator are excluded from prediction inputs.
Combine image information with record fields
Missing numerical fields are filled and rescaled; categorical values become numerical indicator columns. ResNet-18 converts each image into a summary of 512 numerical features, called an embedding. Logistic regression combines these features, the record fields, or both, with weighting to account for class imbalance. Keeping the image model fixed limits compute and focuses the comparison on the information supplied.
Train and test with unavailable fields
The experiment independently hides 0%, 30%, 50% or 70% of test metadata fields. A second fusion model is trained with additional copies that hide 30% and 60% of fields. This is metadata-dropout training. It tests whether training with incomplete records improves performance when fields disappear at test time. Saved data assignments, settings and metric tables record the completed conditions.
The important design choices
Group by patient rather than image
Place every record from a patient in the same partition.
The trade-offThis reduces leakage from seeing the same patient during training and testing. One test group still gives limited evidence of generalisation and does not replace an external patient cohort.
Keep the full and pilot datasets separate
Report the full metadata-only study and smaller image–metadata study as distinct comparisons.
The trade-offTheir scores cannot be compared as if they tested the same images and patients.
Show the cost of missing-data training
Report performance with complete and incomplete records for ordinary and dropout-trained fusion.
The trade-offRobustness training can reduce performance when every field is available. Showing only severe missingness would hide that cost.
Change record availability while keeping the image fixed
Take an illustrative test image and its known label, then change which record fields reach the classifier.
- Evaluate ordinary fusion using the complete record.
- Hide a specified set of fields and apply the unchanged data preparation and classifier.
- Evaluate dropout-trained fusion on that same incomplete record, while retaining its result on the complete version.
What this showsUsing the same image separates missing-field sensitivity from a change in the test population. Keeping both results also exposes the cost of robustness training on complete records.
Illustrative example, not an additional measured result.
How it was tested—and what it shows
The checks
The three image-based approaches use the same 90 test images. Macro-F1 balances missed and incorrect class predictions while giving each diagnosis equal weight; balanced accuracy also weights classes equally. AUROC examines how well scores distinguish diagnoses, while Brier and calibration-error measures assess probability quality. Saved files retain exact sample counts and the patient-grouped assignments.
The pilot uses one seed, one held-out patient group and a generic pretrained image model. Fields disappear independently in the simulation, whereas real missing records may follow clinical workflows or patient characteristics. The report does not establish statistical significance, reliability for patient subgroups or transfer to a clinic. The full metadata test and smaller image test remain separate.
| Missing fields | Ordinary fusion macro-F1 | Dropout fusion macro-F1 |
|---|---|---|
| 0% | 0.550 | 0.500 |
| 30% | 0.535 | 0.460 |
| 50% | 0.486 | 0.491 |
| 70% | 0.424 | 0.464 |
What to take from the results
On the same image pilot, combining metadata with images increased macro-F1 from 0.460 to 0.550 and macro AUROC from 0.795 to 0.844. Dropout-trained fusion achieved 0.500 macro-F1 on complete records. With 70% of metadata missing, it achieved 0.464 versus 0.424 for ordinary fusion: improved performance under severe missingness at a cost on complete records.
The benefit varied with missingness. Dropout training performed worse at 30% masking and only slightly better at 50%. In the separate full-data study, metadata-only macro-F1 fell from 0.586 to 0.196 as missing fields increased to 70%. These results show why input availability belongs in the evaluation, without establishing that one fusion approach is generally superior.
What to keep in mind
- The image/fusion results are from a single seed and fold with 90 held-out images.
- Missing information was simulated; real clinical missingness can follow different patterns.
- These are research results, not evidence of diagnostic safety or clinical usefulness. A local write-up is not a peer-reviewed publication.
Technical notes & source
Built with
Python · PAD-UFES-20 · image features · tabular data
Terms used in this project
- Fusion
- Combining different kinds of input, such as an image and a table of recorded facts.
- Macro-F1
- An average classification score that gives each class equal weight.
- Metadata dropout
- Deliberately hide some recorded fields during training so the model practises working without them.
Read the source material
Local research · available on request. This page shares a summary, not private source code or raw research files.