I’d always wondered if there’s a detectable difference between how men and women write—not the topics they choose, but something in the prose itself. So I built a web application to find out.
The app uses a machine learning text classifier trained on a corpus of over 10,000 text entries split 50/50 male/female.
Once trained, I tested its predictive power against text produced by 400 participants. Each participant wrote at least 200 characters, and the classifier output a percentage confidence for the author’s gender. For example, it might say a text has a 60% chance of having a male author and 40% chance of having a female author.
I then checked the predictions against participants’ self-reported gender. The short version: it works better than chance, but not by a huge margin. Here are the details:
N.B. Tolerance: the classifier outputs its guesses as a percentage of certainty. The certainty has to be above the tolerance level to be deemed “classified”. If the certainty is under the tolerance level, the classifier isn’t sure enough for the text to be considered classified.
| Classified % | Unclassified % | Total % | |
|---|---|---|---|
| Classified to gender (Tolerance set at 65% (n = 400)) | 64 | 36 | 100 |
| Correct % | Incorrect % | Total % | |
|---|---|---|---|
| Classified gender correctly (Tolerance set at 65% (n = 256)) | 66 | 33 | 100 |
The results show that at the set tolerance level the app classified two-thirds of participants, and of those it classified, it was correct two-thirds of the time. That’s 66% accuracy versus 50% for random guessing—not revolutionary, but meaningfully better than chance. The percentage correct also matched the set tolerance, which suggests the classifier’s confidence scores are well-calibrated: when it says a passage has a 67% chance of being written by a female, that 67% is a valid figure.
This is just the beginning, and we can expect lots more of this sort of approach in the future. But a few caveats:
- I’m assuming participants reported their gender accurately—which they may not have. (After all, how valid are survey responses?)
- The classifier was trained on blog posts, because I didn’t have enough first-hand participants available. There may be stylistic differences between blog writing and the text participants submitted, which would reduce accuracy.
- This was a quick experiment, my first step into this area. With a bit of tweaking and better training data, I’m confident the accuracy could improve.
What I don’t know is what the classifier is actually detecting. Is it vocabulary? Sentence structure? Some other pattern I haven’t thought of? The model is a black box, and that’s both the appeal and the limitation of machine learning approaches.
If you can think of a good application for automated text classification, let me know. I’d love to build something genuinely useful out of this.