KwickClips Artificial Intelligence · 42 sec · free
Why does 'the' score zero?
Short answer. It is in every document, so log(1) = 0.
What is the TF-IDF formula?
TF x log(N/DF).
TF-IDF, N = 3
| Word | DF | IDF |
|---|---|---|
| the | 3 | log(1) = 0 |
| cricket | 1 | log(3) = 0.477 |
Remember
| TF x log(N/DF) |
| Rare words score high |
Why does the word the score zero in TF-IDF? TF-IDF is TF times log of N by DF. The is in all three documents, so zero. Cricket is rare, so higher. Remember. TF times log N by DF. Rare words score high.
This clip is from the full lesson: Text Processing: Normalisation, Tokenisation, Bag of Words and TF-IDF — 8 minutes, with the tables, the quick answers and the whole lesson in text.
Useful for: CBSE Class 10 Artificial Intelligence (417), CBSE Class 10 Artificial Intelligence (417), CBSE Class 10 Artificial Intelligence (417)
More KwickClips from this lesson
What comes after sentence segmentation?46 sec
What is the stem of 'studies'?42 sec
What is the first step?38 secVoice-over is AI-generated; the script is written and checked by Kajal Ma'am. Confirm anything you plan around against your official board document. We never ask for a password or an OTP.