{"id":475,"date":"2018-02-15T21:31:59","date_gmt":"2018-02-15T18:31:59","guid":{"rendered":"https:\/\/grechka.family\/dmitry\/blog\/?p=475"},"modified":"2019-09-05T21:04:43","modified_gmt":"2019-09-05T18:04:43","slug":"birdsong-report-b1-model-classification-quality","status":"publish","type":"post","link":"https:\/\/grechka.family\/dmitry\/blog\/2018\/02\/birdsong-report-b1-model-classification-quality\/","title":{"rendered":"Birdsong.report B1 Model Classification Quality"},"content":{"rendered":"

Here are some important notices about the classification quality of B1 model currently deployed at (birdsong.report<\/a>)<\/p>\n

I made per class classification quality analysis (don’t know why I did not do it before!) and the results are both impressive and distressing.
\n<\/p>\n

The metrics are calculated on 2174 validation samples.
\nThey are not class balanced, but it is a random subset of whole available data, it reflects the distribution of classes in whole dataset.<\/p>\n

\"B1<\/a>
B1 model quality evaluation samples<\/figcaption><\/figure>\n

Part 1. Impressive results<\/h2>\n

First of all, some of the birds are classified really well.<\/p>\n

Let’s look at per-bird Precision (Positive Predictive Value) plot, which indicates how many of predicted species are indeed that species.<\/p>\n

\"B1<\/a>
B1 model quality evaluation (PPV)<\/figcaption><\/figure>\n

12 species have the values higher than 0.75<\/strong>.<\/p>\n

Among 87<\/strong> predicted nightingales, 84<\/strong> were indeed nightingales (the highest PPV=0.9655<\/strong> among all birds)<\/p>\n

Sensitivity is even more impressive for some species.<\/p>\n

\"B1<\/a>
B1 model quality evaluation (sensitivity)<\/figcaption><\/figure>\n

The skylark has the highest sensitivity value of 0.988<\/strong> (84<\/strong> out of 85<\/strong> presented skylark songs were detected)!<\/p>\n

But here comes the distressing part: the network did not detect 7 of presented species at all!
\nYou may see them having 0 sensitivity value at the lower part of the plot.
\nBut before proceeding to the discussion about these faults, let’s see the last two plots: F1-scores for different classes and the combined plot for all 3 metrics.<\/p>\n

\"B1<\/a>
B1 model quality evaluation (F1 measure)<\/figcaption><\/figure>\n
\"B1<\/a>
B1 model quality evaluation (combined)<\/figcaption><\/figure>\n

Not bad!<\/p>\n

Interesting that sensitivity is almost always higher than precision. Why is it so? Did it happen by chance?<\/p>\n

Part 2. Distressing<\/h2>\n

The network did not detect 7 known species in any of 243<\/strong> presented recordings.
\nFor instance, we can explore the Woodpecker, Coal Tit and House Sparrow. Validation set contained 70<\/strong> recordings of Great Spotted Woodpecker, 47<\/strong> of House Sparrow, 44
\n<\/strong> of Coal Tit. None of them were classified correctly.<\/p>\n

How did the network classify them?<\/p>\n

\"B1<\/a>
B1 model quality evaluation (some misclassifications)<\/figcaption><\/figure>\n

The network considered all of the Coal Tit samples as Eurasian Blue Tit or Great Tit. They are all tits so maybe it is normal for the network to be confused.
\nSimilar situation is with the House Sparrow, it is classified as Eurasian Tree Sparrow.<\/p>\n

Well, maybe it is too early to train the network to learn spices from the same family.<\/p>\n

The situation is different with the woodpecker, it is classified as Swallow. That is really strange.
\nConsider yourself:<\/p>\n