Supervised Contrastive Loss

[waleeattia]

Reference issue

#426

Type of change

Implementing supervised contrastive loss
Adding plotting script to compare accuracies and transfer efficiencies

What does this implement/fix?

Implementing contrastive loss explicitly learns the progressive learning network transformer by penalizing samples of different classes that are close to one another. The new script enables two dnn algorithms to be compared by plotting the difference between their accuracies and transfer efficiencies. The accuracy of the supervised contrastive loss version improves by 6 percent compared to the PL network with categorical cross entropy.

Additional information

NDD 2021

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.09%. Comparing base (634d4d1) to head (43b05f7).
⚠️ Report is 112 commits behind head on staging.

Additional details and impacted files

@@           Coverage Diff            @@
##           staging     #518   +/-   ##
========================================
  Coverage    90.09%   90.09%           
========================================
  Files            7        7           
  Lines          404      404           
========================================
  Hits           364      364           
  Misses          40       40           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jdey4]

@rflperry Does this PR help your query about contrastive loss?

[rflperry]

Yeah seems like it matches my results here that the transfer ability goes down, which I find interesting but the reason why I'm still a bit intrigued by. Not really worth adding if just always worse? I forget why I had multiple different results with different labels.

[rflperry]

My takeaways/summary:

• Since the decider is k-Nearest Neighbors, we want the learned (penultimate) representation to place samples of the same class close together.
• Contrastive loss learns representations that are close together, and this is validated as we see higher accuracy from our kNN classifier. Softmax worked, but wasn't explicitly tuned to learn what we wanted (see embedding results for various losses here). In a way, the best loss would be a function of the network and decider together.
• One slightly odd thing is that the difference in accuracy is non-monotonic (i.e. goes down then up). Maybe just a result of not running enough simulations?
• Despite the accuracy going up, the transfer efficiencies are slightly worse. I'm a bit fuzzy on the details of the transfer efficiency metric, but potentially the learned embeddings are not good for OOD performance (this has been observed in various learned embedding algorithms like tSNE I believe)

[PSSF23]

@waleeattia Put figure in correct folder & save in pdf format.

[waleeattia]

@PSSF23 fixed!

[PSSF23]

Remove commented code & unnecessary prints. After these LGTM.

[waleeattia]

@PSSF23 Perfect, just made those changes. Thank you!

[PSSF23]

Still some commented code remaining in benchmarks/cifar_exp/plot_compare_two_algos.py @waleeattia

[waleeattia]

@PSSF23 Sorry I missed that, it should be good now.