Transfer Learning Theory Reading Group

Location: 1102

Time: Saturday 2pm (every other week)

In this bi-weekly reading group, we will read classic domain adaptation theory papers discussed in the following textbook:

Advances in Domain Adaptation Theory (ADAT), Redko

Through the readings, we hope to get a basic understanding of how and why domain adaptation algorithms work fundamentally, and in what conditions they would not work.

• ADAT chapter 1 (Learning theory background)
• ADAT chapter 2 (Domain adaptation background)

• ADAT chapter 3.1-3.4
• Ben-David, Shai, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. “A theory of learning from different domains.” Machine learning 79, no. 1 (2010): 151-175 (defines the HΔH-divergence, a preliminary work was published in NIPS 2007)

• ADAT chapter 3.5.1-3.5.2
• Mansour, Yishay, Mehryar Mohri, and Afshin Rostamizadeh. “Domain adaptation: Learning bounds and algorithms.” arXiv preprint arXiv:0902.3430 (2009).improved generalization bound using discrepancy distance

• ADAT chapter 3.5.3 a discrepancy distance based generalization bound for regression problems.
• Cortes, Corinna, and Mehryar Mohri. “Domain adaptation in regression.” In International Conference on Algorithmic Learning Theory, pp. 308-323. Springer, Berlin, Heidelberg, 2011.
• See also:
  • Cortes, Corinna, Mehryar Mohri, and Andrés Munoz Medina. “Adaptation based on generalized discrepancy.” The Journal of Machine Learning Research 20, no. 1 (2019): 1-30.
  • Maurer, Andreas. “Transfer bounds for linear feature learning.” Machine learning 75, no. 3 (2009): 327-350.

• ADAT Chapter 4.1-4.2
• David, Shai Ben, Tyler Lu, Teresa Luu, and Dávid Pál. “Impossibility theorems for domain adaptation.” In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 129-136. JMLR Workshop and Conference Proceedings, 2010. (using the HΔH-divergence)

• ADAT Chapter 4.3-4.4
• Ben-David, Shai, and Ruth Urner. “On the hardness of domain adaptation and the utility of unlabeled target samples.” In International Conference on Algorithmic Learning Theory, pp. 139-153. Springer, Berlin, Heidelberg, 2012.

• ADAT Chapter 5.1-5.3
• Redko, Ievgen, Amaury Habrard, and Marc Sebban. “Theoretical analysis of domain adaptation with optimal transport.” In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 737-753. Springer, Cham, 2017.

• Baxter, Jonathan. “A model of inductive bias learning.” Journal of artificial intelligence research 12 (2000): 149-198. (Show how multi-task learning can improve generalization, assuming the target task is embedded within an environment of related tasks.)
• ERM-based Multi-source Transfer Learning (recent work by Xinyi on the sample complexity of multi-source transfer learning)