94691 Deep Learning

Warning: The information on this page is indicative. The subject outline for a particular session, location and mode of offering is the authoritative source of all information about the subject for that offering. Required texts, recommended texts and references in particular are likely to change. Students will be provided with a subject outline once they enrol in the subject.

Subject handbook information prior to 2024 is available in the Archives.

Description

This subject gives students a holistic understanding of deep learning, based on readings of research papers, coding exercises and practical assignments. Students become familiar with the underlying principles of deep learning neural networks, as well as with commonly used architectures and their applications. On completion of the subject, students are able to build, test and deploy deep learning models using industry standard software.

Subject learning objectives (SLOs)

Course intended learning outcomes (CILOs)

This subject contributes specifically to the development of the following course intended learning outcomes:

• Explore and test models and generalisations for describing the behaviour of sociotechnical systems and selecting data sources, taking into account the needs and values of different contexts and stakeholders (1.2)
• Use transdisciplinary approaches to seeing and doing to uncover underrepresented, or misrepresented, elements of a system (1.4)
• Critique contemporary trends and theoretical frameworks in data science for relevance to one's own practice (2.1)
• Apply and assess data science concepts, theories, practices and tools for designing and managing data discovery investigations in professional environments that draw upon diverse data sources, including efforts to shed light on underrepresented components (2.4)
• Develop, test, justify and deliver data project propositions, methodologies, analytics outcomes and recommendations for informing decision-making, both to specialist and non-specialist audiences (4.3)
• Engage in active, reflective practice that supports flexible navigation of assumptions, alternatives and uncertainty in professional data science contexts (5.1)

Contribution to the development of graduate attributes

The subject gives students a practical introduction to Deep Learning with a focus on applications. Popular architectures are covered and students are given opportunities to apply and interpret them in realistic settings. Technical skills are developed through practical coding labs and assessment tasks. As this area is developing rapidly, students are also encouraged to develop self-learning skills via assessment tasks that require them to research and assimilate recent and historical papers on the topic.

The subject addresses the following graduate attributes (GA):

GA 1 Sociotechnical systems thinking

GA 2 Creative, analytical and rigorous sense making

GA 3 Create value in problem solving and inquiry

GA 4 Persuasive and robust communication

Teaching and learning strategies

This subject is conducted in 7 sessions with weekly activities & readings assigned between classes. The classes are divided into a lecture component and a ‘lab’ component. Each comprising 1-1.5 hours depending on the content for that particular week. The lab components themselves involve two types of activities:

1) 'Code together’ sessions in which the instructor and students build understanding through collaboratively coding solutions to problems or implementing the theoretical content of the week.

2) Practical coding tasks for students to complete themselves and in small groups.

Assignments are a mix of practical coding exercises, report writing (to a business stakeholder) and literature reviews. This means that students get exposed to historical and current academic research in the area while developing tangible skills to implement these technologies and communicate highly technical matters to nonspecialist business audiences.

Due to the rapidly advancing nature of this field it is critical for students to develop skills in quickly absorbing, dissecting and understanding academic research and their value to business problems.

Assessment

Assessment task 1: Assessment task 1: Building Neural Network Architectures

Assessment task 2: Assessment task 2: Deep learning research & its industrial applications

Assessment task 3: Assessment task 3: Deep Learning final project

References

• Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv:1406.1078 [cs, stat].

• Clevert, D.-A., Unterthiner, T., Hochreiter, S., 2015. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). arXiv:1511.07289 [cs].

• Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L., 2009. ImageNet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition. Presented at the 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. https://doi.org/10.1109/CVPR.2009.5206848

• Glorot, X., Bengio, Y., n.d. Understanding the dif?culty of training deep feedforward neural networks 8.

• Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y., 2013. Maxout Networks. arXiv:1302.4389 [cs, stat].

• Greff, K., Srivastava, R.K., Koutník, J., Steunebrink, B.R., Schmidhuber, J., 2017. LSTM: A Search Space Odyssey. IEEE Transactions on Neural Networks and Learning Systems 28, 2222–2232. https://doi.org/10.1109/TNNLS.2016.2582924

• He, K., Zhang, X., Ren, S., Sun, J., 2015a. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs].

• He, K., Zhang, X., Ren, S., Sun, J., 2015b. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. arXiv:1502.01852 [cs].

• Hochreiter, S., Schmidhuber, J., 1997. Long Short-Term Memory. Neural Comput. 9, 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

• Ioffe, S., Szegedy, C., 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03167 [cs].

• Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D., Bernstein, M., Fei-Fei, L., 2015. Image Retrieval Using Scene Graphs. Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3668–3678.

• Kessy, A., Lewin, A., Strimmer, K., 2018. Optimal whitening and decorrelation. The American Statistician 72, 309–314. https://doi.org/10.1080/00031305.2016.1277159

• Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. ImageNet Classification with Deep Convolutional Neural Networks, in: Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12. Curran Associates Inc., USA, pp. 1097–1105.

• Lecun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 2278–2324. https://doi.org/10.1109/5.726791

• Lipton, Z.C., Berkowitz, J., Elkan, C., 2015. A Critical Review of Recurrent Neural Networks for Sequence Learning. arXiv:1506.00019 [cs].

• Maas, A.L., Hannun, A.Y., Ng, A.Y., n.d. Recti?er Nonlinearities Improve Neural Network Acoustic Models 6.

• Rumelhart, D.E., Hinton, G.E., Williams, R.J., 1986. Learning representations by back-propagating errors. Nature 323, 533–536. https://doi.org/10.1038/323533a0

• Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., Bengio, Y., 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. arXiv:1502.03044 [cs].