Data Science at Scale - 2025 entry

Data science and some machine learning technologies rely on large amounts of data to be effective and many commercial and scientific applications require the analysis of large quantities of heterogenous, noisy data on distributed machines. This module will examine the ways in which algorithms for data science can be implemented for large data and will discuss new algorithms specifically designed for large scale data. You will also work with large-scale distributed and cloud systems for storing and computing with big data.

Through theory and practice this module aims to equip you with an understanding of the principles of distributed computing, particularly on cloud-based systems, the ways in which data can be stored and accessed to allow efficient computation, and efficient algorithms for large-scale computation.

Distributed cloud computing will provide you with the underpinning knowledge required to develop and implement machine learning and artificial intelligence algorithms on distributed high-performance computing systems.

On successful completion of this module you should be able to:

Module Specific Skills and Knowledge:

1. Explain the common challenges encountered in large scale data science projects.

2. Demonstrate competence in applying a range of abstraction and programming models for large-scale data processing.

3. Analyse and use a range of data storage models for parallel query processing

4. Apply cloud and distributed systems for data processing.

5. Design and implement algorithms for machine learning on large scale distributed systems.

Discipline Specific Skills and Knowledge:

6. Describe several different programming paradigms and associated data structures.

7. Critically analyse and compare a variety of data science methods and their applications to real problems.

Personal and Key Transferable/ Employment Skills and Knowledge:

8. Plan and write a technical report.

9. Adapt existing technical knowledge to learning new methods.

• Introduction: the size of data and impediments to efficient computation;
• Data storage and retrieval: relational databases and NoSQL systems;
• Distributed systems and data: cloud computing and supercomputing; data distribution and consistency;
• The MapReduce paradigm and implementations;
• Algorithms for large scale learning: stochastic gradient descent, large scale linear algebra;
• Stream processing;
• Future architectures; co-design of hardware and algorithms.

Reassessment will be by coursework and/or written exam in the failed or deferred element only. For referred candidates, the module mark will be capped at 50%. For deferred candidates, the module mark will be uncapped.

Basic reading:

• Kleppmann, M., 2016. Designing data-intensive applications: the big ideas behind reliable, scalable, and maintainable systems. 1st ed. Sebastopol, CA: O'Reilly Media.
• White, T., 2015. Hadoop: the definitive guide: storage and analysis at internet scale. 4th ed. Sebastopol, CA: O'Reilly Media.
• Narkhede, N., Shapira, G. and Polino, T., 2016. Kafka: the definitive guide. 1st ed. Sebastopol, CA: O'Reilly Media.
• Chambers, B., 2018. Spark: the definitive guide. 1st ed. Sebastopol, CA: O'Reilly Media.

Web-based and electronic resources:

• ELE

Reading list for this module: