Skip to main content
B.Eng DegreeFaculty of Faculty of Engineering & TechnologySoftware Engineering

SWE 310: Data Engineering & Scalable Data Pipelines

Architectural principles of large-scale data engineering. Batch vs stream processing, Extract-Transform-Load (ETL) and ELT pipelines, distributed compute frameworks (Apache Spark), distributed messaging, data lakehouse architectures (Apache Parquet, ...

Instructor
Created by PAMCET Learning Team
intermediate2 - 4 weeksEnglish

Course Overview

Architectural principles of large-scale data engineering. Batch vs stream processing, Extract-Transform-Load (ETL) and ELT pipelines, distributed compute frameworks (Apache Spark), distributed messaging, data lakehouse architectures (Apache Parquet, ... Curriculum Structure &...

What You'll Learn

  • Architect resilient, fault-tolerant batch and streaming data pipelines processing high-volume event streams.
  • Write distributed dataset transformations using Apache Spark DataFrames and PySpark.
  • Orchestrate multi-stage data dependencies using Apache Airflow Directed Acyclic Graphs (DAGs).
  • Manage evolving data schemas using Apache Avro and Schema Registry without breaking consumers.
  • Implement automated data quality verification gates using Great Expectations.

Requirements

  • SWE 205
  • SWE 210

Degree Programme Curriculum Context

This course is a curriculum unit of the Bachelor of Engineering (B.Eng.) in Software Engineering degree framework in Nigeria.

Earn Your Certificate

Showcase your skills with a certificate! Complete the course, submit your project, and earn your certificate. Here's a sample of what you'll receive to show off your accomplishments.

course-certificate
Lecture Board & Syllabus

Course Curriculum

Browse complete theoretical foundations, worked calculations, laboratory protocols, and board assessment questions under each topic.

Launch Course Reader →
8 Chapters
•
16 Lecture
•
24h
Subtopic 1Document LectureOpen Access~18 min read

Data Engineering Foundations: OLTP vs OLAP, Data Warehouses, Lakes & Lakehouses: Theoretical Foundations, System Architecture & Subtopic Deep-Dive

SWE 310 • 300L • Software Systems Infrastructure Chapter 1: Data Engineering Foundations: OLTP vs OLAP, Data Warehouses, Lakes & Lakehouses — Theoretical Foundations, System Architecture & Analytical Modeling Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Benchmark: IEEE/ACM CS2023-IM-ENG 1. Executive Overvie...

Module 1 • Subtopic 1•14,665 characters of authentic academic content
Subtopic 2Document LectureOpen Access~8 min read

Data Engineering Foundations: OLTP vs OLAP, Data Warehouses, Lakes & Lakehouses: Worked Engineering Problems, Production Code & Simulation Protocols

SWE 310 • 300L • Software Systems Infrastructure Chapter 1: Data Engineering Foundations: OLTP vs OLAP, Data Warehouses, Lakes & Lakehouses — Applied Engineering Computation, Production Systems & Verification Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Standard: IEEE / ACM SE Practice Guidelines 1. Laborat...

Module 1 • Subtopic 2•6,747 characters of authentic academic content
Formative Assessment10 Points • 15 Mins

Chapter 1: Data Engineering Foundations: OLTP vs OLAP, Data Warehouses, Lakes & Lakehouses — Formative Engineering Quiz

Assessment Protocol:10 Total Points•70% Passing Threshold•15 Mins Time Limit•Bloom's Taxonomy: Analyze & Evaluate

Rigorous formative evaluation assessing theoretical knowledge, mathematical formulations, and engineering principles for Data Engineering Foundations: OLTP vs OLAP, Data Warehouses, Lakes & Lakehouses.

Module 1 • Formative Assessment
Subtopic 1Document LectureOpen Access~19 min read

Batch vs Stream Processing Paradigms: Lambda vs Kappa Architectures: Theoretical Foundations, System Architecture & Subtopic Deep-Dive

SWE 310 • 300L • Software Systems Infrastructure Chapter 2: Batch vs Stream Processing Paradigms: Lambda vs Kappa Architectures — Theoretical Foundations, System Architecture & Analytical Modeling Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Benchmark: IEEE/ACM CS2023-IM-ENG 1. Executive Overview & Theoreti...

Module 2 • Subtopic 1•15,949 characters of authentic academic content
Subtopic 2Document LectureOpen Access~8 min read

Batch vs Stream Processing Paradigms: Lambda vs Kappa Architectures: Worked Engineering Problems, Production Code & Simulation Protocols

SWE 310 • 300L • Software Systems Infrastructure Chapter 2: Batch vs Stream Processing Paradigms: Lambda vs Kappa Architectures — Applied Engineering Computation, Production Systems & Verification Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Standard: IEEE / ACM SE Practice Guidelines 1. Laboratory & Comput...

Module 2 • Subtopic 2•6,687 characters of authentic academic content
Formative Assessment10 Points • 15 Mins

Chapter 2: Batch vs Stream Processing Paradigms: Lambda vs Kappa Architectures — Formative Engineering Quiz

Assessment Protocol:10 Total Points•70% Passing Threshold•15 Mins Time Limit•Bloom's Taxonomy: Analyze & Evaluate

Rigorous formative evaluation assessing theoretical knowledge, mathematical formulations, and engineering principles for Batch vs Stream Processing Paradigms: Lambda vs Kappa Architectures.

Module 2 • Formative Assessment
Subtopic 1Document LectureOpen Access~18 min read

Columnar Storage Formats: Parquet, ORC, Compression Schemes & Predicate Pushdown: Theoretical Foundations, System Architecture & Subtopic Deep-Dive

SWE 310 • 300L • Software Systems Infrastructure Chapter 3: Columnar Storage Formats: Parquet, ORC, Compression Schemes & Predicate Pushdown — Theoretical Foundations, System Architecture & Analytical Modeling Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Benchmark: IEEE/ACM CS2023-IM-ENG 1. Executive Overvi...

Module 3 • Subtopic 1•14,626 characters of authentic academic content
Subtopic 2Document LectureOpen Access~8 min read

Columnar Storage Formats: Parquet, ORC, Compression Schemes & Predicate Pushdown: Worked Engineering Problems, Production Code & Simulation Protocols

SWE 310 • 300L • Software Systems Infrastructure Chapter 3: Columnar Storage Formats: Parquet, ORC, Compression Schemes & Predicate Pushdown — Applied Engineering Computation, Production Systems & Verification Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Standard: IEEE / ACM SE Practice Guidelines 1. Labora...

Module 3 • Subtopic 2•6,751 characters of authentic academic content
Formative Assessment10 Points • 15 Mins

Chapter 3: Columnar Storage Formats: Parquet, ORC, Compression Schemes & Predicate Pushdown — Formative Engineering Quiz

Assessment Protocol:10 Total Points•70% Passing Threshold•15 Mins Time Limit•Bloom's Taxonomy: Analyze & Evaluate

Rigorous formative evaluation assessing theoretical knowledge, mathematical formulations, and engineering principles for Columnar Storage Formats: Parquet, ORC, Compression Schemes & Predicate Pushdown.

Module 3 • Formative Assessment
Subtopic 1Document LectureOpen Access~18 min read

Distributed Computing with Apache Spark: RDDs, DataFrames, Catalyst Optimizer & Shuffling: Theoretical Foundations, System Architecture & Subtopic Deep-Dive

SWE 310 • 300L • Software Systems Infrastructure Chapter 4: Distributed Computing with Apache Spark: RDDs, DataFrames, Catalyst Optimizer & Shuffling — Theoretical Foundations, System Architecture & Analytical Modeling Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Benchmark: IEEE/ACM CS2023-IM-ENG 1. Executi...

Module 4 • Subtopic 1•14,773 characters of authentic academic content
Subtopic 2Document LectureOpen Access~8 min read

Distributed Computing with Apache Spark: RDDs, DataFrames, Catalyst Optimizer & Shuffling: Worked Engineering Problems, Production Code & Simulation Protocols

SWE 310 • 300L • Software Systems Infrastructure Chapter 4: Distributed Computing with Apache Spark: RDDs, DataFrames, Catalyst Optimizer & Shuffling — Applied Engineering Computation, Production Systems & Verification Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Standard: IEEE / ACM SE Practice Guidelines...

Module 4 • Subtopic 2•6,787 characters of authentic academic content
Formative Assessment10 Points • 15 Mins

Chapter 4: Distributed Computing with Apache Spark: RDDs, DataFrames, Catalyst Optimizer & Shuffling — Formative Engineering Quiz

Assessment Protocol:10 Total Points•70% Passing Threshold•15 Mins Time Limit•Bloom's Taxonomy: Analyze & Evaluate

Rigorous formative evaluation assessing theoretical knowledge, mathematical formulations, and engineering principles for Distributed Computing with Apache Spark: RDDs, DataFrames, Catalyst Optimizer & Shuffling.

Module 4 • Formative Assessment
Subtopic 1Document LectureOpen Access~18 min read

Data Pipeline Orchestration: Workflow DAGs, Apache Airflow, Sensors & Operators: Theoretical Foundations, System Architecture & Subtopic Deep-Dive

SWE 310 • 300L • Software Systems Infrastructure Chapter 5: Data Pipeline Orchestration: Workflow DAGs, Apache Airflow, Sensors & Operators — Theoretical Foundations, System Architecture & Analytical Modeling Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Benchmark: IEEE/ACM CS2023-IM-ENG 1. Executive Overvie...

Module 5 • Subtopic 1•14,668 characters of authentic academic content
Subtopic 2Document LectureOpen Access~8 min read

Data Pipeline Orchestration: Workflow DAGs, Apache Airflow, Sensors & Operators: Worked Engineering Problems, Production Code & Simulation Protocols

SWE 310 • 300L • Software Systems Infrastructure Chapter 5: Data Pipeline Orchestration: Workflow DAGs, Apache Airflow, Sensors & Operators — Applied Engineering Computation, Production Systems & Verification Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Standard: IEEE / ACM SE Practice Guidelines 1. Laborat...

Module 5 • Subtopic 2•6,747 characters of authentic academic content
Formative Assessment10 Points • 15 Mins

Chapter 5: Data Pipeline Orchestration: Workflow DAGs, Apache Airflow, Sensors & Operators — Formative Engineering Quiz

Assessment Protocol:10 Total Points•70% Passing Threshold•15 Mins Time Limit•Bloom's Taxonomy: Analyze & Evaluate

Rigorous formative evaluation assessing theoretical knowledge, mathematical formulations, and engineering principles for Data Pipeline Orchestration: Workflow DAGs, Apache Airflow, Sensors & Operators.

Module 5 • Formative Assessment
Subtopic 1Document LectureOpen Access~18 min read

Schema Management: Schema Evolution, Confluent Schema Registry & Apache Avro: Theoretical Foundations, System Architecture & Subtopic Deep-Dive

SWE 310 • 300L • Software Systems Infrastructure Chapter 6: Schema Management: Schema Evolution, Confluent Schema Registry & Apache Avro — Theoretical Foundations, System Architecture & Analytical Modeling Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Benchmark: IEEE/ACM CS2023-IM-ENG 1. Executive Overview &...

Module 6 • Subtopic 1•14,676 characters of authentic academic content
Subtopic 2Document LectureOpen Access~8 min read

Schema Management: Schema Evolution, Confluent Schema Registry & Apache Avro: Worked Engineering Problems, Production Code & Simulation Protocols

SWE 310 • 300L • Software Systems Infrastructure Chapter 6: Schema Management: Schema Evolution, Confluent Schema Registry & Apache Avro — Applied Engineering Computation, Production Systems & Verification Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Standard: IEEE / ACM SE Practice Guidelines 1. Laboratory...

Module 6 • Subtopic 2•6,735 characters of authentic academic content
Formative Assessment10 Points • 15 Mins

Chapter 6: Schema Management: Schema Evolution, Confluent Schema Registry & Apache Avro — Formative Engineering Quiz

Assessment Protocol:10 Total Points•70% Passing Threshold•15 Mins Time Limit•Bloom's Taxonomy: Analyze & Evaluate

Rigorous formative evaluation assessing theoretical knowledge, mathematical formulations, and engineering principles for Schema Management: Schema Evolution, Confluent Schema Registry & Apache Avro.

Module 6 • Formative Assessment
Subtopic 1Document LectureOpen Access~18 min read

Stream Processing: Windowing (Tumbling, Sliding, Session), Watermarks & Exactly-Once Semantics: Theoretical Foundations, System Architecture & Subtopic Deep-Dive

SWE 310 • 300L • Software Systems Infrastructure Chapter 7: Stream Processing: Windowing (Tumbling, Sliding, Session), Watermarks & Exactly-Once Semantics — Theoretical Foundations, System Architecture & Analytical Modeling Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Benchmark: IEEE/ACM CS2023-IM-ENG 1. Ex...

Module 7 • Subtopic 1•14,658 characters of authentic academic content
Subtopic 2Document LectureOpen Access~9 min read

Stream Processing: Windowing (Tumbling, Sliding, Session), Watermarks & Exactly-Once Semantics: Worked Engineering Problems, Production Code & Simulation Protocols

SWE 310 • 300L • Software Systems Infrastructure Chapter 7: Stream Processing: Windowing (Tumbling, Sliding, Session), Watermarks & Exactly-Once Semantics — Applied Engineering Computation, Production Systems & Verification Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Standard: IEEE / ACM SE Practice Guidel...

Module 7 • Subtopic 2•6,807 characters of authentic academic content
Formative Assessment10 Points • 15 Mins

Chapter 7: Stream Processing: Windowing (Tumbling, Sliding, Session), Watermarks & Exactly-Once Semantics — Formative Engineering Quiz

Assessment Protocol:10 Total Points•70% Passing Threshold•15 Mins Time Limit•Bloom's Taxonomy: Analyze & Evaluate

Rigorous formative evaluation assessing theoretical knowledge, mathematical formulations, and engineering principles for Stream Processing: Windowing (Tumbling, Sliding, Session), Watermarks & Exactly-Once Semantics.

Module 7 • Formative Assessment
Subtopic 1Document LectureOpen Access~19 min read

Data Quality, Governance & Data Observability (Great Expectations / dbt): Theoretical Foundations, System Architecture & Subtopic Deep-Dive

SWE 310 • 300L • Software Systems Infrastructure Chapter 8: Data Quality, Governance & Data Observability (Great Expectations / dbt) — Theoretical Foundations, System Architecture & Analytical Modeling Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Benchmark: IEEE/ACM CS2023-IM-ENG 1. Executive Overview &...

Module 8 • Subtopic 1•15,545 characters of authentic academic content
Subtopic 2Document LectureOpen Access~8 min read

Data Quality, Governance & Data Observability (Great Expectations / dbt): Worked Engineering Problems, Production Code & Simulation Protocols

SWE 310 • 300L • Software Systems Infrastructure Chapter 8: Data Quality, Governance & Data Observability (Great Expectations / dbt) — Applied Engineering Computation, Production Systems & Verification Course: Data Engineering & Scalable Data Pipelines Curriculum: CCMAS-SWE-310 Accreditation: COREN-SWE-DAT-03 Standard: IEEE / ACM SE Practice Guidelines 1. Laboratory &am...

Module 8 • Subtopic 2•6,727 characters of authentic academic content
Formative Assessment10 Points • 15 Mins

Chapter 8: Data Quality, Governance & Data Observability (Great Expectations / dbt) — Formative Engineering Quiz

Assessment Protocol:10 Total Points•70% Passing Threshold•15 Mins Time Limit•Bloom's Taxonomy: Analyze & Evaluate

Rigorous formative evaluation assessing theoretical knowledge, mathematical formulations, and engineering principles for Data Quality, Governance & Data Observability (Great Expectations / dbt).

Module 8 • Formative Assessment

Instructor

PAMCET Learning Team

PAMCET Learning Team

Institutional Course Curation · PAMCET

The PAMCET Learning Team is the institutional curriculum and content group behind course publishing on this platform. PAMCET (Professional and Academic Management Centre for Education and Training) is operated by Digitpen Hub Ltd, a company registered in Lagos, Nigeria. Courses credited to the PAMCET Learning Team are structured, organized, and published centrally by the platform rather than authored by a single named individual instructor — each course's syllabus, learning outcomes, and content are reviewed before publication under this institutional account. For questions about a specific course or to report a content issue, please use the platform's Help & Support or Contact Us pages.

Qualifications:

PAMCET Learning Team — Institutional Course Curation, Digitpen Hub Ltd

My Skills:

  • Curriculum Curation
  • Course Content Review
  • Learning Outcome Design
  • Course Publishing & Quality Assurance
  • Platform Content Operations

Reviews

No reviews available
SWE 310: Data Engineering & Scalable Data Pipelines
Free$24.99100% Off
Certificate available separately for $7 after completion

Course Includes

8 Chapters
16 Lectures
2 - 4 weeks
Skill Level intermediate
Taught in English
Lifetime Access
Certification of Completion
Category: University - Software Engineering
Share On

Download Syllabus & Study Guide

Receive the full lecture syllabus, reading list, and milestone schedule for SWE 310: Data Engineering & Scalable Data Pipelines.

By submitting, you agree to receive educational updates. You can unsubscribe anytime. Read our Privacy Policy.