CSC367 Lecture Notes: Part 5
Distributed Systems and Message Passing, MPI

See all lecture notes here.

Key Properties of Distributed Networks

To write good distributed programs, we need to understand some key properties of distributed networks.

Network Topology

Message Passing

Types of Operations

MPI Library

Setup and Teardown

To execute a program with MPI, you must call it with mpirun, passing in the MPI arguments before your program's arguments.

int MPI_init(int *argc, char ***argv)

At the beginning, you must call MPI_init to initialize MPI. After this call, argc and argv will be modified to be the same as if you had called your program normally in serial. MPI_init returns MPI_SUCCESS on success, otherwise it will return an error code.

int MPI_finalize(void)

At the end of main, you must call MPI_finalize to finalize MPI. This will free up any resources that MPI is using. MPI_finalize returns MPI_SUCCESS on success, otherwise it will return an error code. Note that no MPI calls are allowed after this, including MPI_init.

Note that all processes must call MPI_init and MPI_finalize, otherwise there will be undefined behavior.

Communicators

A MPI communication domains is a set of processes that can communicate with each other. MPI_Comm type variables called communicators store info about communication domains. The default communicator is MPI_COMM_WORLD, which includes all processes.

In some cases, we want to perform tasks in disjoint groups of processes, for which we must define custom communicators. MPI provides functions to retrieve the communicator size and the id of the current process.

int MPI_Comm_size(MPI_Comm comm, int *size)

This function returns the number of processes in the communicator comm by reference in size.

int MPI_Comm_rank(MPI_Comm comm, int *rank)

This function returns the rank of the current process in the communicator comm by reference in rank. The rank is a unique integer in the range [0, size) for each communicator.

Here is an example of a "Hello World" function using these functions.

#include "mpi.h"
#include <stdio.h>

int main(int argc, char *argv[])
{
  int rank, size;
  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  printf("Hello world from process %d of %d\n", rank, size);
  MPI_Finalize();
  return 0;
}

Timing

Timing in MPI can be done with the MPI_Wtime function. This function returns the number of seconds since some arbitrary point in the past as a double. This function is guaranteed to be monotonic, so it can be used to measure elapsed time.

Variable Types for Communication

As the systems in the network may have different byte order (endian-ness), MPI has its own types for communication so the byte order is kept consistent internally by MPI. The types (and their corresponding C types) are:

MPI Type	C Type
MPI_CHAR	signed char
MPI_SHORT	short
MPI_INT	int
MPI_LONG	long
MPI_UNSIGNED_CHAR	unsigned char
MPI_UNSIGNED_SHORT	unsigned short
MPI_UNSIGNED	unsigned int
MPI_UNSIGNED_LONG	unsigned long
MPI_FLOAT	float
MPI_DOUBLE	double
MPI_BYTE	N/A
MPI_PACKED	N/A

Part 6 and Onwards

See Part 6.