Both Pthreads and OpenMP are for writing multi-threaded programs.
MPI (Message Passing Interface) is for writing multi-process ones. This is quite a different model:
Hello world in MPI might look like this:
#include <stdio.h>
#include <mpi.h>
int main(int argc, char** argv) {
int rank, size;
/* initialize MPI */
MPI_Init(&argc, &argv);
/* get the rank (process id) and size (number of processes) */
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
/* print a hello message */
printf("Hello from process %d of %d!\n", rank, size);
/* quit MPI */
MPI_Finalize();
return 0;
}
You might notice that, unlike Pthreads or OpenMP, there is no code to create any threads/processes. That is because the entire program is run by each process.
To compile MPI programs you cannot use the standard gcc command. You must use the mpicc command:
$ mpicc hello.c
This command is a wrapper over gcc that adds the necessary compiler and linker flags.
We can then run our program in the standard way:
$ ./a.out
However that runs it only as one process!
In order to launch multiple processes, we use the mpirun command:
$ mpirun -np 8 ./a.out
This tells MPI to launch 8 processes running our program, and give them each a rank.
If you run MPI and get warnings about missing network interfaces such as:
-------------------------------------------------------------------------- [[35292,1],0]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: OpenFabrics (openib) Host: cs Another transport will be used instead, although this may result in lower performance. --------------------------------------------------------------------------
Then add the following to your .bash_profile:
export OMPI_MCA_btl_base_warn_component_unused=0
We can also use MPI with C++ programs.
#include <iostream>
#include <mpi.h>
int main(int argc, char** argv) {
int rank, size;
/* initialize MPI */
MPI_Init(&argc, &argv);
/* get the rank (process id) and size (number of processes) */
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
/* print a hello message */
std::cout << "Hello from process " << rank << " of " << size << "!" << std::endl;
/* quit MPI */
MPI_Finalize();
return 0;
}
The procedure for compiling and running is nearly the same, except we use "mpic++" instead of "mpicc":
$ mpic++ hello.cpp
$ mpirun -np 8 ./a.out
Threads communicate by passing messages amongst themselves. Messages are simply blocks of data.
Messages may be passed "point-to-point", where one process sends a message and another receives it.
Messages may also be broadcast from one process to the rest.
How could write an algorithm to sum a range of numbers, in parallel, using a message passing model?
The following two functions can be used for point-to-point communication in MPI:
MPI_Send(void* buffer, int count, MPI_Datatype type, int destination, int tag, MPI_Comm comm);
MPI_Recv(void* buffer, int count, MPI_Datatype type, int source, int tag, MPI_Comm comm, MPI_Status* status);
These functions have several parameters:
buffer
In a send call, a pointer to the data to be sent. In a receive call, a pointer to the place the data should be written.
count
The number of data elements to be sent or received.
type
The type of data to be sent or received. Along with the count, this tells MPI how many bytes will be sent/received. Below is a partial list:
destination/source
The rank of the process we want to send data to / receive data from.
tag
An arbitrary integer that identifies what we are communicating. MPI uses this to match sends with receives.
comm
The "communication group" of the operation. This is normally MPI_COMM_WORLD.
status
In a receive operation, this allows us to see how many bytes were actually received.
Below is an MPI implementation of the sum program. It uses the simple method of all processes passing their partial sums to the first process.
#include <stdlib.h>
#include <mpi.h>
#include <stdio.h>
#define START 0
#define END 100
int main(int argc, char** argv) {
int rank, size;
/* initialize MPI */
MPI_Init(&argc, &argv);
/* get the rank (process id) and size (number of processes) */
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
/* calculate the start and end points by evenly dividing the range */
int start = ((END - START) / size) * rank;
int end = start + ((END - START) / size) - 1;
/* the last process needs to do all remaining ones */
if (rank == (size - 1)) {
end = END;
}
/* do the calculation */
int sum = 0, i;
for (i = start; i <= end; i++) {
sum += i;
}
/* debugging output */
printf("Process %d: sum(%d, %d) = %d\n", rank, start, end, sum);
/* MPI communication: process 0 receives everyone elses sum */
if (rank == 0) {
/* parent process: receive each partial sum and add it to ours */
int partial, i;
MPI_Status status;
for (i = 1; i < size; i++) {
MPI_Recv(&partial, 1, MPI_INT, i, 0, MPI_COMM_WORLD, &status);
sum += partial;
}
} else {
/* worker process: send sum to process 0 */
MPI_Send(&sum, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);
}
/* now have process 0 display the results */
if (rank == 0) {
printf("The final sum = %d.\n", sum);
}
/* quit MPI */
MPI_Finalize();
return 0;
}
Copyright © 2024 Ian Finlayson | Licensed under a Creative Commons BY-NC-SA 4.0 License.