MPI: Why do processes not always behave as intended by program logic?

Question

C++ / C#

Problem: There are n processes. from zero to first, a message is transmitted. Then from the first to the second, and so on. When a message arrives at n-1 processes, it sends the message to process 0. All this is repeated m times.
The code:

#include <mpi.h>
#include <stdio.h>

void ring( int m, int argc, char** argv );

int main ( int argc, char** argv ) {
  ring( 3, argc, argv );

  return 0;
}

void ring( int m, int argc, char **argv ) {
  int i = 0, ProcNum, ProcRank, done = 0, buf, process;
  MPI_Status status;

  MPI_Init( &argc, &argv );

    //for ( i = 0; i < m; i++ ) {
    while( i != m ) {
        MPI_Comm_rank( MPI_COMM_WORLD, &ProcRank );
        MPI_Comm_size( MPI_COMM_WORLD, &ProcNum  );

        if ( ProcRank == 0 ) {
          MPI_Send( &ProcRank, 1, MPI_INT, 1, 0, MPI_COMM_WORLD );
        }
        else {
          MPI_Recv( &buf, 1, MPI_INT, ProcRank - 1, 0, MPI_COMM_WORLD, &status );
          printf( "bom-bom from %d to %d on itteration: %d\n", buf, ProcRank, i );

          process = ProcRank + 1;
          if ( process >= ProcNum )
            process = 0;

          MPI_Send( &ProcRank, 1, MPI_INT, process, 0, MPI_COMM_WORLD );
        }

        MPI_Barrier( MPI_COMM_WORLD );
        if ( ProcRank == 0 ) {
          MPI_Recv( &buf, 1, MPI_INT, ProcNum - 1, 0, MPI_COMM_WORLD, &status );
          printf( "bom-bom from %d to %d on itteration: %d\n", buf, ProcRank, i );
          printf( "DONE\n\n" );
        }

        i += 1;
        MPI_Barrier( MPI_COMM_WORLD );
    }

  MPI_Finalize();
}

$ mpirun -np 4 ./prog
bom-bom from 0 to 1 on itteration: 0
bom-bom from 1 to 2 on itteration: 0
bom-bom from 2 to 3 on itteration: 0
bom-bom from 2 to 3 on itteration: 1
bom-bom from 3 to 0 on itteration: 0
DONE

bom-bom from 3 to 0 on itteration: 1
DONE

bom-bom from 0 to 1 on itteration: 1
bom-bom from 0 to 1 on itteration: 2
bom-bom from 1 to 2 on itteration: 1
bom-bom from 1 to 2 on itteration: 2
bom-bom from 2 to 3 on itteration: 2
bom-bom from 3 to 0 on itteration: 2
DONE

$ mpirun -np 4 ./prog
bom-bom from 0 to 1 on itteration: 0
bom-bom from 1 to 2 on itteration: 0
bom-bom from 2 to 3 on itteration: 0
bom-bom from 3 to 0 on itteration: 0
DONE

bom-bom from 0 to 1 on itteration: 1
bom-bom from 1 to 2 on itteration: 1
bom-bom from 2 to 3 on itteration: 1
bom-bom from 3 to 0 on itteration: 1
DONE

bom-bom from 1 to 2 on itteration: 2
bom-bom from 2 to 3 on itteration: 2
bom-bom from 0 to 1 on itteration: 2
bom-bom from 3 to 0 on itteration: 2
DONE

As you can see, in the first case, the processes behave incorrectly. Second run: everything is fine.

Answer 1

This may be due to. that MPI does not define the operation of processes with I / O streams. That is, sequential output from different processes is not guaranteed. And flushing the buffer here fflush() won't help here. If deterministic output is important, MPI I/O can be used, but this is an MPI-2 standard and support is not available in many MPI implementations.