OpenMP, file tree walk and map-reduce example

All happens on Linux and we will use GCC and built in OpenMP support and also thread local storage. We will see how easy is to spread workload across threads using OpenMP.
In man pages we find complete example for nftw or file tree walk. That example will print to standard output about everything what it finds while walking directories. Instead of having it printing everything I made it count files:

#define _XOPEN_SOURCE 500 #include <ftw.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <stdint.h> int counter = 0; int count_files(const char *fpath, const struct stat *sb, int tflag, struct FTW *ftwbuf) { if(tflag == FTW_F){ counter++; } return 0; /* To tell nftw() to continue */ } int main(int argc, char *argv[]) { if (nftw((argc < 2) ? "." : argv[1], count_files, 20, 0) == -1) { perror("nftw"); exit(EXIT_FAILURE); } printf("There was %d files in %s\n", counter, (argc > 1)?argv[1]:"."); exit(EXIT_SUCCESS); }

To compile it I used:

gcc -O1 -o simpleftw simpleftw.c

In order to count files across multiple directories we will pass few directories in argument list to program and also turn our static variable counter into thread local storage. We will use pragma OpenMP parallel for directive combined with reduction which is in our case simple addition:

#define _XOPEN_SOURCE 500 #include <ftw.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <stdint.h> __thread int counter = 0; int count_files(const char *fpath, const struct stat *sb, int tflag, struct FTW *ftwbuf) { if(tflag == FTW_F){ counter++; } return 0; /* To tell nftw() to continue */ } int main(int argc, char *argv[]) { int total_fc = 0; #pragma omp parallel for reduction(+:total_fc) for (int i=1; i< argc; i++){ if (nftw((argc < 2) ? "." : argv[i], count_files, 20, 0) == -1) { perror("nftw"); exit(EXIT_FAILURE); } total_fc += counter; printf("Thread %d %d\n", i, counter); } printf("There was %d files in ",total_fc); for (int i=1; i< argc; i++) printf("%s ", argv[i]); printf("\n"); exit(EXIT_SUCCESS); }

That print inside for loop should demonstrate that thread local storage works as expected. C99 switch is required, or we will have to specify that variable i is private. Also switch for OpenMP is required. To compile it I used:

gcc -O1 -fopenmp -o dwalker dwalker.c -std=c99

Execution produces on my box:

./dwalker ~/Documents/ ~/Downloads/
Thread 1 3894
Thread 2 7784
There was 11678 files in ...

Everything works as expected.