Technical News from The Portland Group
Porting the Weather Research and Forecasting Application to Microsoft Windows Using PGI Workstation
During the 2008 Supercomputing conference in Austin last year, I was talking with a weather researcher about a variety of topics. I asked if he ever used Windows in his work. A blank stare, followed by a simple "No", was the answer. This is a typical response from the HPC community when Windows is mentioned. Why should we care about Windows when all of our applications work just fine on Linux?
While this article is not meant to argue that Windows should replace Linux as your main OS platform, Windows does have a time and place. In recent years, Microsoft has made great efforts to produce high quality HPC tools. The Cray CX-1 personal supercomputer's primary OS is Windows. More importantly, the Windows laptop you're using has become powerful enough to handle many small to midsize simulations.
I'm currently writing this article at 35,000 feet on a five hour flight. Should my work simply stop because I don't have access to my office cluster? Why limit where and how I work simply because my application doesn't run on Windows? Why can't my application be OS neutral?
The goal of this article is to make at least one major application, WRF, OS agnostic. While WRF has been ported to Windows before, building it was difficult, it required the entire Cygwin or SUA tool chain, builds on SUA were not MPI-capable, and SUA can inflict a performance penalty. I wanted to make building WRF as simple as it is on Linux, just type ./configure and ./compile em_real. While I came up a bit short, I did come close.
Note that you will need to install the latest PGI Workstation 9.0-4 release for Windows to follow these build instructions yourself. PGI 9.0-4 includes a few minor compiler enhancements and some additions to the Cygwin environment that are used in the instructions below. Also, if you use the Cygwin that ships with the PGI compilers for Windows, you will need to explicitly uninstall Cygwin before installing PGI 9.0-4.
In addition, you will need to have the free ActivePerl installed.
Building the NetCDF Library
Because NectCDF is the only external library required by WRF, we thought to include it here. For the most part, NetCDF 4.0 has been ported to Windows and only requires two minor changes. Note that you must have PGI release 9.0-4 or greater in order to build NetCDF 4.0.
Step 1Download the new crtdefs9.h file and use it to replace the similarly named file included with the PGI 9.0-4 installation package ( $PGI/win64/9.0-4/include/crtdefs9.h). Alternatively, you can just add the following definition to the originally installed file. Enter this new definition just after the definition of "SIZE_T":
#ifndef _SSIZE_T_DEFINED #ifdef _WIN64 #ifdef EDG_STRICT_ANSI typedef long long ssize_t; #else typedef __int64 ssize_t; #endif #else typedef _W64 int ssize_t; #endif #define _SSIZE_T_DEFINED #endif
This will define ssize_t, which is required for a build of NetCDF. This modification to crtdefs9.h will be default in a future release of the PGI compilers for Windows.
Open a PGI Workstation command window and change to your build directory. Note that this directory should not be under $CYG_ROOT because the PGI ar utility (pgar) is yet not able to interpret Cygwin paths.
Untar the NetCDF binary package The package is available directly from UCAR (University Corporation for Atmospheric Research) at http://www.unidata.ucar.edu/downloads/netcdf/netcdf-4_0_1/index.jsp.
Change into the netcdf-4.0 directory
Edit the file "libsrc/v2i.c" and add the line #include "nc.h" right after line 12; save the file and exit the editor.
Working at the PGI Workstation command line, set the following environment variables:
% export NM=pgnm % export AR=pgar % export CC=pgcc % export FC=pgf90 % export LD=pgf90 % export CXX=pgcpp % export CPPFLAGS="-DpgiFortran -D_MSC_VER" % export CFLAGS="-O2" % export FFLAGS="-O2 -Msignextend" % export PGI_OBJSUFFIX=o
Or, if you expect to build NetCDF frequently, save these commands in a file "setnetcdf" and execute the following command:
% source setnetcdf
Run configure. For example, if you want to install NetCDF in the directory /c/Users/<yourusername>/NETCDF, invoke configure as follows:
% ./configure --enable-shared --prefix=/c/Users/
Run make and then make install. Note that the make will fail when trying to build the NetCDF utilities (ncgen). We have not attempted to port these utilities since they are not required to build WRF.
Compiling WRFv3.1.1 on Windows using PGI Workstation 9.0-4
If you plan on running the DM-Par (MPI) version of WRF, you will first need to download and install the Microsoft HPC Pack 2008 SDK. Reboot your system after installation; this ensures the HPC environment variables are available in the PGI shell environment.
Download the WRF modeling system software Version 3.1.1 (July 31, 2009). Note that these instructions have only been tested with Version 3.1.1 and may or may not work with other versions.
Open a PGI Workstation command window and change to the directory where you will be building WRF.
Do not build in the Cygwin /tmp directory. Choose a directory such as /c/Users/<yourusername>. Enter the commands in the following steps using this PGI Workstation command window. The following instructions assume you will build WRF in /c/Users/<yourusername>/WRF311, and that you have previously built and installed NetCDF in the directory /c/Users/<yourusername>/NETCDF.
Unpack the WRF package:
% tar zxvf WRFV3.1.1.tar.gz
% patch -p0 -i wrf311_pgi_win.diff
Note: for details about the changes we made to WRF using this patch file, see Appendix A below.
Change directories into the WRFV3 directory
% cd WRFV3
Set the following environment variables (note the use of forward slashes):
% export PGI_OBJSUFFIX=o % export WRF_OS=PGI_CYGWIN % export WRF_SRC_ROOT_DIR=C:/DOS/path/to/current/directory % export NETCDF=C:/DOS/path/to/netcdf/lib
Note: if pwd within the PGI Workstation command window returns a path /tmp/WRFV3, the DOS path to the directory is C:/cygwin/tmp/WRFV3 (assuming PGI Workstation is installed on drive C:).
Run ./configure and select the appropriate option, either Serial, SM-Par (OpenMP), or DM-Par (MSMPI)
Note that the Hybrid SM/DM-Par option is also available. However, we have not tested it.
Edit the resulting "configure.wrf" file:
- The above NetCDF build uses a separate Fortran interface library (libnetcdff.lib). If your NetCDF build uses only a single library, please remove -lnetcdff from the LDFLAGS_LOCAL variable.
- Find and remove the second definition of the LIB_BUNDLED variable listed just after Registry, including all continuation lines (the next 10 lines)
- If you are compiling on a Penryn- or Nehalem-class CPU, add the option -tp core2-64 to the definition of FCOPTIM around line 107. There is no need to make this edit if you're compiling on another CPU type. Determine your CPU type using the command pgcpuid.
Compile WRF using the following command:
% ./compile em_real >& build.log
Congratulations! You've complied WRF on Windows and can continue with your research on clouds as you soar high above them. Unless of course you really did want to watch the in-flight movie, then please ignore these instructions.
In this article I tried to show how to go about building WRF on Windows using PGI Workstation. Other build variations are possible and I encourage you to try them using this article as a baseline reference. If during your port you have questions or advice to share, please take a minute to visit the PGI User Forums and read what others are doing or to post about your experience. A future article will show how to build, execute and debug WRF source code from within Microsoft Visual Studio using PGI Visual Fortran. Good luck!
Let's look at the details of the changes made to the source and why each environment variable is needed.
The WRF configure script uses uname to determine what OS you're using, and then presents a list of configure options available for this OS. The result of uname when run on Cygwin can return a variety of strings depending upon the exact version of Cygwin being used. While we could have adapted configure to recognize the string for the particular version we ship with PGI Workstation 9.0-4, customers using their own Cygwin installation should also be able to use these instructions. Hence, setting PGI_CYGWIN presents a single configuration for all versions of Cygwin.
The WRF_SRC_ROOT_DIR variable is used throughout WRF compilation as the location of the top of the WRF build tree. If not set on the command line, the compile script sets the variable to the path returned by the command pwd. However, the PGI compilers and tools don't allow interchangeable use of Cygwin vs DOS-style path names (i.e Cygwin /c/tmp versus DOS C:/tmp), resulting in problems locating libraries unless WRF_SRC_ROOT_DIR is set to a DOS path. PGI is working on adding support for interchangeable pathnames to a future release.
WRF and the build infrastructures for many other applications assume that objects end with a .o suffix. However, the convention on Windows is for all suffixes to be three characters. Thus, .o becomes .obj. The proper fix would be to have the WRF build scripts use a variable for the object suffix, but this could require a painful rewrite. To help, PGI allows you to change the default object suffix via the environment variable PGI_OBJSUFFIX.
The one edit you needed to make to your "configure.wrf" file was the removal of the second LIB_BUNDLED variable. The WRF build harness assumes that libraries end with the suffix .a. The flag -llibraryname will be passed to the Microsoft linker (LINK) as -libpath:libraryname and be resolved as "liblibraryname.lib". If your library suffix is .a, then link won't be able to find it. Instead, you need to pass the full name of the library (along with the path if not found in the local directory). Again, the preferred method would be to have the library suffix be a make variable, but in the mean time we needed to change LIB_BUNDLED to use the full path of the .a libraries.
For the most part, Fortran is highly portable. So long as the programmer avoids extensions, Fortran code usually ports easily to most systems. If portability problems do occur, it generally occurs when the Fortran program needs to call C or system functions. For C code, things can be more difficult. For example, Microsoft doesn't support the C99 standard nor POSIX threads. Luckily, WRF doesn't use either of these but does use the BSD C routines rindex, index, and bzero. Most of the source file changes in WRF were simply to define rindex to strrchr, index to strchr, and bzero to memset.
Other minor changes included:
- Change "read_grib.c" to use "localtime_s" instead of "localtime_r"
- Several files needed the inclusion of the "Winsock2.h" header file instead of "netinet/in.h", for the definition of the function gethostname
- Add a macro definition for the C99 lrint routine in "gribputgds.c"
- Add "io.h" in "open_file.c" to define _commit which replaces the fsync function to flush a file to disk
- Adding header files (like "stdio.h") and working around unused UNIX header files (like "sys/resources.h" and "strings.h")
- Clean up some of the C functions that are missing type information and add missing prototypes (the latter just to clean-up some warning messages)
Because Microsoft does not support the the gethostid function, NCAR will need to add code for this function. I simply created a stub that returns 1. This function is currently used only when binding processes to NVIDIA devices on a cluster so may not be needed (yet!)
The final workaround was to avoid the use of rlimit to set the applications stack size.