by Ryan Kitchen

Any software developer knows that developing and maintaining a large software project can be a difficult task, particularly when documentation is sparse and the original developers are unavailable. Sometimes a small change causes unexpected problems elsewhere in a program, and they can be very difficult to track down in a large codebase. Other times, one might want to explain how a program works to a new developer, and it would be very useful to have a visual representation of the program’s internal structure. When adapting existing HPC applications to OpenMP or OpenACC, sometimes it is difficult to prioritize what exactly should be parallelized first. These tasks can be difficult, labor intensive, or even impossible to do by hand.

Compilers, however, contain most of the information needed to solve these problems, locked away from the user in various internal representations. In the latest release of the PGI compilers, a new feature makes this information readily accessible to the user.

Beginning with version 17.7, PGI compilers have the ability to export information gaterhed from the internal representation (IR) of various compilation stages in an organized, human readable format. This new Program Analysis Summary Output (PASO) feature is enabled using a new compiler option (-⁠Msummary). The data provides a detailed summary of the program’s internal structure, spanning not just one but all the files that make up a project. PASO data contains detailed information about every function call site, every place where a variable is declared, read, or modified, OpenACC and OpenMP regions, and much more.

What Can I Do With PASO?

The primary purpose of the PASO feature is to enable the creation of documentation and analysis tools. Some of the things you can do with this data are:

  • Generate call graphs and data flow diagrams
  • Trace every place a variable is used or modified, including aliased variables
  • Identify what functions contain or where they are called from OpenACC and OpenMP regions
  • Find functions that contain nested loops or recursion

How is This Different Than Existing Tools?

One of the key advantages of PASO is that it provides information specifically relevant to parallelized HPC applications. Other tools used for these use cases tend to be suited more for general purpose programming. For example, Doxygen is an industry standard tool for generating call graphs and other kinds of program documentation. PASO dives a little deeper by providing information about pragma-based parallelization, using interprocedural analysis to trace variable usage and function calls between different parts of a program, and keeping track of function and variable aliases. Some of this information is either not available in or not used by existing tools, and could be useful for a variety of purposes.

Other tools exist which can perform a more in-depth program analysis, such as Scitool Understand, which is a reverse engineering program capable of doing most of these tasks and many others. However, these tools are very expensive and may be inaccessible to those in a research setting.

How Does It Work?

PASO works by collecting information from various stages of compilation, and then combining it all together at link time to create an export file. This file represents the entire program, including every user-compiled object file and any standard libraries. PASO organizes all of this information in an unambiguous, hierarchical, human readable format. Performing this analysis at link time enables the tool to make connections between different files in the same project, which makes it possible for users to perform more thorough, detailed kinds of analysis.

PASO is easy to enable for any project. Simply compile your program with the PGI command line flags: -⁠Msummary -⁠Wi,-e⁠xport,export_file.json. This will generate an output file export_file.json, which can then be imported into your user-made tools or used interactively through a scripting environment.

The output from this tool is formatted using the JSON (JavaScript Object Notation) standard. This standard was chosen because many scripting languages already support it natively, and import libraries are available for many compiled languages as well.

The organization of the export data was driven by how programmers logically organizes their code. A program consists of multiple files. Files contain functions, functions contain variables, have parameters, and call other functions. Variables also have bindings, which can be used to help determine where they are used, modified, and possibly what their values are.

PASO file format breakdown

This diagram is a structural overview of how the file format is broken down, and does not reflect the content of individual elements, Fortran specific elements such as modules, or additional features. For more information, please see the PASO documentation.

How Can I Use It?

The JSON format is ideal for use with scripting languages such as Python. To use this JSON file in Python, all you need to do is load it and parse it.

>>> import json
>>> with open("export_file.json") as file:
...  data = json.load(file)

The structure data now contains the entire PASO data export in the form of a key-pair tree structure. Now, if you wanted to find out what files were contained in this program, you could iterate over the first level of this structure and print out the names of the object files and their reference name:

>>> for file in data["files"]:
...  print file+”:”data["files"][file]["filename"]

This produces an output of all files used in the compilation, including any libraries:

file1: test.o
lib465: strtold_l.o
lib464: strtod_l.o
lib426: dl-deps.o

This view includes a list of many, many standard libraries. For the purpose of convenience, the reference name for these libraries is prefixed with lib instead of file. To filter these out, we can use a prefix check:

>>> for file in data["files"]:
...  if "lib" not in file:
...   print file + ":" + data["files"][file]["filename"]

Call Graph Example

Let’s try doing something a little more interesting: building a call graph. The first step for building a call graph for a program is finding the main function. PGI supports both C/C++ and Fortran, so main might not necessarily be called “main”. For this reason, the main attribute is present for the main function in C, and the main program in Fortran, whatever its name may be.

>>> def findmain():                                                             
...  for file in data["files"]:                                                 
...   for function in data["files"][file]["functions"]:                         
...    if "attributes" in data["files"][file]["functions"][function] and "main" in data["files"][file]["functions"][function]["attributes"]:
...     return {"file":file,"function":function}

>>> findmain()
{'function': u'main', 'file': u'file1'}

Of course, in a large program, we may want to find the call graph of a specific function. So we could also select the function based on source filename and function name.

>>> def findfunc(filename, func):	
...  for file in data["files"]:
...   if "source" in data["files"][file] and data["files"][file]["source"] == filename:
...    if func in data["files"][file]["functions"]:
...     return {"file":file,"function":func}
...  print "Function not found!"
...  quit()
>>> findfunc("test.c","main")
{'function': 'main', 'file': u'file1'}

These two functions both produce a function reference, identical in format to the ones used in the summary output. These references, as well as variable references which follow the same pattern, are essentially an address to a function or variable definition. For example, you could use the following to get the function structure of the main function:

>>> funcref = findmain()
>>> funct = data[“files”][funcref[“file”]][“functions”][funcref[“function”]]

Then you can use funct directly without having to include the full path from data. This makes scripting much, much cleaner. Next, we’ll set up our output. To generate some nice graphs, let;s use the GraphViz DOT format, which can be imported into any graphing tool. The format is as follows:

	Digraph callgraph{

To set it up, we simply open a file and print out the header:

>>> def makecallgraph( function_reference ):
...  outfile = open("","w")
...  outfile.write("Digraph callgraph{\n")
...  callgraph(function_reference, outfile)
...  outfile.write("}\n")
...  outfile.close()

Lastly, we define the “callgraph” function:

>>> def callgraph( func, outfile ):
...  if func not in callgraph.visited:
...   callgraph.visited.append(func)
...  function = data["files"][func["file"]]["functions"][func["function"]]
...  if "calls" in function:
...   for call in function["calls"]:
...    target = call["name"]
...    outfile.write(func["function"]+"->"+target["function"]+"\n")
...    if target not in callgraph.visited:
...     callgraph(target,outfile)
>>> callgraph.visited=[]

Now we can generate the call graph simply by calling makecallgraph on the function reference from findmain.

>>> makecallgraph(findmain())

The output file, and the graph created from it look like this:

PASO file format breakdown
Digraph callgraph{

From this graph, we can see that main calls Y, which calls X twice and V three times. Y and V also call W. However, with a little more information, this graph could be a lot more useful.

OpenACC/ OpenMP Annotation of Call Graph

Now that we have a basic call graph, we can annotate it with information about OpenACC and OpenMP regions. To do this, we need to find out if a function call is inside of a parallel region, and then propagate that information to each subsequent call. Fortunately, each call site in the summary output has an attributes field that contains the needed information. If the function call is inside of a region, it will have either an omp or acc attribute, or both. To find all paths, both with and without parallel regions, we’ll add the OMP/ACC attributes to the function reference. We can use the color setting in DOT to represent this on our graph. This requires a very minor change to our program to achieve this functionality.

>>> def callgraph( func, outfile ):
...  if func not in callgraph.visited:
...   callgraph.visited.append(func)
...  function = data["files"][func["file"]]["functions"][func["function"]]
...  if "calls" in function:
...   for call in function["calls"]:
...    target = call["name"]
...    color = ""
...    if "acc" in func or ("attributes" in call and "acc" in call["attributes"]):
...     target["acc"] = True
...     color = "[color=green]"
...    if "omp" in func or ("attributes" in call and "omp" in call["attributes"]):
...     target["omp"] = True
...     if "acc" in target:
...      color = "[color=purple]"
...     else:
...      color = "[color=blue]"
...    outfile.write(func["function"]+"->"+target["function"]+color+"\n")
...    if target not in callgraph.visited:
...     callgraph( target,outfile )
>>> callgraph.visited=[]

Now if we run our new program on the same export file, we get this output:

Subroutine interdependencies
Digraph callgraph{

From this output, we can see that one of Y's calls to X resides inside a blue, OpenMP region. The three different calls to V are a call from an OpenMP region, a call from a green OpenACC region, and a call from an OpenACC region inside of an OpenMP region, which is represented in purple. We can infer all of this from the graph, without ever looking at the program’s source.


Another key feature of PASO is the concept of bindings. Each function’s parameters are bound to every variable, expression, and constant that is passed to it as an argument. These bindings include variables indirectly passed to it, either via assignment or by aliasing.


	void func2(int *a);
	int x = 100;
	int *y = &x;

This example would generate two bindings for func2: one assignment for x, indicating that the value of A would be changed by this function, and one argument of y, which was passed explicitly but not changed.

In the variable record for y, there would also be an assignment to x. This lets you know what variable's address was taken.

Global Variables and Common Blocks

PASO does not directly list global variables or their file scope. However, whenever a nonlocal variable (including globals and Fortran common blocks) is used or modified in a function, a record of its usage is inserted into the function’s nonlocal used or nonlocal modified sections. Got an unexpected value in a global variable at runtime? This feature can be used to trace down every place where it was modified.

	>>> varref = {“file:file1”,”variable”:”xyz”}
	>>> for file in data[“files”]:
	>>>  for function in data[“files”][file][“functions”]:
	>>>   func = data[“files”][file][“functions”][function]
	>>>   if “nonlocal modified” in func:
	>>>    for var in func[“nonlocal modified”]:
	>>>     if var[“name”] == varref:
	>>>      print file+”:”+function


PGI’s new Program Analysis Summary Output feature has lots of unexplored potential, for both debugging and documentation purposes. The export format is simple to use with Python and other scripting languages. Simply enable the export with -⁠Msummary -⁠Wi,-⁠export,[filename.json], and you’re good to go. For reference, see the export file documentation.

Click me
Cookie Consent

This site uses cookies to store information on your computer. See our cookie policy for further details on how to block cookies.