Top Application Performers (TAP)
Frequently Asked Questions
Addressing the need for demonstrating
computer power based on realistic system loads
What is TAP, where can I get information?
TAP stands for Top Application Performers. The TAP List is a rank
list of the fastest computer systems measured by their performance
on large-scale applications. The TAP List homepage is http://www.ece.purdue.edu/ParaMount/TAP
TAP is an effort to encourage the use of real applications for measuring
the performance of high-performance computer (HPC) systems. The need
for evaluating computer systems with realistic benchmarks is evident
and widely acknowledged. However, so far there has not been an effort
to demonstrate HPC system performance based on such metrics. We provide
the TAP list, a new rank list of high-performance computer systems,
based on real, industrial applications and data sets.
 Benchmarks:
The TAP lists are based on benchmark suites developed by the SPEC
High-Performance group ( http://www.spec.org/hpg ), The TAP HPC list
uses the SPEChpc benchmarks, which currently include representative
computational codes in seismic processing, ab-inito molecular modeling,
and mesoscale weather modeling. The benchmarks include several data
sets, called small, medium, large, X-large. TAP HPC uses the
medium data sets. A second list, TAP OMP, has been added more recently
to rank shared-memory systems based on OpenMP application performance.
To obtain benchmark codes, contact: info@spec.org
 Submitting Results,
Run Rules:
The rules for running and reporting benchmarks for the TAP list follow
the same well-defined rules as for publicly reporting SPEC benchmark
results. The run rules for the SPEChpc suite are described at www.spec.org/hpg/runrules.html.
For more information contact: info@spec.org.
The submitter must be member of the SPEC High-Performance Group. SPEC
membership is open to both industrial and nonprofit organizations.
Non-members can find a sponsor to submit the results. This can either
be the vendor company of the system being benchmarked or an academic
SPEC member. Purdue University considers sponsoring academic participants;
send mail to topap@ecn.purdue.edu .
 Ranking:
The TAP HPC list ranks computer systems using a weighted sum of the
performance results reported for the individual benchmarks. The weights
are chosen so that each benchmark adds a comparable contribution to
the overall metric. Benchmark codes for with no results have been reported
add zero to the metric.
 Verifying
Valid Results:
Results must be submitted as SPEC results. SPEC results are reviewed
by a SPEC committee, which checks the submission for feasibility and
completeness. The committee is made up of members from SPEC's High-Performance
Group, which consists of representatives from major computer manufacturers
and academic institutions involved in high-performance computing research.
Each benchmark has a validation procedure built in.
 Benefits
for you
In addition to the obvious benefit of seeing your computer system
and site on a highly ranked position on the TAP list, there are benefits
from being a member of the SPEC HPG benchmarking organization. See http://www.spec.org/hpg/joining.html for
more information (you can join SPEC HPG either as a commercial company
or as non-profit site.)
 To Submit Results:
See the Call For Participation in the TAP50
List or contact: topap@ecn.purdue.edu
 How
to Obtain More Information:
For more information about SPEChpc benchmarks: http://www.spec.org/hpg.
To obtain benchmark codes and for SPEC membership, see SPEC.
For general information about TAP: http://www.ece.purdue.edu/ParaMount/TAP ,
or send mail to: topap@ecn.purdue.edu
Each approved benchmark report, whether or not it makes it to the TAP
list, will be published in SPEC HPG's Web pages of results. See SPEC HPC results and SPEC OMP results
How is performance measured?
The TAP List uses a sum over the SPEC metric for each benchmark, normalized
by a reference run for each application. If benchmark results have
not been reported for all benchmarks in the suite, the contributions
of the missing benchmarks are zero.
The fastest computer systems I know are
not on the TAP List. Why?
Your favorite supercomputer may not be "on TAP" because of time or
capability.
Time: The TAP List is a relatively new effort. To be listed,
one needs to run the SPEC HPC or
OMP benchmarks and report the results through standard SPEC submission
channels. Once the results have been posted by SPEC, they will be included
in TAP. This process takes some time.
Capability: Your favorite supercomputer may not be a Top Application
Performer, and the TAP List simply reflects this fact. One motivation
for creating the TAP List was the fact that the performance of high-performance
computers is often measured with "peak performance" (e.g., Gigaflops)
or "kernel performance" (e.g., speed of a matrix multiplication). Such
metrics do neither reflect real application patterns nor the difficulties
of implementing realistic applications on a computer system. By contrast,
the TAP list uses the SPEC HPC benchmarks, which consist of the largest
publicly available computational applications. These benchmarks have
been run on a large number of systems. Nevertheless, it may take considerable
effort to create a benchmark submission, especially if there is not
yet a report for your machine. Also, we have seen that some benchmarkers
struggle getting the applications to run correctly on large systems.
Benchmark reports can only be submitted if the runs pass a built-in
validation test, defined by an application expert. One reason why you
don't see many large systems on the TAP list is that it appears to
be more difficult for large parallel systems to pass the validation
test.
I know of a better benchmark for the TAP
List. How can I get it included?
The SPEC HPC benchmarks include the largest publicly available high-performance
computer applications. Continuous update of the suite with codes that
deserve this attribute is part of the mission of SPEC's high performance
group. Getting such applications into the SPEC suite is difficult and
your participation in enhancing this suite and making it a better representative
of large-scale applications would be most valuable and welcome. For
information please contact the SPEC
office or the TAP committe.
|