Documentation
AQAvit™ Verification
fooBar

The AQAvit project was created to “make quality certain to happen.” AQAvit verification is achieved through the process of running and passing a prescribed and versioned set of tests in the AQAvit test suite. The Eclipse Foundation Quality Verification Software License (QVSL) requires those who wish to verify their product to share a summary of the test results by which verification was achieved. This document describes how to run the AQAvit verification tests, check verification has passed, and collect the required test results for publication.

AQAvit verification is one of the criteria for listing in the Adoptium Marketplace. Leveraging the AQAvit test suite to ensure the quality of the binaries listed in the Adoptium Marketplace not only communicates to consumers how serious we are about quality, but also consolidates the good verification practices of the Adoptium Working Group members under a centralized effort. AQAvit aligns its test suite standards with the ever-changing requirements of the user base.

Overview

The AQAvit test suite is a large set of tests, many contributed to the AQAvit project and some pulled from a variety of open-source projects, that serve to verify the quality of OpenJDK binaries. The suite is suitable for testing Java SE 8 or higher versions on all supported platforms. To verify binaries, testers clone a specified release of the aqa-tests github repository (the latest stable release of the aqa-tests repository), configure their test environment, execute and pass the required test targets against each binary they plan to verify and make the results of those test runs available as per the QVSL.

AQAvit Test Targets to Run

The tests are divided into different groups and those groups are split into 3 levels. This logical categorization of the tests provides flexibility and granularity and can be visually represented in a grid.

grid view

For the current release of AQAvit, the required set of top-level test targets to run are [sanity.functional, extended.functional, special.functional, sanity.openjdk, extended.openjdk, sanity.system, extended.system, sanity.perf, extended.perf]. In subsequent AQAvit releases, targets will be added to raise quality bar even higher.

Details Regarding Test Execution

AQAvit can be run in various CI/CD environments as well as directly via command-line in a container or on a test machine that has the test prerequisites installed. The basic steps are the same in any environment.

The .tap files and metadata files contain timestamps and information about the binary under test. This information is collected from java -version output, release file information and querying some system properties during the test run. Where applicable, the information should match with the binary listed in the marketplace.

Verifying Results

The AQAvit test suite produces test result files and metadata files at the end of the test execution. Upon running and passing each of the nine required test targets, the result files and metadata files are to be gathered and shared. For test targets that contain failures, the root cause of the failure should be addressed and the target can be rerun and an updated test result file produced and shared.

Test result files that are produced follow a certain naming convention and use a simple TAP (Test Anything Protocol). When top-level targets are run serially, a single .tap file is produced, for example:

Test_openjdk11_hs_sanity.system_aarch64_linux.tap

contains version, impl/distribution, test target and platform information in the name, and its contents look like:

# Timestamp: Wed Mar  2 10:51:55 2022 UTC
1..168
ok 1 - MachineInfo_0
  ---
    duration_ms: 581
  ...
ok 2 - ClassLoadingTest_5m_0
  ---
    duration_ms: 304339
  ...
ok 3 - ClassLoadingTest_5m_1
  ---
    duration_ms: 303883
  ...
etc.
  ...
ok 168 - MauveMultiThrdLoad_5m_1
  ---
    duration_ms: 304296
  ...

One can see in this example that the top-level target sanity.system contains 168 sub-targets. Of the set of expected subtargets, some may be 'skipped' due to being filtered out as not applicable for a particular version or platform, but there must be none that failed. Within the tap file, they will show as 'not ok' if they have failed. Failing subtargets can be rerun individually and the tap file produced for that individual run can be included in the <results>.zip file to indicate that the binary under test was able to pass all expected targets.

Documentation Authors
gdams smlambert llxia tellison
Help us make these docs great!
All Adoptium docs are open source. See something that's wrong or unclear?
gradient overlay mobile

Connect with the community

Eclipse Temurin offers high-performance, cross-platform, open-source Java runtime binaries that are enterprise-ready and Java SE TCK-tested for general use in the Java ecosystem.

Frequently asked questions

Eclipse Temurin offers high-performance, cross-platform, open-source Java runtime.

Have a question that hasn’t been answered? Get in touch

Where are the latest Adoptium® JDKs?

Where are the latest Adoptium® JDKs?

Where are the latest Adoptium® JDKs?

Where are the latest Adoptium® JDKs?

Where are the latest Adoptium® JDKs?

Where are the latest Adoptium® JDKs?

Thank you to our 300+contributors