Release Process

From Crypto++ Wiki
(Redirected from Release Testing)
Jump to navigation Jump to search

This page describes the process used to release a version of the Crypto++ library. The process is a mashup of development, testing and then independent confirmations. The testing occurs on various CPUs, compilers, platforms, using different configurations. In addition, analysis tools are used as a security gate. The testing favors platforms that are used in practice today, like desktop, server and mobile operating systems.

The CPUs include commodity IBM PC compatible processors, like AMD and Intel; and mobile processors like ARM and MIPS. Platforms are selected based on availability and functionality (for example, CentOS 5 and OpenBSD are often used when security is a high priority). Optimizations include both -O2 and -O3 because GCC vectorizes at -O3.

Configurations include Debug (-DDEBUG -Og) and Release (-DNDEBUG -O{2|3|5|s}); C++ standard versions C++03, C++11, C++11, C++17, C++20, C++23; and C++ standard libraries include GNU's libstdc++ and LLVM's libc++. In addition it includes various combinations of defines, like NO_OS_DEPENDENCE.

A related topic is Release Versioning, and it discusses how the project names a version of Crypto++, what to expect when a version number changes, and how to locate a particular state of the source code in Subversion or Git.

Related pages are cryptest.sh and cryptest.nmake, and it discusses the test scripts for Linux, Unix and Windows used in the release process.

The Process

The process has six broad steps:

  1. Identify objectives
  2. Perform the work
  3. Measure performance
  4. Independent cross-validation
  5. Prepare the release
  6. Announce the release

Identify objectives

The community generally sets the course the library is taking. Features can be sensitive to the release cycle, depending on the impact to existing users. For example, a change that requires all client code to be recompiled cannot occur at a maintenance release (a release where the last number changes). Changes that require recompilation usually occurs at a major version bump (a release where the first number changes). See Release Versioning for more details.

Sometimes a release slips a bit as planned features are added. Sometimes features are removed if they could cause us to miss a date. Other times there may be spare cycles to add more than was planned.

A related topic is Goals and Objectives for the library.

Perform the work

The work is performed by folks who have the spare cycles. Sometimes its easy on Wei and the delegates because the only thing needed is a pull or merge request. Other times its more difficult, like adding additional functionality from the ground up. At other times, the work is monotonous, like rounding out test cases and test scripts to ensure a stable release.

The documentation (if any :) is added during the development phase while its fresh. Its not as bad as it sounds, though. At Crypto++ 5.6.3, which was a maintenance release, nearly 17,000 lines of Doxygen documentation was added. In addition, wiki pages covering new classes was added.

Measure performance

Throughout the process we measure progress to ensure we are meeting goals and objectives. Work is performed offline, outside of master. On occasion, a minor commit may occur that precedes a full release check-in.

Measuring performance includes security testing and evaluations. In this step, we ensure specific tools are applied to the work in progress to avoid latent and not so readily apparent bugs. This step often depends on a complete set of positive and negative test cases, and is carried out by tools like Undefined Behavior Sanitizer, Valgrind, Enterprise Analysis and Coverity.

At this stage, the script cryptest.sh (Unix, Linux, OS X) and cryptest.cmd (Windows) is run often on multiple platforms. cryptest.sh and cryptest.cmd are comprehensive testing scripts that ensure the library is well exercised.

The process can take one of two paths at this point. If there are problems or the work is incomplete, then the process loops back to the perform the work stage. If the work is complete and scripts are successful, the process can move towards release approval.

Independent cross-validation

Independent validation is important to ensure biased and myopic views are avoided. At any point in time, anyone is free to test the code and report issues or make suggestions. The mailing list and bug reporter are not a hostile environment to those doing the reporting.

All bugs are available to anyone who wants to see them. In addition, results from tools like Coverity are available to anyone who wants to see them. We take the position because risk cannot be mitigated when critical information is withheld from the folks who need it.

Prepare the release

When the interested parties and stakeholders feel the library is ready for release, a "gold" ZIP is built from sources. The ZIP is made available for download on the website, and the same sources used to build the ZIP are committed to source control.

There may be a few errata items against the ZIP at this point, but all the major work has been completed. If there are errata items then they are cleared, a new ZIP is uploaded and the change is checked into source control.

Once the library is stable in the various environments, we ask the "Crypto++ Primaries" to approve a release. The primaries are like a Board of Directors with Wei Dai as the Chairman and CEO. Folks on the board often have limited cycles to share, so we try not to disturb them too often. We also solicit community feedback.

Announce the release

After approval is provided by the primaries and users, we perform two final tests. First, we test the download of the ZIP and check-out of the code from version control and ensure there are no material differences between them. Second, we ensure the library builds and executes successfully with a make check. We do this for both the ZIP and the check-out to detect potential irregularities that can occur at the "ZIP and commit" stage.

Once the library passes the final validations, the hash of the ZIP is published, the source code is tagged in version control and the release announcement is made. The tag must be created using annotation. The hash of the ZIP is created with a second, distinct security-centric program such as OpenSSL.† The release announcement is made on three lists. The three lists are Crypto++ Announce list, Crypto++ Users list and sci.crypt Usenet group. In addition, an email is sent to a number of distributions and package maintainers.

A crawl request should be made with Google to pick up the new version numbers. The site to request a crawl is Google Search Console.

†The OpenSSL commands are openssl {sha1|sha256|sha512|blake2b512} cryptoppXXX.zip. In the past, Whirlpool was also used, but it appears OpenSSL 3.0 dropped support for it.

Increment version numbers

After releasing the library the project's compatibility should be bumped. For example, if Crypto++ 5.6.4 was released, then the compatibility number should be changed from 4 to 5 in in 5.6.5. And if Crypto++ 5.7 was released, then the version should be changed to 5.8.

This change helps with user questions that start with "I'm using Crypto++ 5.6.3 and I'm having problems with ...". When the project encounters the question, its unclear if its the 5.6.3 release ZIP, or Master without the next version bump. The change also helps determine the actual compatibility level during development. That is, if Master advertises version information X.Y.Z, then it better be compatible with X.Y.

The locations of version numbers are:

  • config_ver.h and CRYPTOPP_VERSION
  • cryptlib.h and Doxygen comment
  • cryptopp.rc and VERSION_INFO block
  • GNUmakefile and libcryptopp.pc recipe
  • Doxyfile and PROJECT_NUMBER
  • Readme.txt and version number
  • History.txt and version number
  • change_version.sh script

Also see Release Versioning and Post Release Increment on the wiki and the discussion Post-release version number increment on the user mailing list.

Self Tests

Crypto++ relies on two sets of self tests to test compatibility and provide functional evaluations. Both are invoked using cryptest.exe.

During comprehensive testing, cryptest.sh repeatedly builds the library and runs the self tests using different configurations.

cryptest.exe

The first set of self tests are the validation tests, and they are run with ./cryptest.exe v. The validation tests include testing the basic configuration, like the word size and endianess, and cpu features. The tests also include basic algorithm tests.

The second set of test are the test vectors, and they are run with ./cryptest.exe tv all. The test vectors are more comprehensive, and include a complete set of tests when available, like Brian Gladman's test suite for AES.

The library attempts to gather test vectors from independent sources to avoid false validations. However, its not always possible since not every standard and algorithm publishes test vectors. If a test vector is not available, then the library will attempt to create them using a different library like Botan. If test vectors are still not available then the library will create its own set and list the Source as Crypto++.

Continuous Integration

The library utilizes Continuous Integration (CI) from AppVeyor and Travis. The AppVeyor and Travis testing occurs on Noloader's GitHub account. The results are available at Noloader | Crypto++ | Commits. A different GitHub is used to provide segregation and guard against a web service compromise that could potentially taint Wei Dai's source code.

Breaks are transient. Before a release the breaks are usually leared.

AppVeyor

AppVeyor test Windows using Visual Studio 2012, 2013, 2015 and 2017. The AppVeyor tests are not as reliable as we hoped. For example, AppVeyor is showing success but we have witnessed Issue 649, Windows and incorrect results for AES when used on CPUs without AES-NI.

Travis

Travis tests Android, iOS, Linux and OS X. Both Debug and release builds are tested for Linux and OS X. Android tests armeabi, armeabi-v7a, arm64-v8a, mips, mips64, i686 and x86_64. iOS tests armv7a, arm64 and i386. iOS testing include building for AppleTV and WatchOS.

Cirrus

Cirrus tests FreeBSD builds. Both Debug and release builds are tested for FreeBSD.

Analysis Tools

The source code is subjected to six different phases of analysis. Loosely speaking, a clean compile with reasonable warnings is a security gate, as is additional static and dynamic analysis. To help ensure a trouble free release, Crypto++ is tested with the following:

  • Compiler warnings
  • Clang and GCC Undefined Behavior Sanitizer (UBsan)
  • Clang and GCC Address Sanitizer (Asan)
  • Clang and GCC Bounds Sanitizer (Bsan)
  • Valgrind at -O1 optimizations
  • Visual Studio Enterprise Analysis
  • Coverity (anyone can view the defects)

Compiler Warnings

The library tests with elevated warnings on a regular basis. What constitutes "reasonable warnings" is subjective, but its roughly -Wall -Wextra under modern GCC (and compatible compilers), and /W4 under Visual Studio and the VC++ compilers.

All warnings are managed but not necessarily cleared. The example below is a classic problem that can be managed but not cleared. The issue is, when T = unsigned then t < 0 is superfluous because t cannot be less than 0. Also see Comparison is always false due to limited range ... on Stack Overflow. 

template <class T>
void DoFoo(const T t)
{
    if (t < 0)
        ...
}

Older compilers, like GCC 4.2.1 on OpenBSD, are not tested with elevated warnings because there's nothing new to find. The issue should have already been found with a modern GCC compiler, and it is hard to manage false positives because older GCC compilers lack the pragma GCC diagnostic {push|pop} and pragma GCC diagnostic ignored.

Clang and GCC Sanitizers

The library regularly tests under sanitizers like Asan and UBsan. The GNUmakefile has targets ubsan and asan so testing support is built in. If you are using Clang 3.2 or GCC 4.8 (or higher), then issue make ubsan or make asan. The makefile will set the appropriate compiler options and Crypto++ defines. Once built, issue ./cryptest.exe v or ./cryptest.exe tv all to execute under the sanitizers.

cryptest.sh also tests with -fsanitize=bounds-strict and Control-flow Enforcement Technology (CET) if they are available. Our CET testing is more like a shadow-stack compatibility test since we don't have ROP gadgets built to exploit the library.

There are no makefile targets to add the flags automatically for Bounds Sanitizer or CET. If the cpu and compiler support the necessary features then you can do the following: 

CXXFLAGS="-DNDEBUG -g2 -O1 -fsanitize=bounds-strict" make -j 4

Or: 

CXXFLAGS="-DNDEBUG -g2 -O1 -fcf-protection=full -mcet" make -j 4

Valgrind

To test the library under Valgrind, build the library as normal using -O1 optimization level: 

export CXXFLAGS="-DNDEBUG -g2 -O1"
make -j 4

Once built, issue ./cryptest.exe v or ./cryptest.exe tv all to execute under Valgrind.

A related page is Profiling on the Crypto++ wiki. It discusses some known leaks on different platforms, like OS X and Microsoft.

Enterprise Analysis

Under Visual Studio when Enterprise Analysis is available, the the library tests both the solution's cryptlib and cryptest projects.

Coverity

Coverity is an offline code analysis service. The project regularly tests under Unix, Linux, OS X and Windows. See the Coverity Scan page for the recipes we use and other details, like OS X workarounds.

The Crypto++ project reports are world readable, so you should be able to view them directly. Crypto++ currently enjoys a 0.01 defect rate based on about 167,000 92,800 lines of code.

libumem

libumem is a Solaris library used to detect memory management bugs in applications. We use it for testing on occasion to ensure we are clean: 

LD_PRELOAD=libumem.so ./cryptest.exe v
LD_PRELOAD=libumem.so ./cryptest.exe tv all

clang-tidy

The project uses clang-tidy on occasion. It was responsible for Commit 58726498f100.

This checker generates too much noise so it is not used in practice. For example, the library still supports C++03 so we can't use auto or std::span.

cppcheck

The project uses cppcheck on occasion. It has not identified any issues (yet).

False Positives

Compiler warnings and Coverity produce false positives on occasion. Sometimes that means Crypto++ has to work around a problem that does not really exist. For example, the following was recently checked-in because Clang flagged k as possibly uninitialized even though Decode initialized the value. 

-   unsigned int k, count, repeater;
+   unsigned int k = 0, count = 0, repeater = 0;
    bool result = codeLengthDecoder.Decode(m_reader, k);
    if (!result)
        throw UnexpectedEndErr();

This is a known issue with Clang's inter-procedural analysis engine, and we simply view it as "playing and working nicely with the tools". If the compiler feels inclined, then it can discard the unneeded initialization during optimization.

UBsan, Asan and Valgrind operate on real data, and they do not produce false positives. That has two implications for Crytpo++. First, the library values complete sets of self tests because its an opportunity to help vet trouble spots. Second, UBsan, Asan and Valgrind are security gates, and the findings must be cleared (or the risk documented) to move beyond them.

The Mashup

The following is the mashup various CPUs, compilers and platforms tested before a release. Keep in mind the mashup implicitly includes variations, like Debug (-DDEBUG -Og) vs Release (-DNDEBUG -O{2|3|s}), C++03 vs C++11, and C++ standard libraries like GNU's -stdlib=libstdc++ vs LLVM's -stdlib=libc++. And these platforms are in addition to those tested by way of GNU's GCC compile farm.

The Mashup Matrix
Platform CPU Details or Comments
AIX 7 POWER8 via GNU's GCC compile farm
CentOS 5 x86_64 GCC 4.1 compiler
Cygwin i686 Running on Windows 8.1
Cygwin64 i686 Running on Windows 8.1
Debian 8 S/390 and S/390x QEMU and embedded S/390 chroot
Fedora 1 i386 GCC 3.2 compiler (requires GNU Make 3.81)
Fedora 33* i686 and x86_64 Latest GCC compiler
Mustang ARM64 ARM64 server board from Robert Nelson's test environment
Raspberry Pi 1 ARM ARMv6
Raspberry Pi 3 ARMHF ARMv7
Raspberry Pi 4 ARM64 ARMv8
iOS 6 & iOS 7 ARM and ARM64 Cross-compile, modified Clang
DragonFly x86_64 GCC compiler
OpenBSD 6.6 x86_64 GCC 4.6 compiler
OS X 10.5 PowerPC Xcode 3.2, GCC 4.2.1
OS X 10.9 x86_64 Modern C++ standard library, Xcode 7
OS X 10.12 x86_64 Modern C++ standard library, Command Line Tools
OS X 11.0 arm64 Apple M1, Command Line Tools
Solaris 9-11 x86 and Sparc via OpenCSW
Solaris 11.3 x86_64 DL360 G5
Ubuntu 4 i386 GCC 3.2 compiler (requires GNU Make 3.81)
Ubuntu 18 x86_64 GCC and Clang compilers
Ubuntu 20 x86_64 GCC and Clang compilers
Windows XP x86_64 Visual Studio 2003 and 2005
Windows Vista x86_64 Visual Studio 2008 and 2010
Windows 7 x86_64 Visual Studio 2010 and 2012
Windows 7 x86_64 Visual Studio 2012 and Enterprise Analysis (/analyze)
Windows 8 x86_64 Visual Studio 2012 and 2015
Windows Phone 8 & 10 ARM Command line build using cryptest.nmake
Windows Store 8 & 10 ARM Command line build using cryptest.nmake
Windows Store 10 ARM64 Command line build using cryptest.nmake
Windows Universal Platform ARM & x86 Command line build using cryptest.nmake

* Though Fedora 33 is listed, the latest Fedora is used because it provides the latest tools (and latest GCC) conveniently packaged into a distribution.

Notably missing from the list is Embarcadero (formerly Borland) C++ Builder. The project asked for a license, but the company decided against providing one. Also missing is the Comeau compiler and the Dinkumware Standard C++ library. The Crypto++ project wrote to both companies and asked for the tools to use for testing, but the emails went unanswered.

CPUs

Crypto++ is tested under the following CPUs:

  • i686, x86_64
  • ARM, ARM64
  • MIPS, MIPS64
  • Sparc, Sparc64
  • PowerPC

i686 and x86_64 are commodity PCs. They are implicitly tested during the course of platform testing.

ARM, ARM64, MIPS and MIPS 64 are tested under cross-compiles for iOS and Android.

The PowerPC is a big endian processor provided by a Apple PowerMac running OS X 10.5. The machine only exists for testing Crypto++ (the OpenSSL and Cryptlib projects use it on occasion).

In addition, the Crypto++ project uses GNU's GCC compile farm.

Compilers

Each compiler has its own personality, and each has different strengths and weaknesses. Microsoft's compiler is very forgiving or accommodating, while Intel's compiler is ruthless when it comes to undefined behavior. To help ensure a trouble free release, Crypto++ is tested under the following compilers:

  • Microsoft's VC++
  • LLVM's Clang
  • GNU's GCC
  • Oracle SunCC
  • Intel's ICC
  • IBM XL C/C++

Platforms

Each platform has its own personality, just like each compiler, and each platform provides a unique environment. To help ensure a trouble free release, Crypto++ is tested under the following platforms:

  • Microsoft Windows
  • OpenBSD, NetBSD and FreeBSD
  • Ubuntu and Fedora
  • Solaris 10 and 11
  • Apple OS X and iOS
  • Solaris
  • Android

In addition, the Crypto++ project uses GNU's GCC compile farm and a number of Debian QEMU/Chroot environments.

Testing Gaps

Testing suffers a few known gaps. The gaps are enumerated below. If you can help with them, please contact us.

  1. "ports" on OS X are not well tested. In particular, only MacPorts is tested; and Fink, Gentoo/Alt, Homebrew, Pkgsrc and Rudix are not tested.
  2. MinGW is not well tested. In particular, MinGW-i686 is tested on a Windows 7 machine, and MinGW-64 is tested on a Windows 8 machine.
  3. Android and iOS mobile platforms are only tested to build. The test case is a default configuration provided by GNUMakefile-cross.
  4. Windows Phone and RT mobile platforms are not tested because the platforms are crippled to the point the library is probably insecure and nearly useless. Its probably insecure because the random number generator does not work. (Don't believe the Channel 9 videos on porting a DLL to Windows Phone).

Downloads

cryptest.sh - archive with script to test Crypto++ using various configurations.

Retrieved from "https://www.cryptopp.com/w/index.php?title=Release_Process&oldid=29530"