Static Initialization Order Fiasco

From Crypto++ Wiki
Jump to navigation Jump to search

The C++ Static Initialization Order Fiasco is a well known problem with C++ programs that use static class objects. The problem does not affect plain old datatypes (POD) (like integers and pointers). The problem is the order of initialization is only guaranteed within a translation unit, and the C++ language provides no facilities to control initialization order when multiple translation units or object files are combined into a library or program.

Crypto++ can suffer the static initialization fiasco on occasion. We can only say can because the problem surfaces under certain conditions, and its not readily apparent to the user that the problem has surfaced. The problem surfaces when the linker creates a library or program, and the order objects are linked violates implicit dependencies (the dependencies themselves are not readily apparent).

The library addressed the static initialization order problem at Crypto++ 5.6.3. This page discusses the static objects, some of the symptoms and the remediations put in place to address the issue. It also discusses some of the things that don't work or leave residual gaps.

Static Objects

The Crypto++ library has three static C++ objects that are sensitive to initialization order. Two of them are std::strings (DEFAULT_CHANNEL and AAD_CHANNEL), and one of them is a Crypto++ NameValuePair class (g_nullNameValuePairs). All three are located in cryptlib.cpp. In addition, the test driver program, cryptest.exe, has one static C++ object (a global random number generator), and it is located in test.cpp. The random number generator uses the string DEFAULT_CHANNEL, so the string must be constructed first.

If the C++ static objects are not initialized in an order consistent with the explicit dependencies, then your program could crash in mysterious and not-so-apparent ways.

Symptoms

Symptoms will usually surface in one of three ways. First, you will see a crash dereferencing a this object because this in NULL. Second, you will witness a crash in std::string. The string one is usually due to DEFAULT_CHANNEL being used before the string is constructed. For example, if you have a global random number generator (like in test.cpp) and it is constructed before the std::string (DEFAULT_CHANNEL), then you will experience a crash in std::string because this pointer is probably NULL. If you get unlucky, then the this pointer will be garbage, which will appear to indicate it has been initialized. Since BufferTrandsformation and DEFAULT_CHANNEL are cornerstones of most Crypto++ classes, the dependency issue almost always exists.

The third symptom is sometimes an uninitialized read on platforms like OS X when using Valgrind. Though not readily apparent, the finding is due to use of file scope bools in cpu.cpp: g_hasMMX, g_hasAESNI, g_hasCLMUL and friends. Valgrind flags them as uninitialized even though they are file scope PODs with 0-value that are supposed to be initialized by the BSS. An example is below. 

==12386== Conditional jump or move depends on uninitialised value(s)
==12386==    at 0x572564C: DetectX86Features() (cpu.cpp:158)
==12386==    by 0x56E253F: HasCLMUL() (cpu.h:163)
==12386==    by 0x56DF340: GCM_Base::SetKeyWithoutResync(unsigned char const*, unsigned long, NameValuePairs const&) (gcm.cpp:141)
==12386==    by 0x563B5F7: AuthenticatedSymmetricCipherBase::SetKey(unsigned char const*, unsigned long, NameValuePairs const&) (authenc.cpp:53)
==12386==    by 0x57275D4: SimpleKeyingInterface::SetKeyWithIV(unsigned char const*, unsigned long, unsigned char const*, unsigned long) (cryptlib.cpp:73)

User Code

The TLDR portion of this page: if you compile the library and the library is Crypto++ 5.6.3 or higher, then you should perform the following.

  1. Uncomment CRYPTOPP_INIT_PRIORITY in config.h. That mostly handles the C++ initialization order fiasco for the library.
  2. If your code has C++ static objects that depend on Crypto++ library components, then start numbering your remediations at CRYPTOPP_USER_PRIORITY. That mostly handles the C++ initialization order fiasco for user code.
  3. If your code does not have C++ static objects that depend on Crypto++ library components, then you are in a "don't care" state and you can do nothing for user code.

If you are using Crypto++ 5.6.2 or less, then you should upgrade to Crypto++ 5.6.3. This is a crummy option, but we don't know how else to improve the situation. There are some tricks discussed below in Remediations, but they are incomplete remdiations.

Remediations

The Crytpo++ library attempts to remediate initialization problems using four strategies. We can only say attempts because the problem is a defect in the C++ language, and the work arounds are not a complete remediation. They are not a complete remediation because some platforms, configurations and use cases are left with residual gaps.

Minimize C++ Statics

The first remediation attempts to avoid or minimize the problem by avoiding the use of static C++ variables. We can only say minimize because some are required (also see Don't Use C++ Statics below). On the other hand, Plain Old Datayptes (PODs) require the use of static because a C++ object can displace when the initialization of the POD occurs.

A POD must have a static qualifier if a C++ object is directly or indirectly interacting with it. For example, static bool flag = false is different than bool flag = false because the initialization order fiasco affects the initialization of the of flag.

Concentrating C++ Statics

The second remediation attempts to gather all C++ static objects and place them in a single source file. The source file is cryptlib.cpp, and the objects are declared in an order that respects their implicit dependency needs.

This is not a complete remediation because a user's C++ static objects are not included in cryptlib.cpp when linking against the static archive.

Ordering of Object Files

The third remediation attempts to ensure the object files are linked in an order that respects implicit dependency needs. That is, the first object file in the static archive is cryptlib.o, the second object file is cpu.o and the remaining object files are "don't cares". When the dynamic library is created from the static archive, the constructors for the C++ objects cryptlib.o are invoked first. This is implemented in the GNUmakefile with: 

# List cryptlib.cpp first in an attempt to tame C++ static initialization problems. The problem spills
#  into POD data types, so cpu.cpp is the next candidate for explicit initialization order.
SRCS := cryptlib.cpp cpu.cpp $(filter-out cryptlib.cpp cpu.cpp pch.cpp,$(wildcard *.cpp))

# List of objects with crytlib.o at the first index position
OBJS := $(SRCS:.cpp=.o)

This is not a complete remediation because a user's C++ static objects are not included in the static archive. Additionally, this is a undocumented feature of the linker that is not guaranteed to work.

Another subtle problem with this remediation is compiler drivers like Clang, GCC and ICPC work against object file ordering because they place the libraries after user object files when invoking the linker. Static archives are just a collection of object files, so they can usually be used interchangeably with them.

As a concrete example, g++ myprogram.cpp -o myprogram.exe -lcryptopp performs the exact opposite of what we want and need. That's because the compiler driver effectively invokes the linker with myprogram.o libcryptopp.a rather than libcryptopp.a myprogram.o. And this is the exact situation that occurs when cryptest.exe (with its global random number generator) is linked against libcryptopp.a (with the static objects that need to be initialized first); and its the reason Valgrind produces the finding under Symptoms.

Attributes and Pragmas

The fourth remediation attempts to ensure C++ static objects are declared with compiler specific decorations so the compiler and linker can tend to them. Under GCC compatible compilers, init_priority is used. Under Visual Studio compatible compilers, a #pragma init_seg is used. They are keyed on a #define introduced at 5.6.3 in config.h: CRYPTOPP_INIT_PRIORITY.

The declarations of objects in cryptlib.cpp under GCC compatible compilers are: 

#define HAVE_GCC_INIT_PRIORITY (__GNUC__ && (CRYPTOPP_INIT_PRIORITY > 0) && !(MACPORTS_GCC_COMPILER > 0))
...

#if HAVE_GCC_INIT_PRIORITY
const std::string DEFAULT_CHANNEL __attribute__ ((init_priority (CRYPTOPP_INIT_PRIORITY + 25)));
const std::string AAD_CHANNEL __attribute__ ((init_priority (CRYPTOPP_INIT_PRIORITY + 26))) = "AAD";
const std::string &BufferedTransformation::NULL_CHANNEL = DEFAULT_CHANNEL;

const simple_ptr<NullNameValuePairs> s_pNullNameValuePairs __attribute__ ((init_priority (CRYPTOPP_INIT_PRIORITY + 30))) = new NullNameValuePairs;
const NameValuePairs &g_nullNameValuePairs = *s_pNullNameValuePairs.m_p;
#endif

The declarations of objects in cryptlib.cpp under Microsoft compatible compilers are: 

#define HAVE_MSC_INIT_PRIORITY (_MSC_VER && (CRYPTOPP_INIT_PRIORITY > 0))
...

#if HAVE_MSC_INIT_PRIORITY
#pragma init_seg(lib)
const std::string DEFAULT_CHANNEL;
const std::string AAD_CHANNEL = "AAD";
const std::string &BufferedTransformation::NULL_CHANNEL = DEFAULT_CHANNEL;

const simple_ptr<NullNameValuePairs> s_pNullNameValuePairs(new NullNameValuePairs);
const NameValuePairs &g_nullNameValuePairs = *s_pNullNameValuePairs.m_p;
#endif

This is not a complete remediation because some platforms and compilers don't provide the ability to declaratively provide an explicit initialization order. For example, Solaris and MacPorts GCC compilers fail to compile a program with init_priority (also see MacPorts Issue 37664).

In the case of GCC provided by MacPorts, use of HAVE_GCC_INIT_PRIORITY is guarded by MACPORTS_GCC_COMPILER. MACPORTS_GCC_COMPILER is set on the command line by the GNUmakefile. Its set by the makefile because MacPorts does not provide a way to detect its compiler in the preprocessor. 

MACPORT_COMPILER := $(shell $(CXX) --version 2>&1 | $(EGREP) -i -c "macports")
...

ifneq ($(MACPORT_COMPILER),0)
ifneq ($(GCC_COMPILER),0)
cryptlib.o:
	$(CXX) $(CXXFLAGS) -DMACPORTS_GCC_COMPILER=1 -c cryptlib.cpp
endif
endif

Other Failures

Other things that don't work when attempting to remediate the issue are discussed below.

Don't Use C++ Statics

The problem with this strategy is its simply not realistic. Crypto++ is a case in point: it has over 1400 class files, and there's no way to reliably and efficiently remove the three C++ static objects. Here, reliably and efficiently implies a few things, like guaranteeing the compiler pools all uses on the string AAD, which is what the AAD_CHANNEL provides. If the pooling does not occur, then the program wastes space as the duplicate string is stored, wastes cycles when the string is reconstructed, and could suffer cache and locality penalties. And there's no way for the compiler to pool the use of a nil NameValuePair because pooling occurs on C-strings.

All that really happens in this case is a slight of hand - one problem was transformed into another problem.

Use C++ Static Locals

The problem with this strategy is the problem still manifests itself in the destructors. That is, if user code needs one of the Crypto++ static objects during exit, then the object may be destroyed too soon. When the user code executes (and presuming one of the objects was destroyed), then the user code could crash in its destructors. In fact, the author experienced this problem in high integrity software. The software had AAA logging requirements, and the program crashed upon exit when logging the shutdown because the channel string was destroyed too soon.

All that really happens in this case is a slight of hand - one problem was transformed into another problem.

Linker Scripts

Some folks use linker scripts under GCC to tame the static initialization problem. Unfortunately, the GCC folks don't recommend it, and the toolchain recently broke some projects that were using the feature. See, for example, Replace .ctors/.dtors with .init_array/.fini_array on targets supporting them and collect2 breaks link order control.

Additionally, its not clear to us how to use linker scripts of other platforms, like Solaris.

Also see Apple's Module Initializers and Finalizers

MacPorts Compilers

MacPorts supplies a GCC compiler that does not consume __attribute__ ((init_priority)) or __attribute__ ((constructor)). See gcc46, gcc47 'init_priority' attribute is not supported on this platform for more details. Note: this problem appears to be local to MacPorts and its GCC. Apple supplied GCC does not suffer the issue. Other ports, like Fink, do not suffer the issue. In addition, other compilers, like Clang, do not suffer the issue.

Additionally, MacPorts provides no way to detect the compiler it supplies when the library and programs are being compiled, so the project is in a uncomfortable position. Also see How to reliably detect a MacPorts-ported compiler? on the MacPorts Users mailing list. If you are a MacPorts user attempting to use the GCC compiler, then you have the following options.

  • Don't use a MacPorts compiler
  • Use a MacPorts supplied Crypto++
  • Set #define SUPPORTS_INIT_PRIORITY=1 in GCC's gcc/config/darwin.h
  • Set #define CRYPTOPP_INIT_PRIORTY=0 in Crypto++'s config.h
  • Don't use MacPorts

Don't use a MacPorts compiler - while this seems like a poor option, it appears to be the option MacPorts itself takes by default. It does not build its ports using its compilers.

Use a MacPorts supplied Crypto++ - MacPorts has worked around the issue, so you should not experience trouble when choosing this option. Note: we take no position on how MacPorts fixed it. In fact, we don't even know what they did (or of they did anything other than avoid their MacPorts compiler).

Set #define SUPPORTS_INIT_PRIORITY=1 in GCC's gcc/config/darwin.h - this option configures GCC before building it to avoid the broken compile. This is a complete remediation to the compile failure, and it only needs to be done once. Also see How to change MacPorts config setting before building a Package? on Super User.

Set #define CRYPTOPP_INIT_PRIORTY=0 in Crypto++'s config.h - setting #define CRYPTOPP_INIT_PRIORTY to 0 disables the feature. It simply allows you to build the library without a compile failure.

Don't use MacPorts - this might be a viable option if you are looking for something that "just works". Other ports, like Fink, do not suffer the issue. However, we view it as a crummy option because a user lost a choice.

Penultimate Fix

The penultimate fix for the library's C++ Static Initialization Order Fiasco is the C++ language to provide the tools we need to declaratively control the order of initialization and destruction. Once the library can control its order of initialization and destruction, then user code will be able to reliably control the initialization and destruction order of their static objects.

If the C++ committee ever addresses the issue (with something other than Ostrich Algorithm), then the library will use it. Until the C++ committee provides the support, it will be a intermittent, hit or miss problem.

Retrieved from "https://www.cryptopp.com/w/index.php?title=Static_Initialization_Order_Fiasco&oldid=28899"