Py_Finalize() Resulting In Segmentation Fault For Python 3.9 But Not For Python 2.7

I'm working on a project which uses this C++ matplotlib wrapper matplotlibcpp.h. A minimal example using this original header file is #include 'matplotlibcpp.h' namespace

Solution 1:

I don't have an easy access to a Linux where I can test it, but I think I now understand what's happening.

1. matplotlibcpp uses a static variable to hold the Python interpreter (see line 129 inside interkeeper(bool should_kill)). Like C++ static function variables, it's initialized on the first time the function is called and destructed on program's exit (reference).

2. When main finishes, libc runs cleanup routines for all the shared libraries and for your program (that's __run_exit_handlers in the stacktrace). Since your program is a C++ program, part of its exit handler is destructing all the static variables that were used. One of them is the Python interpreter. Its destructor calls Py_Finalize() which is Python's cleanup routine. Until now, everything's fine.

3. Python has a similar atexit mechanism that allows Python code from everywhere to register functions that should be called during the interpreter shutdown. Apparently, the backend matplotlib chose to use here is PyQt5. It seems to register such atexit callbacks.

4. PyQt5's callback gets called, and crashes. Notice that this is internal PyQt5 code now. Why does this crash? My "educated" guess is that Qt's library exit handler was already called in step 2, before your program's exit handler was called. This apparently causes some weird state in the library (maybe some objects were freed?) and crashes.

This leaves two interesting questions:

1. How to fix this? The solution should be to destruct ctx before your program exits, so the Python interpreter is destructed before any shared libraries terminate themselves. Static lifetimes are known for causing similar problems. If changing matplotlibcpp's interface to not use global static states is not a possible solution, I think you really have to manually call plt::detail::_interpreter::kill() at the end of your main function. You should be able to use atexit() and register a callback that kills the interpreter before the library teardown - I haven't tested it though.

2. Why did this ever work? My guess is that maybe something in PyQt5's callbacks has changed that now causes this crash, or that you use a different backend in Python 2. If no other library is destructively terminating before the program exits, this is fine.