Why is an exception thrown when taking code out of the __del__ method in change_extension()?

Question

Python

There is a procedure that returns the name of the file with the extension replaced, and depending on the rename flag, can rename an existing file:

def change_extension(file_name, new_extension, rename=False):
    new_file_name = os.extsep.join([os.path.splitext(file_name)[0], new_extension])
    if rename:
        if os.path.exists(new_file_name):
            os.remove(new_file_name)
        os.rename(file_name, new_file_name)
    return new_file_name

There is a class whose __del__ method uses exactly the same code as in the procedure above. The __del__ method always works without errors if the file rename code is written directly in it, and causes an error on the last pass if the code is moved to a separate change_extension() function:

class CSVWriter():
    # Initialize and setup dedicated CSV writer
    def __init__(self, file_name_xl3, output_folder, filter_name, filter_function):
        file_name = '.'.join([os.path.split(file_name_xl3)[1][:-4], filter_name, 'tmp'])
        self.file_name = os.path.join(output_folder, file_name)
        self.file_name_xl3 = file_name_xl3
        self.output_folder = output_folder
        self.file_handle = open(self.file_name, 'wb')
        self.filter_name = filter_name
        self.filter_function = filter_function
        self.writer = csv.writer(self.file_handle, dialect=csv.excel_tab, quoting=csv.QUOTE_NONE, delimiter=CFG_CSV_DELIMITER)
    # Automatically close output file in the end and change extension to 'csv'
    def __del__(self):
        self.file_handle.close()
        # change_extension(self.file_name, 'csv', rename=True)  # Why change_extension with the same code causes error?
        new_file_name = os.extsep.join([os.path.splitext(self.file_name)[0], 'csv'])
        if os.path.exists(new_file_name):
            os.remove(new_file_name)
        os.rename(self.file_name, new_file_name)
    # Write line to output CSV file if xdr filter has been passed
    def write_xdr(self, xdr, bypass=False):
        if bypass or self.filter_function(xdr):
            return self.writer.writerow(xdr)

Exception in console:

Exception TypeError: "'NoneType' object is not callable" in <bound method CSVWri
ter.__del__ of <__main__.CSVWriter instance at 0x020565D0>> ignored
Exception TypeError: "'NoneType' object is not callable" in <bound method CSVWri
ter.__del__ of <__main__.CSVWriter instance at 0x020566C0>> ignored
Exception TypeError: "'NoneType' object is not callable" in <bound method CSVWri
ter.__del__ of <__main__.CSVWriter instance at 0x02056710>> ignored
Exception TypeError: "'NoneType' object is not callable" in <bound method CSVWri
ter.__del__ of <__main__.CSVWriter instance at 0x02056670>> ignored

What could be the problem?

Answer 1

Python is not C++, there are NO destructors in Python (in the C++ sense) that are guaranteed to be called. __del__ is not a direct analogue of the C++ destructor, its behavior depends on the GC, it can be called at any time, and in general there is no guarantee that it will be called at all. In general, __del__ in Python is bad practice.
What to do:

1. We should try to refactor the code so that the object requiring finalization is used via with(your_object). Good examples should be followed, for example, a file object, which is handled through with(open('workfile')).
   
2. If it is not possible to reduce the code logic to such a model at all, then you can declare a method close()and call it manually. But if your code is spread out so that you can't screw with into it, then there are no guarantees that you will forget to call this on some branch of the algorithm close().
   
3. Worst option: leave __del__. Nowhere to call it explicitly, just throw objects after use. At the end of the script execution, the GC will pick them up and call them all __del__, but only if there are no cyclic references between them, but in a sufficiently large program this cannot be guaranteed at all.