Delphi Red Herrings: Hidden Virtual Methods
Trying to Grasp the Depths of a Universe Far, Far Away
Genesis
Working on some development I discovered a memory leak. This is not uncommon, and since I had just made some code changes it should have been easy to track down and fix. "Problem solved!" Unfortunately, it took a lot longer than that and after reading dozens of blog posts, Embarcadero DocWiki pages and many Stackoverflow answers, I was still uncertain of what was happening. Even my favorite Delphi gurus on Stackoverflow had apparently never answered a question quite like mine.
Summary
There's a small demo program at https://github.com/Pasquina/ARCinDelphi.git that you really need to download in order to try some of the simple things I mention here.
There are two classes defined in the program:
- A Parent Class
- A Child Class
The parent is descended from TInterfacedObject and hence the child (as a descendant of Parent) also has TInterfacedObject as an ancestor. A couple of empty interfaces are defined to permit referencing.
The child class uses the inherited directive to invoke the Create and Destroy methods of the parent. When you run the program and push the button (the only object that responds to a click on the small form) instantiation proceeds using interfaces, first a parent concrete instance and then a child concrete instance. As instantiation proceeds, ShowMessage is used so you can follow the events as they take place.
Finally, after both concrete instantiations take place, the routine exits the event handler, causing the instantiated objects to go out of scope. From here, the compiler and ARC take over and dispose of the concrete instances.
As the concrete instantiations are destroyed a ShowMessage tracks the events so you can follow the progress of object lifetimes. The messages should look like this:
Creating Base Class Instance (From Event Handler Code)
Base Object Create (From Base Class Create Method)
Creating Child Class Instance (From Event Handler Code)
Base Object Create (From Base Class Create Method via Inherited Directive)
Child Object Create (From Child Class Create Method)
Exiting Scope (From Event Handler Code at the Very End)
Child Object Destroy (From Child Class Destroy Method)
Base Object Destroy (From Base Class Destroy Method via Inherited Directive)
Base Object Destroy (From Base Class Destroy Method)
Not So Fast
While all of this behavior so far is expected, let's try a small experiment.
Remove the override directive from the Child Class Destroy method and run the program as before.
The first thing you notice is you get that annoying message from the compiler about
method... hides virtual method of base type ...
As far as I can tell, that simply isn't true. You can still "see" the base method with code, reference it, invoke it using inherited and so forth. Hidden from who or what?
Then if you run the code, the ShowMessage trace looks like this:
Creating Base Class Instance (From Event Handler Code)
Base Object Create (From Base Class Create Method)
Creating Child Class Instance (From Event Handler Code)
Base Object Create (From Base Class Create Method via Inherited Directive)
Child Object Create (From Child Class Create Method)
Exiting Scope (From Event Handler Code at the Very End)
Child Object Destroy (Missing! Gone fishing! Disappeared!)
Base Object Destroy (From Base Class Destroy Method via Inherited Directive)
Base Object Destroy (From Base Class Destroy Method)
But wait! There's more. Close the program and you get the following from the ReportMemoryLeaksonShutdown that has been enabled for the program.
An unexpected memory leak has occurred. The unexpected small block leaks are:
Etc., etc.
Put the override directive back on the Child Class destructor and things are fine.
Conclusion
I don't know what the intended behavior is supposed to be for this, but I do know that inadequate documentation of all kinds cost me a lot of time figuring out what was going on. Avoiding the problem is easy enough. Triage is another matter.
If anyone has better answers than the ones I've given, I'd like to hear them. Please leave comments if you have more information on this odd behavior.
To me, it is quite clear what is going on.
ReplyDeleteDestructors in Delphi are virtual methods.
In your case destruction starts at TInterfacedObject when reference count is set to 0 (TInterfacedObject._Release).
TInterfacedObject does not know anything about descendant classes so it calls it's own destructor.
If descendant class does not override the destructor then it will not be visible from TInterfacedObject so it will not be called when TInterfacedObject calls the destructor.
This is why you have to call inherited in destructors - the actual destructor called it the last overridden child's method so to make sure that parent's destructor method is called you have to explicitly call inherited destructors.
As far as I understand your explanation, it is because the child destructor is hidden from the parent, yet that is not what the message says. It says that the parent method is hidden. This is simply incorrect and plainly confusing. Even more troublesome is that there don't seem to be any explanations of the behavior in various scenarios that I was able to find. Hence, the message is a red herring, and distracts you from what's really happening.
DeleteThe message is correct from the viewpoint of the child class where the destructor is declared: Parent virtual method is hidden by non-virtual child method.
DeleteThe implications of this are up to the user to realize and requires understanding of how virtual methods work which is one of the core OO programing concepts - polymorphism.
From the top level view: parent only knows about it's own behavior and child signals that new behavior is required by overriding parent's methods.
From low level view: parent has a virtual method table with pointers to methods it knows how to call and child puts a pointer to it's own method into that table by overriding parent method to signal that parent should call that instead of parent's method. If method is not overridden then it is not visible in parent's VMT so parent cannot call it.
Your facts are correct but I disagree with your reasoning. The message itself is the problem, because it implies that the parent is hidden and by your own text that is only true from the viewpoint of the child. The entire rationale is ambiguous without explanation. That is the point of my blog, not that the concept or implementation details are wrong, but rather the documentation and especially the message is deficient, misleading and ineffective. There are enough questions about this on various forums and Stackoverflow to convince me that the explanation needs to be both improved and made more readily available. How else do we expect to attract new devotees to Delphi, which is often touted as the epitome of ease and clarity?
DeleteI agree that there are many things in Delphi that require better and more detailed documentation.
DeleteI do not agree that this message is the problem of itself but the doc page about it does not explain the consequences so a test program is required to discover what the behavior is.
But if you look at the docs on the web for almost any programming language or OS you will find the same - very basic info with minimal detail and the only way to discover how to use it is to write a test program.
Have you tried "reintroduce" to see if it fixes your issue?
ReplyDeleteAs far as I know, reintroduce merely suppresses the warning message. That really doesn't address my issue. I know I can suppress the message. What I want to know is who the method is hidden from, since as far as I can tell the method remains perfectly visible. Further, if the behavior is changed, what is that change? Documentation I was able to find was never able to explain that. Why do we care about this message?
DeleteXepol, 'reintroduce' will not solve this issue. dmz is correct, the 'override' is required, otherwise the TList member inside of TChildObject will not be freed correctly when ARC destroys the child object.
ReplyDeleteThere is another memory leak to watch out for, too - in TChildObject.SetMyList(), the input TList object is being assigned as-is to the FMyList member, which will leak the previous object and take ownership of the input object. SetMyList() should instead be calling FMyList.Assign() to copy values from the input TList object into the existing TList object, rather than replacing it.
Of course the override is required. That's the point of the demo. In fact, without the override, the entire child destructor is not executed, which you would never know about were it not for the memory leak that occurs. My major issue with this is that none of this is ever really explored by existing documentation (at least that I was able to find.) All the message says is that the parent method is hidden—and it is not hidden. It remains perfectly visible. It is the child method that suddenly becomes impotent and until you realize that, bugs can be difficult to spot.
Delete"All the message says is that the parent method is hidden—and it is not hidden. It remains perfectly visible."
DeleteThis is not correct. If you call Destroy using a variable with the exact type of the class you use, the inherited method won't be called. This is exactly what the message says.
The other way round, that a message won't be called by the parent without override, is common knowledge about inheritance. How could it be possible (without using the slow RTTI or anything like this)?
Both has nothing to do with the destructor, but is just how inheritance works.
I have to disagree with your assessment. The message is clearly incorrect. The parent method is not "hidden", at least from the programmer who writes code. At best the message is ambiguous: Hidden from who? What difference does it make? Why would anyone care?
DeleteThis comment has been removed by the author.
ReplyDeleteThis sounds like the classical interface circular references problem. One of the reference needs to be weak. See https://blog.synopse.info/?post/2012/06/18/Circular-reference-and-zeroing-weak-pointers
ReplyDeleteActually my intent was not to explore circular references, which are their own brand of Delphi annoyance. But rather my intent was to highlight what I consider to be an ambiguous message that incorrectly guides programmers into incorrect paths of thinking and bug resolution. As you can see by the comments, it has definitely provoked some commentary!
DeleteThis is probably best covered by W1010 Method '%s' hides virtual method of base type '%s'
ReplyDeletemethod that hides virtual method is no longer virtual, and it will be statically bound. That mean compiler will resolve it at compile time where it can, comparing to resolving virtual methods that happens at runtime through VMT.
Static resolution, during compile time can only be applied if you directly call such method through object reference that is declared as that particular descendant class. Since destructor will be called through _Release method, compiler will use dynamic dispatching and VMT. It does not know you have static method with the same name.
This is also not related to ARC. If you define Parent and Child classes as TObject descendants you can only call Child destructor through child reference directly
var
Child: TChild;
...
Child.Destroy;
Calling Child.Free will no longer invoke TChild destructor because Free is implemented in TObject and will call virtual destructor chain.
You are absolutely correct, of course. But I stand by my thesis that the message is ambiguous and of little assistance to someone trying to learn the language. It all hinges on the notions of the dispatching mechanism, which seems to be one of the weakest parts of the language documentation that I was able to find.
DeleteIf I ever teach programming to beginning students using Delphi I'll be certain to include this topic for a careful examination (along with the additional kinds of insights described by Remy Lebeau above.)
Thanks for your input.
Yes, message itself carries very little information. But I honestly cannot think of better one, or at least better one that could also more clearly explain the consequences.
DeleteDocumentation could also explain more. Hiding is correct term from the perspective of TChild class because when you refer to the method by name anywhere in the TChild you will call TChild method and not the parent one. Since compiler message is shown only for TChild class hidden is appropriate.
Part that is not explained well, and that happens quite often in real code is that calling the method indirectly from the outside context will actually call virtual TParent (base) method, so it is definitely not always "hidden" as the message may suggests.
One possibility for a better message issued for child declaration... Method duplicates ancestor method of same name and is not declared override. Outside of the current context will reference the ancestor method. The descendant method is not reachable outside of the descendant context. I realize it's a bit long. The second sentence could be removed and only minimally impact the meaning.
Delete