Discussion:
Need help in debugging a stack weirdness
(too old to reply)
mclagett
2010-09-15 17:58:03 UTC
Permalink
Raw Message
Hi --

Here's a message I posted over in the forum for the Visual Studio 2010
debugger. They informed that there is no support over there for dump
analysis and directed me to this forum. I'm not using windbg per se, but
rather the integrated Visual Studio 2010 debugger.

What I can add to my original message below is that the behavior is
consistent across runs, and occurs at the same point every time in a
completely reproducible fashion. This is calling code that gets called
elsewhere successfully, but this time after executing some new code that
hasn't been executed before. So I feel pretty confident that something my
code has done has corrupted some key debugger menory somewhere. The thing
that makes me suspect this is that it is primarily single stepping that
causes this behavior. If I go and set explicit breakpoints on each next line
and step through my code that way, it seems to allow me to do that without
exhibiting the behavior described below. (Although it still hangs eventually
the second time through.)

The weird thing is that after the problem occurs the first time through,
(without crashing and eventually reaching some breakpoint I have set), the
program executes lots of code and allows me to single step or set breakpoints
in exactly the fashion I usually do, until I reach this point in the code
again. Then the same thing happens -- although as i mentioned, the second
time through the program either crashes or eventually hangs if I single step
by setting explicit breakpoints as described above. I don't think it's the
HeapAlloc logic itself, since the problem still occurs in exactly the same
fashion even if I create a heap with twice as much memory.

Could it be a stack depth issue? I do believe this may be the deepest use
of the stack in the program execution so far and the code I mentioned above
that is executed for the first time before this call to HeapAlloc happens
does itself extend the stack by a significant amount. Still, I thought there
was something like 1MB of default stack space. And if there were a stack
overflow, wouldn't I get error messages to that effect? The error message I
do get is a memory access violation on an instruction that is accessing that
portion of the stack that previously was accessible (i.e. the locations that
are the top of stack befor the call I describe below that extends the stack).

Hope this is all clear. I'm sorry for the length, but I am absolutely
totally deperate.

I don't expect people here to be able to diagnose my problem for me, but I
just would like some insight on techniques I might use to do this myself.
This is an application that I have put a tremendous time and effort into and
I desperately need to get past this impasses.

Thanks for any help anyone can provide.

Here's my original message:

I'm developing a virtual machine platform that among other things allows me
to call out to C++ Windows OS functions whose address I pass into it. I have
a lot of code that uses this mechanism, so I'm pretty confident that it
generally functions correctly. But I am now encountering some weird behavior
when I call out to the HeapAlloc function.

When I trace through the call, the stack looks like it is set up correctly
and the stack pointer seems to be correct. Moreover as I start to execute
the HeapAlloc itself, I trace through the code execution and see that the
instructions (this is the disassembly of Microsoft's HeapAlloc
implementation) seem to be manipulating the stack correctly. Here's what the
first eight or so lines of code look like that the initial HeapAlloc entry
point jumps to:

7C9100C4 push 204h
7C9100C9 push 7C9101E0h
7C9100CE call 7C90E8CB
7C9100D3 mov ebx,dword ptr [ebp+8]
7C9100D6 mov dword ptr [ebp-1Ch],ebx
7C9100D9 xor edi,edi
7C9100DB mov dword ptr [ebp-30h],edi
7C9100DE mov byte ptr [ebp-1Eh],0

I step over the first two push instructions and then step into the function
called on line 3. Everything appears normal during this execution (which
basically adjusts the stack frame size -- adding the 0x204 bytes that
specified in the first line above before returning). Instruction by
instruction stepping through this called routine appears to execute correctly
and I observe the changes to the stack induced by this code. But when I
execute the return instruction, I'm brought back to the right place in the
code snippet above (it comes back to the fourth line right after the call, as
it should). But for some reason the yellow symbol that marks one's place in
stepwise debugging has turned from its normal yellow to green. And worse, if
I step over the next instruction (the mov ebx,dword ptr [ebp+8] on the fourth
line above), the first time through this function execution stops executing
line by line and I am proceeds automatically to the next place in code that I
have a breakpoint (which is fairly far away from the code I was stepping
through. So bizarre. The second time through the function when execution
tries beyond the same point in the function an exception is thrown.

Does any of this look familiar to any of you. And do you have any ideas for
me about how to troubleshoot this problem. I'm sure it's something I've done
(it always is), but I don't even begin to have a clue on how to track this
down.

Any thoughts would be greatly appreciated. Thanks.
mclagett
2010-09-20 16:57:03 UTC
Permalink
Raw Message
Figured it out. I was writing to memory I allocated and did not have the
proper strcpy logic in place and was overwriting the bounds of the memory I
allocated. I knew it had to be something like that, but I just kind of
panicked. Sorry for the noise
Post by mclagett
Hi --
Here's a message I posted over in the forum for the Visual Studio 2010
debugger. They informed that there is no support over there for dump
analysis and directed me to this forum. I'm not using windbg per se, but
rather the integrated Visual Studio 2010 debugger.
What I can add to my original message below is that the behavior is
consistent across runs, and occurs at the same point every time in a
completely reproducible fashion. This is calling code that gets called
elsewhere successfully, but this time after executing some new code that
hasn't been executed before. So I feel pretty confident that something my
code has done has corrupted some key debugger menory somewhere. The thing
that makes me suspect this is that it is primarily single stepping that
causes this behavior. If I go and set explicit breakpoints on each next line
and step through my code that way, it seems to allow me to do that without
exhibiting the behavior described below. (Although it still hangs eventually
the second time through.)
The weird thing is that after the problem occurs the first time through,
(without crashing and eventually reaching some breakpoint I have set), the
program executes lots of code and allows me to single step or set breakpoints
in exactly the fashion I usually do, until I reach this point in the code
again. Then the same thing happens -- although as i mentioned, the second
time through the program either crashes or eventually hangs if I single step
by setting explicit breakpoints as described above. I don't think it's the
HeapAlloc logic itself, since the problem still occurs in exactly the same
fashion even if I create a heap with twice as much memory.
Could it be a stack depth issue? I do believe this may be the deepest use
of the stack in the program execution so far and the code I mentioned above
that is executed for the first time before this call to HeapAlloc happens
does itself extend the stack by a significant amount. Still, I thought there
was something like 1MB of default stack space. And if there were a stack
overflow, wouldn't I get error messages to that effect? The error message I
do get is a memory access violation on an instruction that is accessing that
portion of the stack that previously was accessible (i.e. the locations that
are the top of stack befor the call I describe below that extends the stack).
Hope this is all clear. I'm sorry for the length, but I am absolutely
totally deperate.
I don't expect people here to be able to diagnose my problem for me, but I
just would like some insight on techniques I might use to do this myself.
This is an application that I have put a tremendous time and effort into and
I desperately need to get past this impasses.
Thanks for any help anyone can provide.
I'm developing a virtual machine platform that among other things allows me
to call out to C++ Windows OS functions whose address I pass into it. I have
a lot of code that uses this mechanism, so I'm pretty confident that it
generally functions correctly. But I am now encountering some weird behavior
when I call out to the HeapAlloc function.
When I trace through the call, the stack looks like it is set up correctly
and the stack pointer seems to be correct. Moreover as I start to execute
the HeapAlloc itself, I trace through the code execution and see that the
instructions (this is the disassembly of Microsoft's HeapAlloc
implementation) seem to be manipulating the stack correctly. Here's what the
first eight or so lines of code look like that the initial HeapAlloc entry
7C9100C4 push 204h
7C9100C9 push 7C9101E0h
7C9100CE call 7C90E8CB
7C9100D3 mov ebx,dword ptr [ebp+8]
7C9100D6 mov dword ptr [ebp-1Ch],ebx
7C9100D9 xor edi,edi
7C9100DB mov dword ptr [ebp-30h],edi
7C9100DE mov byte ptr [ebp-1Eh],0
I step over the first two push instructions and then step into the function
called on line 3. Everything appears normal during this execution (which
basically adjusts the stack frame size -- adding the 0x204 bytes that
specified in the first line above before returning). Instruction by
instruction stepping through this called routine appears to execute correctly
and I observe the changes to the stack induced by this code. But when I
execute the return instruction, I'm brought back to the right place in the
code snippet above (it comes back to the fourth line right after the call, as
it should). But for some reason the yellow symbol that marks one's place in
stepwise debugging has turned from its normal yellow to green. And worse, if
I step over the next instruction (the mov ebx,dword ptr [ebp+8] on the fourth
line above), the first time through this function execution stops executing
line by line and I am proceeds automatically to the next place in code that I
have a breakpoint (which is fairly far away from the code I was stepping
through. So bizarre. The second time through the function when execution
tries beyond the same point in the function an exception is thrown.
Does any of this look familiar to any of you. And do you have any ideas for
me about how to troubleshoot this problem. I'm sure it's something I've done
(it always is), but I don't even begin to have a clue on how to track this
down.
Any thoughts would be greatly appreciated. Thanks.
Loading...