How do I debug Services.exe (and why and what I've tried so far)

[Johan Johansson]

Hi everyone!

I'm hoping that someone with more experience/knowledge than me will know
what the correct configuration of debuggers on what machines are.

What I want to achieve
----------------------
I want to be able to easily check at what time/in what order what
services started. This is somewhat complicated by the fact that many
services run in the same process.

My idea
-------
Set a breakpoint in Services!ScStartService and print what service is
being started. This part is solved, in other words, once I get to a
point where I can set this breakpoint I'm done.

Practicalities
--------------
Loading symbols from network seems to require services.exe to be
running, so I've opted for a debugging server with a smart client
attached to it to ensure that the symbols are looked up on another
machine. But, as far as I can tell I must then create the debugging
server with DbgSrv and entering that as the debugger for Services.exe
under Windows NT\CurrentVersion\Image Execution Options just seems to
halt the boot.

My first approach was to use a kernel debugger but I did not have much
success going that route either.

Suggestions?

Thanks,
Johan

[Pavel Lebedinsky [MSFT]]

You could try caching symbols locally and removing any network
entries from the symbol path. Then you might be able to use
ntsd -server and connect from another machine.

But generally it's easier to debug things like that from kd.
--
This posting is provided "AS IS" with no warranties, and confers no
rights.

Post by Johan Johansson
What I want to achieve
----------------------
I want to be able to easily check at what time/in what order what
services started. This is somewhat complicated by the fact that many
services run in the same process.
My idea
-------
Set a breakpoint in Services!ScStartService and print what service is
being started. This part is solved, in other words, once I get to a
point where I can set this breakpoint I'm done.
Practicalities
--------------
Loading symbols from network seems to require services.exe to be
running, so I've opted for a debugging server with a smart client
attached to it to ensure that the symbols are looked up on another
machine. But, as far as I can tell I must then create the debugging
server with DbgSrv and entering that as the debugger for Services.exe
under Windows NT\CurrentVersion\Image Execution Options just seems to
halt the boot.
My first approach was to use a kernel debugger but I did not have much
success going that route either.

[Johan Johansson]

Well, if it's easier through kd I should of course use kd. Problem is
I'm a bit of a newbie when it comes to kernel debugging. I'm assuming
there's a way to set breakpoints in user processes then?

Johan

Post by Pavel Lebedinsky [MSFT]
You could try caching symbols locally and removing any network
entries from the symbol path. Then you might be able to use
ntsd -server and connect from another machine.
But generally it's easier to debug things like that from kd.

[Skywing]

Yes, you can set user mode breakpoints from kd in Win2K and later. Be aware
that if you have a user mode debugger attached to the process then it will
receive the breakpoint exceptions (first) and not kd.

BTW, if it is feasible in this situation, I would strongly encourage you to
use VMware or Virtual PC and run kd over a named pipe mapped to a COM port
in the virtual machine. This is much faster (in terms of response times to
things like commands that search memory or otherwise transfer large amounts
of data over the kd connection) and much less of a headache to setup than
using kd on a "real" machine. There is some documentation in the WinDbg
help file about how to do this for both Virtual PC and VMware for recent
versions of WinDbg.

Post by Johan Johansson
Well, if it's easier through kd I should of course use kd. Problem is
I'm a bit of a newbie when it comes to kernel debugging. I'm assuming
there's a way to set breakpoints in user processes then?
Johan

Post by Pavel Lebedinsky [MSFT]
You could try caching symbols locally and removing any network
entries from the symbol path. Then you might be able to use
ntsd -server and connect from another machine.
But generally it's easier to debug things like that from kd.

[Johan Johansson]

Post by Skywing
Yes, you can set user mode breakpoints from kd in Win2K and later. Be aware
that if you have a user mode debugger attached to the process then it will
receive the breakpoint exceptions (first) and not kd.

Yes, tried this. I managed to set the current process context and load
symbols and set the breakpoint - but it was never triggered.
Unfortunately I didn't come up with any better idea on how to set the
breakpoint than to user Image Execution Options to start services.exe in
ntsd and then use .breakin to set the breakpoint.

Post by Skywing
BTW, if it is feasible in this situation, I would strongly encourage you to
use VMware or Virtual PC and run kd over a named pipe mapped to a COM port

This is the setup I am using.

Johan

[Johan Johansson]

Post by Pavel Lebedinsky [MSFT]
But generally it's easier to debug things like that from kd.

I'm running a bit low on unexploded heads at the moment, so if anyone
feels like telling me what's going on instead of what I think is going
on I'd appriciate it. I'm using windbg (latest release) as a kernel
debugger connected to a virtual pc (running a fully patched xp pro)
booted with /debug and so on.

1. I can't set an unresolved breakpoint in a user process that doesn't
exist because I need a process context.

2. I can't get a break when services.exe starts. I've tried fiddling
with the event filters both for create process and load module but
neither seems to have any effect what so ever in kernel mode. I've set
both to enable just to make sure that's the case. Is there btw any
documentation that states how the "exception" that was triggered can be
referenced in the optional command to run? In any event, I suspect this
is related to my next issue:

3. The only loaded user module is ntdll until I set a process context
and issue a .reload. I somehow feel that there is probably a good reason
for this, but at the same time I did expect loaded user modules to be
visible even in the system context.

4. Even once I've set the process context breakpoints are not triggered.
This is what I thought would work:

kd> lm u
start end module name
7c900000 7c9b0000 ntdll (pdb symbols)
ntdll.pdb\36515FB5D04345E491F672FA2E2878C02\ntdll.pdb
kd> !process 0 0

PROCESS ffbbc250 SessionId: 0 Cid: 0298 Peb: 7ffd5000 ParentCid: 0248
DirBase: 068a8000 ObjectTable: e1762458 HandleCount: 246.
Image: services.exe

kd> .process /p ffbbc250
Implicit process is now ffbbc250
.cache forcedecodeuser done

kd> .reload /user
Loading User Symbols
.ModLoad: 01000000 0101c000 C:\WINDOWS\system32\services.exe
...

kd> bp /p ffbbc250 services!ScStartService
Loading symbols for 01000000 services.exe -> services.exe
kd> bl
0 e 01005c3f 0001 (0001) services!ScStartService
Match process data ffbbc250

Looks pretty promising, huh? Unfortunately I can start all the services
I want without any breakpoints being triggered.

5. I then came across the brilliant .send_file -s command which inspired
me to again try using ntsd while controlling it using windbg. Well, no dice:

0:000> .sympath c:\
.sympath c:\
Symbol search path is: c:\
0:000> .breakin
.breakin
Break instruction exception - code 80000003 (first chance)
nt!RtlpBreakWithStatusInstruction:
804e3592 cc int 3
kd> .send_file -s c:\
Unable to create 'c:\\ntoskrnl.pdb', HRESULT 0x80004001

...which means that it's not implemented. Fair enough, it's a kernel
debugger after all, but is there *something* I can do that actually
works? All I want is to

1. Break into services.exe as soon as it starts
2. Lookup symbols on the host rather than the target

There are at least 4 debuggers and 3 kinds of debugging servers. Surely
there must be some combination that does what I need?

Thanks in advance for any insightful comments
Johan

[Pavel Lebedinsky [MSFT]]

Post by Johan Johansson

Post by Pavel Lebedinsky [MSFT]
But generally it's easier to debug things like that from kd.

OK, that might not be true. The easiest way is probably ntsd -d
or ntsd -server with locally cached symbols (provided it actually
works; I haven't tried). But you can do it from kd too - see below.

Post by Johan Johansson
1. I can't set an unresolved breakpoint in a user process that doesn't
exist because I need a process context.
2. I can't get a break when services.exe starts. I've tried fiddling with
the event filters both for create process and load module but neither
seems to have any effect what so ever in kernel mode. I've set both to
enable just to make sure that's the case.

Things like sxe cpr work only in user mode. I don't know if there's
an "official" way to set a breakpoint that fires when a process is
created - I usually use one of the nt!Ps*Process functions or
KeInitializeProcess.

After the system starts, you'll need to skip several breakpoints
(things like smss, csrss etc) until you get to services.exe. Without
private symbols it might be difficult to determine what process is
being started - the easiest way is probably to dump strings on the
stack with dpu esp; dpu, ... until you see a valid looking image
name.

Once the right breakpoint is reached, you need to find the
EPROCESS of the new process. You can step out until the
process has been inserted into the system process list, at which
point it should be picked up by !process 0 0. Or you can get it
(for example) from the first argument of KeInitializeProcess
which on my system is passed in edx (all this is of course
implementation dependant - you need to look around on your
particular system).

Post by Johan Johansson
3. The only loaded user module is ntdll until I set a process context and
issue a .reload. I somehow feel that there is probably a good reason for
this, but at the same time I did expect loaded user modules to be visible
even in the system context.

I think it's because kernel debugger is not notified when user images
are loaded. So basically whenever process context changes or
new DLLs load you need to do .reload /user.

Post by Johan Johansson
4. Even once I've set the process context breakpoints are not triggered.
kd> .process /p ffbbc250
Implicit process is now ffbbc250
.cache forcedecodeuser done
...
kd> bp /p ffbbc250 services!ScStartService
Looks pretty promising, huh? Unfortunately I can start all the services I
want without any breakpoints being triggered.

This does not work because .process (without /i) only changes
debugger's view of the world. Breakpoints are managed by the
kernel on the target system, so you need to be in the right
process context to begin with.

There are several ways to switch to the right process context.
You can do .process /i, or !bpid, or run breakin.exe on the target.
In this case since you want to catch the process as soon as it starts
it's probably better to set a process-specific breakpoint on some
system call that we know will be called during process startup.
NtMapViewOfSection is a good choice since it's called whenever
a DLL is mapped:

kd> bp /p 8259e2d8 nt!NtMapViewOfSection
kd> g
Breakpoint 1 hit
kd> .process
Implicit process is now 8259e2d8

kd> .reload /user
kd> bp services!ScStartService
kd> g

Breakpoint 2 hit

kd> k
ChildEBP RetAddr
0006fe6c 01017257 services!ScStartService
0006fea4 01017887 services!ScStartMarkedServices+0x140
0006fee0 01017f84 services!ScStartServiceAndDependencies+0x2ce
0006ff04 01018009 services!ScStartServiceByName+0x6a
0006ff20 010281c7 services!ScAutoStartServices+0x52
0006ff4c 010284d0 services!SvcctrlMain+0x6a2
0006ff5c 0102bceb services!main+0x19
0006ffa0 77e82ead services!_mainCRTStartup+0xfd

[Drew Bliss [MSFT]]

On the notification of user-mode module loads in kd:

UM notifications are not produced by the target system by default. You can
enable this by OR'ing 0x40000 into nt!NtGlobalFlag before the modules load.
Note that kd will still only keep a single module list, so it'll fill up
with UM modules from all processes, so it isn't necessarily helpful. It
will let sxe ld work, though.

Post by Pavel Lebedinsky [MSFT]

Post by Johan Johansson

Post by Pavel Lebedinsky [MSFT]
But generally it's easier to debug things like that from kd.

OK, that might not be true. The easiest way is probably ntsd -d
or ntsd -server with locally cached symbols (provided it actually
works; I haven't tried). But you can do it from kd too - see below.

Post by Johan Johansson
1. I can't set an unresolved breakpoint in a user process that doesn't
exist because I need a process context.
2. I can't get a break when services.exe starts. I've tried fiddling with
the event filters both for create process and load module but neither
seems to have any effect what so ever in kernel mode. I've set both to
enable just to make sure that's the case.

Things like sxe cpr work only in user mode. I don't know if there's
an "official" way to set a breakpoint that fires when a process is
created - I usually use one of the nt!Ps*Process functions or
KeInitializeProcess.
After the system starts, you'll need to skip several breakpoints
(things like smss, csrss etc) until you get to services.exe. Without
private symbols it might be difficult to determine what process is
being started - the easiest way is probably to dump strings on the
stack with dpu esp; dpu, ... until you see a valid looking image
name.
Once the right breakpoint is reached, you need to find the
EPROCESS of the new process. You can step out until the
process has been inserted into the system process list, at which
point it should be picked up by !process 0 0. Or you can get it
(for example) from the first argument of KeInitializeProcess
which on my system is passed in edx (all this is of course
implementation dependant - you need to look around on your
particular system).

Post by Johan Johansson
3. The only loaded user module is ntdll until I set a process context and
issue a .reload. I somehow feel that there is probably a good reason for
this, but at the same time I did expect loaded user modules to be visible
even in the system context.

I think it's because kernel debugger is not notified when user images
are loaded. So basically whenever process context changes or
new DLLs load you need to do .reload /user.

Post by Johan Johansson
4. Even once I've set the process context breakpoints are not triggered.
kd> .process /p ffbbc250
Implicit process is now ffbbc250
.cache forcedecodeuser done
...
kd> bp /p ffbbc250 services!ScStartService
Looks pretty promising, huh? Unfortunately I can start all the services I
want without any breakpoints being triggered.

This does not work because .process (without /i) only changes
debugger's view of the world. Breakpoints are managed by the
kernel on the target system, so you need to be in the right
process context to begin with.
There are several ways to switch to the right process context.
You can do .process /i, or !bpid, or run breakin.exe on the target.
In this case since you want to catch the process as soon as it starts
it's probably better to set a process-specific breakpoint on some
system call that we know will be called during process startup.
NtMapViewOfSection is a good choice since it's called whenever
kd> bp /p 8259e2d8 nt!NtMapViewOfSection
kd> g
Breakpoint 1 hit
kd> .process
Implicit process is now 8259e2d8
kd> .reload /user
kd> bp services!ScStartService
kd> g
Breakpoint 2 hit
kd> k
ChildEBP RetAddr
0006fe6c 01017257 services!ScStartService
0006fea4 01017887 services!ScStartMarkedServices+0x140
0006fee0 01017f84 services!ScStartServiceAndDependencies+0x2ce
0006ff04 01018009 services!ScStartServiceByName+0x6a
0006ff20 010281c7 services!ScAutoStartServices+0x52
0006ff4c 010284d0 services!SvcctrlMain+0x6a2
0006ff5c 0102bceb services!main+0x19
0006ffa0 77e82ead services!_mainCRTStartup+0xfd
--
This posting is provided "AS IS" with no warranties, and confers no
rights.

[Johan Johansson]

Pavel, Drew - thank you very much. This should keep me busy for a while.

Johan

Post by Drew Bliss [MSFT]
UM notifications are not produced by the target system by default. You can
enable this by OR'ing 0x40000 into nt!NtGlobalFlag before the modules load.
Note that kd will still only keep a single module list, so it'll fill up
with UM modules from all processes, so it isn't necessarily helpful. It
will let sxe ld work, though.

Post by Pavel Lebedinsky [MSFT]

Post by Johan Johansson

Post by Pavel Lebedinsky [MSFT]
But generally it's easier to debug things like that from kd.

OK, that might not be true. The easiest way is probably ntsd -d
or ntsd -server with locally cached symbols (provided it actually
works; I haven't tried). But you can do it from kd too - see below.

Post by Johan Johansson
1. I can't set an unresolved breakpoint in a user process that doesn't
exist because I need a process context.
2. I can't get a break when services.exe starts. I've tried fiddling with
the event filters both for create process and load module but neither
seems to have any effect what so ever in kernel mode. I've set both to
enable just to make sure that's the case.

Things like sxe cpr work only in user mode. I don't know if there's
an "official" way to set a breakpoint that fires when a process is
created - I usually use one of the nt!Ps*Process functions or
KeInitializeProcess.
After the system starts, you'll need to skip several breakpoints
(things like smss, csrss etc) until you get to services.exe. Without
private symbols it might be difficult to determine what process is
being started - the easiest way is probably to dump strings on the
stack with dpu esp; dpu, ... until you see a valid looking image
name.
Once the right breakpoint is reached, you need to find the
EPROCESS of the new process. You can step out until the
process has been inserted into the system process list, at which
point it should be picked up by !process 0 0. Or you can get it
(for example) from the first argument of KeInitializeProcess
which on my system is passed in edx (all this is of course
implementation dependant - you need to look around on your
particular system).

Post by Johan Johansson
3. The only loaded user module is ntdll until I set a process context and
issue a .reload. I somehow feel that there is probably a good reason for
this, but at the same time I did expect loaded user modules to be visible
even in the system context.

I think it's because kernel debugger is not notified when user images
are loaded. So basically whenever process context changes or
new DLLs load you need to do .reload /user.

Post by Johan Johansson
4. Even once I've set the process context breakpoints are not triggered.
kd> .process /p ffbbc250
Implicit process is now ffbbc250
.cache forcedecodeuser done
...
kd> bp /p ffbbc250 services!ScStartService
Looks pretty promising, huh? Unfortunately I can start all the services I
want without any breakpoints being triggered.

This does not work because .process (without /i) only changes
debugger's view of the world. Breakpoints are managed by the
kernel on the target system, so you need to be in the right
process context to begin with.
There are several ways to switch to the right process context.
You can do .process /i, or !bpid, or run breakin.exe on the target.
In this case since you want to catch the process as soon as it starts
it's probably better to set a process-specific breakpoint on some
system call that we know will be called during process startup.
NtMapViewOfSection is a good choice since it's called whenever
kd> bp /p 8259e2d8 nt!NtMapViewOfSection
kd> g
Breakpoint 1 hit
kd> .process
Implicit process is now 8259e2d8
kd> .reload /user
kd> bp services!ScStartService
kd> g
Breakpoint 2 hit
kd> k
ChildEBP RetAddr
0006fe6c 01017257 services!ScStartService
0006fea4 01017887 services!ScStartMarkedServices+0x140
0006fee0 01017f84 services!ScStartServiceAndDependencies+0x2ce
0006ff04 01018009 services!ScStartServiceByName+0x6a
0006ff20 010281c7 services!ScAutoStartServices+0x52
0006ff4c 010284d0 services!SvcctrlMain+0x6a2
0006ff5c 0102bceb services!main+0x19
0006ffa0 77e82ead services!_mainCRTStartup+0xfd
--
This posting is provided "AS IS" with no warranties, and confers no
rights.

[Hans De Smaele]

Johan,

if I don't misunderstand your problem : you just want to know what services
get started and in what sequence on a computer at startup ?
There are several scenario's when you attach the WinDbg debugger for kernel
debugging. Then, you can examine all the running processes afterwards and see
when they got started.
Another approach, slower but giving a hughe bunch of information is to set
some gflags parameters as soon as the debugger breaks in during your kernel
debugging session. Set gflags to "show loader snaps" (!gflag +sls). This
gives you information about everything that happens on your machine. I
suggest that you put logging on, so that you have the output in a textfile
for later examination).
Hope this helps you.
I explained this in my article about WinDbg debugging :
http://www.microsoft.com/belux/nl/msdn/community/articles/feb05_applicationdebugging.mspx

best regards,
Hans.

Post by Johan Johansson
Hi everyone!
I'm hoping that someone with more experience/knowledge than me will know
what the correct configuration of debuggers on what machines are.
What I want to achieve
----------------------
I want to be able to easily check at what time/in what order what
services started. This is somewhat complicated by the fact that many
services run in the same process.
My idea
-------
Set a breakpoint in Services!ScStartService and print what service is
being started. This part is solved, in other words, once I get to a
point where I can set this breakpoint I'm done.
Practicalities
--------------
Loading symbols from network seems to require services.exe to be
running, so I've opted for a debugging server with a smart client
attached to it to ensure that the symbols are looked up on another
machine. But, as far as I can tell I must then create the debugging
server with DbgSrv and entering that as the debugger for Services.exe
under Windows NT\CurrentVersion\Image Execution Options just seems to
halt the boot.
My first approach was to use a kernel debugger but I did not have much
success going that route either.
Suggestions?
Thanks,
Johan

[Johan Johansson]

Post by Hans De Smaele
if I don't misunderstand your problem : you just want to know what services
get started and in what sequence on a computer at startup ?
There are several scenario's when you attach the WinDbg debugger for kernel
debugging. Then, you can examine all the running processes afterwards and see
when they got started.

No, you understand my problem. Unfortunately the suggested method would
require that each service runs as its own process which is just not the
case.

Post by Hans De Smaele
Another approach, slower but giving a hughe bunch of information is to set
some gflags parameters as soon as the debugger breaks in during your kernel
debugging session. Set gflags to "show loader snaps" (!gflag +sls).

Does it not require that I'm running a checked build of the operating
system or that I know which process to apply it to? I need a technique
that I can apply with reasonable swiftness on customer machines, so that
would be a blocker.

Post by Hans De Smaele
http://www.microsoft.com/belux/nl/msdn/community/articles/feb05_applicationdebugging.mspx

An article I am currently reading with interest =)

Johan

[Hans De Smaele]

Johan,

if you just want to check the running services at some point in time, run
"tlist.exe -v". This comes with the debugging tools and gives you the list of
running services, even if these are 'part' of another service.

H.

Post by Johan Johansson

Post by Hans De Smaele
if I don't misunderstand your problem : you just want to know what services
get started and in what sequence on a computer at startup ?
There are several scenario's when you attach the WinDbg debugger for kernel
debugging. Then, you can examine all the running processes afterwards and see
when they got started.

No, you understand my problem. Unfortunately the suggested method would
require that each service runs as its own process which is just not the
case.

Post by Hans De Smaele
Another approach, slower but giving a hughe bunch of information is to set
some gflags parameters as soon as the debugger breaks in during your kernel
debugging session. Set gflags to "show loader snaps" (!gflag +sls).

Does it not require that I'm running a checked build of the operating
system or that I know which process to apply it to? I need a technique
that I can apply with reasonable swiftness on customer machines, so that
would be a blocker.

Post by Hans De Smaele
http://www.microsoft.com/belux/nl/msdn/community/articles/feb05_applicationdebugging.mspx

An article I am currently reading with interest =)
Johan