Getting an accurate call trace of running IOS not so easy

I’ve spent the past little while looking at why my call trace code grows but doesnt subsequently shrink.  I am maintaining a stack of most recent calls (JAL(R) jump and link register).  I pop an element from the stack when the program counter changes to the return address at the time of the JAL(R).  Not perfect, but OK.  The problem is, it naively doesn’t work properly.  It grows without bound.

The reason for this is surely because I’m trying to trace multiple processes, and the scheduler inside IOS.  I was silly for thinking my naive implementation would work.  According to the Cisco documentation, IOS uses ‘run to completion’ scheduling.  I’m not an expert, but this sounds like an implemention of user threads.

Looking at the call trace I’m generating, I’m see a group of calls being repeated.  This would presumably be what causes the trace to grow without bound.  Looking at the functions in IDA, I can see one of them saving state, including the $s registers, the return address, frame pointer, and the stack pointer.  Then restoring some presumabley previously stored state and returning to the new (or old rather) location.

I’ll hazard a guess, and call that trampoline function context_switch().  I imagine what happens in IOS, is a process executes its code as per normal, and then every now and then calls yield() (or maybe reaches a syscall that blocks waiting on io), possibly puts the current process in the wait queue, then finds a process on the ready queue and performs a context switch().

Developing a call trace is a tricky thing, because you really need the state of each process to do so accurately.  In x86, when the frame pointer is used, knowing the stack pointer is enough to build a call trace, but in MIPS its not as easy since the return address is saved in a register, and then optionally stored in a local stack frame at an offset that isnt entirely constant.

It seems like a goal of its own, to develop a call tracer that works in a multi process/threaded environment, so I am not sure entirely what I will do.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s