How to use the watchdog timer in a RTOS?
Assume I have a cooperative scheduler in an embedded environment. I have many processes running. I want to utilize the watchdog timer so that I can detect when a process has stopped behaving for any reason and reset the processor.
In simpler applications with no RTOS I would always touch the watchdog from the main loop and this was always adequate. However, here, there are many processes that could potentially hang. What is a clean method to touch the watchdog timer periodically while ensuring that each process is in good health?
I was thinking that I could provide a callback function to each process so that it could let another function, which oversees all, know it is still alive. The callback would pass a parameter which would be the tasks unique id so the overseer could determine who was calling back.
One common approach is to delegate the watchdog kicking to a specific task (often either the highest-priority or the lowest priority, tradeoffs / motivations for each approach), and then have all other tasks "check in" with this task.
This way:
if an interrupt is hung (100% CPU), the kicker task won't run, you reset
if the kicker task is hung, you reset
if another task is hung, kicker task sees no check in, kicker task doesn't kick WDG, you reset
Now there are of course implementation details to consider. Some people have each task set its own dedicated bit (atomically) in a global variable; the kicker task checks this group of bit flags at a specific rate, and clears/resets when everyone has checked in (along with kicking the WDG, of course.) I eschew globals like the plague and avoid this approach. RTOS event flags provide a somewhat similar mechanism that is more elegant.
I typically design my embedded systems as event-driven systems. In this case, each tasks blocks at one specific place - on a message queue. All tasks (and ISRs) communicate with each other by sending events / messages. This way, you don't have to worry about a task not checking in because it's blocked on a semaphore "way down there" (if that doesn't make sense, sorry, without writing a lot more I can't explain it better).
Also there is the consideration - do tasks check in "autonomously" or do they reply/respond to a request from the kicker task. Autonomous - for example, once a second, each task receives an event in its queue "tell kicker task you're still alive". Reply-request - once a second (or whatever), kicker tasks tells everybody (via queues) "time to check in" - and eventually every task runs its queue, gets the request and replies. Considerations of task priorities, queueing theory, etc. apply.
There are 100 ways to skin this cat, but the basic principle of a single task that is responsible for kicking the WDG and having other tasks funnel up to the kicker task is pretty standard.
There is at least one other aspect to consider - outside the scope of this question - and that's dealing with interrupts. The method I described above will trigger WDG reset if an ISR is hogging the CPU (good), but what about the opposite scenario - an ISR has (sadly) become accidentally and inadvertantly disabled. In many scenarios, this will not be caught, and your system will still kick the WDG, yet part of your system is crippled. Fun stuff, that's why I love embedded development.
One solution pattern:
That way any thread that never returns a healthy state will stall the watchdog task until the hardware watchdog timeout occurs.
In a preemptive OS, the watchdog thread would be the lowest priority or idle thread. In a cooperative scheduler, it should yield between call-back calls.
The design of the callback functions themselves depends on the specific task and its behaviour and periodicity. Each function can be tailored to the needs and characteristic of the task. Tasks of high periodicity might simply increment a counter, which is set to zero when the callback is called. If the counter is zero on entry, the task did not schedule since the last watchdog check. Tasks with low or aperiodic behaviour might time-stamp their scheduling, the callback might then return a failure if the task has not been scheduled for some specified time period. Both tasks and interrupt handlers might be monitored in this way. Moreover because it is the responsibility of a thread to register with the watchdog, you might have some threads that do not register at all.
The traditional method is to have a watchdog process with the lowest possible priority
PROCESS(watchdog, PRIORITY_LOWEST) { while(1){reset_timer(); sleep(1);} }
And where the actual hardware timer resets the CPU every 3 or 5 seconds perhaps.
Tracking individual processes could be achieved by inverse logic: each process would setup a timer whose callback sends the watchdog a 'stop' message. Then each process would need to cancel the previous timer event and setup a new one somewhere in the 'receive event / message from queue' loop.
PROCESS(watchdog, PRIORITY_LOWEST) {
while(1) {
if (!messages_in_queue()) reset_timer();
sleep(1);
}
}
void wdg_callback(int event) {
msg = new Message();
send(&msg, watchdog);
};
PROCESS(foo, PRIORITY_HIGH) {
timer event=new Timer(1000, wdg_callback);
while (1) {
if (receive(msg, TIMEOUT)) {
// handle msg
} else { // TIMEOUT expired
cancel_event(event);
event = new Timer(1000,wdg_callback);
}
}
}
上一篇: 了解POSIX和Linux / glibc调度之间的差异
下一篇: 如何在RTOS中使用看门狗定时器？