In the topic of code analysis or program analysis (Wikipedia article), there is static code analysis (Wikipedia article) and dynamic program analysis (Wikipedia article). This topic overlaps with performance analysis, formal verification, as well as general debugging.
Bounty
There is a FOSS Factory bounty (p276) on some of these tasks.
Static
GCC's warnings. Yes, really.
GCC plugins can be used for additional semantic analysis. For example, http://lwn.net/Articles/457543/, and search for kernel context in the comments.
Have GCC make use of RPC/MIG in/out specifiers, and have it emit useful warnings in case these are pointing to uninitialized data (for in only).
Port Sequence Numbers. If these are used, care must be taken to update them reliably,
id:"1123688017.3905.22.camel@buko.sinrega.org". This could be checked by a static analysis tool.-
For example, Debian's hurd_20110319-2 package (Samuel Thibault, 2011-08-05: I had a look at those, some are spurious; the realloc issues are for real).
Coccinelle
clang
Linux' sparse
Coverity (nonfree?)
Dynamic
IRC, freenode, #glibc, 2011-09-28
<vsrinivas> two things you can do -- there is an environment variable (DEBUG_MALLOC_ iirc?) that can be set to 2 to make ptmalloc (glibc's allocator) more forceful and verbose wrt error checking <vsrinivas> another is to grab a copy of Tor's source tree and copy out OpenBSD's allocator (its a clearly-identifyable file in the tree); LD_PRELOAD it or link it into your app, it is even more aggressive about detecting memory misuse. <vsrinivas> third, Red hat has a gdb python plugin that can instrument glibc's heap structure. its kinda handy, might help? <vsrinivas> MALLOC_CHECK_ was the envvar you want, sorry.Input fuzzing
Not a new topic; has been used (and a paper published) for early UNIX tools, IIRC.
What about some RPC fuzzing?
