In the topic of code analysis or program analysis (Wikipedia article), there is static code analysis (Wikipedia article) and dynamic program analysis (Wikipedia article). This topic overlaps with performance analysis, formal verification, as well as general debugging.

Bounty

There is a FOSS Factory bounty (p276) on some of these tasks.

Static

Dynamic

  • Valgrind

  • http://en.wikipedia.org/wiki/Electric_Fence

  • http://wiki.debian.org/Hardening

  • https://wiki.ubuntu.com/CompilerFlags

  • IRC, freenode, #glibc, 2011-09-28

    <vsrinivas> two things you can do -- there is an environment variable
      (DEBUG_MALLOC_ iirc?) that can be set to 2 to make ptmalloc (glibc's
      allocator) more forceful and verbose wrt error checking
    <vsrinivas> another is to grab a copy of Tor's source tree and copy out
      OpenBSD's allocator (its a clearly-identifyable file in the tree);
      LD_PRELOAD it or link it into your app, it is even more aggressive
      about detecting memory misuse.
    <vsrinivas> third, Red hat has a gdb python plugin that can instrument
      glibc's heap structure. its kinda handy, might help?
    <vsrinivas> MALLOC_CHECK_ was the envvar you want, sorry.
    
  • Input fuzzing

    Not a new topic; has been used (and a paper published) for early UNIX tools, IIRC.

    What about some RPC fuzzing?