Many newer versions of Sendmail come with a restricted execution utility called ``smrsh'', which limits the executables that Sendmail will allow to be used as mail programs. You need to explicitly allow Mailman's wrapper program to be used with smrsh or Mailman will not work. If mail is not getting delivered to Mailman's wrapper program and you're getting an ``operating system error'' in your mail syslog, this could be your problem.
One good way of enabling this is:
% grep smrsh /etc/mail/sendmail.cf
% strings $path_to_smrsh | less
% cd /var/adm/sm.bin
% ln -s /usr/local/mailman/mail/mailman mailman