IRC, freenode, #hurd, 2011-01-06

<antrik> hm, odd... vmstat tells me that ~500 MiB of RAM are in use; but
  the sum of all RSS is <300 MiB... what's the rest?
<braunr> kernel memory ?
<braunr> the zone allocator maybe
<braunr> or the page cache simply
<antrik> braunr: which page cache? AIUI, caches are implemented by the
  individual filesystem servers -- in which case any memory used by them
  should show up in RSS
<antrik> also, gnumach is listed among other tasks, so I'd assume the
  kernel memery also to be accounted for
<braunr> antrik: no, the kernel maintains a page cache, very similar to
  what is done in Linux, and almost the same as in FreeBSD
<braunr> the file system servers are just backing stores
<braunr> the RSS for the gnumach tasks only includes kernel memory
<braunr> I don't think the page cache is accounted for
<braunr> because it's not really kernel memory, it's a cache of user space
<antrik> apparently my understanding of Mach paging is still (or again?)
  rather incomplete :-(
<antrik> BTW, is there any way to find out how much anonymous memory a
  process is using? the "virtual" includes discardable mappings, and is
  thus not very helpful...
<antrik> (that applies to Linux as well though)
<braunr> can you provide an example of the output of vmstat please ?
<braunr> I don't have a Hurd VM near me
<antrik> olaf@alien:~$ vmstat
<antrik> pagesize:          4K
<antrik> size:            501M
<antrik> free:           6.39M
<antrik> active:          155M
<antrik> inactive:        310M
<antrik> wired:          29.4M
<antrik> zero filled:    15.3G
<antrik> reactivated:     708M
<antrik> pageins:        3.43G
<antrik> pageouts:       1.55G
<antrik> page faults: 26844574
<antrik> cow faults:   3736174
<antrik> memobj hit ratio: 92%
<antrik> swap size:       733M
<antrik> swap free:       432M
<antrik> interesting... closing a single screen window temporarily raises
  the "free" value by almost 10 MB
<antrik> I guess bash is rather hungry nowadays ;-)
<braunr> antrik: I guess the only way is using pmap or looking into
<braunr> but it won't give you the amount of physical memory used by
  anonymous mappings
<antrik> nah, I don't even want that... just like to know how much memory
  (RAM+swap) a process is really using
<braunr> antrik: then the RSS field is what you want
<antrik> OTOH, anonymous doesn't include program code or other actively
  used mappings... so not very useful either
<antrik> nah, RSS doesn't count anything that is in swap
<braunr> well
<braunr> don't you have a SWAP column ?
<braunr> hm
<braunr> i guess not
<braunr> antrik: why do you say it doesn't include other actively used
  mappings ?
<braunr> antrik: and the inclusion of program code also depends on the
  implementation of the ELF handler
<braunr> I don't know how the hurd does that, but some ELF loaders use
  anonymous memory for the execution view
<antrik> well, if a program maps a data file, and regularily accesses parts
  of the file, they won't occupy physical RAM all the time (and show up in
  RSS), but they are not anonymous mappings. similar to program code
<braunr> then this anonymous memory is shared by all processes using that
<antrik> oh, interesting
<antrik> is it really a completely distinct mapping, rather than just COW?
<braunr> the first is
<braunr> others are COW
<antrik> so if a program loads 200 MB of libraries, they are all read in on
  startup, and occupy RAM or swap subsequently, even if most of the code is
  never actually run?...
<kilobug> library code should be backed by the library file on disk, not be
<braunr> depends on the implementation
<braunr> I guess most use the file system backend
<braunr> but in the Hurd, ext2fs.static and use anonymous memory
<braunr> (that's the case for another reason, still, I don't think the
  report in top/ps clearly indicates that fact)
<kilobug> braunr: yeah for bootstrapping issues, makes sense
<braunr> it may also depends on the pic/pie options used when building

IRC, freenode, #hurd, 2011-07-24

< braunr> the panic is probably due to memory shortage
< braunr> so as antrik suggested, use more swap
< antrik> gg0: you could run "vmstat 1" in another terminal to watch memory
< antrik> that way we will know for sure whether it's related
< braunr> antrik: it's trickier than that
< braunr> it depends if the zones used are pageable
< antrik> braunr: well, if it's a zone map exhaustion, then the swap size
  won't change anything?...
< braunr> antrik: in this case no, but if the zone is pageable and the
  pager (backing anonymous memory) refuses to create memory because it
  estimates it's full (all swap space is reserved), it will fail to
< braunr> too
< braunr> but i don't think there are much pageable zones in the kernel
< antrik> yes, but in that case we can see the exhaustion in vmstat :-)
< braunr> many*
< braunr> i'm not sure
< braunr> reserved swap space doesn't mean it's used
< braunr> that's one of the major changes in freebsd 4 or 5 i was
< antrik> if it's reserved, it wouldn't show up as "free", would it?...
< braunr> (btw, it's also what makes anonymous memory merging so hard)
< braunr> yes it would
< braunr> well, it could, i'm not sure
< braunr> anonymous memory is considered as a file
< braunr> one big file filled with zeroes, which is the swap partition
< braunr> when you allocate pageable anonymous memory, a part of this
  "file" is reserved
< braunr> but i don't know if the reported number if the reserved
  (allocated) space, or used (actually containing data)
< braunr> is*
< braunr> i also suspect wired allocations can fail because of a full swap
  (because the kernel is unable to make free pages)
< braunr> in this case vmstat will show it
< antrik> what does it matter whether there is data there or not? if it's
  reserved, it's not free. if it behaves differently, I'd consider that a
  serious bug
< braunr> maybe the original developers intended to monitor its actual
< braunr> antrik: i've just checked how the free count gets updated, and it
  looks like it is on both seqnos_memory_object_data_initialize and
< braunr> antrik: so i guess reserved memory is accounted for

IRC, freenode, #hurd, 2013-01-12

<tschwinge> darnassus linking clang: 600 MiB swap in use and 22 MiB RAM
  free, of 2 GiB.  But ps shows a RSS of just 100 MiB, huh?
<tschwinge> Getting "better": near the end of the link, nearly 1 GiB swap
  in use, and 200 KiB (!) RAM free.
<sobhan> can hurd have more than 1GB of ram ?
<tschwinge> And then it completed; 75 MiB swap in use, and 1.2 GiB RAM
<braunr> tschwinge: unless i'm mistaken, mach uses the legacy "swapping"
  bsd mechanism 
<braunr> tschwinge: i.e. when it swaps a process, it swaps all of it
<braunr> tschwinge: the rest is probably one big anonymous vm object
  containing the process space
<braunr> cached objects aren't currently well accounted
<braunr> (well, since youpi got my page cache patches in, they are, but
  procfs isn't yet modified to report them)
<braunr> tschwinge: right, i'm currently looking at the machine and it
  doesn't add up, i suppoe there are some big files still in the cache
<braunr> ah, git packed objects :p
<braunr> and a few llvm .a/.so/executable files too
<braunr> and since they're probably targets, they're built last, which
  explains why they're retained in the cache for a while

msgh id (why on that page?).