IRC, freenode, #hurd, 2011-01-06
<antrik> hm, odd... vmstat tells me that ~500 MiB of RAM are in use; but the sum of all RSS is <300 MiB... what's the rest? <braunr> kernel memory ? <braunr> the zone allocator maybe <braunr> or the page cache simply <antrik> braunr: which page cache? AIUI, caches are implemented by the individual filesystem servers -- in which case any memory used by them should show up in RSS <antrik> also, gnumach is listed among other tasks, so I'd assume the kernel memery also to be accounted for <braunr> antrik: no, the kernel maintains a page cache, very similar to what is done in Linux, and almost the same as in FreeBSD <braunr> the file system servers are just backing stores <braunr> the RSS for the gnumach tasks only includes kernel memory <braunr> I don't think the page cache is accounted for <braunr> because it's not really kernel memory, it's a cache of user space memory <antrik> apparently my understanding of Mach paging is still (or again?) rather incomplete :-( <antrik> BTW, is there any way to find out how much anonymous memory a process is using? the "virtual" includes discardable mappings, and is thus not very helpful... <antrik> (that applies to Linux as well though) <braunr> can you provide an example of the output of vmstat please ? <braunr> I don't have a Hurd VM near me <antrik> olaf@alien:~$ vmstat <antrik> pagesize: 4K <antrik> size: 501M <antrik> free: 6.39M <antrik> active: 155M <antrik> inactive: 310M <antrik> wired: 29.4M <antrik> zero filled: 15.3G <antrik> reactivated: 708M <antrik> pageins: 3.43G <antrik> pageouts: 1.55G <antrik> page faults: 26844574 <antrik> cow faults: 3736174 <antrik> memobj hit ratio: 92% <antrik> swap size: 733M <antrik> swap free: 432M <antrik> interesting... closing a single screen window temporarily raises the "free" value by almost 10 MB <antrik> I guess bash is rather hungry nowadays ;-) <braunr> antrik: I guess the only way is using pmap or looking into /proc/<pid>/maps <braunr> but it won't give you the amount of physical memory used by anonymous mappings <antrik> nah, I don't even want that... just like to know how much memory (RAM+swap) a process is really using <braunr> antrik: then the RSS field is what you want <antrik> OTOH, anonymous doesn't include program code or other actively used mappings... so not very useful either <antrik> nah, RSS doesn't count anything that is in swap <braunr> well <braunr> don't you have a SWAP column ? <braunr> hm <braunr> i guess not <braunr> antrik: why do you say it doesn't include other actively used mappings ? <braunr> antrik: and the inclusion of program code also depends on the implementation of the ELF handler <braunr> I don't know how the hurd does that, but some ELF loaders use anonymous memory for the execution view <antrik> well, if a program maps a data file, and regularily accesses parts of the file, they won't occupy physical RAM all the time (and show up in RSS), but they are not anonymous mappings. similar to program code <braunr> then this anonymous memory is shared by all processes using that code <antrik> oh, interesting <antrik> is it really a completely distinct mapping, rather than just COW? <braunr> the first is <braunr> others are COW <antrik> so if a program loads 200 MB of libraries, they are all read in on startup, and occupy RAM or swap subsequently, even if most of the code is never actually run?... <kilobug> library code should be backed by the library file on disk, not be swap <braunr> depends on the implementation <braunr> I guess most use the file system backend <braunr> but in the Hurd, ext2fs.static and ld.so.1 use anonymous memory <braunr> (that's the case for another reason, still, I don't think the report in top/ps clearly indicates that fact) <kilobug> braunr: yeah for bootstrapping issues, makes sense <braunr> it may also depends on the pic/pie options used when building libraries
IRC, freenode, #hurd, 2011-07-24
< braunr> the panic is probably due to memory shortage < braunr> so as antrik suggested, use more swap < antrik> gg0: you could run "vmstat 1" in another terminal to watch memory usage < antrik> that way we will know for sure whether it's related < braunr> antrik: it's trickier than that < braunr> it depends if the zones used are pageable < antrik> braunr: well, if it's a zone map exhaustion, then the swap size won't change anything?... < braunr> antrik: in this case no, but if the zone is pageable and the pager (backing anonymous memory) refuses to create memory because it estimates it's full (all swap space is reserved), it will fail to < braunr> too < braunr> but i don't think there are much pageable zones in the kernel < antrik> yes, but in that case we can see the exhaustion in vmstat :-) < braunr> many* < braunr> i'm not sure < braunr> reserved swap space doesn't mean it's used < braunr> that's one of the major changes in freebsd 4 or 5 i was mentioning < antrik> if it's reserved, it wouldn't show up as "free", would it?... < braunr> (btw, it's also what makes anonymous memory merging so hard) < braunr> yes it would < braunr> well, it could, i'm not sure < braunr> anonymous memory is considered as a file < braunr> one big file filled with zeroes, which is the swap partition < braunr> when you allocate pageable anonymous memory, a part of this "file" is reserved < braunr> but i don't know if the reported number if the reserved (allocated) space, or used (actually containing data) < braunr> is* < braunr> i also suspect wired allocations can fail because of a full swap (because the kernel is unable to make free pages) < braunr> in this case vmstat will show it < antrik> what does it matter whether there is data there or not? if it's reserved, it's not free. if it behaves differently, I'd consider that a serious bug < braunr> maybe the original developers intended to monitor its actual usage < braunr> antrik: i've just checked how the free count gets updated, and it looks like it is on both seqnos_memory_object_data_initialize and seqnos_memory_object_data_write < braunr> antrik: so i guess reserved memory is accounted for
IRC, freenode, #hurd, 2013-01-12
<tschwinge> darnassus linking clang: 600 MiB swap in use and 22 MiB RAM free, of 2 GiB. But ps shows a RSS of just 100 MiB, huh? <tschwinge> Getting "better": near the end of the link, nearly 1 GiB swap in use, and 200 KiB (!) RAM free. <sobhan> can hurd have more than 1GB of ram ? <tschwinge> And then it completed; 75 MiB swap in use, and 1.2 GiB RAM free. <braunr> tschwinge: unless i'm mistaken, mach uses the legacy "swapping" bsd mechanism <braunr> tschwinge: i.e. when it swaps a process, it swaps all of it <braunr> tschwinge: the rest is probably one big anonymous vm object containing the process space <braunr> cached objects aren't currently well accounted <braunr> (well, since youpi got my page cache patches in, they are, but procfs isn't yet modified to report them) <braunr> tschwinge: right, i'm currently looking at the machine and it doesn't add up, i suppoe there are some big files still in the cache <braunr> ah, git packed objects :p <braunr> and a few llvm .a/.so/executable files too <braunr> and since they're probably targets, they're built last, which explains why they're retained in the cache for a while
msgh id (why on that page?).