open issue documentation

A bunch of this should also be covered in other (introductionary) material, like Bushnell's Hurd paper. All this should be unfied and streamlined.

IRC, freenode, #hurd, 2011-03-08

<foocraft> I've a question on what are the "units" in the hurd project, if
  you were to divide them into units if they aren't, and what are the
  dependency relations between those units(roughly, nothing too pedantic
  for now)
<antrik> there is GNU Mach (the microkernel); there are the server
  libraries in the Hurd package; there are the actual servers in the same;
  and there is the POSIX implementation layer in glibc
<antrik> relations are a bit tricky
<antrik> Mach is the base layer which implements IPC and memory management
<foocraft> hmm I'll probably allocate time for dependency graph generation,
  in the worst case
<antrik> on top of this, the Hurd servers, using the server libraries,
  implement various aspects of the system functionality
<antrik> client programs use libc calls to use the servers
<antrik> (servers also use libc to communicate with other servers and/or
  Mach though)
<foocraft> so every server depends solely on mach, and no other server?
<foocraft> s/mach/mach and/or libc/
<antrik> I think these things should be pretty clear one you are somewhat
  familiar with the Hurd architecture... nothing really tricky there
<antrik> no
<antrik> servers often depend on other servers for certain functionality

IRC, freenode, #hurd, 2011-03-12

<dEhiN> when mach first starts up, does it have some basic i/o or fs
  functionality built into it to start up the initial hurd translators?
<antrik> I/O is presently completely in Mach
<antrik> filesystems are in userspace
<antrik> the root filesystem and exec server are loaded by grub
<dEhiN> o I see
<dEhiN> so in order to start hurd, you would have to start mach and
  simultaneously start the root filesystem and exec server?
<antrik> not exactly
<antrik> GRUB loads all three, and then starts Mach. Mach in turn starts
  the servers according to the multiboot information passed from GRUB
<dEhiN> ok, so does GRUB load them into ram?
<dEhiN> I'm trying to figure out in my mind how hurd is initially started
  up from a low-level pov
<antrik> yes, as I said, GRUB loads them
<dEhiN> ok, thanks antrik...I'm new to the idea of microkernels, but a
  veteran of monolithic kernels
<dEhiN> although I just learned that windows nt is a hybrid kernel which I
  never knew!
<rm> note there's a /hurd/ext2fs.static
<rm> I belive that's what is used initially... right?
<antrik> yes
<antrik> loading the shared libraries in addition to the actual server
  would be unweildy
<antrik> so the root FS server is linked statically instead
<dEhiN> what does the root FS server do?
<antrik> well, it serves the root FS ;-)
<antrik> it also does some bootstrapping work during startup, to bring the
  rest of the system up

Source Code Documentation

Provide a cross-linked sources documentation, including generated files, like RPC stubs.

• http://www.gnu.org/software/global/

Hurd 101

IO path

Need more stuff like that.

IRC, freenode, #hurd, 2011-10-18

<frhodes> what happens @ boot. and which translators are started in what
  order?
<antrik> short version: grub loads mach, ext2, and ld.so/exec; mach starts
  ext2; ext2 starts exec; ext2 execs a few other servers; ext2 execs
  init. from there on, it's just standard UNIX stuff

IRC, OFTC, #debian-hurd, 2011-11-02

<sekon_> is __dir_lookup a RPC ??
<sekon_> where can i find the source of __dir_lookup ??
<sekon_> grepping most gives out rvalue assignments 
<sekon_> -assignments 
<sekon_> but in hurs/fs.h it is used as a function ??
<pinotree> it should be the mig-generated function for that rpc
<sekon_> how do i know how its implemented ??
<sekon_> is there any way to delve deeprer into mig-generated functions 
<tschwinge> sekon_: The MIG-generated stuff will either be found in the
  package's build directory (if it's building it for themselves), or in the
  glibc build directory (libhurduser, libmachuser; which are all the
  available user RPC stubs).
<tschwinge> sekon_: The implementation can be found in the various Hurd
  servers/libraries.
<tschwinge> sekon_: For example, [hurd]/libdiskfs/dir-lookup.c.
<tschwinge> sekon_: What MIG does is provide a function call interface for
  these ``functions'', and the Mach microkernel then dispatches the
  invocation to the corresponding server, for example a /hurd/ext2fs file
  system (via libdiskfs).
<tschwinge> sekon_: This may help a bit:
  http://www.gnu.org/software/hurd/hurd/hurd_hacking_guide.html

IRC, freenode, #hurd, 2012-01-08

<abique> can you tell me how is done in hurd:  "ls | grep x" ?
<abique> in bash
<youpi> ls's standard output is a port to the pflocal server, and grep x's
  standard input is a port to the pflocal server
<youpi> the connexion between both ports inside the pflocal server being
  done by bash when it calls pipe()
<abique> youpi, so STDOUT_FILENO, STDIN_FILENO, STDERR_FILENO still exists
  ?
<youpi> sure, hurd is compatible with posix
<abique> so bash 1) creates T1 (ls) and T2 (grep), then create a pipe at
  the pflocal server, then connects both ends to T1 and T2, then start(T1),
  start(T2) ?
<youpi> not exactly
<youpi> it's like on usual unix, bash creates the pipe before creating the
  tasks
<youpi> then forks to create both of them, handling them each side of the
  pipe
<abique> ok I see
<youpi> s/handling/handing/
<abique> but when you do pipe() on linux, it creates a kernel object, this
  time it's 2 port on the pflocal ?
<youpi> yes
<abique> how are spawned tasks ?
<abique> with fork() ?
<youpi> yes
<youpi> which is just task_create() and duplicating the ports into the new
  task
<abique> ok
<abique> so it's easy to rewrite fork() with a good control of duplicated
  fd
<abique> about threading, mutexes, conditions, etc.. are kernel objects or
  just userland objects ?
<youpi> just ports
<youpi> (only threads are kernel objects)
<abique> so, about efficiency, are pipes and mutexes efficient ?
<youpi> depends what you call "efficient"
<youpi> it's less efficient than on linux, for sure
<youpi> but enough for a workable system
<abique> maybe hurd is the right place for a userland thread library like
  pth or any fiber library
<abique> ?
<youpi> hurd already uses a userland thread library
<youpi> libcthreads
<abique> is it M:N ?
<youpi> libthreads, actually
<youpi> yes

Actually, the Hurd has never used an M:N model. Both libthreads (cthreads) and libpthread use an 1:1 model.

<abique> nice
<abique> is the task scheduler in the kernel ?
<youpi> the kernel thread scheduler, yes, of course
<youpi> there has to be one
<abique> are the posix open()/readdir()/etc... the direct vfs or wraps an
  hurd layer libvfs ?
<youpi> they wrap RPCs to the filesystem servers
<antrik> the Bushnell paper is probably the closest we have to a high-level
  documentation of these concepts...
<antrik> the Hurd does not have a central VFS component at all. name
  lookups are performed directly on the individual FS servers
<antrik> that's probably the most fundamental design feature of the Hurd
<antrik> (all filesystem operations actually, not only lookups)

IRC, freenode, #hurd, 2012-01-09

<braunr> youpi: are you sure cthreads are M:N ? i'm almost sure they're 1:1
<braunr> and no modern OS is a right place for any thread userspace
  library, as they wouldn't have support to run threads on different
  processors (unless processors can be handled by userspace servers, but
  still, it requires intimate cooperation between the threading library and
  the kernel/userspace server in any case
<youpi> braunr: in libthreads, they are M:N
<youpi> you can run threads on different processors by using several kernel
  threads, there's no problem in there, a lot of projects do this
<braunr> a pure userspace library can't use kernel threads
<braunr> at least pth was explacitely used on systems like bsd at a time
  when they didn't have kernel threads exactly for that reason
<braunr> explicitely*
<braunr> and i'm actually quite surprised to learn that we have an M:N
  threading model :/
<youpi> why do you say "can't" ?
<braunr> but i wanted to reply to abique and he's not around
<youpi> of course you need kernel threads
<youpi> but all you need is to bind them
<braunr> well, what i call a userspace threading library is a library that
  completely implement threads without the support of the kernel
<braunr> or only limited support, like signals
<youpi> errr, you can't implement anything with absolutely no support of
  the kernel
<braunr> pth used only SIGALRM iirc
<youpi> asking for more kernel threads to use more processors doesn't seem
  much
<braunr> it's not
<braunr> but i'm refering to what abique said
<braunr> 01:32 < abique> maybe hurd is the right place for a userland
  thread library like pth or any fiber library
<youpi> well, it's indeed more, because the glibc lets external libraries
  provide their mutex
<youpi> while on linux, glibc doesn't
<braunr> i believe he meant removing thread support from the kernel :p
<youpi> ah
<braunr> and replying "nice" to an M:N threading model is also suspicious,
  since experience seems to show 1:1 models are better
<youpi> "better" ????
<braunr> yes
<youpi> well
<youpi> I don't have any time to argue  about that
<youpi> because that'd be extremely long
<braunr> simpler, so far less bugs, and also less headache concerning posix
  conformance
<youpi> but there's no absolute "better" here
<youpi> but less performant
<youpi> less flexible
<braunr> that's why i mention experience :)
<youpi> I mean experience too
<braunr> why less performant ?
<youpi> because you pay kernel transition
<youpi> because you don't know anything about the application threads
<youpi> etc.
<braunr> really ?
<youpi> yes
<braunr> i fail to see where the overhead is
<youpi> I'm not saying m:n is generally better than 1:1 either
<youpi> thread switch, thread creation, etc.
<braunr> creation is slower, i agree, but i'm not sure it's used frequently
  enough to really matter
<youpi> it is sometimes used frequently enough
<youpi> and in those cases it would be a headache to avoid it
<braunr> ok
<braunr> i thought thread pools were used in those cases
<youpi> synchronized with kernel mutexes ?
<youpi> that's still slow
<braunr> it reduces to the thread switch overhead
<braunr> which, i agree is slightly slower
<braunr> ok, i's a bit less performant :)
<braunr> well don't futexes exist just for that too ?
<youpi> yes and no
<youpi> in that case they don't help
<youpi> because they do sleep
<youpi> they help only when the threads are living
<braunr> ok
<youpi> now as I said I don't have to talk much more, I have to leave :)

IRC, freenode, #hurd, 2012-12-06

<braunr> spiderweb: have you read
  http://www.gnu.org/software/hurd/hurd-paper.html ?
<spiderweb> I'll have a look.
<braunr> and also the beginning of
  http://ftp.sceen.net/mach/mach_a_new_kernel_foundation_for_unix_development.pdf
<braunr> these two should provide a good look at the big picture the hurd
  attemtps to achieve
<Tekk_> I can't help but wonder though, what advantages were really
  achieved with early mach?
<Tekk_> weren't they just running a monolithic unix server like osx does?
<braunr> most mach-based systems were
<braunr> but thanks to that, they could provide advanced features over
  other well established unix systems
<braunr> while also being compatible
<Tekk_> so basically it was just an ease of development thing
<braunr> well that's what mach aimed at being
<braunr> same for the hurd
<braunr> making things easy
<Tekk_> but as a side effect hurd actually delivers on the advantages of
  microkernels aside from that, but the older systems wouldn't, correct?
<braunr> that's how there could be network file systems in very short time
  and very scarce resources (i.e. developers working on it), while on other
  systems it required a lot more to accomplish that
<braunr> no, it's not a side effect of the microkernel
<braunr> the hurd retains and extends the concept of flexibility introduced
  by mach
<Tekk_> the improved stability, etc. isn't a side effect of being able to
  restart generally thought of as system-critical processes?
<braunr> no
<braunr> you can't restart system critical processes on the hurd either
<braunr> that's one feature of minix, and they worked hard on it
<Tekk_> ah, okay. so that's currently just the domain of minix
<Tekk_> okay
<Tekk_> spiderweb: well, there's 1 advantage of minix for you :P
<braunr> the main idea of mach is to make it easy to extend unix
<braunr> without having hundreds of system calls
<braunr> the hurd keeps that and extends it by making many operations
  unprivileged
<braunr> you don't need special code for kernel modules any more
<braunr> it's easy
<braunr> you don't need special code to handle suid bits and other ugly
  similar hacks,
<braunr> it's easy
<braunr> you don't need fuse
<braunr> easy
<braunr> etc..

IRC, freenode, #hurd, 2012-12-06

<spiderweb> what is the #1 feature that distinguished hurd from other
  operating systems. the concept of translators. (will read more when I get
  more time).
<braunr> yes, translators
<braunr> using the VFS as a service directory
<braunr> and the VFS permissions to control access to those services

IRC, freenode, #hurd, 2012-12-10

<spiderweb> I want to work on hurd, but I think I'm going to start with
  minix, I own the minix book 3rd ed. it seems like a good intro to
  operating systems in general. like I don't even know what a semaphore is
  yet.
<braunr> well, enjoy learning :)
<spiderweb> once I finish that book, what reading do you guys recommend?
<spiderweb> other than the wiki
<braunr> i wouldn't recommend starting with a book that focuses on one
  operating system anyway
<braunr> you tend to think in terms of what is done in that specific
  implementation and compare everything else to that
<braunr> tannenbaum is not only the main author or minix, but also the one
  of the book http://en.wikipedia.org/wiki/Modern_Operating_Systems
<braunr>
  http://en.wikipedia.org/wiki/List_of_important_publications_in_computer_science#Operating_systems
  should be a pretty good list :)

IRC, freenode, #hurd, 2013-03-12

<mjjc> i have a question regarding ipc in hurd. if a task is created, does
  it contain any default port rights in its space? i am trying to deduce
  how one calls dir_lookup() on the root translator in glibc's open().
<kilobug> mjjc: yes, there are some default port rights, but I don't
  remember the details :/
<mjjc> kilobug: do you know where i should search for details?
<kilobug> mjjc: hum either in the Hurd's hacking guide
  https://www.gnu.org/software/hurd/hacking-guide/ or directly in the
  source code of exec server/libc I would say, or just ask again the
  question here later on to see if someone else has more information
<mjjc> ok, thanks
<pinotree> there's also rpctrace to, as the name says, trace all the rpc's
  executed
<braunr> some ports are introduced in new tasks, yes
<braunr> see
  http://www.gnu.org/software/hurd/hacking-guide/hhg.html#The-main-function
<braunr> and
<braunr>
  http://www.gnu.org/software/hurd/gnumach-doc/Task-Special-Ports.html#Task-Special-Ports
<mjjc> yes, the second link was just what i was looking for, thanks
<braunr> the second is very general
<braunr> also, the first applies to translators only
<braunr> if you're looking for how to do it for a non-translator
  application, the answer is probably somewhere in glibc
<braunr> _hurd_startup i'd guess