It is possible to run Hurd stuff on top of another system instead of on Mach. One obvious variant is emulation (using QEMU, for example), but doing that does not really integratable the Hurd guest into the host system. There is also a more direct way, more powerful, but it also has certain requirements to do it effectively.
See also Mach on top of POSIX.
IRC, freenode, #hurd, August / September 2010
<marcusb> silver_hook: the Hurd can also refer to the interfaces of the filesystems etc, and a lot of that is really just server/client APIs that could be implemented on any system that has transferable rights to message capabilities. <marcusb> silver_hook: it's surprising how few systems *have* transferable rights, though! <marcusb> silver_hook: usually it is added as an afterthought <marcusb> and comes with restriction <youpi> marcusb: there's SCM_RIGHTS to transfer fds, which is quite often available <marcusb> youpi: yes, I know this as "fdpassing" <marcusb> youpi: it's described in the Stevens series even [...] <marcusb> ArneBab: well, let me put it this way. the Linux kernel has no interface to manipulate another tasks's virtual address space, ie you can't map/unmap stuff in another process <marcusb> ArneBab: you would have to use ptrace and load some stub code in that process to make that happen. <marcusb> ArneBab: so for complete transparent manipulation, you need a kernel module <marcusb> that is what the User Mode Linux kernel module does <marcusb> ArneBab: so say you use the User Mode Linux kernel module for that one feature. Then you can do everything that User Mode Linux can do, which, I assure you, includes running subhurds :) <marcusb> it can be a bit tricky to implement those features, but it is not harder than writing a kernel in the first place <ArneBab> So, if I got an admin to install User Mode Linux and Mach emulation, I’d get the flexibility (and independence from admin decisions) I have in the Hurd? <marcusb> ArneBab: one problem is that you still use Linux. For those who want to get rid of Linux for political reasons, that would mean complete failure <marcusb> ArneBab: if you have UML kernel module, you can implement Mach in user space <marcusb> ArneBab: in fact, John Tobey did this a couple of years ago, or started it
<marcusb> ArneBab: or you can just implement parts of it and relay to Linux for the rest <marcusb> the point is, that if you don't care for kernel improvements, and are sufficiently happy with the translator stuff, it's not hard to bring the Hurd to Linux or BSD
Continue reading about the benefits of a native Hurd implementation.
IRC, freenode, #hurd, 2010-12-28
<antrik> kilobug: there is no real requirement for the Hurd to run on a microkernel... as long as the important mechanisms are provided (most notably external pagers and Mach IPC), the Hurd could run an top of pretty much any kernel... <antrik> whether it makes sense is another question of course :-) <antrik> though I must say that I'm more and more convinced running the Hurd on top of a monolithic kernel would actually be a useful approach for the time being...
IRC, freenode, #hurd, 2011-02-11
<neal> marcus and I were discussing how to add Mach to Linux <neal> one could write a module to implement Mach IPC <neal> and another to implement Mach VM <neal> the big thing missing with Mach VM is the ability for a tracing process to easily map or unmap an inferior process's memory <antrik> neal: why would a tracing process need to map the inferior's memory? <neal> the simple answer is that is how it is done on Mach <antrik> neal: is it? not sure we are talking about the same thing here. GDB uses vm_read()/vm_write() to access the inferior's memory AFAIK <neal> on linux? <neal> I think it use /proc/pid/mem <antrik> on Hurd <neal> I'm talking about adding Mach to Linux <neal> by adding some functionality to Linux <neal> and then implementing a bunch in user space <antrik> yeah, but I don't understand the point about mapping inferior's memory :-( <antrik> what would be in user space? <neal> there are a number of different cut points <neal> one could imagine just using Linux's device drivers, CPU scheduler, memory management, etc. <neal> another possibility would be something higher where Hurd processes just use some Hurdish servers <antrik> neal: yeah, these are all options I have been considering... too bad I wasn't able to come to FOSDEM -- I'd love to have participated in this discussion :-( <antrik> neal: BTW, were you just discussing this as a hypothetical idea, or something you are seriously considering? <neal> I'm unlikely to work on it, sorry <antrik> didn't really expect that :-) <antrik> would be nice though if you could write up your conclusions...
IRC, freenode, #hurd, 2012-10-12
<peo-xaci> do hurd system libraries make raw system calls ever (i.e. inlined syscall() / raw assembly)? <braunr> sure <peo-xaci> hmm, so a hurd emulation layer would need to use ptrace if it should be fool proof? :/ <braunr> there is no real need for raw assembly, and the very syscalls are all available through macros <braunr> hum what are you trying to say ? <peo-xaci> well, if they are done through syscall, as a function, not a macro, then they can be intercepted with LD_PRELOAD <peo-xaci> so applications that do Hurd (Mach?) syscalls could work on f.e. Linux, if a special libc is injected into the program with LD_PRELOAD <peo-xaci> same thing with making standard Linux-applications go through the Hurd emulation layer <peo-xaci> without recompilation <mel-_> peo-xaci: the second direction is implemented in glibc. <mel-_> for the other direction, I personally see little use for it <braunr> peo-xaci: ok i misunderstood <braunr> peo-xaci: i don't think there is any truely direct syscall usage in the hurd <peo-xaci> hmm, I'm not sure I understand what directions you are referring to mel-_ <braunr> peo-xaci: what are you trying to achieve ? <peo-xaci> I want to make the Hurd design more accessible by letting Hurd application run on the Linux kernel, preferably without recompilation. This would be done with a daemon that implements Mach and which all syscalls would go to. <peo-xaci> then, I also want so that standard Linux applications can go through that Mach daemon as well, if a special libc is preloaded <braunr> you might want to discuss this with antrik <peo-xaci> what I'm trying to figure out specifically is if there is some library/interface that glue Hurd with Mach and would be better suited to emulate than Mach? Mach seems to be more of an implementation detail to the hurd and not something an application would directly use. <braunr> yes, the various hurd libraries (libports and libpager mostly) <peo-xaci> From [http://www.gnu.org/software/hurd/hurd/libports.html]: "libports is not (at least, not for now) a generalization / abstraction of Mach ports to the functionality the Hurd needs, that is, it is not meant to provide an interface independently of the underlying microkernel." <peo-xaci> Is this still true? <peo-xaci> Does libpager abstract the rest? <peo-xaci> (and the other hurd libraries) <braunr> there is nothing that really abstracts the hurd from mach <braunr> for example, reference counting often happens here and there <braunr> and core libraries like glibc and libpthread heavily rely on it (through sysdeps specific code though) <braunr> libports and libpager are meant to simplify object manipulation for the former, and pager operations for the latter <peo-xaci> and applications, such as translators, often use Mach interfaces directly? <peo-xaci> correct? <braunr> depends on what often means <braunr> let's say they do <peo-xaci> :/ then it probably is better to emulate Mach after all <braunr> there was a mach on posix port a long time ago <peo-xaci> I thought applications were completely separated from the microkernel in use by the Hurd <braunr> that level of abstraction is pretty new <braunr> genode is the only system i know which does that
<braunr> and it's still for "l4 variants" <pinotree> ah, thanks (i forgot that name) <antrik> braunr: Genode also runs on Linux and a few other non-L4 environments IIRC <antrik> peo-xaci: I'm not sure binary emulation is really useful. rather, I'd recompile stuff as "regular" Linux executables, only using a special glibc <antrik> where the special glibc could be basically a port of the Hurd glibc communicating with the Mach emulation instead of real Mach; or it could do emulation at a higher level <antrik> a higher level emulation would be more complicated to implement, but more efficient, and allow better integration with the ordinary GNU/Linux environment <antrik> also note that any regular program could be recompiled against the HELL glibc to run in the Hurdish environment... <antrik> (well, glibc + hurd server libraries) <peo-xaci> I'm willing to accept that Hurd-application would need to be recompiled to work on the HELL <peo-xaci> but not Linux-applications :) <antrik> peo-xaci: if you happen to understand German, there is a fairly good overview in my thesis report ;-) <antrik> peo-xaci: there are no "Hurd applications" or "Linux applications" <peo-xaci> well, let me define what I mean by the terms: Hurd applications use Hurd-specific interfaces/syscalls, and Linux applications use Linux-specific interfaces/syscalls <antrik> a few programs use Linux-specific interfaces (and we probably can't run them in HELL just as we can't run them on actual Hurd); but all other programs work in any glibc environment <antrik> (usually in any POSIX environment in fact...) <antrik> peo-xaci: no sane application uses syscalls <peo-xaci> they do under the hood <peo-xaci> I have read about inlined syscalls <antrik> again, there are *some* applications using Linux-specific interfaces (sometimes because they are inherently bound to Linux features, sometimes unnecessarily) <antrik> so far there are no applications using Hurd-specific interfaces <peo-xaci> translators do? <peo-xaci> they are standard executables are they not? <peo-xaci> I would like so that translators also can be run in the HELL <antrik> I wouldn't consider them applications. all existing translators are pretty much components of the Hurd itself <peo-xaci> okay, it's a question about semantics, perhaps I should use another word than "applications" :) <peo-xaci> for me, applications are what have a main-function, or similar single entry point <braunr> hum <braunr> that's not a good enough definition <antrik> anyways, as I said, I think recompiling translators against a Hurdish glibc and ported translator libraries seems the most reasonable approach to me <braunr> let's say applications are userspace processes that make use of services provided by the operating system <braunr> translators being part of the operating system here <antrik> braunr: do you know whether the Mach-on-POSIX was actually functional, or just an abandoned experiment?... <antrik> (I don't remember hearing of it before...) <braunr> incomplete iirc <peo-xaci> braunr: still, when I've explained what I meant, even if I used the wrong term, then my previous statements should come in another light <peo-xaci> antrik / braunr: are you still interested in hearing my thoughts/ideas about HELL? <antrik> oh, there is more to come? ;-) <peo-xaci> yes! I don't think I have made myself completely understood :/ <peo-xaci> what I envision is a HELL system that works on as low level as feasible, to make it possible to do almost anything that can be done on the real Hurd (except possibly testing hardware drivers and such very low level stuff). <braunr> sure <peo-xaci> I want it to be more than just allowing programs to access a virtual filesystem à la FUSE. My idea is that all user space system libraries/programs of the Hurd should be inside the HELL as well, and they should not be emulated. <peo-xaci> The system should at the very least be API compatible, so at the very most a recompilation is necessary. <peo-xaci> I also want so that GNU/Linux-programs can access the features of the HELL with little effort on the user. At most perhaps a script that wraps LD_PRELOADing has to be run on the binary. Best would be if it could work also with insane assembly programs using raw system calls, or if glibc happens to have some well hidden syscall being inlined to raw assembly code. <peo-xaci> And I think I have an idea on how an implementation could satisfy these things! <peo-xaci> By modifying the kernel and replace those syscalls that make sense for the Hurd/Mach <peo-xaci> with "the kernel", I meant Linux <braunr> it's possible but tedious and not very useful so better do that later <braunr> mach did something similar at its time <braunr> there was a syscall emulation library <peo-xaci> but isn't it about as much work as emulating the interface on user-level? <braunr> and the kernel cooperated so that unmodified unix binaries performing syscalls would actually jump to functions provided by that library, which generally made an RPC <peo-xaci> instead of a bunch of extern-declerations, one would put the symbols in the syscall table <braunr> define what "those syscalls that make sense for the Hurd/Mach" actually means <peo-xaci> open/close, for example <braunr> otherwise i don't see another better way than what the old mach folks did <braunr> well, with that old, but existing support, your open would perform a syscall <braunr> the kernel would catch it and redirect the caller to its syscall emulation library <braunr> which would call the open RPC instead <peo-xaci> wait, so this "existing support" you're talking about; is this a module for the Linux kernel (or a fork, or something else)? <peo-xaci> where can I find it? <braunr> no <braunr> it was for mach <braunr> in order to run unmodified unix binaries <braunr> the opposite of what you're trying to do <peo-xaci> ah okay <braunr> well <braunr> not really either :) <peo-xaci> does posix/unix define a standard for how a syscall table should look like, to allow binary syscall compatibility? <braunr> absolutely not <peo-xaci> so how could this mach module run any unmodified unix binary? if they expected different sys calls at different offsets? <braunr> posix specifically (and very early) states that it almost forbids itself to deal with anything regarding to ABIs <braunr> depends <braunr> since it was old, there weren't that many unix systems <braunr> and even today, there are techniques like those used by netbsd (and many other actually) <braunr> that are able to inspect the binary and load a syscall emulation environment depending on its exposed ABI <braunr> e.g. file on an executable states which system it's for <peo-xaci> hmm, I'm not sure how a kernel would implement that in practice.. I thought these things were so hard coded and dependent on raw memory reads that it would not be possible <braunr> but i really think it's not worth the time for your project <peo-xaci> to be honest I have virtually no experience of practical kernel programming <braunr> with an LDT on x86 for example <braunr> no, there is really not that much hardcoded <braunr> quite the contrary <braunr> there is a lot of runtime detection today <peo-xaci> well I mean how the syscall table is read <braunr> it's not read <peo-xaci> it's read to find the function pointer to the syscall handler in the kernel? <braunr> no <braunr> that's the really basic approach <braunr> (and in practice it can happen of course) <braunr> what really happens is that, for example, on linux, the user space system call code is loaded as a virtual shared library <braunr> use ldd on an executable to see it <braunr> this virtual object provides code that, depending on what the kernel has detected, will use the appropriate method to perform a system call <peo-xaci> but this user space system calls need to make some kind of cpu interupt to communicate with the kernel, right? <braunr> the glibc itself has no idea how a system call will look like in the end <braunr> yes <peo-xaci> an assembler programmer would be able to get around this glue code? <braunr> that's precisely what is embedded in this virtual library <braunr> it could yes <braunr> i think even when sysenter/sysexit is supported, legacy traps are still implemented to support old binaries <braunr> but then all these different entry points will lead to the same code inside the kernel <peo-xaci> but when the glue code is used, then its API compatible, and then I can understand that the kernel can allow different syscall implementations for different executables <braunr> what glue code ? <peo-xaci> what you talked about above "the user space system call code is loaded as a virtual shared library" <braunr> let's call it vdso
<peo-xaci> thanks, I looked it up on Wikipedia and understand immediately :P <peo-xaci> so VDSOs are provided by the kernel, not a regular library file, right? <vdox2> What does HELL stand for :) ? <dardevelin> vdox2, Hurd Emulation Layer for Linux <vdox2> dardevelin: thanks <braunr> peo-xaci: yes <antrik> peo-xaci: I believe your goals are conflicting. a low-level implementation makes it basically impossible to interact between the HELL environment and the GNU/Linux environment in any meaningful way. to allow such interaction, you *have* to have some glue at a higher semantic level <braunr> agreed <antrik> peo-xaci: BTW, if you want regular Linux binaries to get somehow redirected to access HELL facilities, there is already a framework (don't remember the name right now) that allows this kind of system call redirection on Linux <antrik> (it can run both through LD_PRELOAD or as a kernel module -- where obviously only the latter would allow raw system call redirection... but TBH, I don't think that's worthwhile anyways. the rare cases where programs use raw system calls are usually for extremely system-specific stuff anyways...) <antrik> ViewOS is the name <antrik> err... View-OS I mean <antrik> or maybe View OS ? ;-) <antrik> whatever, you'll find it :-)
<antrik> I'm not sure it's really worthwhile to use this either though... the most meaningful interaction is probably at the FS level, and that can be done with FUSE <antrik> OHOH, View-OS probably allows doing more interesting stuff that FUSE, such as modyfing the way the VFS works... <antrik> OTOH <antrik> so it could expose more of the Hurd features, at least in theory
IRC, freenode, #hurd, 2012-10-13
<peo-xaci> antrik / braunr: thanks for your input! I'm not entirely convinced though. :) I will probably return to this project once I have acquired a lot more knowledge about low level stuff. I want to see for myself whether a low level HELL is not feasible. :P <braunr> peo-xaci: what's the point of a low level hell ? <peo-xaci> more Hurd code can be tested in the hell, if the hell is at a low level <peo-xaci> at a higher level, some Hurd code cannot run, because the interfaces they use would not be accessible from the higher level emulation <antrik> peo-xaci: I never said it's not possible. I actually said it would be easier to do. I just said you can't do it low level *and* have meaningful interaction with the host system <peo-xaci> I don't understand why <braunr> peo-xaci: i really don't see what you want to achieve with low level support <braunr> what would be unavailable with a higher level approach ?