What you already can do is attaching gdb -- either live or at a core dump. Please see https://github.com/Solo5/solo5/blob/master/docs/debugging.md for detailed instructions ;)

What you as well can do is record-replay -- well, at least there used to be this option, I'm now lost whether it has never been merged anywhere, or it is stuck in some PR somewhere. This was truly great - since the external interface is so thin, it is easily doable to dump all external events (API calls and returns) onto disk and replay one-by-one, inspecting the state.

To be a bit more precise, what we achieved so far: apart from web servers and TLS reverse proxies (with DNS-backed let's encrypt provisioning), DNS authoritative servers, a CalDAV server, git client and servers, SMTP stack (including dkim etc.), OpenVPN implementation, archive mirror (using tar as file system), monitoring with syslog and influx, ...

A big milestone was binary releases and deployment instructions, see https://robur.coop/Projects/Reproducible_builds

see more at https://blog.robur.coop https://hannes.robur.coop https://blog.osau.re/ https://reyn.ir/archive.html

If you're curious how/what to integrate MirageOS into your infrastructure, please reach out (best via email).

good news is that MirageOS can as well be executed in a seccomp context with only 5 or 7 system calls allowed -- see the spt target of solo5 https://github.com/solo5/solo5 -- also this great talk by mato https://archive.fosdem.org/2019/schedule/event/solo5_unikern...

Some MirageOS developers were hired/acquired by docker -- though not the (open source) project, beither the code... There's still quite some work on MirageOS itself, including reproducible binary builds, VPN, DNS services, orchestration solutions, ...

Amazing how little code is needed for that, and how clear the interface is... documentation and tutorials at https://docs.osau.re/miou/

Scanning IPv6 is effectively impossible.

We use asynchronous tasks in MirageOS (cooperative multitasking) using lwt http://ocsigen.org/lwt/latest/manual/manual, so you can serve multiple network connections at the same time.

Since there are no processes, there's no concept of "fork", but indeed you can run multiple tasks at the same time (using the same address space). Why again would you need multiple processes? Since the programming language OCaml has a semantics and is memory-safe, there's no strong reason for isolation at execution time (apart from the C bits, which we try to keep to a minimum and compile with runtime safety flags (red-zone for stack protection, mapping execute-only (seems to only reliably work on OpenBSD), etc.)).

What about multiprocessing? I assume that MirageOS can take advantage of multiple cores (or do you need separate instances per core?). In this case is the system still shared memory?

Also I would say there are reasons for isolation beyond memory safety.

MirageOS is - similar to the latest OCaml release - only using a single CPU core. You can run multiple unikernels, one on each core. If doing that, you can use Xen vchan (shared memory), or TCP for marshalling.

> Also I would say there are reasons for isolation beyond memory safety.

Would you mind to elaborate which reasons you are thinking of?

> MirageOS is - similar to the latest OCaml release - only using a single CPU core.

Thanks. Is that going to change now that OCaml is finally getting proper multicore support?

>> Also I would say there are reasons for isolation beyond memory safety. >Would you mind to elaborate which reasons you are thinking of?

Memory safety in a sense protects the integrity of the 'runtime', but only partially help to protect business level integrity. A task might still tricked (by mistake or malice) to access objects it is not supposed to. I'm sure that OCaml has enough abstractions to help prevent that, but full isolation of tasks is a blunt and effective tool.

> only using a single CPU core

How does it go when you deploy a unikernel on EC2 (or on any IaaS where the hypervisor is managed unlike bare-metal) with multiple cores? Is there a way to start a unikernel per core on a single instance, or are you bound to use single core instance types only?

Is this also how you deal with logging/monitoring?

Do you have a concept of logging daemon? How would I run prometheus exporter-like things?

Is there a FAQ where questions along these lines are answered?

Just substitute 'microservice' with 'unikernel' and you do broadly the same things. There's a prometheus library that you link with the MirageOS unikernel and it exports using that: https://github.com/mirage/prometheus

No FAQ for this sort of thing yet, but we should start assembling one sometime soon. Questions like this very welcome on the discussion forums: https://discuss.ocaml.org/t/ann-mirageos-4-0/9598 to help us get started.

There's a nice collection of unikernels over at: https://github.com/roburio/unikernels and https://github.com/tarides/unikernels for various infrastructure pieces (like https, smtp, dns, ip filters, etc) that are good to crib from for your own infrastructure.

Maybe https://hannes.robur.coop/Posts/Monitoring sheds some light how to monitor MirageOS unikernels ;)

Conurrency instead of parallelism.

No amount of concurrency on a single core can equal the performance of multiple cores.

That is why you deploy a process per core, with affinity so that it doesn't jump around.

You can run more than one unikernel per machine. That's why things like KVM or Bhyve are targets instead of just bare metal.

In ancient times, only Xen was supported. Nowadays, the support moved to:

  - Xen (PVH)
  - Linux KVM, FreeBSD BHyve, OpenBSD VMM
  - SPT (seccomp, no hardware virtualization)
  - virtio (GCE, ..)
  - muen (muen.sk)

Take a look at https://github.com/solo5/solo5 which is used as the low level bits to run OCaml.

> - SPT (seccomp, no hardware virtualization)

To be clear, this means running an ordinary Linux binary, right?

I'm curious about the remaining advantages in practice of running multiple processes on a shared kernel rather than using hardware virtualization. I guess the main one is more efficient resource pooling, particularly memory and the page cache. I understand the primary advantage of virtualization is a smaller attack surface.

Yes, this is an ordinary Linux binary.

The attack surface is different, you may be interested in https://archive.fosdem.org/2019/schedule/event/solo5_unikern... and/or https://archive.fosdem.org/2019/schedule/event/solo5_unikern... :)

TL;DR: hardware virtualization is pushing trust into hardware -- but can you trust the hardware implementation (to isolate memory)?

One of the stated advantages various places of having a unikernel build as a regular ELF binary on Linux or BSD (including the semi-BSD macOS) is that you can use it as your instrumented debug build very easily. Yes, you can instrument your app heavily. You can have console output. You can log to a mounted volume, to an object store, or to a remote log server with rsyslog or something like Elastic. But with an ELF binary on an OS you can strace, dtrace, fence it, run it under gdb or another debugger, instrument the OS under it, or whatever. Then the same source code builds essentially the same program to run right on KVM or Xen with no OS under it for security and efficiency.

Highly recommend the history of English podcast, which goes deep into the topics https://historyofenglishpodcast.com/

Processes? Seriously if there's only a single one, why would you waste resources on process management and process information? Just get over it ;) http://unikernel.org https://mirage.io :D

There isn't a single one, there's at least one per container.