On a desktop computer, you think of an operating system as a big piece of complex software. For small systems (like an Arduino) you might want something a lot simpler. Object Oriented State Machine Operating System (OOSMOS) is a single-file and highly portable operating system, and it recently went open source.
OOSMOS has a unique approach because it is threadless, which makes it easy to use in memory constrained systems because there is no stack required for threads that don’t exist. The unit of execution is a C++ object (although you can use C) that contains a state machine.
You can read the API documentation online. Just remember that this is not an end user OS like Windows or Linux, but an operating environment for managing multiple tasks. You can, though, use OOSMOS under Windows or Linux as well as many other host systems.