The hard disk. The hard disk is used for long-term storage of all software, including the operating system.
When a computer is started up, the hardware will automatically load the operating system and start it running. This process is called booting. The reason for this odd term is that the operating system is itself involved in getting itself running---a process that is like someone "pulling themselves up by their bootstraps." Once the operating system is running, it is used to start up any other program.
Here is a (simplified) list of what happens when the user (you) starts up an application. Assume that the operating system (OS) is already running.
As the application runs, the OS is there in the background managing resources, doing input and output for the application, and keeping everything else running.