systemd status=203/EXEC: The Service That Never Started
Logged into a box today and systemctl status greeted me with State: degraded and Failed: 1 units. One service had died, and it had died in a very specific way: status=203/EXEC. That status means the program never ran at all. Here is how to read it, why systemd is so quiet about the reason, and the one command that tells you straight away.
The broken service comes from the exercise "The service that never starts" (linux-002) in awesome_labs. That's my collection of hands-on labs that run on local Lima VMs. Each exercise breaks something on purpose, you fix it in a real shell, and the lab checks your work.
The failure
$ systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● kl-report.service loaded failed failed kubelab report heartbeat
$ systemctl status kl-report.service
× kl-report.service - kubelab report heartbeat
Loaded: loaded (/etc/systemd/system/kl-report.service; disabled; preset: enabled)
Active: failed (Result: exit-code) since Mon 2026-11-16 15:30:36 EET; 2min 30s ago
Duration: 235us
Process: 16216 ExecStart=/opt/kl-report/reportd (code=exited, status=203/EXEC)
Main PID: 16216 (code=exited, status=203/EXEC)
CPU: 198us
Nov 16 15:30:36 docker-host systemd[1]: Started kl-report.service - kubelab report heartbeat.
Nov 16 15:30:36 docker-host systemd[1]: kl-report.service: Main process exited, code=exited, status=203/EXEC
Nov 16 15:30:36 docker-host systemd[1]: kl-report.service: Failed with result 'exit-code'.Three things in that output already point at the answer:
status=203/EXEC: the program did not exit with 203. systemd did. Exit codes from 200 up are systemd's own codes, and they mean "I failed while setting up your process, before your code ran".Duration: 235us: a quarter of a millisecond. Nothing real runs that fast. The service never got going.Process: ... ExecStart=/opt/kl-report/reportd: the exact path systemd tried to run is printed right there.
What 203/EXEC means
systemd starts a service in a few steps. It forks, then sets up the working directory, user, namespaces and stdout, and finally calls execve() on the ExecStart= binary. If one of those steps fails, the child exits with a code that names the step. You can look any of them up with systemd-analyze:
$ systemd-analyze exit-status 200 203 209 217 226
NAME STATUS CLASS
CHDIR 200 systemd
EXEC 203 systemd
STDOUT 209 systemd
USER 217 systemd
NAMESPACE 226 systemdSo 203 means the final step, execve(), failed. The kernel refused to run the file. It might not exist, it might not be executable, or the kernel might not understand how to start it.
systemd won't tell you why (most of the time)
You might expect the journal to have the real error. I reproduced each cause with test units on Ubuntu 24.04 (systemd 255), and for the two most common ones, a missing file and a file without the execute bit, the journal shows the same three lines as above and nothing else, even when read as root:
$ sudo journalctl -b -u kl-report.service -o cat
Started kl-report.service - kubelab report heartbeat.
kl-report.service: Main process exited, code=exited, status=203/EXEC
kl-report.service: Failed with result 'exit-code'.Staring at the journal won't help, so check the path instead.
Finding the cause
The quickest check is to list the path from ExecStart=:
$ cat /etc/systemd/system/kl-report.service
[Unit]
Description=kubelab report heartbeat
[Service]
ExecStart=/opt/kl-report/reportd
Restart=no
[Install]
WantedBy=multi-user.target
$ ls /opt/kl-report/
report.shThere it is. The unit wants reportd, but the directory only has report.sh. Someone renamed the program, or the unit was written for a version that was never deployed.
The better tool is systemd-analyze verify. It checks the unit file and, unlike the journal, actually says what is wrong with the executable:
$ systemd-analyze verify /etc/systemd/system/kl-report.service
kl-report.service: Command /opt/kl-report/reportd is not executable: No such file or directoryFor a file that exists but has no execute bit it says is not executable: Permission denied. It exits non-zero in both cases, so it works in scripts and in CI that ships unit files too.
The fix
Point ExecStart= at the file that is really there. The cleanest way to edit a unit is systemctl edit --full. It opens the file in your editor as root and reloads systemd when you save:
sudo systemctl edit --full kl-report.service[Service]
ExecStart=/opt/kl-report/report.sh
Restart=noIf you edit the file directly with sudo vim, you have to reload yourself, and that needs root too. Without sudo, polkit stops and asks for a password:
sudo vim /etc/systemd/system/kl-report.service
sudo systemctl daemon-reload
sudo systemctl restart kl-report.serviceAnd now:
$ sudo systemctl status kl-report.service
● kl-report.service - kubelab report heartbeat
Loaded: loaded (/etc/systemd/system/kl-report.service; disabled; preset: enabled)
Active: active (running) since Mon 2026-11-16 15:36:28 EET; 7s ago
Main PID: 16383 (report.sh)
Tasks: 2 (limit: 3392)
CGroup: /system.slice/kl-report.service
├─16383 /bin/bash /opt/kl-report/report.sh
└─16391 sleep 2The main process shows up as /bin/bash /opt/kl-report/report.sh. That's the kernel reading the script's #!/bin/bash line and running bash with the script as its argument.
Other things that cause 203/EXEC
A wrong path is the most common cause, but these all give the same status code. For the last three, systemd does log a Failed to execute line, and the error text is misleading:
- No execute bit. The file is there, but
chmod +xwas never run (very common aftergit cloneon a repo that didn't keep the mode, or after copying from a Windows share). Fix:chmod +x, or run it through its interpreter withExecStart=/bin/bash /opt/kl-report/report.sh. - Interpreter in the shebang doesn't exist, e.g.
#!/usr/bin/python9. The journal saysFailed to execute /path/script.sh: No such file or directory. The script is right there. The missing file is the interpreter. - No shebang at all. The file is executable, but the kernel has no idea how to run it. The journal says
Failed to execute /path/script.sh: Exec format error. A binary built for the wrong CPU (an x86_64 build on an ARM box) gives the same error. - Filesystem mounted
noexec. Scripts under/tmpor a data mount withnoexecfail with permission denied even with+x. Check withfindmnt -T /path/to/file.
Windows line endings. The first line is really #!/bin/bash\r, so the kernel looks for a program called bash\r. You get the same misleading No such file or directory. file shows it straight away:
$ file report.sh
report.sh: Bourne-Again shell script, ASCII text executable, with CRLF line terminators
$ sed -i 's/\r$//' report.sh # or: dos2unix report.shWhen it isn't obvious, run the exact ExecStart= command by hand as the service's user. The shell gives a clearer error than systemd does. For the CRLF case, bash says cannot execute: required file not found.
One more thing: it won't survive a reboot
Look at the Loaded: line again: disabled. The service runs now because it was started by hand, but nothing starts it at boot. That's the second half of the exercise, and the grader checks for it: the service has to be both active and enabled. If it should always run:
sudo systemctl enable --now kl-report.serviceFor a heartbeat-style service it is also worth swapping Restart=no for Restart=on-failure (with a RestartSec=) so a crash doesn't leave it dead until someone next runs systemctl --failed.
Checklist
status=2xxmeans systemd failed before your code ran. Look the code up withsystemd-analyze exit-status.- For 203/EXEC, check the
ExecStart=path first:ls -lit. - Run
systemd-analyze verifyon the unit. It names the problem when the journal doesn't. - If the journal says "No such file or directory" for a file that exists, check the shebang and the line endings (
head -1 file | od -c). - Edit with
sudo systemctl edit --full, thenenable --nowif it should come back after a reboot.
Try it yourself
Reading about a broken service isn't the same as fixing one. This exercise and many more are in awesome_labs: over 1,300 exercises across Linux, Docker, Kubernetes, networking, Git, Terraform, Ansible, cloud, databases and security. Each has a scenario, hints you reveal one at a time, and a full solution if you get stuck. It runs on macOS and Linux:
git clone https://github.com/kpatr1981/awesome_labs.git
cd awesome_labs
make build
./awesome_labs install # sets up Lima and anything else that's missing
awesome_labs # browse the tracks and pick an exerciseThis one is linux-002 in Linux · Linux essentials. Reset it and see if you can fix it without scrolling back up.
$ ls ./see-also/
$ ./buy-me-a-coffee ☕ // found this useful? I write these in my spare time