diagnose-crash Skill

1---2name: diagnose-crash3description: >4  Diagnose why a program crashed on this machine, from a systemd-coredump core dump.5  Use when a process has segfaulted, aborted, or otherwise dumped core, when asked6  why an application crashed or disappeared, or when a "Process crashed:" desktop7  notification is acted on. Triggers: crash, segfault, SIGSEGV, SIGABRT, core dump,8  coredumpctl, "why did X crash", "X keeps crashing", backtrace symbolization.9  Covers reporting a confirmed Omarchy bug upstream — see reporting.md.10---11 12# Diagnosing a Crash13 14Work from evidence. The goal is an honest account of what happened, not a15plausible-sounding story.16 17## Establish the facts18 19`coredumpctl info <pid>` is the starting point. Beyond the backtrace, note the20**command line** the process was started with — it usually reveals what the21program was working on when it died, which is often the whole answer.22 23`coredumpctl list` shows whether this crash is a one-off or a pattern. Repeated24crashes of the same program, or several programs dying together, point somewhere25different than a single failure does.26 27## Rule out the boring causes first28 29Check resource exhaustion before blaming the program: `free -h`, and the journal30for OOM kills. A process killed by the OOM killer is not a bug in that process.31 32## Correlate against the timeline33 34The crash timestamp is the most underused piece of evidence. Compare it against:35 36- **Filesystem mtimes.** A directory or file whose mtime lands on the same second37  as the crash strongly suggests what triggered it.38- **The journal** around that moment, for related warnings from the same or39  neighbouring processes.40- **Recent package updates.** A crash that starts right after an update points at41  the update.42 43## Read the whole core, not just frame 044 45Thread stacks other than the crashing one show what work was **in flight** —46thumbnailers, image loaders, IPC readers, GPU queues. That context often explains47the trigger even when the crashing frame itself cannot be symbolized.48 49Note any third-party code in the address space: file-manager or browser50extensions, plugins, out-of-tree drivers. In-process third-party code is a common51crash source and worth flagging — but do not pin blame on it without evidence52that it is actually implicated.53 54## Symbolize when you can55 56This is Arch, which runs a public debuginfod server:57 58```bash59core=$(mktemp -t crash-XXXXXX.core)60trap 'rm -f "$core"' EXIT61coredumpctl dump <pid> --output="$core"62DEBUGINFOD_URLS="https://debuginfod.archlinux.org" \63  gdb -q <executable> "$core" \64  -batch -ex 'set debuginfod enabled on' -ex 'bt'65```66 67A core is a verbatim copy of the process's memory and can hold passwords, tokens,68and private documents. Write it to a fresh `mktemp` path rather than a predictable69shared one, and delete it when you are done — never leave it lying in `/tmp`.70 71Many packages publish no debug symbols. When frames stay unresolved, say so —72never invent function names to fill the gap. An unsymbolized stack still has73shape: which library each frame belongs to, and whether the crash came from a74signal handler, a main loop, or a worker thread.75 76## Report77 781. What crashed, and what it was doing at the time.792. The most likely mechanism — separating clearly what the evidence **proves**80   from what you are **inferring**.813. Whether any user data was lost, and where it can be recovered from. Check the82   trash before concluding anything is gone.834. Whether it is likely to recur, and what would avoid or fix it.84 85Be straight about the limits of the evidence. If the cause is genuinely86ambiguous, say so rather than assembling confidence out of guesswork.87 88**Leave the system as you found it.** Diagnosis reads; it does not fix, tidy, or89reconfigure. The one thing to clean up is your own: delete the core you extracted90above, which is a copy of the crashed process's memory. The single change a91diagnosis may make is the mute below, and only when the user asks for it.92 93## Offer to stop the notifications for this program94 95A crash you have explained often keeps happening anyway. Finish by offering to96silence notifications for **that one program**, and never run it unprompted. Say97how to lift it in the same breath, so it is not a one-way door.98 99```bash100omarchy-crash-mute '<program>'        # silence it101omarchy-crash-mute '<program>' off    # let it speak again102omarchy-crash-mute                    # list what is muted103```104 105Pass the `binary:` path from the crash facts, or the `process:` name where no106binary was recorded; the command reduces either to the name the watcher keys on.107A diagnosis run by hand from `omarchy agent crash <pid>` has neither, so take108them from `coredumpctl info`. Prefer the binary: a process name is truncated to10915 characters and a basename is not, so muting the truncated form matches110nothing, forever, while looking like it worked.111 112Quote it. The name is whatever the crashed program's author called a file, and a113single quote inside one closes yours and runs the rest as your shell.114 115The key is a bare name, so anything run through an interpreter is keyed as the116interpreter: muting `python3.13` silences every Python program on the machine.117Say so rather than quietly doing it.118 119None of this fixes anything, and a mute offered in place of a fix that was within120reach is the wrong answer. For every program rather than one, the switch is121_Trigger > Toggle > Crash Capture_.122 123## If it is an Omarchy bug124 125Most application crashes are upstream bugs in those applications, not Omarchy's126doing. In the minority of cases where the cause really does sit within Omarchy's127sphere of control, read [`reporting.md`](reporting.md) before offering to file128anything.129