In ptrace requests that can also deliver a signal to the affected process, the request itself can be ignored if the signal delivery fails. E.g., in kerne/ptrace.c, the code for PTRACE_DETACH is: ret = ptrace_induce_signal(child, engine, data); if (!ret) ret = ptrace_detach(child, engine); Similar code exists in other places such as that for PTRACE_CONT. This code is in linux-2.6.17 with the utrace patches applied. The behaviour can be exercised by, e.g., calling ptrace(PTRACE_DETACH, pid, NULL, sig) where sig either is an invalid signal, or some internal conditions in ptrace.c aren't met. Also https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=206475
Created attachment 1316 [details] Testcase demonstrating the failure. This t/c does two ptrace(PTRACE_DETACH...) operations. The first tries to deliver an invalid signal--999--thereby failing (rc == -1) and inhibiting the intended detach. The second delivers a valid SIGKILL and succeeds (rc == 0).
vanilla kernel's ptrace_detach also fails for invalid signal what is the errno in the real failure case?
This is really "NOTABUG" but there's no option under "Resolve bug" for that.