This is the mail archive of the cygwin mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

RE: 1.7.5: Occasional failure of CreatePipe or signal handing due to thread-unsafe code in cwdstuff::set


On Sep 04 02:26 Corinna Vinschen wrote:
> On Sep  3 16:18, John Carey wrote:
> > On Sep 03 12:37 Corinna Vinschen wrote:
> > > On Sep  2 23:32, John Carey wrote:
> > > > In Aug 17 10:15, Corinna Vinschen wrote:
> > > > > I just released 1.7.6-1.
> > > > ...
> > > > > What changed since Cygwin 1.7.5:
> > > > > ================================
> > > > >
> > > > > - Cygwin handles the current working directory entirely on its own.  The
> > > > >   Win32 current working directory is set to an invalid path to be out of
> > > > >   the way.  This affects calls to the Win32 file API (CreateFile, etc.).
> > > > >   See http://cygwin.com/htdocs/cygwin-ug-net/using.html#pathnames-win32-api
> > > >
> > > > Thank you very much for the fix!
> > > >
> > > > I've been running tests against Cygwin 1.7.6, and then 1.7.7,
> > > > and those sporadic, non-deterministic failures in CreatePipe
> > > > did stop after the 1.7.6 upgrade, and are still gone in 1.7.7.
> > > > I think it's been long enough to conclude that it is not just
> > > > the non-determinism--it really is fixed, as expected.
> > > >
> > > > I understand that this issue opened a can of worms;
> > > > thanks again for your efforts to overcome those difficulties.
> > >
> > > I still don't like the final workaround, which is, to set the Win32 CWD
> > > to the Cygwin CWD.  It would be nice if we could revert that change to
> > > the pre-1.7.6 behaviour in a Vista-friendly way.  If you ever find out
> > > how to make sure that the new handle in the PEB's user parameter block
> > > is used even on Vista and later, pray tell me.
> >
> > Thus far the only ideas I have come up with are somewhat
> > shaky and go well beyond the documented Win32 API.
> > (If only there was the equivalent of dup2(), but for Win32
> > handles!!!)
>
> ACK.
>
> > Just how much undocumented behavior is
> > tolerable, do you think?
>
> Up to XP/2K3, we can simply set the handle in the user parameter block
> and be done with it, just as in 1.7.5, but without the Vista workaround.
>
> The problem only starts with Vista.  I have no objections to use
> undocumented features, if they work.  If there's any way to replace the
> cwd handle with our own *and* keep the Win32 API happy, I'll take it.

I think I've found a way to open the directory handle for backup intent
on Vista and later versions.  Essentially, I emulate the new things that
SetCurrentDirectory() is doing, but in order to get the necessary
addresses, I have to do some very ugly hacks.

The proof-of-concept code follows (and is also attached).  It includes
an analysis of what is going on--to the extent that I know what is going
on, of course.  Please let me know what you think.

  - - - - - - BEGIN INCLUSION - - - - - - -

// Compile with Cygwin G++, and include a '-I' argument that
// specifies the "winsup" directory of the Cygwin source tree.
//
// Run with two arguments:
//
//   1. An absolute Windows path to a directory (can use forward slashes).
//
//   2. The relative name of a file within that directory.
//
// The purpose of this source code is to discuss Win32 CWD
// issues and to compile into a proof-of-concept program.
// Please read the interleaved comments.

#include <w32api/include/windows.h>
#include <w32api/include/ntdef.h>
#include <w32api/include/winnt.h>
#include <cygwin/ntdll.h>
#include <algorithm>
#include <cstdlib>
#include <cstring>
#include <iostream>
#include <locale>
#include <string>

using namespace std;

/****************************************

Primary research was on Windows 7 x64, but there appears to be
at least superficial similarity on Windows 2008 (x32 and x64)
and Windows Vista (x32 and x64).

Let "Params" be an alias for *NtCurrentTeb ()->Peb->ProcessParameters.

Let "HeapHandle" be an alias for *(PVOID*)((char*)NtCurrentTeb ()->Peb + 0x18)
(which is the second 32-bit word after ProcessParameters.)

Let "DismountCount" be an alias for the user space mapping of
KUSER_SHARED_DATA::DismountCount: namely, *(ULONG*)0x7ffe02dc.
See: http://www.nirsoft.net/kernel_struct/vista/KUSER_SHARED_DATA.html

In the implementation of SetCurrentDirectory (),
Params.CurrentDirectoryName.MaximumLength is read but NOT modified.
Its value determines the sizes of various buffers, including the
new buffer that will hold the new current directory pathname.

The constant value we have observed for
Params.CurrentDirectoryName.MaximumLength
is 520, which is sizeof(wchar_t [MAX_PATH]).

In addition to the PEB, there is an allocated memory block describing
the current directory.  Its lifetime is controlled by thread-safe
reference counting.  Call its type "VistaCwd"; more details follow.

The PEB does NOT seem to point to any VistaCwd instances.  Instead,
there is a global pointer outside of the PEB, which we call "Cwd",
and it is protected by a critical section, which we call "CwdCS",
which is also outside of the PEB.  Apparently these globals are
in the ntdll.dll data space, but their symbols are not exported.
Later we will discuss how to compute their addresses.

NOTE: The SetCurrentDirectory () implementation appears to update the
Win32 CWD *without* locking the PEB as such.  It locks CwdCS instead.

Structure:

****************************************/

  struct VistaCwd {
    // Use bus-locked increment/decrement to adjust this reference count.
    // When the reference count drops to zero, destroy and deallocate.

    volatile LONG ReferenceCount;

    // The open handle to the current directory.

    HANDLE DirectoryHandle;

    // A snapshot of the global DismountCount, updated at various
    // times during the lifetime of this struct.  It appears that
    // reads of and writes to this member are locked by CwdCS,
    // except for initialization (see below).
    //
    // What is this member for?  Following things in a debugger,
    // it seems likely that whenever the current VistaCwd is used,
    // its current value of OldDismountCount is compared against the
    // current (or perhaps slightly stale?) value of DismountCount,
    // and if it does not match then some extra action is taken.
    // Such extra action seems to include checking if the volume
    // is mounted.
    //
    // Presumably the global DismountCount increases monotonically.
    // Reads of the global DismountCount appear to be unlocked.
    // If a stale value is stored in OldDismountCount, then it
    // may be harmless in the sense that it just fails to prevent
    // unnecessary computation later.  But a stale read during
    // a comparison with OldDismountCount could prevent the extra
    // actions, which could be more serious.  At a guess, that
    // problem is prevented by the following code structure,
    // observed in at least one place that uses the CWD setting:
    // the value of OldDismountCount is read under a CwdCS lock,
    // and only *after* that lock is released is the global
    // DismountCount read.  The lock-unlock probably flushes
    // writes to the global DismountCount.

    ULONG OldDismountCount;

    // The path of the current directory.  Always ends in a backslash;
    // one is appended if the given path did not already end in one.
    //
    // The "MaximumLength" and "Buffer" members always specify the size
    // and starting address of the "Buffer" sibling member (see below).

    UNICODE_STRING Path;

    // The actual size of this member, and therefore of the struct as
    // a whole, is computed dynamically so that this member has the
    // same size as the previous value of
    //
    //   Params.CurrentDirectoryName.MaximumLength
    //
    // But since SetCurrentDirectory () never seems to modify that
    // PEB datum, in practice the dimension is always MAX_PATH.

    wchar_t Buffer[MAX_PATH];
  };

/****************************************

Allocation:

Before modifying the PEB data, the implementation
of SetCurrentDirectory () allocates a new instance
of VistaCwd, as follows:

    RtlAllocateHeap (HeapHandle, 0,
      offsetof (VistaCwd, Buffer) + Params.CurrentDirectoryName.MaximumLength);

Initialization:

The SetCurrentDirectory () implementation
then initializes that instance as follows:

  // Read DismountCount before calling NtOpenFile().
  // If the relative order matters at all, then it is
  // probably following the general principle that
  // a stale value copied from DismountCount to the
  // OldDismountCount member (see below) just causes
  // extra work but is otherwise harmless, whereas
  // a too-recent value might prevent necessary work.

  ULONG DismountCountBeforeNtOpenFile = DismountCount;

  ReferenceCount = 1;

  DirectoryHandle = NtOpenFile (...);

  // NOTE: There is no locking around this initialization.
  // But the SetCurrentDirectory () implementation locks CwdCS
  // while updating Cwd to point to new instances of VistaCwd,
  // and presumably that suffices to flush this initialization
  // to other CPUs.  Other threads would not see the new VistaCwd
  // instance before that point.  As mentioned before, copying of
  // a stale value of the global DismountCount is probably harmless,
  // in that it would merely trigger an unnecessary update.

  OldDismountCount = DismountCountBeforeNtOpenFile;

  // The length of the path in bytes, not counting the terminator.

  Path.Length = ...;

  // As mentioned before, in practice this is sizeof(wchar_t [MAX_PATH]).

  Path.MaximumLength = Params.CurrentDirectoryName.MaximumLength;

  Path.Buffer = Buffer;

  // The wide path for the directory; if necessary a backslash is
  // appended so that the final character (before the terminator)
  // is always a backslash.

  memcpy (Buffer, ...);
  Buffer[Path.Length / sizeof(wchar_t)] = L'\0';

Usage:

After creating the new VistaCwd instance; call it newCwd, the
SetCurrentDirectory () implementation modifies the PEB and Cwd
under a lock on CwdCS, as follows:

  Params.CurrentDirectoryHandle = newCwd.DirectoryHandle;

  Params.CurrentDirectoryName.Buffer = newCwd.Path.Buffer;

  Params.CurrentDirectoryName.Length = newCwd.Path.Length;

  VistaCwd *oldCwd = Cwd;

  Cwd = newCwd;

Note that as mentioned before,
Params.CurrentDirectoryName.MaximumLength
is NOT modified.

Cleanup:

After the SetCurrentDirectory () implementation releases
its lock on CwdCS, it performs a bus-locked decrement on
oldCwd->ReferenceCount (unless oldCwd == NULL).  If the
result is zero, it then destroys and deallocates the
old VistaCwd instance as follows:

  NtClose (oldCwd->DirectoryHandle);

  RtlFreeHeap (HeapHandle, 0, oldCwd);

Addresses of Cwd and CwdCS:

In order to imitate the way in which SetCurrentDirectory () updates
the Win32 CWD data, we must learn the addresses of Cwd and CwdCS.

The following program performs code analysis to discover those.
It then changes the Win32 CWD directly and tests the result
with a relative CreateFile ().  It is intended as
a quick-and-dirty stand-alone proof-of-concept program;
final code to be included in cygwin1.dll would differ greatly.

****************************************/

typedef unsigned char code_t;

unsigned peek32 (const unsigned char *p)
{
  return p[0] + (p[1] << 8) + (p[2] << 16) + (p[3] << 24);
}

void badCode ()
{
  cerr << "Code looks different than expected; bailing out." << endl;
  exit (1);
}

ostream& winError (ostream& os, DWORD code = GetLastError())
{
  LPSTR msg = 0;
  if (FormatMessageA (FORMAT_MESSAGE_FROM_SYSTEM
                      | FORMAT_MESSAGE_ALLOCATE_BUFFER,
                      0,
                      code,
                      MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
                      (LPSTR)&msg,
                      1,
                      NULL)) {
    os << "error " << code << ": " << msg;
    LocalFree (msg);
  } else {
    os << "???";
  }
  return os;
}

#define DDKAPI __stdcall

typedef CONST char *PCSZ;

typedef
NTSTATUS
DDKAPI
(* NtClose_t)(
  /*IN*/ HANDLE  Handle);

typedef
NTSTATUS
DDKAPI
(* NtOpenFile_t) (
  /*OUT*/ PHANDLE  FileHandle,
  /*IN*/ ACCESS_MASK  DesiredAccess,
  /*IN*/ POBJECT_ATTRIBUTES  ObjectAttributes,
  /*OUT*/ PIO_STATUS_BLOCK  IoStatusBlock,
  /*IN*/ ULONG  ShareAccess,
  /*IN*/ ULONG  OpenOptions);

typedef
PVOID
NTAPI
(* RtlAllocateHeap_t) (
  /*IN*/ HANDLE  HeapHandle,
  /*IN*/ ULONG   Flags,
  /*IN*/ ULONG   Size
);

typedef
BOOLEAN
NTAPI
(* RtlFreeHeap_t) (
  /*IN*/ HANDLE  HeapHandle,
  /*IN*/ ULONG   Flags,
  /*IN*/ PVOID   P
);

typedef
VOID
DDKAPI
(* RtlInitString_t) (
  /*IN OUT*/ PSTRING  DestinationString,
  /*IN*/ PCSZ  SourceString);

typedef
NTSTATUS
DDKAPI
(* RtlAnsiStringToUnicodeString_t) (
  /*IN OUT*/ PUNICODE_STRING  DestinationString,
  /*IN*/ PANSI_STRING  SourceString,
  /*IN*/ BOOLEAN  AllocateDestinationString);

typedef
VOID
DDKAPI
(* RtlFreeUnicodeString_t) (
  /*IN*/ PUNICODE_STRING  UnicodeString);

typedef
NTSTATUS
DDKAPI
(* RtlUnicodeStringToAnsiString_t)(
  /*IN OUT*/ PANSI_STRING  DestinationString,
  /*IN*/ PUNICODE_STRING  SourceString,
  /*IN*/ BOOLEAN  AllocateDestinationString);

typedef
VOID
DDKAPI
(* RtlFreeAnsiString_t)(
  /*IN*/ PANSI_STRING  AnsiString);

#define NTFUNC_DECL(F) F ## _t _ ## F

struct NtFuncs
{
  NTFUNC_DECL(NtClose);
  NTFUNC_DECL(NtOpenFile);
  NTFUNC_DECL(RtlAllocateHeap);
  NTFUNC_DECL(RtlFreeHeap);
  NTFUNC_DECL(RtlInitString);
  NTFUNC_DECL(RtlAnsiStringToUnicodeString);
  NTFUNC_DECL(RtlFreeUnicodeString);
  NTFUNC_DECL(RtlUnicodeStringToAnsiString);
  NTFUNC_DECL(RtlFreeAnsiString);

  NtFuncs(HMODULE module);
};

#define NTFUNC_BIND(M, F) _ ## F = (F ## _t) GetProcAddress(M, # F)

NtFuncs::NtFuncs(HMODULE module)
{
  NTFUNC_BIND(module, NtClose);
  NTFUNC_BIND(module, NtOpenFile);
  NTFUNC_BIND(module, RtlAllocateHeap);
  NTFUNC_BIND(module, RtlFreeHeap);
  NTFUNC_BIND(module, RtlInitString);
  NTFUNC_BIND(module, RtlAnsiStringToUnicodeString);
  NTFUNC_BIND(module, RtlFreeUnicodeString);
  NTFUNC_BIND(module, RtlUnicodeStringToAnsiString);
  NTFUNC_BIND(module, RtlFreeAnsiString);
}

void modifiedSetCurrentDirectory (
    const NtFuncs& nt,
    CRITICAL_SECTION *CwdCS,
    VistaCwd **Cwd,
    UNICODE_STRING& upath)
{
  // The real SetCurrentDirectory () implementation calls
  // a non-exported function that appears to expand relative
  // paths to absolute paths and convert / to \.  It might
  // also do other things.
  //
  // There appear to be no locks on the PEB or Cwd that
  // persist between this conversion and the actual Win32
  // CWD change, so presumably the following race is possible:
  //
  //   1. Thread A calls SetCurrentDirectory("C:\\a\\b").
  //
  //   2. Thread A creates Thread B.
  //
  //   3. Thread A starts calling SetCurrentDirectory("..")
  //      and gets through relative-to-absolute conversion,
  //      yielding "C:\\a" as the desired CWD.
  //
  //   4. Thread B calls SetCurrentDirectory("C:\\d\\e").
  //
  //   5. Thread A acquires CwdCS and sets Cwd to a new
  //      VistaCwd instance that specifies "C:\\a".
  //
  // Holding a lock between (3) and (5) would force
  // (4) to come either before (3) or after (5), but
  // there would still be a race: if (4) comes first,
  // then the final CWD is "C:\\d", whereas if (4)
  // comes last, then the final CWD is "C:\\d\\e".
  // Probably that is why no lock is held: it would
  // increase contention but not eliminate races.
  //
  // On the other hand, it appears that at least some
  // other versions of Windows that do not use the
  // VistaCwd mechanism, such as Windows Server 2003 R2 x64,
  // might actually acquire a lock before relative-to-absolute
  // path conversion.  Perhaps Microsoft rethought the races
  // when they changed to the newer mechanism?
  //
  // Anyway, the output of relative-to-absolute conversion
  // is run through another, non-exported path conversion
  // function that prefixes \??\ and presumably does
  // other normalizations appropriate to such paths.
  // However, that special form is used only as an argument
  // to NtOpenFile()--it does NOT get put into the new
  // VistaCwd instance.  Instead, the VistaCwd instance
  // gets a separate copy of the output of the
  // relative-to-absolute conversion, and in this copy only,
  // a backslash is appended if the absolute path did not
  // already end in a backslash.
  //
  // In this implementation we do not bother to perform
  // the relative-to-absolute path conversion; instead
  // we just warn if the path is not absolute.
  // But we do convert / to \.

  replace (upath.Buffer, upath.Buffer + upath.Length / sizeof(wchar_t),
           L'/', L'\\');
  if (upath.Length < 3 || upath.Buffer[1] != L':' || upath.Buffer[2] != L'\\'){
    cerr << "Relative paths not implemented." << endl;
    exit (1);
  }

  // Prefix \??\ to the path before passing it to NtOpenFile().
  if (upath.Length > (MAX_PATH - 5) * sizeof(wchar_t)) {
    cerr << "Path too long." << endl;
    exit(1);
  }
  wchar_t ntupath_buf[MAX_PATH];
  UNICODE_STRING ntupath;
  ntupath.Length = upath.Length + 4 * sizeof(wchar_t);
  ntupath.MaximumLength = sizeof ntupath_buf;
  ntupath.Buffer = ntupath_buf;
  memcpy(ntupath_buf, L"\\??\\", 4 * sizeof(wchar_t));
  memcpy(ntupath_buf + 4, upath.Buffer, upath.Length);
  ntupath_buf[4 + (upath.Length / sizeof(wchar_t))] = L'\0';

  RTL_USER_PROCESS_PARAMETERS& Params
    = *NtCurrentTeb ()->Peb->ProcessParameters;

  PVOID HeapHandle = *(PVOID*)((char*)NtCurrentTeb ()->Peb + 0x18);

  volatile ULONG& DismountCount = *(volatile ULONG*)0x7ffe02dc;

  // Do this before the NtOpenFile, just in case that order matters.

  ULONG DismountCountBeforeNtOpenFile = DismountCount;

  HANDLE h;
  NTSTATUS status;
  IO_STATUS_BLOCK io;
  OBJECT_ATTRIBUTES attr;

  InitializeObjectAttributes (&attr, &ntupath,
                              OBJ_CASE_INSENSITIVE | OBJ_INHERIT,
                              NULL, NULL);
  status = nt._NtOpenFile (&h, SYNCHRONIZE | FILE_TRAVERSE, &attr, &io,
                           FILE_SHARE_VALID_FLAGS,
                           FILE_DIRECTORY_FILE
                           | FILE_SYNCHRONOUS_IO_NONALERT
                           | FILE_OPEN_FOR_BACKUP_INTENT);
  if (!NT_SUCCESS (status))
  {
    ANSI_STRING narrow;
    nt._RtlUnicodeStringToAnsiString(&narrow, &ntupath, TRUE);

    cerr << "Failed to open directory '";
    cerr.write(narrow.Buffer, narrow.Length);
    cerr << "': NTSTATUS " << showbase << hex << status << endl;

    nt._RtlFreeAnsiString(&narrow);

    exit(1);
  }

  // As mentioned before, in practice this is sizeof (wchar_t [MAX_PATH]).

  USHORT MaximumLength = Params.CurrentDirectoryName.MaximumLength;

  VistaCwd *newCwd = (VistaCwd *) nt._RtlAllocateHeap (HeapHandle, 0,
      offsetof (VistaCwd, Buffer) + MaximumLength);

  newCwd->ReferenceCount = 1;

  newCwd->DirectoryHandle = h;

  newCwd->OldDismountCount = DismountCountBeforeNtOpenFile;

  newCwd->Path.Length = upath.Length;

  newCwd->Path.MaximumLength = MaximumLength;

  newCwd->Path.Buffer = newCwd->Buffer;

  memcpy (newCwd->Buffer, upath.Buffer, upath.Length);
  newCwd->Buffer[upath.Length / sizeof(wchar_t)] = L'\0';

  // In new VistaCwd instance, ensure that the final path character is L'\\':

  if (newCwd->Buffer[(upath.Length / sizeof(wchar_t)) - 1] != L'\\') {
    if (upath.Length > newCwd->Path.MaximumLength - (2 * sizeof(wchar_t))) {
      cerr << "Path too long." << endl;
      exit(1);
    }
    newCwd->Buffer[upath.Length / sizeof(wchar_t)] = L'\\';
    newCwd->Buffer[(upath.Length / sizeof(wchar_t)) + 1] = L'\0';
    upath.Length += sizeof(wchar_t);
  }

  // NOTE: We never acquire the PEB lock, only the Vista++ CWD lock:

  EnterCriticalSection(CwdCS);

  Params.CurrentDirectoryHandle = newCwd->DirectoryHandle;

  Params.CurrentDirectoryName.Buffer = newCwd->Path.Buffer;

  Params.CurrentDirectoryName.Length = newCwd->Path.Length;

  VistaCwd *oldCwd = *Cwd;

  *Cwd = newCwd;

  LeaveCriticalSection(CwdCS);

  if (InterlockedDecrement(&oldCwd->ReferenceCount) == 0) {
    nt._NtClose (oldCwd->DirectoryHandle);

    nt._RtlFreeHeap (HeapHandle, 0, oldCwd);
  }
}

int main (int argc, char **argv)
{
  HMODULE module = GetModuleHandle ("ntdll.dll");

  // Reading the CWD is simpler than writing the CWD,
  // which makes it easier to find the addresses we need,
  // and might also tend to make code changes less frequent.
  const code_t *get_dir = (const code_t*) GetProcAddress
    (module, "RtlGetCurrentDirectory_U");

  const code_t *ent_crit = (const code_t*) GetProcAddress
    (module, "RtlEnterCriticalSection");

  // Find first relative call instruction.
  const code_t *rcall = (const code_t *) memchr (get_dir, 0xE8, 32);
  if (! rcall) {
    badCode ();
  }

  // Compute the address, use_cwd, of the function being called.
  // This function actually fetches the current VistaCwd instance
  // and performs actions conditioned upon the freshness of its
  // OldDismountCount member.
  ptrdiff_t offset = peek32 (rcall + 1);
  const code_t *use_cwd = rcall + 5 + offset;

  // Find the first "push edi" instruction...
  const code_t *movedi = (const code_t *) memchr (use_cwd, 0x57, 32);
  ++movedi;
  // ...which should be followed by "mov edi, crit-sect-addr" then "push edi".
  // (Ideally we should not depend upon %EDI being the register,
  // but this is a proof of concept, and even with more flexibility
  // we are still depending heavily upon code structure here.)
  if (movedi[0] != 0xBF || movedi[5] != 0x57) {
    badCode ();
  }
  // Get the address of the critical section for the CWD.
  CRITICAL_SECTION *CwdCS = (CRITICAL_SECTION *) peek32 (movedi + 1);

  // To check we are seeing the right code, we check our expectation that
  // the next instruction is a relative call into RtlEnterCriticalSection.
  rcall = movedi + 6;
  if (rcall[0] != 0xe8) {
    badCode ();
  }
  offset = peek32 (rcall + 1);
  if (rcall + 5 + offset != ent_crit) {
    badCode ();
  }

  // After locking the critical section, the code should read the
  // global CWD block pointer that is guarded by that critical section.
  const code_t *movesi = rcall + 5;
  if (movesi[0] != 0x8b) {
    badCode ();
  }
  VistaCwd **Cwd = (VistaCwd **) peek32 (movesi + 2);

  cout << showbase << hex << (size_t)CwdCS
       << "  <== critical section" << endl;
  cout << showbase << hex << (size_t)Cwd
       << "  <== Vista++ CWD struct pointer" << endl;

  if (argc >= 2) {
    NtFuncs nt(module);

    STRING npath;
    nt._RtlInitString (&npath, argv[1]);

    UNICODE_STRING upath;
    nt._RtlAnsiStringToUnicodeString (&upath, &npath, TRUE);

    modifiedSetCurrentDirectory (nt, CwdCS, Cwd, upath);

    nt._RtlFreeUnicodeString (&upath);

    cout << "Changed directory." << endl;
  }

  if (argc >= 3) {
    HANDLE h = CreateFile (argv[2], GENERIC_READ, 0, NULL, OPEN_EXISTING,
                           FILE_ATTRIBUTE_NORMAL, NULL);
    if (h == INVALID_HANDLE_VALUE) {
      winError(cerr << "Failed to open file: ") << endl;
      return 1;
    }

    cout << "Successfully opened file." << endl;
    if (! CloseHandle (h)) {
      winError(cerr << "Failed to close file: ") << endl;
      return 1;
    }
  }

  return 0;
}
// Compile with Cygwin G++, and include a '-I' argument that
// specifies the "winsup" directory of the Cygwin source tree.
//
// Run with two arguments:
//
//   1. An absolute Windows path to a directory (can use forward slashes).
//
//   2. The relative name of a file within that directory.
//
// The purpose of this source code is to discuss Win32 CWD
// issues and to compile into a proof-of-concept program.
// Please read the interleaved comments.

#include <w32api/include/windows.h>
#include <w32api/include/ntdef.h>
#include <w32api/include/winnt.h>
#include <cygwin/ntdll.h>
#include <algorithm>
#include <cstdlib>
#include <cstring>
#include <iostream>
#include <locale>
#include <string>

using namespace std;

/****************************************

Primary research was on Windows 7 x64, but there appears to be
at least superficial similarity on Windows 2008 (x32 and x64)
and Windows Vista (x32 and x64).

Let "Params" be an alias for *NtCurrentTeb ()->Peb->ProcessParameters.

Let "HeapHandle" be an alias for *(PVOID*)((char*)NtCurrentTeb ()->Peb + 0x18)
(which is the second 32-bit word after ProcessParameters.)

Let "DismountCount" be an alias for the user space mapping of
KUSER_SHARED_DATA::DismountCount: namely, *(ULONG*)0x7ffe02dc.
See: http://www.nirsoft.net/kernel_struct/vista/KUSER_SHARED_DATA.html

In the implementation of SetCurrentDirectory (),
Params.CurrentDirectoryName.MaximumLength is read but NOT modified.
Its value determines the sizes of various buffers, including the
new buffer that will hold the new current directory pathname.

The constant value we have observed for
Params.CurrentDirectoryName.MaximumLength
is 520, which is sizeof(wchar_t [MAX_PATH]).

In addition to the PEB, there is an allocated memory block describing
the current directory.  Its lifetime is controlled by thread-safe
reference counting.  Call its type "VistaCwd"; more details follow.

The PEB does NOT seem to point to any VistaCwd instances.  Instead,
there is a global pointer outside of the PEB, which we call "Cwd",
and it is protected by a critical section, which we call "CwdCS",
which is also outside of the PEB.  Apparently these globals are
in the ntdll.dll data space, but their symbols are not exported.
Later we will discuss how to compute their addresses.

NOTE: The SetCurrentDirectory () implementation appears to update the
Win32 CWD *without* locking the PEB as such.  It locks CwdCS instead.

Structure:

****************************************/

  struct VistaCwd {
    // Use bus-locked increment/decrement to adjust this reference count.
    // When the reference count drops to zero, destroy and deallocate.

    volatile LONG ReferenceCount;

    // The open handle to the current directory.

    HANDLE DirectoryHandle;

    // A snapshot of the global DismountCount, updated at various
    // times during the lifetime of this struct.  It appears that
    // reads of and writes to this member are locked by CwdCS,
    // except for initialization (see below).
    //
    // What is this member for?  Following things in a debugger,
    // it seems likely that whenever the current VistaCwd is used,
    // its current value of OldDismountCount is compared against the
    // current (or perhaps slightly stale?) value of DismountCount,
    // and if it does not match then some extra action is taken.
    // Such extra action seems to include checking if the volume
    // is mounted.
    //
    // Presumably the global DismountCount increases monotonically.
    // Reads of the global DismountCount appear to be unlocked.
    // If a stale value is stored in OldDismountCount, then it
    // may be harmless in the sense that it just fails to prevent
    // unnecessary computation later.  But a stale read during
    // a comparison with OldDismountCount could prevent the extra
    // actions, which could be more serious.  At a guess, that
    // problem is prevented by the following code structure,
    // observed in at least one place that uses the CWD setting:
    // the value of OldDismountCount is read under a CwdCS lock,
    // and only *after* that lock is released is the global
    // DismountCount read.  The lock-unlock probably flushes
    // writes to the global DismountCount.

    ULONG OldDismountCount;

    // The path of the current directory.  Always ends in a backslash;
    // one is appended if the given path did not already end in one.
    //
    // The "MaximumLength" and "Buffer" members always specify the size
    // and starting address of the "Buffer" sibling member (see below).

    UNICODE_STRING Path;

    // The actual size of this member, and therefore of the struct as
    // a whole, is computed dynamically so that this member has the
    // same size as the previous value of
    //
    //   Params.CurrentDirectoryName.MaximumLength
    //
    // But since SetCurrentDirectory () never seems to modify that
    // PEB datum, in practice the dimension is always MAX_PATH.

    wchar_t Buffer[MAX_PATH];
  };

/****************************************

Allocation:

Before modifying the PEB data, the implementation
of SetCurrentDirectory () allocates a new instance
of VistaCwd, as follows:

    RtlAllocateHeap (HeapHandle, 0,
      offsetof (VistaCwd, Buffer) + Params.CurrentDirectoryName.MaximumLength);

Initialization:

The SetCurrentDirectory () implementation
then initializes that instance as follows:

  // Read DismountCount before calling NtOpenFile().
  // If the relative order matters at all, then it is
  // probably following the general principle that
  // a stale value copied from DismountCount to the
  // OldDismountCount member (see below) just causes
  // extra work but is otherwise harmless, whereas
  // a too-recent value might prevent necessary work.

  ULONG DismountCountBeforeNtOpenFile = DismountCount;

  ReferenceCount = 1;

  DirectoryHandle = NtOpenFile (...);

  // NOTE: There is no locking around this initialization.
  // But the SetCurrentDirectory () implementation locks CwdCS
  // while updating Cwd to point to new instances of VistaCwd,
  // and presumably that suffices to flush this initialization
  // to other CPUs.  Other threads would not see the new VistaCwd
  // instance before that point.  As mentioned before, copying of
  // a stale value of the global DismountCount is probably harmless,
  // in that it would merely trigger an unnecessary update.

  OldDismountCount = DismountCountBeforeNtOpenFile;

  // The length of the path in bytes, not counting the terminator.

  Path.Length = ...;

  // As mentioned before, in practice this is sizeof(wchar_t [MAX_PATH]).

  Path.MaximumLength = Params.CurrentDirectoryName.MaximumLength;

  Path.Buffer = Buffer;

  // The wide path for the directory; if necessary a backslash is
  // appended so that the final character (before the terminator)
  // is always a backslash.

  memcpy (Buffer, ...);
  Buffer[Path.Length / sizeof(wchar_t)] = L'\0';

Usage:

After creating the new VistaCwd instance; call it newCwd, the
SetCurrentDirectory () implementation modifies the PEB and Cwd
under a lock on CwdCS, as follows:

  Params.CurrentDirectoryHandle = newCwd.DirectoryHandle;

  Params.CurrentDirectoryName.Buffer = newCwd.Path.Buffer;

  Params.CurrentDirectoryName.Length = newCwd.Path.Length;

  VistaCwd *oldCwd = Cwd;

  Cwd = newCwd;

Note that as mentioned before,
Params.CurrentDirectoryName.MaximumLength
is NOT modified.

Cleanup:

After the SetCurrentDirectory () implementation releases
its lock on CwdCS, it performs a bus-locked decrement on
oldCwd->ReferenceCount (unless oldCwd == NULL).  If the
result is zero, it then destroys and deallocates the
old VistaCwd instance as follows:

  NtClose (oldCwd->DirectoryHandle);

  RtlFreeHeap (HeapHandle, 0, oldCwd);

Addresses of Cwd and CwdCS:

In order to imitate the way in which SetCurrentDirectory () updates
the Win32 CWD data, we must learn the addresses of Cwd and CwdCS.

The following program performs code analysis to discover those.
It then changes the Win32 CWD directly and tests the result
with a relative CreateFile ().  It is intended as
a quick-and-dirty stand-alone proof-of-concept program;
final code to be included in cygwin1.dll would differ greatly.

****************************************/

typedef unsigned char code_t;

unsigned peek32 (const unsigned char *p)
{
  return p[0] + (p[1] << 8) + (p[2] << 16) + (p[3] << 24);
}

void badCode ()
{
  cerr << "Code looks different than expected; bailing out." << endl;
  exit (1);
}

ostream& winError (ostream& os, DWORD code = GetLastError())
{
  LPSTR msg = 0;
  if (FormatMessageA (FORMAT_MESSAGE_FROM_SYSTEM
                      | FORMAT_MESSAGE_ALLOCATE_BUFFER,
                      0,
                      code,
                      MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
                      (LPSTR)&msg,
                      1,
                      NULL)) {
    os << "error " << code << ": " << msg;
    LocalFree (msg);
  } else {
    os << "???";
  }
  return os;
}

#define DDKAPI __stdcall

typedef CONST char *PCSZ;

typedef
NTSTATUS
DDKAPI
(* NtClose_t)(
  /*IN*/ HANDLE  Handle);

typedef
NTSTATUS
DDKAPI
(* NtOpenFile_t) (
  /*OUT*/ PHANDLE  FileHandle,
  /*IN*/ ACCESS_MASK  DesiredAccess,
  /*IN*/ POBJECT_ATTRIBUTES  ObjectAttributes,
  /*OUT*/ PIO_STATUS_BLOCK  IoStatusBlock,
  /*IN*/ ULONG  ShareAccess,
  /*IN*/ ULONG  OpenOptions);

typedef
PVOID
NTAPI
(* RtlAllocateHeap_t) (
  /*IN*/ HANDLE  HeapHandle,
  /*IN*/ ULONG   Flags,
  /*IN*/ ULONG   Size
);

typedef
BOOLEAN
NTAPI
(* RtlFreeHeap_t) (
  /*IN*/ HANDLE  HeapHandle,
  /*IN*/ ULONG   Flags,
  /*IN*/ PVOID   P
);

typedef
VOID
DDKAPI
(* RtlInitString_t) (
  /*IN OUT*/ PSTRING  DestinationString,
  /*IN*/ PCSZ  SourceString);

typedef
NTSTATUS
DDKAPI
(* RtlAnsiStringToUnicodeString_t) (
  /*IN OUT*/ PUNICODE_STRING  DestinationString,
  /*IN*/ PANSI_STRING  SourceString,
  /*IN*/ BOOLEAN  AllocateDestinationString);

typedef
VOID
DDKAPI
(* RtlFreeUnicodeString_t) (
  /*IN*/ PUNICODE_STRING  UnicodeString);

typedef
NTSTATUS
DDKAPI
(* RtlUnicodeStringToAnsiString_t)(
  /*IN OUT*/ PANSI_STRING  DestinationString,
  /*IN*/ PUNICODE_STRING  SourceString,
  /*IN*/ BOOLEAN  AllocateDestinationString);

typedef
VOID
DDKAPI
(* RtlFreeAnsiString_t)(
  /*IN*/ PANSI_STRING  AnsiString);

#define NTFUNC_DECL(F) F ## _t _ ## F

struct NtFuncs
{
  NTFUNC_DECL(NtClose);
  NTFUNC_DECL(NtOpenFile);
  NTFUNC_DECL(RtlAllocateHeap);
  NTFUNC_DECL(RtlFreeHeap);
  NTFUNC_DECL(RtlInitString);
  NTFUNC_DECL(RtlAnsiStringToUnicodeString);
  NTFUNC_DECL(RtlFreeUnicodeString);
  NTFUNC_DECL(RtlUnicodeStringToAnsiString);
  NTFUNC_DECL(RtlFreeAnsiString);

  NtFuncs(HMODULE module);
};

#define NTFUNC_BIND(M, F) _ ## F = (F ## _t) GetProcAddress(M, # F)

NtFuncs::NtFuncs(HMODULE module)
{
  NTFUNC_BIND(module, NtClose);
  NTFUNC_BIND(module, NtOpenFile);
  NTFUNC_BIND(module, RtlAllocateHeap);
  NTFUNC_BIND(module, RtlFreeHeap);
  NTFUNC_BIND(module, RtlInitString);
  NTFUNC_BIND(module, RtlAnsiStringToUnicodeString);
  NTFUNC_BIND(module, RtlFreeUnicodeString);
  NTFUNC_BIND(module, RtlUnicodeStringToAnsiString);
  NTFUNC_BIND(module, RtlFreeAnsiString);
}

void modifiedSetCurrentDirectory (
    const NtFuncs& nt,
    CRITICAL_SECTION *CwdCS,
    VistaCwd **Cwd,
    UNICODE_STRING& upath)
{
  // The real SetCurrentDirectory () implementation calls
  // a non-exported function that appears to expand relative
  // paths to absolute paths and convert / to \.  It might
  // also do other things.
  //
  // There appear to be no locks on the PEB or Cwd that
  // persist between this conversion and the actual Win32
  // CWD change, so presumably the following race is possible:
  //
  //   1. Thread A calls SetCurrentDirectory("C:\\a\\b").
  //
  //   2. Thread A creates Thread B.
  //
  //   3. Thread A starts calling SetCurrentDirectory("..")
  //      and gets through relative-to-absolute conversion,
  //      yielding "C:\\a" as the desired CWD.
  //
  //   4. Thread B calls SetCurrentDirectory("C:\\d\\e").
  //
  //   5. Thread A acquires CwdCS and sets Cwd to a new
  //      VistaCwd instance that specifies "C:\\a".
  //
  // Holding a lock between (3) and (5) would force
  // (4) to come either before (3) or after (5), but
  // there would still be a race: if (4) comes first,
  // then the final CWD is "C:\\d", whereas if (4)
  // comes last, then the final CWD is "C:\\d\\e".
  // Probably that is why no lock is held: it would
  // increase contention but not eliminate races.
  //
  // On the other hand, it appears that at least some
  // other versions of Windows that do not use the
  // VistaCwd mechanism, such as Windows Server 2003 R2 x64,
  // might actually acquire a lock before relative-to-absolute
  // path conversion.  Perhaps Microsoft rethought the races
  // when they changed to the newer mechanism?
  //
  // Anyway, the output of relative-to-absolute conversion
  // is run through another, non-exported path conversion
  // function that prefixes \??\ and presumably does
  // other normalizations appropriate to such paths.
  // However, that special form is used only as an argument
  // to NtOpenFile()--it does NOT get put into the new
  // VistaCwd instance.  Instead, the VistaCwd instance
  // gets a separate copy of the output of the
  // relative-to-absolute conversion, and in this copy only,
  // a backslash is appended if the absolute path did not
  // already end in a backslash.
  //
  // In this implementation we do not bother to perform
  // the relative-to-absolute path conversion; instead
  // we just warn if the path is not absolute.
  // But we do convert / to \.

  replace (upath.Buffer, upath.Buffer + upath.Length / sizeof(wchar_t),
           L'/', L'\\');
  if (upath.Length < 3 || upath.Buffer[1] != L':' || upath.Buffer[2] != L'\\'){
    cerr << "Relative paths not implemented." << endl;
    exit (1);
  }

  // Prefix \??\ to the path before passing it to NtOpenFile().
  if (upath.Length > (MAX_PATH - 5) * sizeof(wchar_t)) {
    cerr << "Path too long." << endl;
    exit(1);
  }
  wchar_t ntupath_buf[MAX_PATH];
  UNICODE_STRING ntupath;
  ntupath.Length = upath.Length + 4 * sizeof(wchar_t);
  ntupath.MaximumLength = sizeof ntupath_buf;
  ntupath.Buffer = ntupath_buf;
  memcpy(ntupath_buf, L"\\??\\", 4 * sizeof(wchar_t));
  memcpy(ntupath_buf + 4, upath.Buffer, upath.Length);
  ntupath_buf[4 + (upath.Length / sizeof(wchar_t))] = L'\0';

  RTL_USER_PROCESS_PARAMETERS& Params
    = *NtCurrentTeb ()->Peb->ProcessParameters;

  PVOID HeapHandle = *(PVOID*)((char*)NtCurrentTeb ()->Peb + 0x18);

  volatile ULONG& DismountCount = *(volatile ULONG*)0x7ffe02dc;

  // Do this before the NtOpenFile, just in case that order matters.

  ULONG DismountCountBeforeNtOpenFile = DismountCount;

  HANDLE h;
  NTSTATUS status;
  IO_STATUS_BLOCK io;
  OBJECT_ATTRIBUTES attr;

  InitializeObjectAttributes (&attr, &ntupath,
                              OBJ_CASE_INSENSITIVE | OBJ_INHERIT,
                              NULL, NULL);
  status = nt._NtOpenFile (&h, SYNCHRONIZE | FILE_TRAVERSE, &attr, &io,
                           FILE_SHARE_VALID_FLAGS,
                           FILE_DIRECTORY_FILE
                           | FILE_SYNCHRONOUS_IO_NONALERT
                           | FILE_OPEN_FOR_BACKUP_INTENT);
  if (!NT_SUCCESS (status))
  {
    ANSI_STRING narrow;
    nt._RtlUnicodeStringToAnsiString(&narrow, &ntupath, TRUE);

    cerr << "Failed to open directory '";
    cerr.write(narrow.Buffer, narrow.Length);
    cerr << "': NTSTATUS " << showbase << hex << status << endl;

    nt._RtlFreeAnsiString(&narrow);

    exit(1);
  }

  // As mentioned before, in practice this is sizeof (wchar_t [MAX_PATH]).

  USHORT MaximumLength = Params.CurrentDirectoryName.MaximumLength;

  VistaCwd *newCwd = (VistaCwd *) nt._RtlAllocateHeap (HeapHandle, 0,
      offsetof (VistaCwd, Buffer) + MaximumLength);

  newCwd->ReferenceCount = 1;

  newCwd->DirectoryHandle = h;

  newCwd->OldDismountCount = DismountCountBeforeNtOpenFile;

  newCwd->Path.Length = upath.Length;

  newCwd->Path.MaximumLength = MaximumLength;

  newCwd->Path.Buffer = newCwd->Buffer;

  memcpy (newCwd->Buffer, upath.Buffer, upath.Length);
  newCwd->Buffer[upath.Length / sizeof(wchar_t)] = L'\0';

  // In new VistaCwd instance, ensure that the final path character is L'\\':

  if (newCwd->Buffer[(upath.Length / sizeof(wchar_t)) - 1] != L'\\') {
    if (upath.Length > newCwd->Path.MaximumLength - (2 * sizeof(wchar_t))) {
      cerr << "Path too long." << endl;
      exit(1);
    }
    newCwd->Buffer[upath.Length / sizeof(wchar_t)] = L'\\';
    newCwd->Buffer[(upath.Length / sizeof(wchar_t)) + 1] = L'\0';
    upath.Length += sizeof(wchar_t);
  }

  // NOTE: We never acquire the PEB lock, only the Vista++ CWD lock:

  EnterCriticalSection(CwdCS);

  Params.CurrentDirectoryHandle = newCwd->DirectoryHandle;

  Params.CurrentDirectoryName.Buffer = newCwd->Path.Buffer;

  Params.CurrentDirectoryName.Length = newCwd->Path.Length;

  VistaCwd *oldCwd = *Cwd;

  *Cwd = newCwd;

  LeaveCriticalSection(CwdCS);

  if (InterlockedDecrement(&oldCwd->ReferenceCount) == 0) {
    nt._NtClose (oldCwd->DirectoryHandle);

    nt._RtlFreeHeap (HeapHandle, 0, oldCwd);
  }
}

int main (int argc, char **argv)
{
  HMODULE module = GetModuleHandle ("ntdll.dll");

  // Reading the CWD is simpler than writing the CWD,
  // which makes it easier to find the addresses we need,
  // and might also tend to make code changes less frequent.
  const code_t *get_dir = (const code_t*) GetProcAddress
    (module, "RtlGetCurrentDirectory_U");

  const code_t *ent_crit = (const code_t*) GetProcAddress
    (module, "RtlEnterCriticalSection");

  // Find first relative call instruction.
  const code_t *rcall = (const code_t *) memchr (get_dir, 0xE8, 32);
  if (! rcall) {
    badCode ();
  }

  // Compute the address, use_cwd, of the function being called.
  // This function actually fetches the current VistaCwd instance
  // and performs actions conditioned upon the freshness of its
  // OldDismountCount member.
  ptrdiff_t offset = peek32 (rcall + 1);
  const code_t *use_cwd = rcall + 5 + offset;

  // Find the first "push edi" instruction...
  const code_t *movedi = (const code_t *) memchr (use_cwd, 0x57, 32);
  ++movedi;
  // ...which should be followed by "mov edi, crit-sect-addr" then "push edi".
  // (Ideally we should not depend upon %EDI being the register,
  // but this is a proof of concept, and even with more flexibility
  // we are still depending heavily upon code structure here.)
  if (movedi[0] != 0xBF || movedi[5] != 0x57) {
    badCode ();
  }
  // Get the address of the critical section for the CWD.
  CRITICAL_SECTION *CwdCS = (CRITICAL_SECTION *) peek32 (movedi + 1);

  // To check we are seeing the right code, we check our expectation that
  // the next instruction is a relative call into RtlEnterCriticalSection.
  rcall = movedi + 6;
  if (rcall[0] != 0xe8) {
    badCode ();
  }
  offset = peek32 (rcall + 1);
  if (rcall + 5 + offset != ent_crit) {
    badCode ();
  }

  // After locking the critical section, the code should read the
  // global CWD block pointer that is guarded by that critical section.
  const code_t *movesi = rcall + 5;
  if (movesi[0] != 0x8b) {
    badCode ();
  }
  VistaCwd **Cwd = (VistaCwd **) peek32 (movesi + 2);

  cout << showbase << hex << (size_t)CwdCS
       << "  <== critical section" << endl;
  cout << showbase << hex << (size_t)Cwd
       << "  <== Vista++ CWD struct pointer" << endl;

  if (argc >= 2) {
    NtFuncs nt(module);

    STRING npath;
    nt._RtlInitString (&npath, argv[1]);

    UNICODE_STRING upath;
    nt._RtlAnsiStringToUnicodeString (&upath, &npath, TRUE);

    modifiedSetCurrentDirectory (nt, CwdCS, Cwd, upath);

    nt._RtlFreeUnicodeString (&upath);

    cout << "Changed directory." << endl;
  }

  if (argc >= 3) {
    HANDLE h = CreateFile (argv[2], GENERIC_READ, 0, NULL, OPEN_EXISTING,
                           FILE_ATTRIBUTE_NORMAL, NULL);
    if (h == INVALID_HANDLE_VALUE) {
      winError(cerr << "Failed to open file: ") << endl;
      return 1;
    }

    cout << "Successfully opened file." << endl;
    if (! CloseHandle (h)) {
      winError(cerr << "Failed to close file: ") << endl;
      return 1;
    }
  }

  return 0;
}
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]