Re: Extending /proc/*/maps

[Corinna Vinschen <corinna-cygwin at cygwin dot com>]

On May 12 13:53, Ryan Johnson wrote:
> On 12/05/2011 12:55 PM, Corinna Vinschen wrote:
> >>>    struct heap
> >>>    {
> >>>      heap *next;
> >>>-    void *base;
> >>>+    unsigned heap_id;
> >>>+    uintptr_t base;
> >>>+    uintptr_t end;
> >>>+    unsigned long flags;
> >>>    };
> >>We don't actually need the end pointer: we're trying to match an
> >No, we need it.  The heaps consist of reserved and committed memory
> >blocks, as well as of shareable and non-shareable blocks.  Thus you
> >get multiple VirtualQuery calls per heap, thus you have to check for
> >the address within the entire heap(*).
> The code which queries heap_info already deals with allocations that
> take multiple VirtualQuery calls to traverse, and only calls into
> heap_info for the base of any given allocation. 

However, at least heap 2 consists of multiple allocated memory areas,
which are separately VirtualAlloc'ed or NtMapViewOfSection'ed.  Therefore
the first VirtualQuery returns AllocationBase = BaseAddress = 0x20000
and the next VirtualQuery returns AllocationBase = BaseAddress = 0x30000.
However, all the memory from 0x20000 up to 0x230000 constitutes a single
start block in heap 2!

> CAVEAT: According to MSDN's documentation for HeapWalk, "a heap
> consists of one or more regions of virtual memory, each with a
> unique region index." During heap walking, wFlags is set to
> PROCESS_HEAP_REGION whenever "the heap element is located at the
> beginning of a region of contiguous virtual memory in use by the
> heap." However, "large block regions" (those allocated directly via
> VirtualAlloc) do not set the PROCESS_HEAP_REGION flag. If this
> PROCESS_HEAP_REGION flag corresponds to your (flags & 2), then we
> won't report any large blocks (however, it's documented to have the
> value 0x0001, with 0x0002 being PROCESS_HEAP_UNCOMMITTED_RANGE).

I created a test application with several heaps and I create large
blocks in two of them.  They also have the flags&2 value set.  The
problem for heap walk is to identify blocks with a valid address.  Keep
in mind that all subsequent blocks within an allocated heap area do
*not* have an addres, but only a size, with address == 0.  You only get
their address by iterating from the start block to the block you're
looking for and adding up all sizes of the blocks up to there.

Consequentially a start block of another VirtualAlloc'ed area needs a
marker that the address is valid.  That marker is the flags value 2.
Which is kind of weird, actually, since the existence of a non-0 address
in the block should have been enough of a hint that this is a start
block...

As for the big blocks, they are apparently identified by the value in
the "Unknown" member of the DEBUG_HEAP_BLOCK structure.  Here's what I
figured out so far as far as "Unknown" is concerned:

   0x1000  All normal heaps
   0x3000  The process default heap (heap 0)
   0xc000  A low-frag heap
  0x10000  Heap 2, perhaps the meaning is "stack"?
  0x32000  Subsequently allocated block of a growable heap
 0x1e9000  Large block

I don't claim to understand the values, especially the reason for
setting several bits.

> >>>    heap *heaps;
> >>This is a misnomer now -- it's really a list of heap regions/blocks.
> >I don't think so.  The loop stores only the blocks which constitute
> >the original VirtualAlloc'ed memory regions.  They are not the real
> >heap blocks returned by Heap32First/Heap32Next.  These are filtered
> >out by checking for flags&  2 (**).
> Sorry, I cut too much context out of the diff. That's
> heap_info::heaps, which indeed refers to heap regions which we
> identified by checking flags&2 (that's why it needs the heap_id
> inside it now, where it didn't before) (++)

So you think something like heap_chunks is better?

> >>>+		heap *h = (heap *) cmalloc (HEAP_FHANDLER, sizeof (heap));
> >>>+		*h = (heap) {heaps, hcnt, barray[bcnt].Address,
> >>>+			     barray[bcnt].Address + barray[bcnt].Size,
> >>>+			     harray->Heaps[hcnt].Flags};
> >>>+		heaps = h;
> >>Given that the number of heap blocks is potentially large, I think
> >Not really.  See (**).  3 heaps ->  3 blocks, or only slightly more
> >if a growable heap got expanded.
> See (++). When I point my essentially-identical version of the code
> at emacs, it reports 8 heaps, each with 2-4 regions. The list
> traversed by fill_on_match has ~20 entries.

Oh, ok.  From my POV 20 is not a large number.  Ordering might take
more time than scanning.  I don't think it's worth the effort.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat