Repo created

This commit is contained in:
Fr4nz D13trich 2025-11-22 14:04:28 +01:00
parent 81b91f4139
commit f8c34fa5ee
22732 changed files with 4815320 additions and 2 deletions

View file

@ -0,0 +1,4 @@
primiano@chromium.org
wfh@chromium.org
# COMPONENT: Internals

View file

@ -0,0 +1,197 @@
This document describes how malloc / new calls are routed in the various Chrome
platforms.
Bare in mind that the chromium codebase does not always just use `malloc()`.
Some examples:
- Large parts of the renderer (Blink) use two home-brewed allocators,
PartitionAlloc and BlinkGC (Oilpan).
- Some subsystems, such as the V8 JavaScript engine, handle memory management
autonomously.
- Various parts of the codebase use abstractions such as `SharedMemory` or
`DiscardableMemory` which, similarly to the above, have their own page-level
memory management.
Background
----------
The `allocator` target defines at compile-time the platform-specific choice of
the allocator and extra-hooks which services calls to malloc/new. The relevant
build-time flags involved are `use_allocator` and `use_allocator_shim`.
The default choices are as follows:
**Windows**
`use_allocator: winheap`, the default Windows heap.
Additionally, `static_library` (i.e. non-component) builds have a shim
layer wrapping malloc/new, which is controlled by `use_allocator_shim`.
The shim layer provides extra security features, such as preventing large
allocations that can hit signed vs. unsigned bugs in third_party code.
**Linux Desktop / CrOS**
`use_allocator: tcmalloc`, a forked copy of tcmalloc which resides in
`third_party/tcmalloc/chromium`. Setting `use_allocator: none` causes the build
to fall back to the system (Glibc) symbols.
**Android**
`use_allocator: none`, always use the allocator symbols coming from Android's
libc (Bionic). As it is developed as part of the OS, it is considered to be
optimized for small devices and more memory-efficient than other choices.
The actual implementation backing malloc symbols in Bionic is up to the board
config and can vary (typically *dlmalloc* or *jemalloc* on most Nexus devices).
**Mac/iOS**
`use_allocator: none`, we always use the system's allocator implementation.
In addition, when building for `asan` / `msan` both the allocator and the shim
layer are disabled.
Layering and build deps
-----------------------
The `allocator` target provides both the source files for tcmalloc (where
applicable) and the linker flags required for the Windows shim layer.
The `base` target is (almost) the only one depending on `allocator`. No other
targets should depend on it, with the exception of the very few executables /
dynamic libraries that don't depend, either directly or indirectly, on `base`
within the scope of a linker unit.
More importantly, **no other place outside of `/base` should depend on the
specific allocator** (e.g., directly include `third_party/tcmalloc`).
If such a functional dependency is required that should be achieved using
abstractions in `base` (see `/base/allocator/allocator_extension.h` and
`/base/memory/`)
**Why `base` depends on `allocator`?**
Because it needs to provide services that depend on the actual allocator
implementation. In the past `base` used to pretend to be allocator-agnostic
and get the dependencies injected by other layers. This ended up being an
inconsistent mess.
See the [allocator cleanup doc][url-allocator-cleanup] for more context.
Linker unit targets (executables and shared libraries) that depend in some way
on `base` (most of the targets in the codebase) get automatically the correct
set of linker flags to pull in tcmalloc or the Windows shim-layer.
Source code
-----------
This directory contains just the allocator (i.e. shim) layer that switches
between the different underlying memory allocation implementations.
The tcmalloc library originates outside of Chromium and exists in
`../../third_party/tcmalloc` (currently, the actual location is defined in the
allocator.gyp file). The third party sources use a vendor-branch SCM pattern to
track Chromium-specific changes independently from upstream changes.
The general intent is to push local changes upstream so that over
time we no longer need any forked files.
Unified allocator shim
----------------------
On most platforms, Chrome overrides the malloc / operator new symbols (and
corresponding free / delete and other variants). This is to enforce security
checks and lately to enable the
[memory-infra heap profiler][url-memory-infra-heap-profiler].
Historically each platform had its special logic for defining the allocator
symbols in different places of the codebase. The unified allocator shim is
a project aimed to unify the symbol definition and allocator routing logic in
a central place.
- Full documentation: [Allocator shim design doc][url-allocator-shim].
- Current state: Available and enabled by default on Android, CrOS, Linux,
Mac OS and Windows.
- Tracking bug: [https://crbug.com/550886][crbug.com/550886].
- Build-time flag: `use_allocator_shim`.
**Overview of the unified allocator shim**
The allocator shim consists of three stages:
```
+-------------------------+ +-----------------------+ +----------------+
| malloc & friends | -> | shim layer | -> | Routing to |
| symbols definition | | implementation | | allocator |
+-------------------------+ +-----------------------+ +----------------+
| - libc symbols (malloc, | | - Security checks | | - tcmalloc |
| calloc, free, ...) | | - Chain of dispatchers| | - glibc |
| - C++ symbols (operator | | that can intercept | | - Android |
| new, delete, ...) | | and override | | bionic |
| - glibc weak symbols | | allocations | | - WinHeap |
| (__libc_malloc, ...) | +-----------------------+ +----------------+
+-------------------------+
```
**1. malloc symbols definition**
This stage takes care of overriding the symbols `malloc`, `free`,
`operator new`, `operator delete` and friends and routing those calls inside the
allocator shim (next point).
This is taken care of by the headers in `allocator_shim_override_*`.
*On Linux/CrOS*: the allocator symbols are defined as exported global symbols
in `allocator_shim_override_libc_symbols.h` (for `malloc`, `free` and friends)
and in `allocator_shim_override_cpp_symbols.h` (for `operator new`,
`operator delete` and friends).
This enables proper interposition of malloc symbols referenced by the main
executable and any third party libraries. Symbol resolution on Linux is a breadth first search that starts from the root link unit, that is the executable
(see EXECUTABLE AND LINKABLE FORMAT (ELF) - Portable Formats Specification).
Additionally, when tcmalloc is the default allocator, some extra glibc symbols
are also defined in `allocator_shim_override_glibc_weak_symbols.h`, for subtle
reasons explained in that file.
The Linux/CrOS shim was introduced by
[crrev.com/1675143004](https://crrev.com/1675143004).
*On Android*: load-time symbol interposition (unlike the Linux/CrOS case) is not
possible. This is because Android processes are `fork()`-ed from the Android
zygote, which pre-loads libc.so and only later native code gets loaded via
`dlopen()` (symbols from `dlopen()`-ed libraries get a different resolution
scope).
In this case, the approach instead of wrapping symbol resolution at link time
(i.e. during the build), via the `--Wl,-wrap,malloc` linker flag.
The use of this wrapping flag causes:
- All references to allocator symbols in the Chrome codebase to be rewritten as
references to `__wrap_malloc` and friends. The `__wrap_malloc` symbols are
defined in the `allocator_shim_override_linker_wrapped_symbols.h` and
route allocator calls inside the shim layer.
- The reference to the original `malloc` symbols (which typically is defined by
the system's libc.so) are accessible via the special `__real_malloc` and
friends symbols (which will be relocated, at load time, against `malloc`).
In summary, this approach is transparent to the dynamic loader, which still sees
undefined symbol references to malloc symbols.
These symbols will be resolved against libc.so as usual.
More details in [crrev.com/1719433002](https://crrev.com/1719433002).
**2. Shim layer implementation**
This stage contains the actual shim implementation. This consists of:
- A singly linked list of dispatchers (structs with function pointers to `malloc`-like functions). Dispatchers can be dynamically inserted at runtime
(using the `InsertAllocatorDispatch` API). They can intercept and override
allocator calls.
- The security checks (suicide on malloc-failure via `std::new_handler`, etc).
This happens inside `allocator_shim.cc`
**3. Final allocator routing**
The final element of the aforementioned dispatcher chain is statically defined
at build time and ultimately routes the allocator calls to the actual allocator
(as described in the *Background* section above). This is taken care of by the
headers in `allocator_shim_default_dispatch_to_*` files.
Appendixes
----------
**How does the Windows shim layer replace the malloc symbols?**
The mechanism for hooking LIBCMT in Windows is rather tricky. The core
problem is that by default, the Windows library does not declare malloc and
free as weak symbols. Because of this, they cannot be overridden. To work
around this, we start with the LIBCMT.LIB, and manually remove all allocator
related functions from it using the visual studio library tool. Once removed,
we can now link against the library and provide custom versions of the
allocator related functionality.
See the script `preb_libc.py` in this folder.
Related links
-------------
- [Unified allocator shim doc - Feb 2016][url-allocator-shim]
- [Allocator cleanup doc - Jan 2016][url-allocator-cleanup]
- [Proposal to use PartitionAlloc as default allocator](https://crbug.com/339604)
- [Memory-Infra: Tools to profile memory usage in Chrome](/docs/memory-infra/README.md)
[url-allocator-cleanup]: https://docs.google.com/document/d/1V77Kgp_4tfaaWPEZVxNevoD02wXiatnAv7Ssgr0hmjg/edit?usp=sharing
[url-memory-infra-heap-profiler]: /docs/memory-infra/heap_profiler.md
[url-allocator-shim]: https://docs.google.com/document/d/1yKlO1AO4XjpDad9rjcBOI15EKdAGsuGO_IeZy0g0kxo/edit?usp=sharing

View file

@ -0,0 +1,46 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/allocator_check.h"
#include "base/allocator/buildflags.h"
#include "build/build_config.h"
#if defined(OS_WIN)
#include "base/allocator/winheap_stubs_win.h"
#endif
#if defined(OS_LINUX)
#include <malloc.h>
#endif
#if defined(OS_MACOSX)
#include "base/allocator/allocator_interception_mac.h"
#endif
namespace base {
namespace allocator {
bool IsAllocatorInitialized() {
#if defined(OS_WIN) && BUILDFLAG(USE_ALLOCATOR_SHIM)
// Set by allocator_shim_override_ucrt_symbols_win.h when the
// shimmed _set_new_mode() is called.
return g_is_win_shim_layer_initialized;
#elif defined(OS_LINUX) && BUILDFLAG(USE_TCMALLOC) && \
!defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
// From third_party/tcmalloc/chromium/src/gperftools/tcmalloc.h.
// TODO(primiano): replace with an include once base can depend on allocator.
#define TC_MALLOPT_IS_OVERRIDDEN_BY_TCMALLOC 0xbeef42
return (mallopt(TC_MALLOPT_IS_OVERRIDDEN_BY_TCMALLOC, 0) ==
TC_MALLOPT_IS_OVERRIDDEN_BY_TCMALLOC);
#elif defined(OS_MACOSX) && !defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
// From allocator_interception_mac.mm.
return base::allocator::g_replaced_default_zone;
#else
return true;
#endif
}
} // namespace allocator
} // namespace base

View file

@ -0,0 +1,18 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_ALLOCATOR_ALLOCATOR_CHECK_H_
#define BASE_ALLOCATOR_ALLOCATOR_ALLOCATOR_CHECK_H_
#include "base/base_export.h"
namespace base {
namespace allocator {
BASE_EXPORT bool IsAllocatorInitialized();
} // namespace allocator
} // namespace base
#endif // BASE_ALLOCATOR_ALLOCATOR_ALLOCATOR_CHECK_H_

View file

@ -0,0 +1,73 @@
// Copyright (c) 2012 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/allocator_extension.h"
#include "base/allocator/buildflags.h"
#include "base/logging.h"
#if BUILDFLAG(USE_TCMALLOC)
#include "third_party/tcmalloc/chromium/src/gperftools/heap-profiler.h"
#include "third_party/tcmalloc/chromium/src/gperftools/malloc_extension.h"
#include "third_party/tcmalloc/chromium/src/gperftools/malloc_hook.h"
#endif
namespace base {
namespace allocator {
void ReleaseFreeMemory() {
#if BUILDFLAG(USE_TCMALLOC)
::MallocExtension::instance()->ReleaseFreeMemory();
#endif
}
bool GetNumericProperty(const char* name, size_t* value) {
#if BUILDFLAG(USE_TCMALLOC)
return ::MallocExtension::instance()->GetNumericProperty(name, value);
#endif
return false;
}
bool SetNumericProperty(const char* name, size_t value) {
#if BUILDFLAG(USE_TCMALLOC)
return ::MallocExtension::instance()->SetNumericProperty(name, value);
#endif
return false;
}
void GetHeapSample(std::string* writer) {
#if BUILDFLAG(USE_TCMALLOC)
::MallocExtension::instance()->GetHeapSample(writer);
#endif
}
bool IsHeapProfilerRunning() {
#if BUILDFLAG(USE_TCMALLOC) && defined(ENABLE_PROFILING)
return ::IsHeapProfilerRunning();
#endif
return false;
}
void SetHooks(AllocHookFunc alloc_hook, FreeHookFunc free_hook) {
// TODO(sque): Use allocator shim layer instead.
#if BUILDFLAG(USE_TCMALLOC)
// Make sure no hooks get overwritten.
auto prev_alloc_hook = MallocHook::SetNewHook(alloc_hook);
if (alloc_hook)
DCHECK(!prev_alloc_hook);
auto prev_free_hook = MallocHook::SetDeleteHook(free_hook);
if (free_hook)
DCHECK(!prev_free_hook);
#endif
}
int GetCallStack(void** stack, int max_stack_size) {
#if BUILDFLAG(USE_TCMALLOC)
return MallocHook::GetCallerStackTrace(stack, max_stack_size, 0);
#endif
return 0;
}
} // namespace allocator
} // namespace base

View file

@ -0,0 +1,67 @@
// Copyright (c) 2012 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_ALLOCATOR_EXTENSION_H_
#define BASE_ALLOCATOR_ALLOCATOR_EXTENSION_H_
#include <stddef.h> // for size_t
#include <string>
#include "base/base_export.h"
#include "build/build_config.h"
namespace base {
namespace allocator {
// Callback types for alloc and free.
using AllocHookFunc = void (*)(const void*, size_t);
using FreeHookFunc = void (*)(const void*);
// Request that the allocator release any free memory it knows about to the
// system.
BASE_EXPORT void ReleaseFreeMemory();
// Get the named property's |value|. Returns true if the property is known.
// Returns false if the property is not a valid property name for the current
// allocator implementation.
// |name| or |value| cannot be NULL
BASE_EXPORT bool GetNumericProperty(const char* name, size_t* value);
// Set the named property's |value|. Returns true if the property is known and
// writable. Returns false if the property is not a valid property name for the
// current allocator implementation, or is not writable. |name| cannot be NULL.
BASE_EXPORT bool SetNumericProperty(const char* name, size_t value);
// Outputs to |writer| a sample of live objects and the stack traces
// that allocated these objects. The format of the returned output
// is equivalent to the output of the heap profiler and can
// therefore be passed to "pprof".
// NOTE: by default, the allocator does not do any heap sampling, and this
// function will always return an empty sample. To get useful
// data from GetHeapSample, you must also set the numeric property
// "tcmalloc.sampling_period_bytes" to a value such as 524288.
BASE_EXPORT void GetHeapSample(std::string* writer);
BASE_EXPORT bool IsHeapProfilerRunning();
// Register callbacks for alloc and free. Can only store one callback at a time
// for each of alloc and free.
BASE_EXPORT void SetHooks(AllocHookFunc alloc_hook, FreeHookFunc free_hook);
// Attempts to unwind the call stack from the current location where this
// function is being called from. Must be called from a hook function registered
// by calling SetSingle{Alloc,Free}Hook, directly or indirectly.
//
// Arguments:
// stack: pointer to a pre-allocated array of void*'s.
// max_stack_size: indicates the size of the array in |stack|.
//
// Returns the number of call stack frames stored in |stack|, or 0 if no call
// stack information is available.
BASE_EXPORT int GetCallStack(void** stack, int max_stack_size);
} // namespace allocator
} // namespace base
#endif // BASE_ALLOCATOR_ALLOCATOR_EXTENSION_H_

View file

@ -0,0 +1,56 @@
// Copyright 2017 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_ALLOCATOR_INTERCEPTION_MAC_H_
#define BASE_ALLOCATOR_ALLOCATOR_INTERCEPTION_MAC_H_
#include <stddef.h>
#include "base/base_export.h"
#include "third_party/apple_apsl/malloc.h"
namespace base {
namespace allocator {
struct MallocZoneFunctions;
// Saves the function pointers currently used by the default zone.
void StoreFunctionsForDefaultZone();
// Same as StoreFunctionsForDefaultZone, but for all malloc zones.
void StoreFunctionsForAllZones();
// For all malloc zones that have been stored, replace their functions with
// |functions|.
void ReplaceFunctionsForStoredZones(const MallocZoneFunctions* functions);
extern bool g_replaced_default_zone;
// Calls the original implementation of malloc/calloc prior to interception.
bool UncheckedMallocMac(size_t size, void** result);
bool UncheckedCallocMac(size_t num_items, size_t size, void** result);
// Intercepts calls to default and purgeable malloc zones. Intercepts Core
// Foundation and Objective-C allocations.
// Has no effect on the default malloc zone if the allocator shim already
// performs that interception.
BASE_EXPORT void InterceptAllocationsMac();
// Updates all malloc zones to use their original functions.
// Also calls ClearAllMallocZonesForTesting.
BASE_EXPORT void UninterceptMallocZonesForTesting();
// Periodically checks for, and shims new malloc zones. Stops checking after 1
// minute.
BASE_EXPORT void PeriodicallyShimNewMallocZones();
// Exposed for testing.
BASE_EXPORT void ShimNewMallocZones();
BASE_EXPORT void ReplaceZoneFunctions(ChromeMallocZone* zone,
const MallocZoneFunctions* functions);
} // namespace allocator
} // namespace base
#endif // BASE_ALLOCATOR_ALLOCATOR_INTERCEPTION_MAC_H_

View file

@ -0,0 +1,568 @@
// Copyright 2017 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// This file contains all the logic necessary to intercept allocations on
// macOS. "malloc zones" are an abstraction that allows the process to intercept
// all malloc-related functions. There is no good mechanism [short of
// interposition] to determine new malloc zones are added, so there's no clean
// mechanism to intercept all malloc zones. This file contains logic to
// intercept the default and purgeable zones, which always exist. A cursory
// review of Chrome seems to imply that non-default zones are almost never used.
//
// This file also contains logic to intercept Core Foundation and Objective-C
// allocations. The implementations forward to the default malloc zone, so the
// only reason to intercept these calls is to re-label OOM crashes with slightly
// more details.
#include "base/allocator/allocator_interception_mac.h"
#include <CoreFoundation/CoreFoundation.h>
#import <Foundation/Foundation.h>
#include <errno.h>
#include <mach/mach.h>
#include <mach/mach_vm.h>
#import <objc/runtime.h>
#include <stddef.h>
#include <new>
#include "base/allocator/buildflags.h"
#include "base/allocator/malloc_zone_functions_mac.h"
#include "base/bind.h"
#include "base/bits.h"
#include "base/logging.h"
#include "base/mac/mac_util.h"
#include "base/mac/mach_logging.h"
#include "base/process/memory.h"
#include "base/threading/sequenced_task_runner_handle.h"
#include "build/build_config.h"
#include "third_party/apple_apsl/CFBase.h"
namespace base {
namespace allocator {
bool g_replaced_default_zone = false;
namespace {
bool g_oom_killer_enabled;
// Starting with Mac OS X 10.7, the zone allocators set up by the system are
// read-only, to prevent them from being overwritten in an attack. However,
// blindly unprotecting and reprotecting the zone allocators fails with
// GuardMalloc because GuardMalloc sets up its zone allocator using a block of
// memory in its bss. Explicit saving/restoring of the protection is required.
//
// This function takes a pointer to a malloc zone, de-protects it if necessary,
// and returns (in the out parameters) a region of memory (if any) to be
// re-protected when modifications are complete. This approach assumes that
// there is no contention for the protection of this memory.
void DeprotectMallocZone(ChromeMallocZone* default_zone,
mach_vm_address_t* reprotection_start,
mach_vm_size_t* reprotection_length,
vm_prot_t* reprotection_value) {
mach_port_t unused;
*reprotection_start = reinterpret_cast<mach_vm_address_t>(default_zone);
struct vm_region_basic_info_64 info;
mach_msg_type_number_t count = VM_REGION_BASIC_INFO_COUNT_64;
kern_return_t result = mach_vm_region(
mach_task_self(), reprotection_start, reprotection_length,
VM_REGION_BASIC_INFO_64, reinterpret_cast<vm_region_info_t>(&info),
&count, &unused);
MACH_CHECK(result == KERN_SUCCESS, result) << "mach_vm_region";
// The kernel always returns a null object for VM_REGION_BASIC_INFO_64, but
// balance it with a deallocate in case this ever changes. See 10.9.2
// xnu-2422.90.20/osfmk/vm/vm_map.c vm_map_region.
mach_port_deallocate(mach_task_self(), unused);
// Does the region fully enclose the zone pointers? Possibly unwarranted
// simplification used: using the size of a full version 8 malloc zone rather
// than the actual smaller size if the passed-in zone is not version 8.
CHECK(*reprotection_start <=
reinterpret_cast<mach_vm_address_t>(default_zone));
mach_vm_size_t zone_offset =
reinterpret_cast<mach_vm_size_t>(default_zone) -
reinterpret_cast<mach_vm_size_t>(*reprotection_start);
CHECK(zone_offset + sizeof(ChromeMallocZone) <= *reprotection_length);
if (info.protection & VM_PROT_WRITE) {
// No change needed; the zone is already writable.
*reprotection_start = 0;
*reprotection_length = 0;
*reprotection_value = VM_PROT_NONE;
} else {
*reprotection_value = info.protection;
result = mach_vm_protect(mach_task_self(), *reprotection_start,
*reprotection_length, false,
info.protection | VM_PROT_WRITE);
MACH_CHECK(result == KERN_SUCCESS, result) << "mach_vm_protect";
}
}
#if !defined(ADDRESS_SANITIZER)
MallocZoneFunctions g_old_zone;
MallocZoneFunctions g_old_purgeable_zone;
void* oom_killer_malloc(struct _malloc_zone_t* zone, size_t size) {
void* result = g_old_zone.malloc(zone, size);
if (!result && size)
TerminateBecauseOutOfMemory(size);
return result;
}
void* oom_killer_calloc(struct _malloc_zone_t* zone,
size_t num_items,
size_t size) {
void* result = g_old_zone.calloc(zone, num_items, size);
if (!result && num_items && size)
TerminateBecauseOutOfMemory(num_items * size);
return result;
}
void* oom_killer_valloc(struct _malloc_zone_t* zone, size_t size) {
void* result = g_old_zone.valloc(zone, size);
if (!result && size)
TerminateBecauseOutOfMemory(size);
return result;
}
void oom_killer_free(struct _malloc_zone_t* zone, void* ptr) {
g_old_zone.free(zone, ptr);
}
void* oom_killer_realloc(struct _malloc_zone_t* zone, void* ptr, size_t size) {
void* result = g_old_zone.realloc(zone, ptr, size);
if (!result && size)
TerminateBecauseOutOfMemory(size);
return result;
}
void* oom_killer_memalign(struct _malloc_zone_t* zone,
size_t alignment,
size_t size) {
void* result = g_old_zone.memalign(zone, alignment, size);
// Only die if posix_memalign would have returned ENOMEM, since there are
// other reasons why NULL might be returned (see
// http://opensource.apple.com/source/Libc/Libc-583/gen/malloc.c ).
if (!result && size && alignment >= sizeof(void*) &&
base::bits::IsPowerOfTwo(alignment)) {
TerminateBecauseOutOfMemory(size);
}
return result;
}
void* oom_killer_malloc_purgeable(struct _malloc_zone_t* zone, size_t size) {
void* result = g_old_purgeable_zone.malloc(zone, size);
if (!result && size)
TerminateBecauseOutOfMemory(size);
return result;
}
void* oom_killer_calloc_purgeable(struct _malloc_zone_t* zone,
size_t num_items,
size_t size) {
void* result = g_old_purgeable_zone.calloc(zone, num_items, size);
if (!result && num_items && size)
TerminateBecauseOutOfMemory(num_items * size);
return result;
}
void* oom_killer_valloc_purgeable(struct _malloc_zone_t* zone, size_t size) {
void* result = g_old_purgeable_zone.valloc(zone, size);
if (!result && size)
TerminateBecauseOutOfMemory(size);
return result;
}
void oom_killer_free_purgeable(struct _malloc_zone_t* zone, void* ptr) {
g_old_purgeable_zone.free(zone, ptr);
}
void* oom_killer_realloc_purgeable(struct _malloc_zone_t* zone,
void* ptr,
size_t size) {
void* result = g_old_purgeable_zone.realloc(zone, ptr, size);
if (!result && size)
TerminateBecauseOutOfMemory(size);
return result;
}
void* oom_killer_memalign_purgeable(struct _malloc_zone_t* zone,
size_t alignment,
size_t size) {
void* result = g_old_purgeable_zone.memalign(zone, alignment, size);
// Only die if posix_memalign would have returned ENOMEM, since there are
// other reasons why NULL might be returned (see
// http://opensource.apple.com/source/Libc/Libc-583/gen/malloc.c ).
if (!result && size && alignment >= sizeof(void*) &&
base::bits::IsPowerOfTwo(alignment)) {
TerminateBecauseOutOfMemory(size);
}
return result;
}
#endif // !defined(ADDRESS_SANITIZER)
#if !defined(ADDRESS_SANITIZER)
// === Core Foundation CFAllocators ===
bool CanGetContextForCFAllocator() {
return !base::mac::IsOSLaterThan10_15_DontCallThis();
}
CFAllocatorContext* ContextForCFAllocator(CFAllocatorRef allocator) {
ChromeCFAllocatorLions* our_allocator = const_cast<ChromeCFAllocatorLions*>(
reinterpret_cast<const ChromeCFAllocatorLions*>(allocator));
return &our_allocator->_context;
}
CFAllocatorAllocateCallBack g_old_cfallocator_system_default;
CFAllocatorAllocateCallBack g_old_cfallocator_malloc;
CFAllocatorAllocateCallBack g_old_cfallocator_malloc_zone;
void* oom_killer_cfallocator_system_default(CFIndex alloc_size,
CFOptionFlags hint,
void* info) {
void* result = g_old_cfallocator_system_default(alloc_size, hint, info);
if (!result)
TerminateBecauseOutOfMemory(alloc_size);
return result;
}
void* oom_killer_cfallocator_malloc(CFIndex alloc_size,
CFOptionFlags hint,
void* info) {
void* result = g_old_cfallocator_malloc(alloc_size, hint, info);
if (!result)
TerminateBecauseOutOfMemory(alloc_size);
return result;
}
void* oom_killer_cfallocator_malloc_zone(CFIndex alloc_size,
CFOptionFlags hint,
void* info) {
void* result = g_old_cfallocator_malloc_zone(alloc_size, hint, info);
if (!result)
TerminateBecauseOutOfMemory(alloc_size);
return result;
}
#endif // !defined(ADDRESS_SANITIZER)
// === Cocoa NSObject allocation ===
typedef id (*allocWithZone_t)(id, SEL, NSZone*);
allocWithZone_t g_old_allocWithZone;
id oom_killer_allocWithZone(id self, SEL _cmd, NSZone* zone) {
id result = g_old_allocWithZone(self, _cmd, zone);
if (!result)
TerminateBecauseOutOfMemory(0);
return result;
}
void UninterceptMallocZoneForTesting(struct _malloc_zone_t* zone) {
ChromeMallocZone* chrome_zone = reinterpret_cast<ChromeMallocZone*>(zone);
if (!IsMallocZoneAlreadyStored(chrome_zone))
return;
MallocZoneFunctions& functions = GetFunctionsForZone(zone);
ReplaceZoneFunctions(chrome_zone, &functions);
}
} // namespace
bool UncheckedMallocMac(size_t size, void** result) {
#if defined(ADDRESS_SANITIZER)
*result = malloc(size);
#else
if (g_old_zone.malloc) {
*result = g_old_zone.malloc(malloc_default_zone(), size);
} else {
*result = malloc(size);
}
#endif // defined(ADDRESS_SANITIZER)
return *result != NULL;
}
bool UncheckedCallocMac(size_t num_items, size_t size, void** result) {
#if defined(ADDRESS_SANITIZER)
*result = calloc(num_items, size);
#else
if (g_old_zone.calloc) {
*result = g_old_zone.calloc(malloc_default_zone(), num_items, size);
} else {
*result = calloc(num_items, size);
}
#endif // defined(ADDRESS_SANITIZER)
return *result != NULL;
}
void StoreFunctionsForDefaultZone() {
ChromeMallocZone* default_zone = reinterpret_cast<ChromeMallocZone*>(
malloc_default_zone());
StoreMallocZone(default_zone);
}
void StoreFunctionsForAllZones() {
// This ensures that the default zone is always at the front of the array,
// which is important for performance.
StoreFunctionsForDefaultZone();
vm_address_t* zones;
unsigned int count;
kern_return_t kr = malloc_get_all_zones(mach_task_self(), 0, &zones, &count);
if (kr != KERN_SUCCESS)
return;
for (unsigned int i = 0; i < count; ++i) {
ChromeMallocZone* zone = reinterpret_cast<ChromeMallocZone*>(zones[i]);
StoreMallocZone(zone);
}
}
void ReplaceFunctionsForStoredZones(const MallocZoneFunctions* functions) {
// The default zone does not get returned in malloc_get_all_zones().
ChromeMallocZone* default_zone =
reinterpret_cast<ChromeMallocZone*>(malloc_default_zone());
if (DoesMallocZoneNeedReplacing(default_zone, functions)) {
ReplaceZoneFunctions(default_zone, functions);
}
vm_address_t* zones;
unsigned int count;
kern_return_t kr =
malloc_get_all_zones(mach_task_self(), nullptr, &zones, &count);
if (kr != KERN_SUCCESS)
return;
for (unsigned int i = 0; i < count; ++i) {
ChromeMallocZone* zone = reinterpret_cast<ChromeMallocZone*>(zones[i]);
if (DoesMallocZoneNeedReplacing(zone, functions)) {
ReplaceZoneFunctions(zone, functions);
}
}
g_replaced_default_zone = true;
}
void InterceptAllocationsMac() {
if (g_oom_killer_enabled)
return;
g_oom_killer_enabled = true;
// === C malloc/calloc/valloc/realloc/posix_memalign ===
// This approach is not perfect, as requests for amounts of memory larger than
// MALLOC_ABSOLUTE_MAX_SIZE (currently SIZE_T_MAX - (2 * PAGE_SIZE)) will
// still fail with a NULL rather than dying (see
// http://opensource.apple.com/source/Libc/Libc-583/gen/malloc.c for details).
// Unfortunately, it's the best we can do. Also note that this does not affect
// allocations from non-default zones.
#if !defined(ADDRESS_SANITIZER)
// Don't do anything special on OOM for the malloc zones replaced by
// AddressSanitizer, as modifying or protecting them may not work correctly.
ChromeMallocZone* default_zone =
reinterpret_cast<ChromeMallocZone*>(malloc_default_zone());
if (!IsMallocZoneAlreadyStored(default_zone)) {
StoreZoneFunctions(default_zone, &g_old_zone);
MallocZoneFunctions new_functions = {};
new_functions.malloc = oom_killer_malloc;
new_functions.calloc = oom_killer_calloc;
new_functions.valloc = oom_killer_valloc;
new_functions.free = oom_killer_free;
new_functions.realloc = oom_killer_realloc;
new_functions.memalign = oom_killer_memalign;
ReplaceZoneFunctions(default_zone, &new_functions);
g_replaced_default_zone = true;
}
ChromeMallocZone* purgeable_zone =
reinterpret_cast<ChromeMallocZone*>(malloc_default_purgeable_zone());
if (purgeable_zone && !IsMallocZoneAlreadyStored(purgeable_zone)) {
StoreZoneFunctions(purgeable_zone, &g_old_purgeable_zone);
MallocZoneFunctions new_functions = {};
new_functions.malloc = oom_killer_malloc_purgeable;
new_functions.calloc = oom_killer_calloc_purgeable;
new_functions.valloc = oom_killer_valloc_purgeable;
new_functions.free = oom_killer_free_purgeable;
new_functions.realloc = oom_killer_realloc_purgeable;
new_functions.memalign = oom_killer_memalign_purgeable;
ReplaceZoneFunctions(purgeable_zone, &new_functions);
}
#endif
// === C malloc_zone_batch_malloc ===
// batch_malloc is omitted because the default malloc zone's implementation
// only supports batch_malloc for "tiny" allocations from the free list. It
// will fail for allocations larger than "tiny", and will only allocate as
// many blocks as it's able to from the free list. These factors mean that it
// can return less than the requested memory even in a non-out-of-memory
// situation. There's no good way to detect whether a batch_malloc failure is
// due to these other factors, or due to genuine memory or address space
// exhaustion. The fact that it only allocates space from the "tiny" free list
// means that it's likely that a failure will not be due to memory exhaustion.
// Similarly, these constraints on batch_malloc mean that callers must always
// be expecting to receive less memory than was requested, even in situations
// where memory pressure is not a concern. Finally, the only public interface
// to batch_malloc is malloc_zone_batch_malloc, which is specific to the
// system's malloc implementation. It's unlikely that anyone's even heard of
// it.
#ifndef ADDRESS_SANITIZER
// === Core Foundation CFAllocators ===
// This will not catch allocation done by custom allocators, but will catch
// all allocation done by system-provided ones.
CHECK(!g_old_cfallocator_system_default && !g_old_cfallocator_malloc &&
!g_old_cfallocator_malloc_zone)
<< "Old allocators unexpectedly non-null";
bool cf_allocator_internals_known = CanGetContextForCFAllocator();
if (cf_allocator_internals_known) {
CFAllocatorContext* context =
ContextForCFAllocator(kCFAllocatorSystemDefault);
CHECK(context) << "Failed to get context for kCFAllocatorSystemDefault.";
g_old_cfallocator_system_default = context->allocate;
CHECK(g_old_cfallocator_system_default)
<< "Failed to get kCFAllocatorSystemDefault allocation function.";
context->allocate = oom_killer_cfallocator_system_default;
context = ContextForCFAllocator(kCFAllocatorMalloc);
CHECK(context) << "Failed to get context for kCFAllocatorMalloc.";
g_old_cfallocator_malloc = context->allocate;
CHECK(g_old_cfallocator_malloc)
<< "Failed to get kCFAllocatorMalloc allocation function.";
context->allocate = oom_killer_cfallocator_malloc;
context = ContextForCFAllocator(kCFAllocatorMallocZone);
CHECK(context) << "Failed to get context for kCFAllocatorMallocZone.";
g_old_cfallocator_malloc_zone = context->allocate;
CHECK(g_old_cfallocator_malloc_zone)
<< "Failed to get kCFAllocatorMallocZone allocation function.";
context->allocate = oom_killer_cfallocator_malloc_zone;
} else {
DLOG(WARNING) << "Internals of CFAllocator not known; out-of-memory "
"failures via CFAllocator will not result in termination. "
"http://crbug.com/45650";
}
#endif
// === Cocoa NSObject allocation ===
// Note that both +[NSObject new] and +[NSObject alloc] call through to
// +[NSObject allocWithZone:].
CHECK(!g_old_allocWithZone) << "Old allocator unexpectedly non-null";
Class nsobject_class = [NSObject class];
Method orig_method =
class_getClassMethod(nsobject_class, @selector(allocWithZone:));
g_old_allocWithZone =
reinterpret_cast<allocWithZone_t>(method_getImplementation(orig_method));
CHECK(g_old_allocWithZone)
<< "Failed to get allocWithZone allocation function.";
method_setImplementation(orig_method,
reinterpret_cast<IMP>(oom_killer_allocWithZone));
}
void UninterceptMallocZonesForTesting() {
UninterceptMallocZoneForTesting(malloc_default_zone());
vm_address_t* zones;
unsigned int count;
kern_return_t kr = malloc_get_all_zones(mach_task_self(), 0, &zones, &count);
CHECK(kr == KERN_SUCCESS);
for (unsigned int i = 0; i < count; ++i) {
UninterceptMallocZoneForTesting(
reinterpret_cast<struct _malloc_zone_t*>(zones[i]));
}
ClearAllMallocZonesForTesting();
}
namespace {
void ShimNewMallocZonesAndReschedule(base::Time end_time,
base::TimeDelta delay) {
ShimNewMallocZones();
if (base::Time::Now() > end_time)
return;
base::TimeDelta next_delay = delay * 2;
SequencedTaskRunnerHandle::Get()->PostDelayedTask(
FROM_HERE,
base::BindOnce(&ShimNewMallocZonesAndReschedule, end_time, next_delay),
delay);
}
} // namespace
void PeriodicallyShimNewMallocZones() {
base::Time end_time = base::Time::Now() + base::TimeDelta::FromMinutes(1);
base::TimeDelta initial_delay = base::TimeDelta::FromSeconds(1);
ShimNewMallocZonesAndReschedule(end_time, initial_delay);
}
void ShimNewMallocZones() {
StoreFunctionsForAllZones();
// Use the functions for the default zone as a template to replace those
// new zones.
ChromeMallocZone* default_zone =
reinterpret_cast<ChromeMallocZone*>(malloc_default_zone());
DCHECK(IsMallocZoneAlreadyStored(default_zone));
MallocZoneFunctions new_functions;
StoreZoneFunctions(default_zone, &new_functions);
ReplaceFunctionsForStoredZones(&new_functions);
}
void ReplaceZoneFunctions(ChromeMallocZone* zone,
const MallocZoneFunctions* functions) {
// Remove protection.
mach_vm_address_t reprotection_start = 0;
mach_vm_size_t reprotection_length = 0;
vm_prot_t reprotection_value = VM_PROT_NONE;
DeprotectMallocZone(zone, &reprotection_start, &reprotection_length,
&reprotection_value);
CHECK(functions->malloc && functions->calloc && functions->valloc &&
functions->free && functions->realloc);
zone->malloc = functions->malloc;
zone->calloc = functions->calloc;
zone->valloc = functions->valloc;
zone->free = functions->free;
zone->realloc = functions->realloc;
if (functions->batch_malloc)
zone->batch_malloc = functions->batch_malloc;
if (functions->batch_free)
zone->batch_free = functions->batch_free;
if (functions->size)
zone->size = functions->size;
if (zone->version >= 5 && functions->memalign) {
zone->memalign = functions->memalign;
}
if (zone->version >= 6 && functions->free_definite_size) {
zone->free_definite_size = functions->free_definite_size;
}
// Restore protection if it was active.
if (reprotection_start) {
kern_return_t result =
mach_vm_protect(mach_task_self(), reprotection_start,
reprotection_length, false, reprotection_value);
MACH_CHECK(result == KERN_SUCCESS, result) << "mach_vm_protect";
}
}
} // namespace allocator
} // namespace base

View file

@ -0,0 +1,386 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/allocator_shim.h"
#include <errno.h>
#include <atomic>
#include <new>
#include "base/allocator/buildflags.h"
#include "base/atomicops.h"
#include "base/bits.h"
#include "base/logging.h"
#include "base/macros.h"
#include "base/process/process_metrics.h"
#include "base/threading/platform_thread.h"
#include "build/build_config.h"
#if !defined(OS_WIN)
#include <unistd.h>
#else
#include "base/allocator/winheap_stubs_win.h"
#endif
#if defined(OS_MACOSX)
#include <malloc/malloc.h>
#include "base/allocator/allocator_interception_mac.h"
#endif
// No calls to malloc / new in this file. They would would cause re-entrancy of
// the shim, which is hard to deal with. Keep this code as simple as possible
// and don't use any external C++ object here, not even //base ones. Even if
// they are safe to use today, in future they might be refactored.
namespace {
base::subtle::AtomicWord g_chain_head =
reinterpret_cast<base::subtle::AtomicWord>(
&base::allocator::AllocatorDispatch::default_dispatch);
bool g_call_new_handler_on_malloc_failure = false;
inline size_t GetCachedPageSize() {
static size_t pagesize = 0;
if (!pagesize)
pagesize = base::GetPageSize();
return pagesize;
}
// Calls the std::new handler thread-safely. Returns true if a new_handler was
// set and called, false if no new_handler was set.
bool CallNewHandler(size_t size) {
#if defined(OS_WIN)
return base::allocator::WinCallNewHandler(size);
#else
std::new_handler nh = std::get_new_handler();
if (!nh)
return false;
(*nh)();
// Assume the new_handler will abort if it fails. Exception are disabled and
// we don't support the case of a new_handler throwing std::bad_balloc.
return true;
#endif
}
inline const base::allocator::AllocatorDispatch* GetChainHead() {
// TODO(primiano): Just use NoBarrier_Load once crbug.com/593344 is fixed.
// Unfortunately due to that bug NoBarrier_Load() is mistakenly fully
// barriered on Linux+Clang, and that causes visible perf regressons.
return reinterpret_cast<const base::allocator::AllocatorDispatch*>(
#if defined(OS_LINUX) && defined(__clang__)
*static_cast<const volatile base::subtle::AtomicWord*>(&g_chain_head)
#else
base::subtle::NoBarrier_Load(&g_chain_head)
#endif
);
}
} // namespace
namespace base {
namespace allocator {
void SetCallNewHandlerOnMallocFailure(bool value) {
g_call_new_handler_on_malloc_failure = value;
}
void* UncheckedAlloc(size_t size) {
const allocator::AllocatorDispatch* const chain_head = GetChainHead();
return chain_head->alloc_function(chain_head, size, nullptr);
}
void InsertAllocatorDispatch(AllocatorDispatch* dispatch) {
// Loop in case of (an unlikely) race on setting the list head.
size_t kMaxRetries = 7;
for (size_t i = 0; i < kMaxRetries; ++i) {
const AllocatorDispatch* chain_head = GetChainHead();
dispatch->next = chain_head;
// This function guarantees to be thread-safe w.r.t. concurrent
// insertions. It also has to guarantee that all the threads always
// see a consistent chain, hence the atomic_thread_fence() below.
// InsertAllocatorDispatch() is NOT a fastpath, as opposite to malloc(), so
// we don't really want this to be a release-store with a corresponding
// acquire-load during malloc().
std::atomic_thread_fence(std::memory_order_seq_cst);
subtle::AtomicWord old_value =
reinterpret_cast<subtle::AtomicWord>(chain_head);
// Set the chain head to the new dispatch atomically. If we lose the race,
// the comparison will fail, and the new head of chain will be returned.
if (subtle::NoBarrier_CompareAndSwap(
&g_chain_head, old_value,
reinterpret_cast<subtle::AtomicWord>(dispatch)) == old_value) {
// Success.
return;
}
}
CHECK(false); // Too many retries, this shouldn't happen.
}
void RemoveAllocatorDispatchForTesting(AllocatorDispatch* dispatch) {
DCHECK_EQ(GetChainHead(), dispatch);
subtle::NoBarrier_Store(&g_chain_head,
reinterpret_cast<subtle::AtomicWord>(dispatch->next));
}
} // namespace allocator
} // namespace base
// The Shim* functions below are the entry-points into the shim-layer and
// are supposed to be invoked by the allocator_shim_override_*
// headers to route the malloc / new symbols through the shim layer.
// They are defined as ALWAYS_INLINE in order to remove a level of indirection
// between the system-defined entry points and the shim implementations.
extern "C" {
// The general pattern for allocations is:
// - Try to allocate, if succeded return the pointer.
// - If the allocation failed:
// - Call the std::new_handler if it was a C++ allocation.
// - Call the std::new_handler if it was a malloc() (or calloc() or similar)
// AND SetCallNewHandlerOnMallocFailure(true).
// - If the std::new_handler is NOT set just return nullptr.
// - If the std::new_handler is set:
// - Assume it will abort() if it fails (very likely the new_handler will
// just suicide priting a message).
// - Assume it did succeed if it returns, in which case reattempt the alloc.
ALWAYS_INLINE void* ShimCppNew(size_t size) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
void* ptr;
do {
void* context = nullptr;
#if defined(OS_MACOSX)
context = malloc_default_zone();
#endif
ptr = chain_head->alloc_function(chain_head, size, context);
} while (!ptr && CallNewHandler(size));
return ptr;
}
ALWAYS_INLINE void* ShimCppAlignedNew(size_t size, size_t alignment) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
void* ptr;
do {
void* context = nullptr;
#if defined(OS_MACOSX)
context = malloc_default_zone();
#endif
ptr = chain_head->alloc_aligned_function(chain_head, alignment, size,
context);
} while (!ptr && CallNewHandler(size));
return ptr;
}
ALWAYS_INLINE void ShimCppDelete(void* address) {
void* context = nullptr;
#if defined(OS_MACOSX)
context = malloc_default_zone();
#endif
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
return chain_head->free_function(chain_head, address, context);
}
ALWAYS_INLINE void* ShimMalloc(size_t size, void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
void* ptr;
do {
ptr = chain_head->alloc_function(chain_head, size, context);
} while (!ptr && g_call_new_handler_on_malloc_failure &&
CallNewHandler(size));
return ptr;
}
ALWAYS_INLINE void* ShimCalloc(size_t n, size_t size, void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
void* ptr;
do {
ptr = chain_head->alloc_zero_initialized_function(chain_head, n, size,
context);
} while (!ptr && g_call_new_handler_on_malloc_failure &&
CallNewHandler(size));
return ptr;
}
ALWAYS_INLINE void* ShimRealloc(void* address, size_t size, void* context) {
// realloc(size == 0) means free() and might return a nullptr. We should
// not call the std::new_handler in that case, though.
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
void* ptr;
do {
ptr = chain_head->realloc_function(chain_head, address, size, context);
} while (!ptr && size && g_call_new_handler_on_malloc_failure &&
CallNewHandler(size));
return ptr;
}
ALWAYS_INLINE void* ShimMemalign(size_t alignment, size_t size, void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
void* ptr;
do {
ptr = chain_head->alloc_aligned_function(chain_head, alignment, size,
context);
} while (!ptr && g_call_new_handler_on_malloc_failure &&
CallNewHandler(size));
return ptr;
}
ALWAYS_INLINE int ShimPosixMemalign(void** res, size_t alignment, size_t size) {
// posix_memalign is supposed to check the arguments. See tc_posix_memalign()
// in tc_malloc.cc.
if (((alignment % sizeof(void*)) != 0) ||
!base::bits::IsPowerOfTwo(alignment)) {
return EINVAL;
}
void* ptr = ShimMemalign(alignment, size, nullptr);
*res = ptr;
return ptr ? 0 : ENOMEM;
}
ALWAYS_INLINE void* ShimValloc(size_t size, void* context) {
return ShimMemalign(GetCachedPageSize(), size, context);
}
ALWAYS_INLINE void* ShimPvalloc(size_t size) {
// pvalloc(0) should allocate one page, according to its man page.
if (size == 0) {
size = GetCachedPageSize();
} else {
size = (size + GetCachedPageSize() - 1) & ~(GetCachedPageSize() - 1);
}
// The third argument is nullptr because pvalloc is glibc only and does not
// exist on OSX/BSD systems.
return ShimMemalign(GetCachedPageSize(), size, nullptr);
}
ALWAYS_INLINE void ShimFree(void* address, void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
return chain_head->free_function(chain_head, address, context);
}
ALWAYS_INLINE size_t ShimGetSizeEstimate(const void* address, void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
return chain_head->get_size_estimate_function(
chain_head, const_cast<void*>(address), context);
}
ALWAYS_INLINE unsigned ShimBatchMalloc(size_t size,
void** results,
unsigned num_requested,
void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
return chain_head->batch_malloc_function(chain_head, size, results,
num_requested, context);
}
ALWAYS_INLINE void ShimBatchFree(void** to_be_freed,
unsigned num_to_be_freed,
void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
return chain_head->batch_free_function(chain_head, to_be_freed,
num_to_be_freed, context);
}
ALWAYS_INLINE void ShimFreeDefiniteSize(void* ptr, size_t size, void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
return chain_head->free_definite_size_function(chain_head, ptr, size,
context);
}
ALWAYS_INLINE void* ShimAlignedMalloc(size_t size,
size_t alignment,
void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
void* ptr = nullptr;
do {
ptr = chain_head->aligned_malloc_function(chain_head, size, alignment,
context);
} while (!ptr && g_call_new_handler_on_malloc_failure &&
CallNewHandler(size));
return ptr;
}
ALWAYS_INLINE void* ShimAlignedRealloc(void* address,
size_t size,
size_t alignment,
void* context) {
// _aligned_realloc(size == 0) means _aligned_free() and might return a
// nullptr. We should not call the std::new_handler in that case, though.
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
void* ptr = nullptr;
do {
ptr = chain_head->aligned_realloc_function(chain_head, address, size,
alignment, context);
} while (!ptr && size && g_call_new_handler_on_malloc_failure &&
CallNewHandler(size));
return ptr;
}
ALWAYS_INLINE void ShimAlignedFree(void* address, void* context) {
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
return chain_head->aligned_free_function(chain_head, address, context);
}
} // extern "C"
#if !defined(OS_WIN) && !defined(OS_MACOSX)
// Cpp symbols (new / delete) should always be routed through the shim layer
// except on Windows and macOS where the malloc intercept is deep enough that it
// also catches the cpp calls.
#include "base/allocator/allocator_shim_override_cpp_symbols.h"
#endif
#if defined(OS_ANDROID)
// Android does not support symbol interposition. The way malloc symbols are
// intercepted on Android is by using link-time -wrap flags.
#include "base/allocator/allocator_shim_override_linker_wrapped_symbols.h"
#elif defined(OS_WIN)
// On Windows we use plain link-time overriding of the CRT symbols.
#include "base/allocator/allocator_shim_override_ucrt_symbols_win.h"
#elif defined(OS_MACOSX)
#include "base/allocator/allocator_shim_default_dispatch_to_mac_zoned_malloc.h"
#include "base/allocator/allocator_shim_override_mac_symbols.h"
#else
#include "base/allocator/allocator_shim_override_libc_symbols.h"
#endif
// In the case of tcmalloc we also want to plumb into the glibc hooks
// to avoid that allocations made in glibc itself (e.g., strdup()) get
// accidentally performed on the glibc heap instead of the tcmalloc one.
#if BUILDFLAG(USE_TCMALLOC)
#include "base/allocator/allocator_shim_override_glibc_weak_symbols.h"
#endif
#if defined(OS_MACOSX)
namespace base {
namespace allocator {
void InitializeAllocatorShim() {
// Prepares the default dispatch. After the intercepted malloc calls have
// traversed the shim this will route them to the default malloc zone.
InitializeDefaultDispatchToMacAllocator();
MallocZoneFunctions functions = MallocZoneFunctionsToReplaceDefault();
// This replaces the default malloc zone, causing calls to malloc & friends
// from the codebase to be routed to ShimMalloc() above.
base::allocator::ReplaceFunctionsForStoredZones(&functions);
}
} // namespace allocator
} // namespace base
#endif
// Cross-checks.
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
#error The allocator shim should not be compiled when building for memory tools.
#endif
#if (defined(__GNUC__) && defined(__EXCEPTIONS)) || \
(defined(_MSC_VER) && defined(_CPPUNWIND))
#error This code cannot be used when exceptions are turned on.
#endif

View file

@ -0,0 +1,152 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_ALLOCATOR_SHIM_H_
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_H_
#include <stddef.h>
#include "base/base_export.h"
#include "build/build_config.h"
namespace base {
namespace allocator {
// Allocator Shim API. Allows to:
// - Configure the behavior of the allocator (what to do on OOM failures).
// - Install new hooks (AllocatorDispatch) in the allocator chain.
// When this shim layer is enabled, the route of an allocation is as-follows:
//
// [allocator_shim_override_*.h] Intercept malloc() / operator new calls:
// The override_* headers define the symbols required to intercept calls to
// malloc() and operator new (if not overridden by specific C++ classes).
//
// [allocator_shim.cc] Routing allocation calls to the shim:
// The headers above route the calls to the internal ShimMalloc(), ShimFree(),
// ShimCppNew() etc. methods defined in allocator_shim.cc.
// These methods will: (1) forward the allocation call to the front of the
// AllocatorDispatch chain. (2) perform security hardenings (e.g., might
// call std::new_handler on OOM failure).
//
// [allocator_shim_default_dispatch_to_*.cc] The AllocatorDispatch chain:
// It is a singly linked list where each element is a struct with function
// pointers (|malloc_function|, |free_function|, etc). Normally the chain
// consists of a single AllocatorDispatch element, herein called
// the "default dispatch", which is statically defined at build time and
// ultimately routes the calls to the actual allocator defined by the build
// config (tcmalloc, glibc, ...).
//
// It is possible to dynamically insert further AllocatorDispatch stages
// to the front of the chain, for debugging / profiling purposes.
//
// All the functions must be thread safe. The shim does not enforce any
// serialization. This is to route to thread-aware allocators (e.g, tcmalloc)
// wihout introducing unnecessary perf hits.
struct AllocatorDispatch {
using AllocFn = void*(const AllocatorDispatch* self,
size_t size,
void* context);
using AllocZeroInitializedFn = void*(const AllocatorDispatch* self,
size_t n,
size_t size,
void* context);
using AllocAlignedFn = void*(const AllocatorDispatch* self,
size_t alignment,
size_t size,
void* context);
using ReallocFn = void*(const AllocatorDispatch* self,
void* address,
size_t size,
void* context);
using FreeFn = void(const AllocatorDispatch* self,
void* address,
void* context);
// Returns the best available estimate for the actual amount of memory
// consumed by the allocation |address|. If possible, this should include
// heap overhead or at least a decent estimate of the full cost of the
// allocation. If no good estimate is possible, returns zero.
using GetSizeEstimateFn = size_t(const AllocatorDispatch* self,
void* address,
void* context);
using BatchMallocFn = unsigned(const AllocatorDispatch* self,
size_t size,
void** results,
unsigned num_requested,
void* context);
using BatchFreeFn = void(const AllocatorDispatch* self,
void** to_be_freed,
unsigned num_to_be_freed,
void* context);
using FreeDefiniteSizeFn = void(const AllocatorDispatch* self,
void* ptr,
size_t size,
void* context);
using AlignedMallocFn = void*(const AllocatorDispatch* self,
size_t size,
size_t alignment,
void* context);
using AlignedReallocFn = void*(const AllocatorDispatch* self,
void* address,
size_t size,
size_t alignment,
void* context);
using AlignedFreeFn = void(const AllocatorDispatch* self,
void* address,
void* context);
AllocFn* const alloc_function;
AllocZeroInitializedFn* const alloc_zero_initialized_function;
AllocAlignedFn* const alloc_aligned_function;
ReallocFn* const realloc_function;
FreeFn* const free_function;
GetSizeEstimateFn* const get_size_estimate_function;
// batch_malloc, batch_free, and free_definite_size are specific to the OSX
// and iOS allocators.
BatchMallocFn* const batch_malloc_function;
BatchFreeFn* const batch_free_function;
FreeDefiniteSizeFn* const free_definite_size_function;
// _aligned_malloc, _aligned_realloc, and _aligned_free are specific to the
// Windows allocator.
AlignedMallocFn* const aligned_malloc_function;
AlignedReallocFn* const aligned_realloc_function;
AlignedFreeFn* const aligned_free_function;
const AllocatorDispatch* next;
// |default_dispatch| is statically defined by one (and only one) of the
// allocator_shim_default_dispatch_to_*.cc files, depending on the build
// configuration.
static const AllocatorDispatch default_dispatch;
};
// When true makes malloc behave like new, w.r.t calling the new_handler if
// the allocation fails (see set_new_mode() in Windows).
BASE_EXPORT void SetCallNewHandlerOnMallocFailure(bool value);
// Allocates |size| bytes or returns nullptr. It does NOT call the new_handler,
// regardless of SetCallNewHandlerOnMallocFailure().
BASE_EXPORT void* UncheckedAlloc(size_t size);
// Inserts |dispatch| in front of the allocator chain. This method is
// thread-safe w.r.t concurrent invocations of InsertAllocatorDispatch().
// The callers have responsibility for inserting a single dispatch no more
// than once.
BASE_EXPORT void InsertAllocatorDispatch(AllocatorDispatch* dispatch);
// Test-only. Rationale: (1) lack of use cases; (2) dealing safely with a
// removal of arbitrary elements from a singly linked list would require a lock
// in malloc(), which we really don't want.
BASE_EXPORT void RemoveAllocatorDispatchForTesting(AllocatorDispatch* dispatch);
#if defined(OS_MACOSX)
// On macOS, the allocator shim needs to be turned on during runtime.
BASE_EXPORT void InitializeAllocatorShim();
#endif // defined(OS_MACOSX)
} // namespace allocator
} // namespace base
#endif // BASE_ALLOCATOR_ALLOCATOR_SHIM_H_

View file

@ -0,0 +1,88 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/allocator_shim.h"
#include "base/compiler_specific.h"
#include <dlfcn.h>
#include <malloc.h>
// This translation unit defines a default dispatch for the allocator shim which
// routes allocations to libc functions.
// The code here is strongly inspired from tcmalloc's libc_override_glibc.h.
extern "C" {
void* __libc_malloc(size_t size);
void* __libc_calloc(size_t n, size_t size);
void* __libc_realloc(void* address, size_t size);
void* __libc_memalign(size_t alignment, size_t size);
void __libc_free(void* ptr);
} // extern "C"
namespace {
using base::allocator::AllocatorDispatch;
void* GlibcMalloc(const AllocatorDispatch*, size_t size, void* context) {
return __libc_malloc(size);
}
void* GlibcCalloc(const AllocatorDispatch*,
size_t n,
size_t size,
void* context) {
return __libc_calloc(n, size);
}
void* GlibcRealloc(const AllocatorDispatch*,
void* address,
size_t size,
void* context) {
return __libc_realloc(address, size);
}
void* GlibcMemalign(const AllocatorDispatch*,
size_t alignment,
size_t size,
void* context) {
return __libc_memalign(alignment, size);
}
void GlibcFree(const AllocatorDispatch*, void* address, void* context) {
__libc_free(address);
}
NO_SANITIZE("cfi-icall")
size_t GlibcGetSizeEstimate(const AllocatorDispatch*,
void* address,
void* context) {
// glibc does not expose an alias to resolve malloc_usable_size. Dynamically
// resolve it instead. This should be safe because glibc (and hence dlfcn)
// does not use malloc_size internally and so there should not be a risk of
// recursion.
using MallocUsableSizeFunction = decltype(malloc_usable_size)*;
static MallocUsableSizeFunction fn_ptr =
reinterpret_cast<MallocUsableSizeFunction>(
dlsym(RTLD_NEXT, "malloc_usable_size"));
return fn_ptr(address);
}
} // namespace
const AllocatorDispatch AllocatorDispatch::default_dispatch = {
&GlibcMalloc, /* alloc_function */
&GlibcCalloc, /* alloc_zero_initialized_function */
&GlibcMemalign, /* alloc_aligned_function */
&GlibcRealloc, /* realloc_function */
&GlibcFree, /* free_function */
&GlibcGetSizeEstimate, /* get_size_estimate_function */
nullptr, /* batch_malloc_function */
nullptr, /* batch_free_function */
nullptr, /* free_definite_size_function */
nullptr, /* aligned_malloc_function */
nullptr, /* aligned_realloc_function */
nullptr, /* aligned_free_function */
nullptr, /* next */
};

View file

@ -0,0 +1,76 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include <malloc.h>
#include "base/allocator/allocator_shim.h"
#include "build/build_config.h"
// This translation unit defines a default dispatch for the allocator shim which
// routes allocations to the original libc functions when using the link-time
// -Wl,-wrap,malloc approach (see README.md).
// The __real_X functions here are special symbols that the linker will relocate
// against the real "X" undefined symbol, so that __real_malloc becomes the
// equivalent of what an undefined malloc symbol reference would have been.
// This is the counterpart of allocator_shim_override_linker_wrapped_symbols.h,
// which routes the __wrap_X functions into the shim.
extern "C" {
void* __real_malloc(size_t);
void* __real_calloc(size_t, size_t);
void* __real_realloc(void*, size_t);
void* __real_memalign(size_t, size_t);
void* __real_free(void*);
} // extern "C"
namespace {
using base::allocator::AllocatorDispatch;
void* RealMalloc(const AllocatorDispatch*, size_t size, void* context) {
return __real_malloc(size);
}
void* RealCalloc(const AllocatorDispatch*,
size_t n,
size_t size,
void* context) {
return __real_calloc(n, size);
}
void* RealRealloc(const AllocatorDispatch*,
void* address,
size_t size,
void* context) {
return __real_realloc(address, size);
}
void* RealMemalign(const AllocatorDispatch*,
size_t alignment,
size_t size,
void* context) {
return __real_memalign(alignment, size);
}
void RealFree(const AllocatorDispatch*, void* address, void* context) {
__real_free(address);
}
} // namespace
const AllocatorDispatch AllocatorDispatch::default_dispatch = {
&RealMalloc, /* alloc_function */
&RealCalloc, /* alloc_zero_initialized_function */
&RealMemalign, /* alloc_aligned_function */
&RealRealloc, /* realloc_function */
&RealFree, /* free_function */
nullptr, /* get_size_estimate_function */
nullptr, /* batch_malloc_function */
nullptr, /* batch_free_function */
nullptr, /* free_definite_size_function */
nullptr, /* aligned_malloc_function */
nullptr, /* aligned_realloc_function */
nullptr, /* aligned_free_function */
nullptr, /* next */
};

View file

@ -0,0 +1,112 @@
// Copyright 2017 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/allocator_shim_default_dispatch_to_mac_zoned_malloc.h"
#include <utility>
#include "base/allocator/allocator_interception_mac.h"
#include "base/allocator/allocator_shim.h"
#include "base/allocator/malloc_zone_functions_mac.h"
namespace base {
namespace allocator {
namespace {
void* MallocImpl(const AllocatorDispatch*, size_t size, void* context) {
MallocZoneFunctions& functions = GetFunctionsForZone(context);
return functions.malloc(reinterpret_cast<struct _malloc_zone_t*>(context),
size);
}
void* CallocImpl(const AllocatorDispatch*,
size_t n,
size_t size,
void* context) {
MallocZoneFunctions& functions = GetFunctionsForZone(context);
return functions.calloc(reinterpret_cast<struct _malloc_zone_t*>(context), n,
size);
}
void* MemalignImpl(const AllocatorDispatch*,
size_t alignment,
size_t size,
void* context) {
MallocZoneFunctions& functions = GetFunctionsForZone(context);
return functions.memalign(reinterpret_cast<struct _malloc_zone_t*>(context),
alignment, size);
}
void* ReallocImpl(const AllocatorDispatch*,
void* ptr,
size_t size,
void* context) {
MallocZoneFunctions& functions = GetFunctionsForZone(context);
return functions.realloc(reinterpret_cast<struct _malloc_zone_t*>(context),
ptr, size);
}
void FreeImpl(const AllocatorDispatch*, void* ptr, void* context) {
MallocZoneFunctions& functions = GetFunctionsForZone(context);
functions.free(reinterpret_cast<struct _malloc_zone_t*>(context), ptr);
}
size_t GetSizeEstimateImpl(const AllocatorDispatch*, void* ptr, void* context) {
MallocZoneFunctions& functions = GetFunctionsForZone(context);
return functions.size(reinterpret_cast<struct _malloc_zone_t*>(context), ptr);
}
unsigned BatchMallocImpl(const AllocatorDispatch* self,
size_t size,
void** results,
unsigned num_requested,
void* context) {
MallocZoneFunctions& functions = GetFunctionsForZone(context);
return functions.batch_malloc(
reinterpret_cast<struct _malloc_zone_t*>(context), size, results,
num_requested);
}
void BatchFreeImpl(const AllocatorDispatch* self,
void** to_be_freed,
unsigned num_to_be_freed,
void* context) {
MallocZoneFunctions& functions = GetFunctionsForZone(context);
functions.batch_free(reinterpret_cast<struct _malloc_zone_t*>(context),
to_be_freed, num_to_be_freed);
}
void FreeDefiniteSizeImpl(const AllocatorDispatch* self,
void* ptr,
size_t size,
void* context) {
MallocZoneFunctions& functions = GetFunctionsForZone(context);
functions.free_definite_size(
reinterpret_cast<struct _malloc_zone_t*>(context), ptr, size);
}
} // namespace
void InitializeDefaultDispatchToMacAllocator() {
StoreFunctionsForAllZones();
}
const AllocatorDispatch AllocatorDispatch::default_dispatch = {
&MallocImpl, /* alloc_function */
&CallocImpl, /* alloc_zero_initialized_function */
&MemalignImpl, /* alloc_aligned_function */
&ReallocImpl, /* realloc_function */
&FreeImpl, /* free_function */
&GetSizeEstimateImpl, /* get_size_estimate_function */
&BatchMallocImpl, /* batch_malloc_function */
&BatchFreeImpl, /* batch_free_function */
&FreeDefiniteSizeImpl, /* free_definite_size_function */
nullptr, /* aligned_malloc_function */
nullptr, /* aligned_realloc_function */
nullptr, /* aligned_free_function */
nullptr, /* next */
};
} // namespace allocator
} // namespace base

View file

@ -0,0 +1,19 @@
// Copyright 2017 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_ALLOCATOR_SHIM_DEFAULT_DISPATCH_TO_ZONED_MALLOC_H_
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_DEFAULT_DISPATCH_TO_ZONED_MALLOC_H_
namespace base {
namespace allocator {
// This initializes AllocatorDispatch::default_dispatch by saving pointers to
// the functions in the current default malloc zone. This must be called before
// the default malloc zone is changed to have its intended effect.
void InitializeDefaultDispatchToMacAllocator();
} // namespace allocator
} // namespace base
#endif // BASE_ALLOCATOR_ALLOCATOR_SHIM_DEFAULT_DISPATCH_TO_ZONED_MALLOC_H_

View file

@ -0,0 +1,84 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/allocator_shim.h"
#include "base/allocator/allocator_shim_internals.h"
#include "third_party/tcmalloc/chromium/src/config.h"
#include "third_party/tcmalloc/chromium/src/gperftools/tcmalloc.h"
namespace {
using base::allocator::AllocatorDispatch;
void* TCMalloc(const AllocatorDispatch*, size_t size, void* context) {
return tc_malloc(size);
}
void* TCCalloc(const AllocatorDispatch*, size_t n, size_t size, void* context) {
return tc_calloc(n, size);
}
void* TCMemalign(const AllocatorDispatch*,
size_t alignment,
size_t size,
void* context) {
return tc_memalign(alignment, size);
}
void* TCRealloc(const AllocatorDispatch*,
void* address,
size_t size,
void* context) {
return tc_realloc(address, size);
}
void TCFree(const AllocatorDispatch*, void* address, void* context) {
tc_free(address);
}
size_t TCGetSizeEstimate(const AllocatorDispatch*,
void* address,
void* context) {
return tc_malloc_size(address);
}
} // namespace
const AllocatorDispatch AllocatorDispatch::default_dispatch = {
&TCMalloc, /* alloc_function */
&TCCalloc, /* alloc_zero_initialized_function */
&TCMemalign, /* alloc_aligned_function */
&TCRealloc, /* realloc_function */
&TCFree, /* free_function */
&TCGetSizeEstimate, /* get_size_estimate_function */
nullptr, /* batch_malloc_function */
nullptr, /* batch_free_function */
nullptr, /* free_definite_size_function */
nullptr, /* aligned_malloc_function */
nullptr, /* aligned_realloc_function */
nullptr, /* aligned_free_function */
nullptr, /* next */
};
// In the case of tcmalloc we have also to route the diagnostic symbols,
// which are not part of the unified shim layer, to tcmalloc for consistency.
extern "C" {
SHIM_ALWAYS_EXPORT void malloc_stats(void) __THROW {
return tc_malloc_stats();
}
SHIM_ALWAYS_EXPORT int mallopt(int cmd, int value) __THROW {
return tc_mallopt(cmd, value);
}
#ifdef HAVE_STRUCT_MALLINFO
SHIM_ALWAYS_EXPORT struct mallinfo mallinfo(void) __THROW {
return tc_mallinfo();
}
#endif
} // extern "C"

View file

@ -0,0 +1,103 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/allocator_shim.h"
#include "base/allocator/winheap_stubs_win.h"
#include "base/logging.h"
namespace {
using base::allocator::AllocatorDispatch;
void* DefaultWinHeapMallocImpl(const AllocatorDispatch*,
size_t size,
void* context) {
return base::allocator::WinHeapMalloc(size);
}
void* DefaultWinHeapCallocImpl(const AllocatorDispatch* self,
size_t n,
size_t elem_size,
void* context) {
// Overflow check.
const size_t size = n * elem_size;
if (elem_size != 0 && size / elem_size != n)
return nullptr;
void* result = DefaultWinHeapMallocImpl(self, size, context);
if (result) {
memset(result, 0, size);
}
return result;
}
void* DefaultWinHeapMemalignImpl(const AllocatorDispatch* self,
size_t alignment,
size_t size,
void* context) {
CHECK(false) << "The windows heap does not support memalign.";
return nullptr;
}
void* DefaultWinHeapReallocImpl(const AllocatorDispatch* self,
void* address,
size_t size,
void* context) {
return base::allocator::WinHeapRealloc(address, size);
}
void DefaultWinHeapFreeImpl(const AllocatorDispatch*,
void* address,
void* context) {
base::allocator::WinHeapFree(address);
}
size_t DefaultWinHeapGetSizeEstimateImpl(const AllocatorDispatch*,
void* address,
void* context) {
return base::allocator::WinHeapGetSizeEstimate(address);
}
void* DefaultWinHeapAlignedMallocImpl(const AllocatorDispatch*,
size_t size,
size_t alignment,
void* context) {
return base::allocator::WinHeapAlignedMalloc(size, alignment);
}
void* DefaultWinHeapAlignedReallocImpl(const AllocatorDispatch*,
void* ptr,
size_t size,
size_t alignment,
void* context) {
return base::allocator::WinHeapAlignedRealloc(ptr, size, alignment);
}
void DefaultWinHeapAlignedFreeImpl(const AllocatorDispatch*,
void* ptr,
void* context) {
base::allocator::WinHeapAlignedFree(ptr);
}
} // namespace
// Guarantee that default_dispatch is compile-time initialized to avoid using
// it before initialization (allocations before main in release builds with
// optimizations disabled).
constexpr AllocatorDispatch AllocatorDispatch::default_dispatch = {
&DefaultWinHeapMallocImpl,
&DefaultWinHeapCallocImpl,
&DefaultWinHeapMemalignImpl,
&DefaultWinHeapReallocImpl,
&DefaultWinHeapFreeImpl,
&DefaultWinHeapGetSizeEstimateImpl,
nullptr, /* batch_malloc_function */
nullptr, /* batch_free_function */
nullptr, /* free_definite_size_function */
&DefaultWinHeapAlignedMallocImpl,
&DefaultWinHeapAlignedReallocImpl,
&DefaultWinHeapAlignedFreeImpl,
nullptr, /* next */
};

View file

@ -0,0 +1,44 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_ALLOCATOR_SHIM_INTERNALS_H_
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_INTERNALS_H_
#if defined(__GNUC__)
#include <sys/cdefs.h> // for __THROW
#ifndef __THROW // Not a glibc system
#ifdef _NOEXCEPT // LLVM libc++ uses noexcept instead
#define __THROW _NOEXCEPT
#else
#define __THROW
#endif // !_NOEXCEPT
#endif
// Shim layer symbols need to be ALWAYS exported, regardless of component build.
//
// If an exported symbol is linked into a DSO, it may be preempted by a
// definition in the main executable. If this happens to an allocator symbol, it
// will mean that the DSO will use the main executable's allocator. This is
// normally relatively harmless -- regular allocations should all use the same
// allocator, but if the DSO tries to hook the allocator it will not see any
// allocations.
//
// However, if LLVM LTO is enabled, the compiler may inline the shim layer
// symbols into callers. The end result is that allocator calls in DSOs may use
// either the main executable's allocator or the DSO's allocator, depending on
// whether the call was inlined. This is arguably a bug in LLVM caused by its
// somewhat irregular handling of symbol interposition (see llvm.org/PR23501).
// To work around the bug we use noinline to prevent the symbols from being
// inlined.
//
// In the long run we probably want to avoid linking the allocator bits into
// DSOs altogether. This will save a little space and stop giving DSOs the false
// impression that they can hook the allocator.
#define SHIM_ALWAYS_EXPORT __attribute__((visibility("default"), noinline))
#endif // __GNUC__
#endif // BASE_ALLOCATOR_ALLOCATOR_SHIM_INTERNALS_H_

View file

@ -0,0 +1,152 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_CPP_SYMBOLS_H_
#error This header is meant to be included only once by allocator_shim.cc
#endif
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_CPP_SYMBOLS_H_
// Preempt the default new/delete C++ symbols so they call the shim entry
// points. This file is strongly inspired by tcmalloc's
// libc_override_redefine.h.
#include <new>
#include "base/allocator/allocator_shim_internals.h"
#include "build/build_config.h"
// std::align_val_t isn't available until C++17, but we want to override aligned
// new/delete anyway to prevent a possible situation where a library gets loaded
// in that uses the aligned operators. We want to avoid a situation where
// separate heaps are used.
// TODO(thomasanderson): Remove this once building with C++17 or later.
#if defined(__cpp_aligned_new) && __cpp_aligned_new >= 201606
#define ALIGN_VAL_T std::align_val_t
#define ALIGN_LINKAGE
#define ALIGN_NEW operator new
#define ALIGN_NEW_NOTHROW operator new
#define ALIGN_DEL operator delete
#define ALIGN_DEL_SIZED operator delete
#define ALIGN_DEL_NOTHROW operator delete
#define ALIGN_NEW_ARR operator new[]
#define ALIGN_NEW_ARR_NOTHROW operator new[]
#define ALIGN_DEL_ARR operator delete[]
#define ALIGN_DEL_ARR_SIZED operator delete[]
#define ALIGN_DEL_ARR_NOTHROW operator delete[]
#else
#define ALIGN_VAL_T size_t
#define ALIGN_LINKAGE extern "C"
#if defined(OS_MACOSX) || defined(OS_WIN)
#error "Mangling is different on these platforms."
#else
#define ALIGN_NEW _ZnwmSt11align_val_t
#define ALIGN_NEW_NOTHROW _ZnwmSt11align_val_tRKSt9nothrow_t
#define ALIGN_DEL _ZdlPvSt11align_val_t
#define ALIGN_DEL_SIZED _ZdlPvmSt11align_val_t
#define ALIGN_DEL_NOTHROW _ZdlPvSt11align_val_tRKSt9nothrow_t
#define ALIGN_NEW_ARR _ZnamSt11align_val_t
#define ALIGN_NEW_ARR_NOTHROW _ZnamSt11align_val_tRKSt9nothrow_t
#define ALIGN_DEL_ARR _ZdaPvSt11align_val_t
#define ALIGN_DEL_ARR_SIZED _ZdaPvmSt11align_val_t
#define ALIGN_DEL_ARR_NOTHROW _ZdaPvSt11align_val_tRKSt9nothrow_t
#endif
#endif
SHIM_ALWAYS_EXPORT void* operator new(size_t size) {
return ShimCppNew(size);
}
SHIM_ALWAYS_EXPORT void operator delete(void* p) __THROW {
ShimCppDelete(p);
}
SHIM_ALWAYS_EXPORT void* operator new[](size_t size) {
return ShimCppNew(size);
}
SHIM_ALWAYS_EXPORT void operator delete[](void* p) __THROW {
ShimCppDelete(p);
}
SHIM_ALWAYS_EXPORT void* operator new(size_t size,
const std::nothrow_t&) __THROW {
return ShimCppNew(size);
}
SHIM_ALWAYS_EXPORT void* operator new[](size_t size,
const std::nothrow_t&) __THROW {
return ShimCppNew(size);
}
SHIM_ALWAYS_EXPORT void operator delete(void* p, const std::nothrow_t&) __THROW {
ShimCppDelete(p);
}
SHIM_ALWAYS_EXPORT void operator delete[](void* p,
const std::nothrow_t&) __THROW {
ShimCppDelete(p);
}
SHIM_ALWAYS_EXPORT void operator delete(void* p, size_t) __THROW {
ShimCppDelete(p);
}
SHIM_ALWAYS_EXPORT void operator delete[](void* p, size_t) __THROW {
ShimCppDelete(p);
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void* ALIGN_NEW(std::size_t size,
ALIGN_VAL_T alignment) {
return ShimCppAlignedNew(size, static_cast<size_t>(alignment));
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void* ALIGN_NEW_NOTHROW(
std::size_t size,
ALIGN_VAL_T alignment,
const std::nothrow_t&) __THROW {
return ShimCppAlignedNew(size, static_cast<size_t>(alignment));
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void ALIGN_DEL(void* p, ALIGN_VAL_T) __THROW {
ShimCppDelete(p);
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void ALIGN_DEL_SIZED(void* p,
std::size_t size,
ALIGN_VAL_T) __THROW {
ShimCppDelete(p);
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void
ALIGN_DEL_NOTHROW(void* p, ALIGN_VAL_T, const std::nothrow_t&) __THROW {
ShimCppDelete(p);
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void* ALIGN_NEW_ARR(std::size_t size,
ALIGN_VAL_T alignment) {
return ShimCppAlignedNew(size, static_cast<size_t>(alignment));
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void* ALIGN_NEW_ARR_NOTHROW(
std::size_t size,
ALIGN_VAL_T alignment,
const std::nothrow_t&) __THROW {
return ShimCppAlignedNew(size, static_cast<size_t>(alignment));
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void ALIGN_DEL_ARR(void* p,
ALIGN_VAL_T) __THROW {
ShimCppDelete(p);
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void ALIGN_DEL_ARR_SIZED(void* p,
std::size_t size,
ALIGN_VAL_T) __THROW {
ShimCppDelete(p);
}
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void
ALIGN_DEL_ARR_NOTHROW(void* p, ALIGN_VAL_T, const std::nothrow_t&) __THROW {
ShimCppDelete(p);
}

View file

@ -0,0 +1,119 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_GLIBC_WEAK_SYMBOLS_H_
#error This header is meant to be included only once by allocator_shim.cc
#endif
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_GLIBC_WEAK_SYMBOLS_H_
// Alias the internal Glibc symbols to the shim entry points.
// This file is strongly inspired by tcmalloc's libc_override_glibc.h.
// Effectively this file does two things:
// 1) Re-define the __malloc_hook & co symbols. Those symbols are defined as
// weak in glibc and are meant to be defined strongly by client processes
// to hook calls initiated from within glibc.
// 2) Re-define Glibc-specific symbols (__libc_malloc). The historical reason
// is that in the past (in RedHat 9) we had instances of libraries that were
// allocating via malloc() and freeing using __libc_free().
// See tcmalloc's libc_override_glibc.h for more context.
#include <features.h> // for __GLIBC__
#include <malloc.h>
#include <unistd.h>
#include <new>
#include "base/allocator/allocator_shim_internals.h"
// __MALLOC_HOOK_VOLATILE not defined in all Glibc headers.
#if !defined(__MALLOC_HOOK_VOLATILE)
#define MALLOC_HOOK_MAYBE_VOLATILE /**/
#else
#define MALLOC_HOOK_MAYBE_VOLATILE __MALLOC_HOOK_VOLATILE
#endif
extern "C" {
// 1) Re-define malloc_hook weak symbols.
namespace {
void* GlibcMallocHook(size_t size, const void* caller) {
return ShimMalloc(size, nullptr);
}
void* GlibcReallocHook(void* ptr, size_t size, const void* caller) {
return ShimRealloc(ptr, size, nullptr);
}
void GlibcFreeHook(void* ptr, const void* caller) {
return ShimFree(ptr, nullptr);
}
void* GlibcMemalignHook(size_t align, size_t size, const void* caller) {
return ShimMemalign(align, size, nullptr);
}
} // namespace
__attribute__((visibility("default"))) void* (
*MALLOC_HOOK_MAYBE_VOLATILE __malloc_hook)(size_t,
const void*) = &GlibcMallocHook;
__attribute__((visibility("default"))) void* (
*MALLOC_HOOK_MAYBE_VOLATILE __realloc_hook)(void*, size_t, const void*) =
&GlibcReallocHook;
__attribute__((visibility("default"))) void (
*MALLOC_HOOK_MAYBE_VOLATILE __free_hook)(void*,
const void*) = &GlibcFreeHook;
__attribute__((visibility("default"))) void* (
*MALLOC_HOOK_MAYBE_VOLATILE __memalign_hook)(size_t, size_t, const void*) =
&GlibcMemalignHook;
// 2) Redefine libc symbols themselves.
SHIM_ALWAYS_EXPORT void* __libc_malloc(size_t size) {
return ShimMalloc(size, nullptr);
}
SHIM_ALWAYS_EXPORT void __libc_free(void* ptr) {
ShimFree(ptr, nullptr);
}
SHIM_ALWAYS_EXPORT void* __libc_realloc(void* ptr, size_t size) {
return ShimRealloc(ptr, size, nullptr);
}
SHIM_ALWAYS_EXPORT void* __libc_calloc(size_t n, size_t size) {
return ShimCalloc(n, size, nullptr);
}
SHIM_ALWAYS_EXPORT void __libc_cfree(void* ptr) {
return ShimFree(ptr, nullptr);
}
SHIM_ALWAYS_EXPORT void* __libc_memalign(size_t align, size_t s) {
return ShimMemalign(align, s, nullptr);
}
SHIM_ALWAYS_EXPORT void* __libc_valloc(size_t size) {
return ShimValloc(size, nullptr);
}
SHIM_ALWAYS_EXPORT void* __libc_pvalloc(size_t size) {
return ShimPvalloc(size);
}
SHIM_ALWAYS_EXPORT int __posix_memalign(void** r, size_t a, size_t s) {
return ShimPosixMemalign(r, a, s);
}
} // extern "C"
// Safety check.
#if !defined(__GLIBC__)
#error The target platform does not seem to use Glibc. Disable the allocator \
shim by setting use_allocator_shim=false in GN args.
#endif

View file

@ -0,0 +1,69 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// Its purpose is to preempt the Libc symbols for malloc/new so they call the
// shim layer entry points.
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_LIBC_SYMBOLS_H_
#error This header is meant to be included only once by allocator_shim.cc
#endif
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_LIBC_SYMBOLS_H_
#include <malloc.h>
#include "base/allocator/allocator_shim_internals.h"
extern "C" {
SHIM_ALWAYS_EXPORT void* malloc(size_t size) __THROW {
return ShimMalloc(size, nullptr);
}
SHIM_ALWAYS_EXPORT void free(void* ptr) __THROW {
ShimFree(ptr, nullptr);
}
SHIM_ALWAYS_EXPORT void* realloc(void* ptr, size_t size) __THROW {
return ShimRealloc(ptr, size, nullptr);
}
SHIM_ALWAYS_EXPORT void* calloc(size_t n, size_t size) __THROW {
return ShimCalloc(n, size, nullptr);
}
SHIM_ALWAYS_EXPORT void cfree(void* ptr) __THROW {
ShimFree(ptr, nullptr);
}
SHIM_ALWAYS_EXPORT void* memalign(size_t align, size_t s) __THROW {
return ShimMemalign(align, s, nullptr);
}
SHIM_ALWAYS_EXPORT void* valloc(size_t size) __THROW {
return ShimValloc(size, nullptr);
}
SHIM_ALWAYS_EXPORT void* pvalloc(size_t size) __THROW {
return ShimPvalloc(size);
}
SHIM_ALWAYS_EXPORT int posix_memalign(void** r, size_t a, size_t s) __THROW {
return ShimPosixMemalign(r, a, s);
}
SHIM_ALWAYS_EXPORT size_t malloc_size(void* address) __THROW {
return ShimGetSizeEstimate(address, nullptr);
}
SHIM_ALWAYS_EXPORT size_t malloc_usable_size(void* address) __THROW {
return ShimGetSizeEstimate(address, nullptr);
}
// The default dispatch translation unit has to define also the following
// symbols (unless they are ultimately routed to the system symbols):
// void malloc_stats(void);
// int mallopt(int, int);
// struct mallinfo mallinfo(void);
} // extern "C"

View file

@ -0,0 +1,54 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_LINKER_WRAPPED_SYMBOLS_H_
#error This header is meant to be included only once by allocator_shim.cc
#endif
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_LINKER_WRAPPED_SYMBOLS_H_
// This header overrides the __wrap_X symbols when using the link-time
// -Wl,-wrap,malloc shim-layer approach (see README.md).
// All references to malloc, free, etc. within the linker unit that gets the
// -wrap linker flags (e.g., libchrome.so) will be rewritten to the
// linker as references to __wrap_malloc, __wrap_free, which are defined here.
#include "base/allocator/allocator_shim_internals.h"
extern "C" {
SHIM_ALWAYS_EXPORT void* __wrap_calloc(size_t n, size_t size) {
return ShimCalloc(n, size, nullptr);
}
SHIM_ALWAYS_EXPORT void __wrap_free(void* ptr) {
ShimFree(ptr, nullptr);
}
SHIM_ALWAYS_EXPORT void* __wrap_malloc(size_t size) {
return ShimMalloc(size, nullptr);
}
SHIM_ALWAYS_EXPORT void* __wrap_memalign(size_t align, size_t size) {
return ShimMemalign(align, size, nullptr);
}
SHIM_ALWAYS_EXPORT int __wrap_posix_memalign(void** res,
size_t align,
size_t size) {
return ShimPosixMemalign(res, align, size);
}
SHIM_ALWAYS_EXPORT void* __wrap_pvalloc(size_t size) {
return ShimPvalloc(size);
}
SHIM_ALWAYS_EXPORT void* __wrap_realloc(void* address, size_t size) {
return ShimRealloc(address, size, nullptr);
}
SHIM_ALWAYS_EXPORT void* __wrap_valloc(size_t size) {
return ShimValloc(size, nullptr);
}
} // extern "C"

View file

@ -0,0 +1,60 @@
// Copyright 2017 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_MAC_SYMBOLS_H_
#error This header is meant to be included only once by allocator_shim.cc
#endif
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_MAC_SYMBOLS_H_
#include "base/allocator/malloc_zone_functions_mac.h"
#include "third_party/apple_apsl/malloc.h"
namespace base {
namespace allocator {
MallocZoneFunctions MallocZoneFunctionsToReplaceDefault() {
MallocZoneFunctions new_functions;
memset(&new_functions, 0, sizeof(MallocZoneFunctions));
new_functions.size = [](malloc_zone_t* zone, const void* ptr) -> size_t {
return ShimGetSizeEstimate(ptr, zone);
};
new_functions.malloc = [](malloc_zone_t* zone, size_t size) -> void* {
return ShimMalloc(size, zone);
};
new_functions.calloc = [](malloc_zone_t* zone, size_t n,
size_t size) -> void* {
return ShimCalloc(n, size, zone);
};
new_functions.valloc = [](malloc_zone_t* zone, size_t size) -> void* {
return ShimValloc(size, zone);
};
new_functions.free = [](malloc_zone_t* zone, void* ptr) {
ShimFree(ptr, zone);
};
new_functions.realloc = [](malloc_zone_t* zone, void* ptr,
size_t size) -> void* {
return ShimRealloc(ptr, size, zone);
};
new_functions.batch_malloc = [](struct _malloc_zone_t* zone, size_t size,
void** results,
unsigned num_requested) -> unsigned {
return ShimBatchMalloc(size, results, num_requested, zone);
};
new_functions.batch_free = [](struct _malloc_zone_t* zone, void** to_be_freed,
unsigned num_to_be_freed) -> void {
ShimBatchFree(to_be_freed, num_to_be_freed, zone);
};
new_functions.memalign = [](malloc_zone_t* zone, size_t alignment,
size_t size) -> void* {
return ShimMemalign(alignment, size, zone);
};
new_functions.free_definite_size = [](malloc_zone_t* zone, void* ptr,
size_t size) {
ShimFreeDefiniteSize(ptr, size, zone);
};
return new_functions;
}
} // namespace allocator
} // namespace base

View file

@ -0,0 +1,174 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// This header defines symbols to override the same functions in the Visual C++
// CRT implementation.
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_UCRT_SYMBOLS_WIN_H_
#error This header is meant to be included only once by allocator_shim.cc
#endif
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_UCRT_SYMBOLS_WIN_H_
#include <malloc.h>
#include <windows.h>
extern "C" {
void* (*malloc_unchecked)(size_t) = &base::allocator::UncheckedAlloc;
namespace {
int win_new_mode = 0;
} // namespace
// This function behaves similarly to MSVC's _set_new_mode.
// If flag is 0 (default), calls to malloc will behave normally.
// If flag is 1, calls to malloc will behave like calls to new,
// and the std_new_handler will be invoked on failure.
// Returns the previous mode.
//
// Replaces _set_new_mode in ucrt\heap\new_mode.cpp
int _set_new_mode(int flag) {
// The MS CRT calls this function early on in startup, so this serves as a low
// overhead proof that the allocator shim is in place for this process.
base::allocator::g_is_win_shim_layer_initialized = true;
int old_mode = win_new_mode;
win_new_mode = flag;
base::allocator::SetCallNewHandlerOnMallocFailure(win_new_mode != 0);
return old_mode;
}
// Replaces _query_new_mode in ucrt\heap\new_mode.cpp
int _query_new_mode() {
return win_new_mode;
}
// These symbols override the CRT's implementation of the same functions.
__declspec(restrict) void* malloc(size_t size) {
return ShimMalloc(size, nullptr);
}
void free(void* ptr) {
ShimFree(ptr, nullptr);
}
__declspec(restrict) void* realloc(void* ptr, size_t size) {
return ShimRealloc(ptr, size, nullptr);
}
__declspec(restrict) void* calloc(size_t n, size_t size) {
return ShimCalloc(n, size, nullptr);
}
// _msize() is the Windows equivalent of malloc_size().
size_t _msize(void* memblock) {
return ShimGetSizeEstimate(memblock, nullptr);
}
__declspec(restrict) void* _aligned_malloc(size_t size, size_t alignment) {
return ShimAlignedMalloc(size, alignment, nullptr);
}
__declspec(restrict) void* _aligned_realloc(void* address,
size_t size,
size_t alignment) {
return ShimAlignedRealloc(address, size, alignment, nullptr);
}
void _aligned_free(void* address) {
ShimAlignedFree(address, nullptr);
}
// _recalloc_base is called by CRT internally.
__declspec(restrict) void* _recalloc_base(void* block,
size_t count,
size_t size) {
const size_t old_block_size = (block != nullptr) ? _msize(block) : 0;
base::CheckedNumeric<size_t> new_block_size_checked = count;
new_block_size_checked *= size;
const size_t new_block_size = new_block_size_checked.ValueOrDie();
void* const new_block = realloc(block, new_block_size);
if (new_block != nullptr && old_block_size < new_block_size) {
memset(static_cast<char*>(new_block) + old_block_size, 0,
new_block_size - old_block_size);
}
return new_block;
}
__declspec(restrict) void* _recalloc(void* block, size_t count, size_t size) {
return _recalloc_base(block, count, size);
}
// The following uncommon _aligned_* routines are not used in Chromium and have
// been shimmed to immediately crash to ensure that implementations are added if
// uses are introduced.
__declspec(restrict) void* _aligned_recalloc(void* address,
size_t num,
size_t size,
size_t alignment) {
CHECK(false) << "This routine has not been implemented";
__builtin_unreachable();
}
size_t _aligned_msize(void* address, size_t alignment, size_t offset) {
CHECK(false) << "This routine has not been implemented";
__builtin_unreachable();
}
__declspec(restrict) void* _aligned_offset_malloc(size_t size,
size_t alignment,
size_t offset) {
CHECK(false) << "This routine has not been implemented";
__builtin_unreachable();
}
__declspec(restrict) void* _aligned_offset_realloc(void* address,
size_t size,
size_t alignment,
size_t offset) {
CHECK(false) << "This routine has not been implemented";
__builtin_unreachable();
}
__declspec(restrict) void* _aligned_offset_recalloc(void* address,
size_t num,
size_t size,
size_t alignment,
size_t offset) {
CHECK(false) << "This routine has not been implemented";
__builtin_unreachable();
}
// The symbols
// * __acrt_heap
// * __acrt_initialize_heap
// * __acrt_uninitialize_heap
// * _get_heap_handle
// must be overridden all or none, as they are otherwise supplied
// by heap_handle.obj in the ucrt.lib file.
HANDLE __acrt_heap = nullptr;
bool __acrt_initialize_heap() {
__acrt_heap = ::HeapCreate(0, 0, 0);
return true;
}
bool __acrt_uninitialize_heap() {
::HeapDestroy(__acrt_heap);
__acrt_heap = nullptr;
return true;
}
intptr_t _get_heap_handle(void) {
return reinterpret_cast<intptr_t>(__acrt_heap);
}
} // extern "C"

View file

@ -0,0 +1,21 @@
// Copyright (c) 2012 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// Workaround for crosbug:629593. Using AFDO on the tcmalloc files is
// causing problems. The tcmalloc files depend on stack layouts and
// AFDO can mess with them. Better not to use AFDO there. This is a
// temporary hack. We will add a mechanism in the build system to
// avoid using -fauto-profile for tcmalloc files.
#if !defined(__clang__) && \
(defined(OS_CHROMEOS) || (__GNUC__ > 5 && __GNUC__ < 7))
// Note that this option only seems to be available in the chromeos GCC 4.9
// toolchain, and stock GCC 5 upto 7.
#pragma GCC optimize ("no-auto-profile")
#endif
#if defined(TCMALLOC_FOR_DEBUGALLOCATION)
#include "third_party/tcmalloc/chromium/src/debugallocation.cc"
#else
#include "third_party/tcmalloc/chromium/src/tcmalloc.cc"
#endif

View file

@ -0,0 +1,115 @@
// Copyright 2017 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/malloc_zone_functions_mac.h"
#include <atomic>
#include "base/synchronization/lock.h"
namespace base {
namespace allocator {
MallocZoneFunctions g_malloc_zones[kMaxZoneCount];
static_assert(std::is_pod<MallocZoneFunctions>::value,
"MallocZoneFunctions must be POD");
void StoreZoneFunctions(const ChromeMallocZone* zone,
MallocZoneFunctions* functions) {
memset(functions, 0, sizeof(MallocZoneFunctions));
functions->malloc = zone->malloc;
functions->calloc = zone->calloc;
functions->valloc = zone->valloc;
functions->free = zone->free;
functions->realloc = zone->realloc;
functions->size = zone->size;
CHECK(functions->malloc && functions->calloc && functions->valloc &&
functions->free && functions->realloc && functions->size);
// These functions might be nullptr.
functions->batch_malloc = zone->batch_malloc;
functions->batch_free = zone->batch_free;
if (zone->version >= 5) {
// Not all custom malloc zones have a memalign.
functions->memalign = zone->memalign;
}
if (zone->version >= 6) {
// This may be nullptr.
functions->free_definite_size = zone->free_definite_size;
}
functions->context = zone;
}
namespace {
// All modifications to g_malloc_zones are gated behind this lock.
// Dispatch to a malloc zone does not need to acquire this lock.
base::Lock& GetLock() {
static base::Lock* g_lock = new base::Lock;
return *g_lock;
}
void EnsureMallocZonesInitializedLocked() {
GetLock().AssertAcquired();
}
int g_zone_count = 0;
bool IsMallocZoneAlreadyStoredLocked(ChromeMallocZone* zone) {
EnsureMallocZonesInitializedLocked();
GetLock().AssertAcquired();
for (int i = 0; i < g_zone_count; ++i) {
if (g_malloc_zones[i].context == reinterpret_cast<void*>(zone))
return true;
}
return false;
}
} // namespace
bool StoreMallocZone(ChromeMallocZone* zone) {
base::AutoLock l(GetLock());
EnsureMallocZonesInitializedLocked();
if (IsMallocZoneAlreadyStoredLocked(zone))
return false;
if (g_zone_count == kMaxZoneCount)
return false;
StoreZoneFunctions(zone, &g_malloc_zones[g_zone_count]);
++g_zone_count;
// No other thread can possibly see these stores at this point. The code that
// reads these values is triggered after this function returns. so we want to
// guarantee that they are committed at this stage"
std::atomic_thread_fence(std::memory_order_seq_cst);
return true;
}
bool IsMallocZoneAlreadyStored(ChromeMallocZone* zone) {
base::AutoLock l(GetLock());
return IsMallocZoneAlreadyStoredLocked(zone);
}
bool DoesMallocZoneNeedReplacing(ChromeMallocZone* zone,
const MallocZoneFunctions* functions) {
return IsMallocZoneAlreadyStored(zone) && zone->malloc != functions->malloc;
}
int GetMallocZoneCountForTesting() {
base::AutoLock l(GetLock());
return g_zone_count;
}
void ClearAllMallocZonesForTesting() {
base::AutoLock l(GetLock());
EnsureMallocZonesInitializedLocked();
memset(g_malloc_zones, 0, kMaxZoneCount * sizeof(MallocZoneFunctions));
g_zone_count = 0;
}
} // namespace allocator
} // namespace base

View file

@ -0,0 +1,103 @@
// Copyright 2017 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_MALLOC_ZONE_FUNCTIONS_MAC_H_
#define BASE_ALLOCATOR_MALLOC_ZONE_FUNCTIONS_MAC_H_
#include <malloc/malloc.h>
#include <stddef.h>
#include "base/base_export.h"
#include "base/logging.h"
#include "third_party/apple_apsl/malloc.h"
namespace base {
namespace allocator {
typedef void* (*malloc_type)(struct _malloc_zone_t* zone, size_t size);
typedef void* (*calloc_type)(struct _malloc_zone_t* zone,
size_t num_items,
size_t size);
typedef void* (*valloc_type)(struct _malloc_zone_t* zone, size_t size);
typedef void (*free_type)(struct _malloc_zone_t* zone, void* ptr);
typedef void* (*realloc_type)(struct _malloc_zone_t* zone,
void* ptr,
size_t size);
typedef void* (*memalign_type)(struct _malloc_zone_t* zone,
size_t alignment,
size_t size);
typedef unsigned (*batch_malloc_type)(struct _malloc_zone_t* zone,
size_t size,
void** results,
unsigned num_requested);
typedef void (*batch_free_type)(struct _malloc_zone_t* zone,
void** to_be_freed,
unsigned num_to_be_freed);
typedef void (*free_definite_size_type)(struct _malloc_zone_t* zone,
void* ptr,
size_t size);
typedef size_t (*size_fn_type)(struct _malloc_zone_t* zone, const void* ptr);
struct MallocZoneFunctions {
malloc_type malloc;
calloc_type calloc;
valloc_type valloc;
free_type free;
realloc_type realloc;
memalign_type memalign;
batch_malloc_type batch_malloc;
batch_free_type batch_free;
free_definite_size_type free_definite_size;
size_fn_type size;
const ChromeMallocZone* context;
};
BASE_EXPORT void StoreZoneFunctions(const ChromeMallocZone* zone,
MallocZoneFunctions* functions);
static constexpr int kMaxZoneCount = 30;
BASE_EXPORT extern MallocZoneFunctions g_malloc_zones[kMaxZoneCount];
// The array g_malloc_zones stores all information about malloc zones before
// they are shimmed. This information needs to be accessed during dispatch back
// into the zone, and additional zones may be added later in the execution fo
// the program, so the array needs to be both thread-safe and high-performance.
//
// We begin by creating an array of MallocZoneFunctions of fixed size. We will
// never modify the container, which provides thread-safety to iterators. When
// we want to add a MallocZoneFunctions to the container, we:
// 1. Fill in all the fields.
// 2. Update the total zone count.
// 3. Insert a memory barrier.
// 4. Insert our shim.
//
// Each MallocZoneFunctions is uniquely identified by |context|, which is a
// pointer to the original malloc zone. When we wish to dispatch back to the
// original malloc zones, we iterate through the array, looking for a matching
// |context|.
//
// Most allocations go through the default allocator. We will ensure that the
// default allocator is stored as the first MallocZoneFunctions.
//
// Returns whether the zone was successfully stored.
BASE_EXPORT bool StoreMallocZone(ChromeMallocZone* zone);
BASE_EXPORT bool IsMallocZoneAlreadyStored(ChromeMallocZone* zone);
BASE_EXPORT bool DoesMallocZoneNeedReplacing(
ChromeMallocZone* zone,
const MallocZoneFunctions* functions);
BASE_EXPORT int GetMallocZoneCountForTesting();
BASE_EXPORT void ClearAllMallocZonesForTesting();
inline MallocZoneFunctions& GetFunctionsForZone(void* zone) {
for (unsigned int i = 0; i < kMaxZoneCount; ++i) {
if (g_malloc_zones[i].context == zone)
return g_malloc_zones[i];
}
IMMEDIATE_CRASH();
}
} // namespace allocator
} // namespace base
#endif // BASE_ALLOCATOR_MALLOC_ZONE_FUNCTIONS_MAC_H_

View file

@ -0,0 +1,8 @@
ajwong@chromium.org
haraken@chromium.org
palmer@chromium.org
tsepez@chromium.org
# TEAM: platform-architecture-dev@chromium.org
# Also: security-dev@chromium.org
# COMPONENT: Blink>MemoryAllocator>Partition

View file

@ -0,0 +1,102 @@
# PartitionAlloc Design
This document describes PartitionAlloc at a high level. For documentation about
its implementation, see the comments in `partition_alloc.h`.
[TOC]
## Overview
PartitionAlloc is a memory allocator optimized for security, low allocation
latency (when called appropriately), and good space efficiency (when called
appropriately). This document aims to help you understand how PartitionAlloc
works so that you can use it effectively.
## Partitions And Buckets
A *partition* is a heap that contains certain object types, objects of certain
sizes, or objects of a certain lifetime (as the caller prefers). Callers can
create as many partitions as they need. Each partition is separate and protected
from any other partitions.
Each partition holds multiple buckets. A *bucket* is a region in a partition
that contains similar-sized objects.
PartitionAlloc aligns each object allocation with the closest bucket size. For
example, if a partition has 3 buckets for 64 bytes, 256 bytes, and 1024 bytes,
then PartitionAlloc will satisfy an allocation request for 128 bytes by rounding
it up to 256 bytes and allocating from the second bucket.
The special allocator class `template <size_t N> class
SizeSpecificPartitionAllocator` will satisfy allocations only of size
`kMaxAllocation = N - kAllocationGranularity` or less, and contains buckets for
all `n * kAllocationGranularity` (n = 1, 2, ..., `kMaxAllocation`). Attempts to
allocate more than `kMaxAllocation` will fail.
## Performance
The current implementation is optimized for the main thread use-case. For
example, PartitionAlloc doesn't have threaded caches.
PartitionAlloc is designed to be extremely fast in its fast paths. The fast
paths of allocation and deallocation require just 2 (reasonably predictable)
branches. The number of operations in the fast paths is minimal, leading to the
possibility of inlining.
For an example of how to use partitions to get good performance and good safety,
see Blink's usage, as described in `wtf/allocator/Allocator.md`.
Large allocations (> kGenericMaxBucketed == 960KB) are realized by direct
memory mmapping. This size makes sense because 960KB = 0xF0000. The next larger
bucket size is 1MB = 0x100000 which is greater than 1/2 the available space in
a SuperPage meaning it would not be possible to pack even 2 sequential
allocations in a SuperPage.
`PartitionRootGeneric::Alloc()` acquires a lock for thread safety. (The current
implementation uses a spin lock on the assumption that thread contention will be
rare in its callers. The original caller was Blink, where this is generally
true. Spin locks also have the benefit of simplicity.)
Callers can get thread-unsafe performance using a
`SizeSpecificPartitionAllocator` or otherwise using `PartitionAlloc` (instead of
`PartitionRootGeneric::Alloc()`). Callers can also arrange for low contention,
such as by using a dedicated partition for single-threaded, latency-critical
allocations.
Because PartitionAlloc guarantees that address space regions used for one
partition are never reused for other partitions, partitions can eat a large
amount of virtual address space (even if not of actual memory).
Mixing various random objects in the same partition will generally lead to lower
efficiency. For good performance, group similar objects into the same partition.
## Security
Security is one of the most important goals of PartitionAlloc.
PartitionAlloc guarantees that different partitions exist in different regions
of the process' address space. When the caller has freed all objects contained
in a page in a partition, PartitionAlloc returns the physical memory to the
operating system, but continues to reserve the region of address space.
PartitionAlloc will only reuse an address space region for the same partition.
PartitionAlloc also guarantees that:
* Linear overflows cannot corrupt into the partition. (There is a guard page at
the beginning of each partition.)
* Linear overflows cannot corrupt out of the partition. (There is a guard page
at the end of each partition.)
* Linear overflow or underflow cannot corrupt the allocation metadata.
PartitionAlloc records metadata in a dedicated region out-of-line (not adjacent
to objects).
* Objects of different sizes will likely be allocated in different buckets, and
hence at different addresses. One page can contain only similar-sized objects.
* Dereference of a freelist pointer should fault.
* Partial pointer overwrite of freelist pointer should fault.
* Large allocations have guard pages at the beginning and end.

View file

@ -0,0 +1,68 @@
// Copyright 2014 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/address_space_randomization.h"
#include "base/allocator/partition_allocator/page_allocator.h"
#include "base/allocator/partition_allocator/random.h"
#include "base/allocator/partition_allocator/spin_lock.h"
#include "base/logging.h"
#include "build/build_config.h"
#if defined(OS_WIN)
#include <windows.h> // Must be in front of other Windows header files.
#include <VersionHelpers.h>
#endif
namespace base {
void* GetRandomPageBase() {
uintptr_t random = static_cast<uintptr_t>(RandomValue());
#if defined(ARCH_CPU_64_BITS)
random <<= 32ULL;
random |= static_cast<uintptr_t>(RandomValue());
// The kASLRMask and kASLROffset constants will be suitable for the
// OS and build configuration.
#if defined(OS_WIN) && !defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
// Windows >= 8.1 has the full 47 bits. Use them where available.
static bool windows_81 = false;
static bool windows_81_initialized = false;
if (!windows_81_initialized) {
windows_81 = IsWindows8Point1OrGreater();
windows_81_initialized = true;
}
if (!windows_81) {
random &= internal::kASLRMaskBefore8_10;
} else {
random &= internal::kASLRMask;
}
random += internal::kASLROffset;
#else
random &= internal::kASLRMask;
random += internal::kASLROffset;
#endif // defined(OS_WIN) && !defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
#else // defined(ARCH_CPU_32_BITS)
#if defined(OS_WIN)
// On win32 host systems the randomization plus huge alignment causes
// excessive fragmentation. Plus most of these systems lack ASLR, so the
// randomization isn't buying anything. In that case we just skip it.
// TODO(palmer): Just dump the randomization when HE-ASLR is present.
static BOOL is_wow64 = -1;
if (is_wow64 == -1 && !IsWow64Process(GetCurrentProcess(), &is_wow64))
is_wow64 = FALSE;
if (!is_wow64)
return nullptr;
#endif // defined(OS_WIN)
random &= internal::kASLRMask;
random += internal::kASLROffset;
#endif // defined(ARCH_CPU_32_BITS)
DCHECK_EQ(0ULL, (random & kPageAllocationGranularityOffsetMask));
return reinterpret_cast<void*>(random);
}
} // namespace base

View file

@ -0,0 +1,201 @@
// Copyright 2014 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_ADDRESS_SPACE_RANDOMIZATION_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_ADDRESS_SPACE_RANDOMIZATION_H_
#include "base/allocator/partition_allocator/page_allocator.h"
#include "base/base_export.h"
#include "build/build_config.h"
namespace base {
// Calculates a random preferred mapping address. In calculating an address, we
// balance good ASLR against not fragmenting the address space too badly.
BASE_EXPORT void* GetRandomPageBase();
namespace internal {
constexpr uintptr_t AslrAddress(uintptr_t mask) {
return mask & kPageAllocationGranularityBaseMask;
}
constexpr uintptr_t AslrMask(uintptr_t bits) {
return AslrAddress((1ULL << bits) - 1ULL);
}
// Turn off formatting, because the thicket of nested ifdefs below is
// incomprehensible without indentation. It is also incomprehensible with
// indentation, but the only other option is a combinatorial explosion of
// *_{win,linux,mac,foo}_{32,64}.h files.
//
// clang-format off
#if defined(ARCH_CPU_64_BITS)
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
// We shouldn't allocate system pages at all for sanitizer builds. However,
// we do, and if random hint addresses interfere with address ranges
// hard-coded in those tools, bad things happen. This address range is
// copied from TSAN source but works with all tools. See
// https://crbug.com/539863.
constexpr uintptr_t kASLRMask = AslrAddress(0x007fffffffffULL);
constexpr uintptr_t kASLROffset = AslrAddress(0x7e8000000000ULL);
#elif defined(OS_WIN)
// Windows 8.10 and newer support the full 48 bit address range. Older
// versions of Windows only support 44 bits. Since kASLROffset is non-zero
// and may cause a carry, use 47 and 43 bit masks. See
// http://www.alex-ionescu.com/?p=246
constexpr uintptr_t kASLRMask = AslrMask(47);
constexpr uintptr_t kASLRMaskBefore8_10 = AslrMask(43);
// Try not to map pages into the range where Windows loads DLLs by default.
constexpr uintptr_t kASLROffset = 0x80000000ULL;
#elif defined(OS_MACOSX)
// macOS as of 10.12.5 does not clean up entries in page map levels 3/4
// [PDP/PML4] created from mmap or mach_vm_allocate, even after the region
// is destroyed. Using a virtual address space that is too large causes a
// leak of about 1 wired [can never be paged out] page per call to mmap. The
// page is only reclaimed when the process is killed. Confine the hint to a
// 39-bit section of the virtual address space.
//
// This implementation adapted from
// https://chromium-review.googlesource.com/c/v8/v8/+/557958. The difference
// is that here we clamp to 39 bits, not 32.
//
// TODO(crbug.com/738925): Remove this limitation if/when the macOS behavior
// changes.
constexpr uintptr_t kASLRMask = AslrMask(38);
constexpr uintptr_t kASLROffset = AslrAddress(0x1000000000ULL);
#elif defined(OS_POSIX) || defined(OS_FUCHSIA)
#if defined(ARCH_CPU_X86_64)
// Linux (and macOS) support the full 47-bit user space of x64 processors.
// Use only 46 to allow the kernel a chance to fulfill the request.
constexpr uintptr_t kASLRMask = AslrMask(46);
constexpr uintptr_t kASLROffset = AslrAddress(0);
#elif defined(ARCH_CPU_ARM64)
#if defined(OS_ANDROID)
// Restrict the address range on Android to avoid a large performance
// regression in single-process WebViews. See https://crbug.com/837640.
constexpr uintptr_t kASLRMask = AslrMask(30);
constexpr uintptr_t kASLROffset = AslrAddress(0x20000000ULL);
#else
// ARM64 on Linux has 39-bit user space. Use 38 bits since kASLROffset
// could cause a carry.
constexpr uintptr_t kASLRMask = AslrMask(38);
constexpr uintptr_t kASLROffset = AslrAddress(0x1000000000ULL);
#endif
#elif defined(ARCH_CPU_PPC64)
#if defined(OS_AIX)
// AIX has 64 bits of virtual addressing, but we limit the address range
// to (a) minimize segment lookaside buffer (SLB) misses; and (b) use
// extra address space to isolate the mmap regions.
constexpr uintptr_t kASLRMask = AslrMask(30);
constexpr uintptr_t kASLROffset = AslrAddress(0x400000000000ULL);
#elif defined(ARCH_CPU_BIG_ENDIAN)
// Big-endian Linux PPC has 44 bits of virtual addressing. Use 42.
constexpr uintptr_t kASLRMask = AslrMask(42);
constexpr uintptr_t kASLROffset = AslrAddress(0);
#else // !defined(OS_AIX) && !defined(ARCH_CPU_BIG_ENDIAN)
// Little-endian Linux PPC has 48 bits of virtual addressing. Use 46.
constexpr uintptr_t kASLRMask = AslrMask(46);
constexpr uintptr_t kASLROffset = AslrAddress(0);
#endif // !defined(OS_AIX) && !defined(ARCH_CPU_BIG_ENDIAN)
#elif defined(ARCH_CPU_S390X)
// Linux on Z uses bits 22 - 32 for Region Indexing, which translates to
// 42 bits of virtual addressing. Truncate to 40 bits to allow kernel a
// chance to fulfill the request.
constexpr uintptr_t kASLRMask = AslrMask(40);
constexpr uintptr_t kASLROffset = AslrAddress(0);
#elif defined(ARCH_CPU_S390)
// 31 bits of virtual addressing. Truncate to 29 bits to allow the kernel
// a chance to fulfill the request.
constexpr uintptr_t kASLRMask = AslrMask(29);
constexpr uintptr_t kASLROffset = AslrAddress(0);
#else // !defined(ARCH_CPU_X86_64) && !defined(ARCH_CPU_PPC64) &&
// !defined(ARCH_CPU_S390X) && !defined(ARCH_CPU_S390)
// For all other POSIX variants, use 30 bits.
constexpr uintptr_t kASLRMask = AslrMask(30);
#if defined(OS_SOLARIS)
// For our Solaris/illumos mmap hint, we pick a random address in the
// bottom half of the top half of the address space (that is, the third
// quarter). Because we do not MAP_FIXED, this will be treated only as a
// hint -- the system will not fail to mmap because something else
// happens to already be mapped at our random address. We deliberately
// set the hint high enough to get well above the system's break (that
// is, the heap); Solaris and illumos will try the hint and if that
// fails allocate as if there were no hint at all. The high hint
// prevents the break from getting hemmed in at low values, ceding half
// of the address space to the system heap.
constexpr uintptr_t kASLROffset = AslrAddress(0x80000000ULL);
#elif defined(OS_AIX)
// The range 0x30000000 - 0xD0000000 is available on AIX; choose the
// upper range.
constexpr uintptr_t kASLROffset = AslrAddress(0x90000000ULL);
#else // !defined(OS_SOLARIS) && !defined(OS_AIX)
// The range 0x20000000 - 0x60000000 is relatively unpopulated across a
// variety of ASLR modes (PAE kernel, NX compat mode, etc) and on macOS
// 10.6 and 10.7.
constexpr uintptr_t kASLROffset = AslrAddress(0x20000000ULL);
#endif // !defined(OS_SOLARIS) && !defined(OS_AIX)
#endif // !defined(ARCH_CPU_X86_64) && !defined(ARCH_CPU_PPC64) &&
// !defined(ARCH_CPU_S390X) && !defined(ARCH_CPU_S390)
#endif // defined(OS_POSIX)
#elif defined(ARCH_CPU_32_BITS)
// This is a good range on 32-bit Windows and Android (the only platforms on
// which we support 32-bitness). Allocates in the 0.5 - 1.5 GiB region. There
// is no issue with carries here.
constexpr uintptr_t kASLRMask = AslrMask(30);
constexpr uintptr_t kASLROffset = AslrAddress(0x20000000ULL);
#else
#error Please tell us about your exotic hardware! Sounds interesting.
#endif // defined(ARCH_CPU_32_BITS)
// clang-format on
} // namespace internal
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_ADDRESS_SPACE_RANDOMIZATION_H_

View file

@ -0,0 +1,127 @@
// Copyright 2019 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/memory_reclaimer.h"
#include "base/allocator/partition_allocator/partition_alloc.h"
#include "base/bind.h"
#include "base/location.h"
#include "base/metrics/histogram_functions.h"
#include "base/timer/elapsed_timer.h"
#include "base/trace_event/trace_event.h"
namespace base {
constexpr TimeDelta PartitionAllocMemoryReclaimer::kStatsRecordingTimeDelta;
// static
PartitionAllocMemoryReclaimer* PartitionAllocMemoryReclaimer::Instance() {
static NoDestructor<PartitionAllocMemoryReclaimer> instance;
return instance.get();
}
void PartitionAllocMemoryReclaimer::RegisterPartition(
internal::PartitionRootBase* partition) {
AutoLock lock(lock_);
DCHECK(partition);
auto it_and_whether_inserted = partitions_.insert(partition);
DCHECK(it_and_whether_inserted.second);
}
void PartitionAllocMemoryReclaimer::UnregisterPartition(
internal::PartitionRootBase* partition) {
AutoLock lock(lock_);
DCHECK(partition);
size_t erased_count = partitions_.erase(partition);
DCHECK_EQ(1u, erased_count);
}
void PartitionAllocMemoryReclaimer::Start(
scoped_refptr<SequencedTaskRunner> task_runner) {
DCHECK(!timer_);
DCHECK(task_runner);
{
AutoLock lock(lock_);
DCHECK(!partitions_.empty());
}
// This does not need to run on the main thread, however there are a few
// reasons to do it there:
// - Most of PartitionAlloc's usage is on the main thread, hence PA's metadata
// is more likely in cache when executing on the main thread.
// - Memory reclaim takes the partition lock for each partition. As a
// consequence, while reclaim is running, the main thread is unlikely to be
// able to make progress, as it would be waiting on the lock.
// - Finally, this runs in idle time only, so there should be no visible
// impact.
//
// From local testing, time to reclaim is 100us-1ms, and reclaiming every few
// seconds is useful. Since this is meant to run during idle time only, it is
// a reasonable starting point balancing effectivenes vs cost. See
// crbug.com/942512 for details and experimental results.
constexpr TimeDelta kInterval = TimeDelta::FromSeconds(4);
timer_ = std::make_unique<RepeatingTimer>();
timer_->SetTaskRunner(task_runner);
// Here and below, |Unretained(this)| is fine as |this| lives forever, as a
// singleton.
timer_->Start(
FROM_HERE, kInterval,
BindRepeating(&PartitionAllocMemoryReclaimer::Reclaim, Unretained(this)));
task_runner->PostDelayedTask(
FROM_HERE,
BindOnce(&PartitionAllocMemoryReclaimer::RecordStatistics,
Unretained(this)),
kStatsRecordingTimeDelta);
}
PartitionAllocMemoryReclaimer::PartitionAllocMemoryReclaimer() = default;
PartitionAllocMemoryReclaimer::~PartitionAllocMemoryReclaimer() = default;
void PartitionAllocMemoryReclaimer::Reclaim() {
TRACE_EVENT0("base", "PartitionAllocMemoryReclaimer::Reclaim()");
// Reclaim will almost always call into the kernel, so tail latency of this
// task would likely be affected by descheduling.
//
// On Linux (and Android) at least, ThreadTicks also includes kernel time, so
// this is a good measure of the true cost of decommit.
ElapsedThreadTimer timer;
constexpr int kFlags =
PartitionPurgeDecommitEmptyPages | PartitionPurgeDiscardUnusedSystemPages;
{
AutoLock lock(lock_); // Has to protect from concurrent (Un)Register calls.
for (auto* partition : partitions_)
partition->PurgeMemory(kFlags);
}
has_called_reclaim_ = true;
if (timer.is_supported())
total_reclaim_thread_time_ += timer.Elapsed();
}
void PartitionAllocMemoryReclaimer::RecordStatistics() {
if (!ElapsedThreadTimer().is_supported())
return;
if (!has_called_reclaim_)
return;
UmaHistogramTimes("Memory.PartitionAlloc.MainThreadTime.5min",
total_reclaim_thread_time_);
has_called_reclaim_ = false;
total_reclaim_thread_time_ = TimeDelta();
}
void PartitionAllocMemoryReclaimer::ResetForTesting() {
AutoLock lock(lock_);
has_called_reclaim_ = false;
total_reclaim_thread_time_ = TimeDelta();
timer_ = nullptr;
partitions_.clear();
}
} // namespace base

View file

@ -0,0 +1,79 @@
// Copyright 2019 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_MEMORY_RECLAIMER_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_MEMORY_RECLAIMER_H_
#include <memory>
#include <set>
#include "base/bind.h"
#include "base/callback.h"
#include "base/location.h"
#include "base/no_destructor.h"
#include "base/single_thread_task_runner.h"
#include "base/thread_annotations.h"
#include "base/time/time.h"
#include "base/timer/elapsed_timer.h"
#include "base/timer/timer.h"
namespace base {
namespace internal {
struct PartitionRootBase;
} // namespace internal
// Posts and handles memory reclaim tasks for PartitionAlloc.
//
// Thread safety: |RegisterPartition()| and |UnregisterPartition()| can be
// called from any thread, concurrently with reclaim. Reclaim itself runs in the
// context of the provided |SequencedTaskRunner|, meaning that the caller must
// take care of this runner being compatible with the various partitions.
//
// Singleton as this runs as long as the process is alive, and
// having multiple instances would be wasteful.
class BASE_EXPORT PartitionAllocMemoryReclaimer {
public:
static PartitionAllocMemoryReclaimer* Instance();
// Internal. Do not use.
// Registers a partition to be tracked by the reclaimer.
void RegisterPartition(internal::PartitionRootBase* partition);
// Internal. Do not use.
// Unregisters a partition to be tracked by the reclaimer.
void UnregisterPartition(internal::PartitionRootBase* partition);
// Starts the periodic reclaim. Should be called once.
void Start(scoped_refptr<SequencedTaskRunner> task_runner);
// Triggers an explicit reclaim now.
void Reclaim();
static constexpr TimeDelta kStatsRecordingTimeDelta =
TimeDelta::FromMinutes(5);
private:
PartitionAllocMemoryReclaimer();
~PartitionAllocMemoryReclaimer();
void ReclaimAndReschedule();
void RecordStatistics();
void ResetForTesting();
// Total time spent in |Reclaim()|.
bool has_called_reclaim_ = false;
TimeDelta total_reclaim_thread_time_;
// Schedules periodic |Reclaim()|.
std::unique_ptr<RepeatingTimer> timer_;
Lock lock_;
std::set<internal::PartitionRootBase*> partitions_ GUARDED_BY(lock_);
friend class NoDestructor<PartitionAllocMemoryReclaimer>;
friend class PartitionAllocMemoryReclaimerTest;
DISALLOW_COPY_AND_ASSIGN(PartitionAllocMemoryReclaimer);
};
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_MEMORY_RECLAIMER_H_

View file

@ -0,0 +1,36 @@
// Copyright (c) 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_H_
#include "base/allocator/partition_allocator/oom_callback.h"
#include "base/logging.h"
#include "base/process/memory.h"
#include "build/build_config.h"
#if defined(OS_WIN)
#include <windows.h>
#endif
namespace {
// The crash is generated in a NOINLINE function so that we can classify the
// crash as an OOM solely by analyzing the stack trace.
NOINLINE void OnNoMemory(size_t size) {
base::internal::RunPartitionAllocOomCallback();
base::internal::OnNoMemoryInternal(size);
IMMEDIATE_CRASH();
}
} // namespace
// OOM_CRASH(size) - Specialization of IMMEDIATE_CRASH which will raise a custom
// exception on Windows to signal this is OOM and not a normal assert.
// OOM_CRASH(size) is called by users of PageAllocator (including
// PartitionAlloc) to signify an allocation failure from the platform.
#define OOM_CRASH(size) \
do { \
OnNoMemory(size); \
} while (0)
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_H_

View file

@ -0,0 +1,27 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/oom_callback.h"
#include "base/logging.h"
namespace base {
namespace {
PartitionAllocOomCallback g_oom_callback;
} // namespace
void SetPartitionAllocOomCallback(PartitionAllocOomCallback callback) {
DCHECK(!g_oom_callback);
g_oom_callback = callback;
}
namespace internal {
void RunPartitionAllocOomCallback() {
if (g_oom_callback)
g_oom_callback();
}
} // namespace internal
} // namespace base

View file

@ -0,0 +1,24 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_CALLBACK_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_CALLBACK_H_
#include "base/base_export.h"
namespace base {
typedef void (*PartitionAllocOomCallback)();
// Registers a callback to be invoked during an OOM_CRASH(). OOM_CRASH is
// invoked by users of PageAllocator (including PartitionAlloc) to signify an
// allocation failure from the platform.
BASE_EXPORT void SetPartitionAllocOomCallback(
PartitionAllocOomCallback callback);
namespace internal {
BASE_EXPORT void RunPartitionAllocOomCallback();
} // namespace internal
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_CALLBACK_H_

View file

@ -0,0 +1,263 @@
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/page_allocator.h"
#include <limits.h>
#include <atomic>
#include "base/allocator/partition_allocator/address_space_randomization.h"
#include "base/allocator/partition_allocator/page_allocator_internal.h"
#include "base/allocator/partition_allocator/spin_lock.h"
#include "base/bits.h"
#include "base/logging.h"
#include "base/no_destructor.h"
#include "base/numerics/checked_math.h"
#include "build/build_config.h"
#if defined(OS_WIN)
#include <windows.h>
#endif
#if defined(OS_WIN)
#include "base/allocator/partition_allocator/page_allocator_internals_win.h"
#elif defined(OS_POSIX)
#include "base/allocator/partition_allocator/page_allocator_internals_posix.h"
#elif defined(OS_FUCHSIA)
#include "base/allocator/partition_allocator/page_allocator_internals_fuchsia.h"
#else
#error Platform not supported.
#endif
namespace base {
namespace {
// We may reserve/release address space on different threads.
subtle::SpinLock& GetReserveLock() {
static NoDestructor<subtle::SpinLock> s_reserveLock;
return *s_reserveLock;
}
// We only support a single block of reserved address space.
void* s_reservation_address = nullptr;
size_t s_reservation_size = 0;
void* AllocPagesIncludingReserved(void* address,
size_t length,
PageAccessibilityConfiguration accessibility,
PageTag page_tag,
bool commit) {
void* ret =
SystemAllocPages(address, length, accessibility, page_tag, commit);
if (ret == nullptr) {
const bool cant_alloc_length = kHintIsAdvisory || address == nullptr;
if (cant_alloc_length) {
// The system cannot allocate |length| bytes. Release any reserved address
// space and try once more.
ReleaseReservation();
ret = SystemAllocPages(address, length, accessibility, page_tag, commit);
}
}
return ret;
}
// Trims |base| to given |trim_length| and |alignment|.
//
// On failure, on Windows, this function returns nullptr and frees |base|.
void* TrimMapping(void* base,
size_t base_length,
size_t trim_length,
uintptr_t alignment,
PageAccessibilityConfiguration accessibility,
bool commit) {
size_t pre_slack = reinterpret_cast<uintptr_t>(base) & (alignment - 1);
if (pre_slack) {
pre_slack = alignment - pre_slack;
}
size_t post_slack = base_length - pre_slack - trim_length;
DCHECK(base_length >= trim_length || pre_slack || post_slack);
DCHECK(pre_slack < base_length);
DCHECK(post_slack < base_length);
return TrimMappingInternal(base, base_length, trim_length, accessibility,
commit, pre_slack, post_slack);
}
} // namespace
void* SystemAllocPages(void* hint,
size_t length,
PageAccessibilityConfiguration accessibility,
PageTag page_tag,
bool commit) {
DCHECK(!(length & kPageAllocationGranularityOffsetMask));
DCHECK(!(reinterpret_cast<uintptr_t>(hint) &
kPageAllocationGranularityOffsetMask));
DCHECK(commit || accessibility == PageInaccessible);
return SystemAllocPagesInternal(hint, length, accessibility, page_tag,
commit);
}
void* AllocPages(void* address,
size_t length,
size_t align,
PageAccessibilityConfiguration accessibility,
PageTag page_tag,
bool commit) {
DCHECK(length >= kPageAllocationGranularity);
DCHECK(!(length & kPageAllocationGranularityOffsetMask));
DCHECK(align >= kPageAllocationGranularity);
// Alignment must be power of 2 for masking math to work.
DCHECK(base::bits::IsPowerOfTwo(align));
DCHECK(!(reinterpret_cast<uintptr_t>(address) &
kPageAllocationGranularityOffsetMask));
uintptr_t align_offset_mask = align - 1;
uintptr_t align_base_mask = ~align_offset_mask;
DCHECK(!(reinterpret_cast<uintptr_t>(address) & align_offset_mask));
// If the client passed null as the address, choose a good one.
if (address == nullptr) {
address = GetRandomPageBase();
address = reinterpret_cast<void*>(reinterpret_cast<uintptr_t>(address) &
align_base_mask);
}
// First try to force an exact-size, aligned allocation from our random base.
#if defined(ARCH_CPU_32_BITS)
// On 32 bit systems, first try one random aligned address, and then try an
// aligned address derived from the value of |ret|.
constexpr int kExactSizeTries = 2;
#else
// On 64 bit systems, try 3 random aligned addresses.
constexpr int kExactSizeTries = 3;
#endif
for (int i = 0; i < kExactSizeTries; ++i) {
void* ret = AllocPagesIncludingReserved(address, length, accessibility,
page_tag, commit);
if (ret != nullptr) {
// If the alignment is to our liking, we're done.
if (!(reinterpret_cast<uintptr_t>(ret) & align_offset_mask))
return ret;
// Free the memory and try again.
FreePages(ret, length);
} else {
// |ret| is null; if this try was unhinted, we're OOM.
if (kHintIsAdvisory || address == nullptr)
return nullptr;
}
#if defined(ARCH_CPU_32_BITS)
// For small address spaces, try the first aligned address >= |ret|. Note
// |ret| may be null, in which case |address| becomes null.
address = reinterpret_cast<void*>(
(reinterpret_cast<uintptr_t>(ret) + align_offset_mask) &
align_base_mask);
#else // defined(ARCH_CPU_64_BITS)
// Keep trying random addresses on systems that have a large address space.
address = GetRandomPageBase();
address = reinterpret_cast<void*>(reinterpret_cast<uintptr_t>(address) &
align_base_mask);
#endif
}
// Make a larger allocation so we can force alignment.
size_t try_length = length + (align - kPageAllocationGranularity);
CHECK(try_length >= length);
void* ret;
do {
// Continue randomizing only on POSIX.
address = kHintIsAdvisory ? GetRandomPageBase() : nullptr;
ret = AllocPagesIncludingReserved(address, try_length, accessibility,
page_tag, commit);
// The retries are for Windows, where a race can steal our mapping on
// resize.
} while (ret != nullptr &&
(ret = TrimMapping(ret, try_length, length, align, accessibility,
commit)) == nullptr);
return ret;
}
void FreePages(void* address, size_t length) {
DCHECK(!(reinterpret_cast<uintptr_t>(address) &
kPageAllocationGranularityOffsetMask));
DCHECK(!(length & kPageAllocationGranularityOffsetMask));
FreePagesInternal(address, length);
}
bool TrySetSystemPagesAccess(void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
DCHECK(!(length & kSystemPageOffsetMask));
return TrySetSystemPagesAccessInternal(address, length, accessibility);
}
void SetSystemPagesAccess(void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
DCHECK(!(length & kSystemPageOffsetMask));
SetSystemPagesAccessInternal(address, length, accessibility);
}
void DecommitSystemPages(void* address, size_t length) {
DCHECK_EQ(0UL, length & kSystemPageOffsetMask);
DecommitSystemPagesInternal(address, length);
}
bool RecommitSystemPages(void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
DCHECK_EQ(0UL, length & kSystemPageOffsetMask);
DCHECK_NE(PageInaccessible, accessibility);
return RecommitSystemPagesInternal(address, length, accessibility);
}
void DiscardSystemPages(void* address, size_t length) {
DCHECK_EQ(0UL, length & kSystemPageOffsetMask);
DiscardSystemPagesInternal(address, length);
}
bool ReserveAddressSpace(size_t size) {
// To avoid deadlock, call only SystemAllocPages.
subtle::SpinLock::Guard guard(GetReserveLock());
if (s_reservation_address == nullptr) {
void* mem = SystemAllocPages(nullptr, size, PageInaccessible,
PageTag::kChromium, false);
if (mem != nullptr) {
// We guarantee this alignment when reserving address space.
DCHECK(!(reinterpret_cast<uintptr_t>(mem) &
kPageAllocationGranularityOffsetMask));
s_reservation_address = mem;
s_reservation_size = size;
return true;
}
}
return false;
}
bool ReleaseReservation() {
// To avoid deadlock, call only FreePages.
subtle::SpinLock::Guard guard(GetReserveLock());
if (!s_reservation_address)
return false;
FreePages(s_reservation_address, s_reservation_size);
s_reservation_address = nullptr;
s_reservation_size = 0;
return true;
}
bool HasReservationForTesting() {
subtle::SpinLock::Guard guard(GetReserveLock());
return s_reservation_address != nullptr;
}
uint32_t GetAllocPageErrorCode() {
return s_allocPageErrorCode;
}
} // namespace base

View file

@ -0,0 +1,198 @@
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_H_
#include <stdint.h>
#include <cstddef>
#include "base/allocator/partition_allocator/page_allocator_constants.h"
#include "base/base_export.h"
#include "base/compiler_specific.h"
#include "build/build_config.h"
namespace base {
enum PageAccessibilityConfiguration {
PageInaccessible,
PageRead,
PageReadWrite,
PageReadExecute,
// This flag is deprecated and will go away soon.
// TODO(bbudge) Remove this as soon as V8 doesn't need RWX pages.
PageReadWriteExecute,
};
// macOS supports tagged memory regions, to help in debugging. On Android,
// these tags are used to name anonymous mappings.
enum class PageTag {
kFirst = 240, // Minimum tag value.
kBlinkGC = 252, // Blink GC pages.
kPartitionAlloc = 253, // PartitionAlloc, no matter the partition.
kChromium = 254, // Chromium page.
kV8 = 255, // V8 heap pages.
kLast = kV8 // Maximum tag value.
};
// Allocate one or more pages.
//
// The requested |address| is just a hint; the actual address returned may
// differ. The returned address will be aligned at least to |align| bytes.
// |length| is in bytes, and must be a multiple of |kPageAllocationGranularity|.
// |align| is in bytes, and must be a power-of-two multiple of
// |kPageAllocationGranularity|.
//
// If |address| is null, then a suitable and randomized address will be chosen
// automatically.
//
// |page_accessibility| controls the permission of the allocated pages.
// |page_tag| is used on some platforms to identify the source of the
// allocation. Use PageTag::kChromium as a catch-all category.
//
// This call will return null if the allocation cannot be satisfied.
BASE_EXPORT void* AllocPages(void* address,
size_t length,
size_t align,
PageAccessibilityConfiguration page_accessibility,
PageTag tag,
bool commit = true);
// Free one or more pages starting at |address| and continuing for |length|
// bytes.
//
// |address| and |length| must match a previous call to |AllocPages|. Therefore,
// |address| must be aligned to |kPageAllocationGranularity| bytes, and |length|
// must be a multiple of |kPageAllocationGranularity|.
BASE_EXPORT void FreePages(void* address, size_t length);
// Mark one or more system pages, starting at |address| with the given
// |page_accessibility|. |length| must be a multiple of |kSystemPageSize| bytes.
//
// Returns true if the permission change succeeded. In most cases you must
// |CHECK| the result.
BASE_EXPORT WARN_UNUSED_RESULT bool TrySetSystemPagesAccess(
void* address,
size_t length,
PageAccessibilityConfiguration page_accessibility);
// Mark one or more system pages, starting at |address| with the given
// |page_accessibility|. |length| must be a multiple of |kSystemPageSize| bytes.
//
// Performs a CHECK that the operation succeeds.
BASE_EXPORT void SetSystemPagesAccess(
void* address,
size_t length,
PageAccessibilityConfiguration page_accessibility);
// Decommit one or more system pages starting at |address| and continuing for
// |length| bytes. |length| must be a multiple of |kSystemPageSize|.
//
// Decommitted means that physical resources (RAM or swap) backing the allocated
// virtual address range are released back to the system, but the address space
// is still allocated to the process (possibly using up page table entries or
// other accounting resources). Any access to a decommitted region of memory
// is an error and will generate a fault.
//
// This operation is not atomic on all platforms.
//
// Note: "Committed memory" is a Windows Memory Subsystem concept that ensures
// processes will not fault when touching a committed memory region. There is
// no analogue in the POSIX memory API where virtual memory pages are
// best-effort allocated resources on the first touch. To create a
// platform-agnostic abstraction, this API simulates the Windows "decommit"
// state by both discarding the region (allowing the OS to avoid swap
// operations) and changing the page protections so accesses fault.
//
// TODO(ajwong): This currently does not change page protections on POSIX
// systems due to a perf regression. Tracked at http://crbug.com/766882.
BASE_EXPORT void DecommitSystemPages(void* address, size_t length);
// Recommit one or more system pages, starting at |address| and continuing for
// |length| bytes with the given |page_accessibility|. |length| must be a
// multiple of |kSystemPageSize|.
//
// Decommitted system pages must be recommitted with their original permissions
// before they are used again.
//
// Returns true if the recommit change succeeded. In most cases you must |CHECK|
// the result.
BASE_EXPORT WARN_UNUSED_RESULT bool RecommitSystemPages(
void* address,
size_t length,
PageAccessibilityConfiguration page_accessibility);
// Discard one or more system pages starting at |address| and continuing for
// |length| bytes. |length| must be a multiple of |kSystemPageSize|.
//
// Discarding is a hint to the system that the page is no longer required. The
// hint may:
// - Do nothing.
// - Discard the page immediately, freeing up physical pages.
// - Discard the page at some time in the future in response to memory
// pressure.
//
// Only committed pages should be discarded. Discarding a page does not decommit
// it, and it is valid to discard an already-discarded page. A read or write to
// a discarded page will not fault.
//
// Reading from a discarded page may return the original page content, or a page
// full of zeroes.
//
// Writing to a discarded page is the only guaranteed way to tell the system
// that the page is required again. Once written to, the content of the page is
// guaranteed stable once more. After being written to, the page content may be
// based on the original page content, or a page of zeroes.
BASE_EXPORT void DiscardSystemPages(void* address, size_t length);
// Rounds up |address| to the next multiple of |kSystemPageSize|. Returns
// 0 for an |address| of 0.
constexpr ALWAYS_INLINE uintptr_t RoundUpToSystemPage(uintptr_t address) {
return (address + kSystemPageOffsetMask) & kSystemPageBaseMask;
}
// Rounds down |address| to the previous multiple of |kSystemPageSize|. Returns
// 0 for an |address| of 0.
constexpr ALWAYS_INLINE uintptr_t RoundDownToSystemPage(uintptr_t address) {
return address & kSystemPageBaseMask;
}
// Rounds up |address| to the next multiple of |kPageAllocationGranularity|.
// Returns 0 for an |address| of 0.
constexpr ALWAYS_INLINE uintptr_t
RoundUpToPageAllocationGranularity(uintptr_t address) {
return (address + kPageAllocationGranularityOffsetMask) &
kPageAllocationGranularityBaseMask;
}
// Rounds down |address| to the previous multiple of
// |kPageAllocationGranularity|. Returns 0 for an |address| of 0.
constexpr ALWAYS_INLINE uintptr_t
RoundDownToPageAllocationGranularity(uintptr_t address) {
return address & kPageAllocationGranularityBaseMask;
}
// Reserves (at least) |size| bytes of address space, aligned to
// |kPageAllocationGranularity|. This can be called early on to make it more
// likely that large allocations will succeed. Returns true if the reservation
// succeeded, false if the reservation failed or a reservation was already made.
BASE_EXPORT bool ReserveAddressSpace(size_t size);
// Releases any reserved address space. |AllocPages| calls this automatically on
// an allocation failure. External allocators may also call this on failure.
//
// Returns true when an existing reservation was released.
BASE_EXPORT bool ReleaseReservation();
// Returns true if there is currently an address space reservation.
BASE_EXPORT bool HasReservationForTesting();
// Returns |errno| (POSIX) or the result of |GetLastError| (Windows) when |mmap|
// (POSIX) or |VirtualAlloc| (Windows) fails.
BASE_EXPORT uint32_t GetAllocPageErrorCode();
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_H_

View file

@ -0,0 +1,48 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_CONSTANTS_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_CONSTANTS_H_
#include <stddef.h>
#include "build/build_config.h"
namespace base {
#if defined(OS_WIN) || defined(ARCH_CPU_PPC64)
static constexpr size_t kPageAllocationGranularityShift = 16; // 64KB
#elif defined(_MIPS_ARCH_LOONGSON)
static constexpr size_t kPageAllocationGranularityShift = 14; // 16KB
#else
static constexpr size_t kPageAllocationGranularityShift = 12; // 4KB
#endif
static constexpr size_t kPageAllocationGranularity =
1 << kPageAllocationGranularityShift;
static constexpr size_t kPageAllocationGranularityOffsetMask =
kPageAllocationGranularity - 1;
static constexpr size_t kPageAllocationGranularityBaseMask =
~kPageAllocationGranularityOffsetMask;
#if defined(_MIPS_ARCH_LOONGSON)
static constexpr size_t kSystemPageSize = 16384;
#elif defined(ARCH_CPU_PPC64)
// Modern ppc64 systems support 4KB and 64KB page sizes.
// Since 64KB is the de-facto standard on the platform
// and binaries compiled for 64KB are likely to work on 4KB systems,
// 64KB is a good choice here.
static constexpr size_t kSystemPageSize = 65536;
#else
static constexpr size_t kSystemPageSize = 4096;
#endif
static constexpr size_t kSystemPageOffsetMask = kSystemPageSize - 1;
static_assert((kSystemPageSize & (kSystemPageSize - 1)) == 0,
"kSystemPageSize must be power of 2");
static constexpr size_t kSystemPageBaseMask = ~kSystemPageOffsetMask;
static constexpr size_t kPageMetadataShift = 5; // 32 bytes per partition page.
static constexpr size_t kPageMetadataSize = 1 << kPageMetadataShift;
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_CONSTANTS_H_

View file

@ -0,0 +1,18 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNAL_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNAL_H_
namespace base {
void* SystemAllocPages(void* hint,
size_t length,
PageAccessibilityConfiguration accessibility,
PageTag page_tag,
bool commit);
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNAL_H_

View file

@ -0,0 +1,202 @@
// Copyright 2019 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
//
// This file implements memory allocation primitives for PageAllocator using
// Fuchsia's VMOs (Virtual Memory Objects). VMO API is documented in
// https://fuchsia.dev/fuchsia-src/zircon/objects/vm_object . A VMO is a kernel
// object that corresponds to a set of memory pages. VMO pages may be mapped
// to an address space. The code below creates VMOs for each memory allocations
// and maps them to the default address space of the current process.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_FUCHSIA_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_FUCHSIA_H_
#include <lib/zx/vmar.h>
#include <lib/zx/vmo.h>
#include "base/allocator/partition_allocator/page_allocator.h"
#include "base/fuchsia/fuchsia_logging.h"
#include "base/logging.h"
namespace base {
namespace {
// Returns VMO name for a PageTag.
const char* PageTagToName(PageTag tag) {
switch (tag) {
case PageTag::kBlinkGC:
return "cr_blink_gc";
case PageTag::kPartitionAlloc:
return "cr_partition_alloc";
case PageTag::kChromium:
return "cr_chromium";
case PageTag::kV8:
return "cr_v8";
default:
DCHECK(false);
return "";
}
}
zx_vm_option_t PageAccessibilityToZxVmOptions(
PageAccessibilityConfiguration accessibility) {
switch (accessibility) {
case PageRead:
return ZX_VM_PERM_READ;
case PageReadWrite:
return ZX_VM_PERM_READ | ZX_VM_PERM_WRITE;
case PageReadExecute:
return ZX_VM_PERM_READ | ZX_VM_PERM_EXECUTE;
case PageReadWriteExecute:
return ZX_VM_PERM_READ | ZX_VM_PERM_WRITE | ZX_VM_PERM_EXECUTE;
default:
NOTREACHED();
FALLTHROUGH;
case PageInaccessible:
return 0;
}
}
} // namespace
// zx_vmar_map() will fail if the VMO cannot be mapped at |vmar_offset|, i.e.
// |hint| is not advisory.
constexpr bool kHintIsAdvisory = false;
std::atomic<int32_t> s_allocPageErrorCode{0};
void* SystemAllocPagesInternal(void* hint,
size_t length,
PageAccessibilityConfiguration accessibility,
PageTag page_tag,
bool commit) {
zx::vmo vmo;
zx_status_t status = zx::vmo::create(length, 0, &vmo);
if (status != ZX_OK) {
ZX_DLOG(INFO, status) << "zx_vmo_create";
return nullptr;
}
const char* vmo_name = PageTagToName(page_tag);
status = vmo.set_property(ZX_PROP_NAME, vmo_name, strlen(vmo_name));
// VMO names are used only for debugging, so failure to set a name is not
// fatal.
ZX_DCHECK(status == ZX_OK, status);
if (page_tag == PageTag::kV8) {
// V8 uses JIT. Call zx_vmo_replace_as_executable() to allow code execution
// in the new VMO.
status = vmo.replace_as_executable(zx::resource(), &vmo);
if (status != ZX_OK) {
ZX_DLOG(INFO, status) << "zx_vmo_replace_as_executable";
return nullptr;
}
}
zx_vm_option_t options = PageAccessibilityToZxVmOptions(accessibility);
uint64_t vmar_offset = 0;
if (hint) {
vmar_offset = reinterpret_cast<uint64_t>(hint);
options |= ZX_VM_SPECIFIC;
}
uint64_t address;
status =
zx::vmar::root_self()->map(vmar_offset, vmo,
/*vmo_offset=*/0, length, options, &address);
if (status != ZX_OK) {
// map() is expected to fail if |hint| is set to an already-in-use location.
if (!hint) {
ZX_DLOG(ERROR, status) << "zx_vmar_map";
}
return nullptr;
}
return reinterpret_cast<void*>(address);
}
void* TrimMappingInternal(void* base,
size_t base_length,
size_t trim_length,
PageAccessibilityConfiguration accessibility,
bool commit,
size_t pre_slack,
size_t post_slack) {
DCHECK_EQ(base_length, trim_length + pre_slack + post_slack);
uint64_t base_address = reinterpret_cast<uint64_t>(base);
// Unmap head if necessary.
if (pre_slack) {
zx_status_t status = zx::vmar::root_self()->unmap(base_address, pre_slack);
ZX_CHECK(status == ZX_OK, status);
}
// Unmap tail if necessary.
if (post_slack) {
zx_status_t status = zx::vmar::root_self()->unmap(
base_address + pre_slack + trim_length, post_slack);
ZX_CHECK(status == ZX_OK, status);
}
return reinterpret_cast<void*>(base_address + pre_slack);
}
bool TrySetSystemPagesAccessInternal(
void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
zx_status_t status = zx::vmar::root_self()->protect(
reinterpret_cast<uint64_t>(address), length,
PageAccessibilityToZxVmOptions(accessibility));
return status == ZX_OK;
}
void SetSystemPagesAccessInternal(
void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
zx_status_t status = zx::vmar::root_self()->protect(
reinterpret_cast<uint64_t>(address), length,
PageAccessibilityToZxVmOptions(accessibility));
ZX_CHECK(status == ZX_OK, status);
}
void FreePagesInternal(void* address, size_t length) {
uint64_t address_int = reinterpret_cast<uint64_t>(address);
zx_status_t status = zx::vmar::root_self()->unmap(address_int, length);
ZX_CHECK(status == ZX_OK, status);
}
void DiscardSystemPagesInternal(void* address, size_t length) {
// TODO(https://crbug.com/1022062): Mark pages as discardable, rather than
// forcibly de-committing them immediately, when Fuchsia supports it.
uint64_t address_int = reinterpret_cast<uint64_t>(address);
zx_status_t status = zx::vmar::root_self()->op_range(
ZX_VMO_OP_DECOMMIT, address_int, length, nullptr, 0);
ZX_CHECK(status == ZX_OK, status);
}
void DecommitSystemPagesInternal(void* address, size_t length) {
// TODO(https://crbug.com/1022062): Review whether this implementation is
// still appropriate once DiscardSystemPagesInternal() migrates to a "lazy"
// discardable API.
DiscardSystemPagesInternal(address, length);
SetSystemPagesAccessInternal(address, length, PageInaccessible);
}
bool RecommitSystemPagesInternal(void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
SetSystemPagesAccessInternal(address, length, accessibility);
return true;
}
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_FUCHSIA_H_

View file

@ -0,0 +1,244 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_POSIX_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_POSIX_H_
#include <errno.h>
#include <sys/mman.h>
#include "base/logging.h"
#include "build/build_config.h"
#if defined(OS_MACOSX)
#include "base/mac/foundation_util.h"
#include "base/mac/mac_util.h"
#include "base/mac/scoped_cftyperef.h"
#include <Security/Security.h>
#include <mach/mach.h>
#endif
#if defined(OS_ANDROID)
#include <sys/prctl.h>
#endif
#if defined(OS_LINUX)
#include <sys/resource.h>
#include <algorithm>
#endif
#include "base/allocator/partition_allocator/page_allocator.h"
#ifndef MAP_ANONYMOUS
#define MAP_ANONYMOUS MAP_ANON
#endif
namespace base {
namespace {
#if defined(OS_ANDROID)
const char* PageTagToName(PageTag tag) {
// Important: All the names should be string literals. As per prctl.h in
// //third_party/android_ndk the kernel keeps a pointer to the name instead
// of copying it.
//
// Having the name in .rodata ensures that the pointer remains valid as
// long as the mapping is alive.
switch (tag) {
case PageTag::kBlinkGC:
return "blink_gc";
case PageTag::kPartitionAlloc:
return "partition_alloc";
case PageTag::kChromium:
return "chromium";
case PageTag::kV8:
return "v8";
default:
DCHECK(false);
return "";
}
}
#endif // defined(OS_ANDROID)
#if defined(OS_MACOSX)
// Tests whether the version of macOS supports the MAP_JIT flag and if the
// current process is signed with the allow-jit entitlement.
bool UseMapJit() {
if (!mac::IsAtLeastOS10_14())
return false;
ScopedCFTypeRef<SecTaskRef> task(SecTaskCreateFromSelf(kCFAllocatorDefault));
ScopedCFTypeRef<CFErrorRef> error;
ScopedCFTypeRef<CFTypeRef> value(SecTaskCopyValueForEntitlement(
task.get(), CFSTR("com.apple.security.cs.allow-jit"),
error.InitializeInto()));
if (error)
return false;
return mac::CFCast<CFBooleanRef>(value.get()) == kCFBooleanTrue;
}
#endif // defined(OS_MACOSX)
} // namespace
// |mmap| uses a nearby address if the hint address is blocked.
constexpr bool kHintIsAdvisory = true;
std::atomic<int32_t> s_allocPageErrorCode{0};
int GetAccessFlags(PageAccessibilityConfiguration accessibility) {
switch (accessibility) {
case PageRead:
return PROT_READ;
case PageReadWrite:
return PROT_READ | PROT_WRITE;
case PageReadExecute:
return PROT_READ | PROT_EXEC;
case PageReadWriteExecute:
return PROT_READ | PROT_WRITE | PROT_EXEC;
default:
NOTREACHED();
FALLTHROUGH;
case PageInaccessible:
return PROT_NONE;
}
}
void* SystemAllocPagesInternal(void* hint,
size_t length,
PageAccessibilityConfiguration accessibility,
PageTag page_tag,
bool commit) {
#if defined(OS_MACOSX)
// Use a custom tag to make it easier to distinguish Partition Alloc regions
// in vmmap(1). Tags between 240-255 are supported.
DCHECK_LE(PageTag::kFirst, page_tag);
DCHECK_GE(PageTag::kLast, page_tag);
int fd = VM_MAKE_TAG(static_cast<int>(page_tag));
#else
int fd = -1;
#endif
int access_flag = GetAccessFlags(accessibility);
int map_flags = MAP_ANONYMOUS | MAP_PRIVATE;
#if defined(OS_MACOSX)
// On macOS 10.14 and higher, executables that are code signed with the
// "runtime" option cannot execute writable memory by default. They can opt
// into this capability by specifying the "com.apple.security.cs.allow-jit"
// code signing entitlement and allocating the region with the MAP_JIT flag.
static const bool kUseMapJit = UseMapJit();
if (page_tag == PageTag::kV8 && kUseMapJit) {
map_flags |= MAP_JIT;
}
#endif
void* ret =
mmap(hint, length, access_flag, map_flags, fd, 0);
if (ret == MAP_FAILED) {
s_allocPageErrorCode = errno;
ret = nullptr;
}
#if defined(OS_ANDROID)
// On Android, anonymous mappings can have a name attached to them. This is
// useful for debugging, and double-checking memory attribution.
if (ret) {
// No error checking on purpose, testing only.
prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, ret, length,
PageTagToName(page_tag));
}
#endif
return ret;
}
void* TrimMappingInternal(void* base,
size_t base_length,
size_t trim_length,
PageAccessibilityConfiguration accessibility,
bool commit,
size_t pre_slack,
size_t post_slack) {
void* ret = base;
// We can resize the allocation run. Release unneeded memory before and after
// the aligned range.
if (pre_slack) {
int res = munmap(base, pre_slack);
CHECK(!res);
ret = reinterpret_cast<char*>(base) + pre_slack;
}
if (post_slack) {
int res = munmap(reinterpret_cast<char*>(ret) + trim_length, post_slack);
CHECK(!res);
}
return ret;
}
bool TrySetSystemPagesAccessInternal(
void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
return 0 == mprotect(address, length, GetAccessFlags(accessibility));
}
void SetSystemPagesAccessInternal(
void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
CHECK_EQ(0, mprotect(address, length, GetAccessFlags(accessibility)));
}
void FreePagesInternal(void* address, size_t length) {
CHECK(!munmap(address, length));
}
void DecommitSystemPagesInternal(void* address, size_t length) {
// In POSIX, there is no decommit concept. Discarding is an effective way of
// implementing the Windows semantics where the OS is allowed to not swap the
// pages in the region.
//
// TODO(ajwong): Also explore setting PageInaccessible to make the protection
// semantics consistent between Windows and POSIX. This might have a perf cost
// though as both decommit and recommit would incur an extra syscall.
// http://crbug.com/766882
DiscardSystemPages(address, length);
}
bool RecommitSystemPagesInternal(void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
#if defined(OS_MACOSX)
// On macOS, to update accounting, we need to make another syscall. For more
// details, see https://crbug.com/823915.
madvise(address, length, MADV_FREE_REUSE);
#endif
// On POSIX systems, the caller need simply read the memory to recommit it.
// This has the correct behavior because the API requires the permissions to
// be the same as before decommitting and all configurations can read.
return true;
}
void DiscardSystemPagesInternal(void* address, size_t length) {
#if defined(OS_MACOSX)
int ret = madvise(address, length, MADV_FREE_REUSABLE);
if (ret) {
// MADV_FREE_REUSABLE sometimes fails, so fall back to MADV_DONTNEED.
ret = madvise(address, length, MADV_DONTNEED);
}
CHECK(0 == ret);
#else
// We have experimented with other flags, but with suboptimal results.
//
// MADV_FREE (Linux): Makes our memory measurements less predictable;
// performance benefits unclear.
//
// Therefore, we just do the simple thing: MADV_DONTNEED.
CHECK(!madvise(address, length, MADV_DONTNEED));
#endif
}
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_POSIX_H_

View file

@ -0,0 +1,144 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_WIN_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_WIN_H_
#include "base/allocator/partition_allocator/oom.h"
#include "base/allocator/partition_allocator/page_allocator_internal.h"
#include "base/logging.h"
namespace base {
// |VirtualAlloc| will fail if allocation at the hint address is blocked.
constexpr bool kHintIsAdvisory = false;
std::atomic<int32_t> s_allocPageErrorCode{ERROR_SUCCESS};
int GetAccessFlags(PageAccessibilityConfiguration accessibility) {
switch (accessibility) {
case PageRead:
return PAGE_READONLY;
case PageReadWrite:
return PAGE_READWRITE;
case PageReadExecute:
return PAGE_EXECUTE_READ;
case PageReadWriteExecute:
return PAGE_EXECUTE_READWRITE;
default:
NOTREACHED();
FALLTHROUGH;
case PageInaccessible:
return PAGE_NOACCESS;
}
}
void* SystemAllocPagesInternal(void* hint,
size_t length,
PageAccessibilityConfiguration accessibility,
PageTag page_tag,
bool commit) {
DWORD access_flag = GetAccessFlags(accessibility);
const DWORD type_flags = commit ? (MEM_RESERVE | MEM_COMMIT) : MEM_RESERVE;
void* ret = VirtualAlloc(hint, length, type_flags, access_flag);
if (ret == nullptr) {
s_allocPageErrorCode = GetLastError();
}
return ret;
}
void* TrimMappingInternal(void* base,
size_t base_length,
size_t trim_length,
PageAccessibilityConfiguration accessibility,
bool commit,
size_t pre_slack,
size_t post_slack) {
void* ret = base;
if (pre_slack || post_slack) {
// We cannot resize the allocation run. Free it and retry at the aligned
// address within the freed range.
ret = reinterpret_cast<char*>(base) + pre_slack;
FreePages(base, base_length);
ret = SystemAllocPages(ret, trim_length, accessibility, PageTag::kChromium,
commit);
}
return ret;
}
bool TrySetSystemPagesAccessInternal(
void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
if (accessibility == PageInaccessible)
return VirtualFree(address, length, MEM_DECOMMIT) != 0;
return nullptr != VirtualAlloc(address, length, MEM_COMMIT,
GetAccessFlags(accessibility));
}
void SetSystemPagesAccessInternal(
void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
if (accessibility == PageInaccessible) {
if (!VirtualFree(address, length, MEM_DECOMMIT)) {
// We check `GetLastError` for `ERROR_SUCCESS` here so that in a crash
// report we get the error number.
CHECK_EQ(static_cast<uint32_t>(ERROR_SUCCESS), GetLastError());
}
} else {
if (!VirtualAlloc(address, length, MEM_COMMIT,
GetAccessFlags(accessibility))) {
int32_t error = GetLastError();
if (error == ERROR_COMMITMENT_LIMIT)
OOM_CRASH(length);
// We check `GetLastError` for `ERROR_SUCCESS` here so that in a crash
// report we get the error number.
CHECK_EQ(ERROR_SUCCESS, error);
}
}
}
void FreePagesInternal(void* address, size_t length) {
CHECK(VirtualFree(address, 0, MEM_RELEASE));
}
void DecommitSystemPagesInternal(void* address, size_t length) {
SetSystemPagesAccess(address, length, PageInaccessible);
}
bool RecommitSystemPagesInternal(void* address,
size_t length,
PageAccessibilityConfiguration accessibility) {
return TrySetSystemPagesAccess(address, length, accessibility);
}
void DiscardSystemPagesInternal(void* address, size_t length) {
// On Windows, discarded pages are not returned to the system immediately and
// not guaranteed to be zeroed when returned to the application.
using DiscardVirtualMemoryFunction =
DWORD(WINAPI*)(PVOID virtualAddress, SIZE_T size);
static DiscardVirtualMemoryFunction discard_virtual_memory =
reinterpret_cast<DiscardVirtualMemoryFunction>(-1);
if (discard_virtual_memory ==
reinterpret_cast<DiscardVirtualMemoryFunction>(-1))
discard_virtual_memory =
reinterpret_cast<DiscardVirtualMemoryFunction>(GetProcAddress(
GetModuleHandle(L"Kernel32.dll"), "DiscardVirtualMemory"));
// Use DiscardVirtualMemory when available because it releases faster than
// MEM_RESET.
DWORD ret = 1;
if (discard_virtual_memory) {
ret = discard_virtual_memory(address, length);
}
// DiscardVirtualMemory is buggy in Win10 SP0, so fall back to MEM_RESET on
// failure.
if (ret) {
void* ptr = VirtualAlloc(address, length, MEM_RESET, PAGE_READWRITE);
CHECK(ptr);
}
}
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_WIN_H_

View file

@ -0,0 +1,849 @@
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/partition_alloc.h"
#include <string.h>
#include <memory>
#include <type_traits>
#include "base/allocator/partition_allocator/partition_direct_map_extent.h"
#include "base/allocator/partition_allocator/partition_oom.h"
#include "base/allocator/partition_allocator/partition_page.h"
#include "base/allocator/partition_allocator/spin_lock.h"
#include "base/logging.h"
#include "base/no_destructor.h"
#include "base/synchronization/lock.h"
namespace base {
// Two partition pages are used as guard / metadata page so make sure the super
// page size is bigger.
static_assert(kPartitionPageSize * 4 <= kSuperPageSize, "ok super page size");
static_assert(!(kSuperPageSize % kPartitionPageSize), "ok super page multiple");
// Four system pages gives us room to hack out a still-guard-paged piece
// of metadata in the middle of a guard partition page.
static_assert(kSystemPageSize * 4 <= kPartitionPageSize,
"ok partition page size");
static_assert(!(kPartitionPageSize % kSystemPageSize),
"ok partition page multiple");
static_assert(sizeof(internal::PartitionPage) <= kPageMetadataSize,
"PartitionPage should not be too big");
static_assert(sizeof(internal::PartitionBucket) <= kPageMetadataSize,
"PartitionBucket should not be too big");
static_assert(sizeof(internal::PartitionSuperPageExtentEntry) <=
kPageMetadataSize,
"PartitionSuperPageExtentEntry should not be too big");
static_assert(kPageMetadataSize * kNumPartitionPagesPerSuperPage <=
kSystemPageSize,
"page metadata fits in hole");
// Limit to prevent callers accidentally overflowing an int size.
static_assert(kGenericMaxDirectMapped <=
(1UL << 31) + kPageAllocationGranularity,
"maximum direct mapped allocation");
// Check that some of our zanier calculations worked out as expected.
static_assert(kGenericSmallestBucket == 8, "generic smallest bucket");
static_assert(kGenericMaxBucketed == 983040, "generic max bucketed");
static_assert(kMaxSystemPagesPerSlotSpan < (1 << 8),
"System pages per slot span must be less than 128.");
internal::PartitionRootBase::PartitionRootBase() = default;
internal::PartitionRootBase::~PartitionRootBase() = default;
PartitionRoot::PartitionRoot() = default;
PartitionRoot::~PartitionRoot() = default;
PartitionRootGeneric::PartitionRootGeneric() = default;
PartitionRootGeneric::~PartitionRootGeneric() = default;
PartitionAllocatorGeneric::PartitionAllocatorGeneric() = default;
Lock& GetLock() {
static NoDestructor<Lock> s_initialized_lock;
return *s_initialized_lock;
}
static bool g_initialized = false;
Lock& GetHooksLock() {
static NoDestructor<Lock> lock;
return *lock;
}
OomFunction internal::PartitionRootBase::g_oom_handling_function = nullptr;
std::atomic<bool> PartitionAllocHooks::hooks_enabled_(false);
std::atomic<PartitionAllocHooks::AllocationObserverHook*>
PartitionAllocHooks::allocation_observer_hook_(nullptr);
std::atomic<PartitionAllocHooks::FreeObserverHook*>
PartitionAllocHooks::free_observer_hook_(nullptr);
std::atomic<PartitionAllocHooks::AllocationOverrideHook*>
PartitionAllocHooks::allocation_override_hook_(nullptr);
std::atomic<PartitionAllocHooks::FreeOverrideHook*>
PartitionAllocHooks::free_override_hook_(nullptr);
std::atomic<PartitionAllocHooks::ReallocOverrideHook*>
PartitionAllocHooks::realloc_override_hook_(nullptr);
void PartitionAllocHooks::SetObserverHooks(AllocationObserverHook* alloc_hook,
FreeObserverHook* free_hook) {
AutoLock guard(GetHooksLock());
// Chained hooks are not supported. Registering a non-null hook when a
// non-null hook is already registered indicates somebody is trying to
// overwrite a hook.
CHECK((!allocation_observer_hook_ && !free_observer_hook_) ||
(!alloc_hook && !free_hook))
<< "Overwriting already set observer hooks";
allocation_observer_hook_ = alloc_hook;
free_observer_hook_ = free_hook;
hooks_enabled_ = allocation_observer_hook_ || allocation_override_hook_;
}
void PartitionAllocHooks::SetOverrideHooks(AllocationOverrideHook* alloc_hook,
FreeOverrideHook* free_hook,
ReallocOverrideHook realloc_hook) {
AutoLock guard(GetHooksLock());
CHECK((!allocation_override_hook_ && !free_override_hook_ &&
!realloc_override_hook_) ||
(!alloc_hook && !free_hook && !realloc_hook))
<< "Overwriting already set override hooks";
allocation_override_hook_ = alloc_hook;
free_override_hook_ = free_hook;
realloc_override_hook_ = realloc_hook;
hooks_enabled_ = allocation_observer_hook_ || allocation_override_hook_;
}
void PartitionAllocHooks::AllocationObserverHookIfEnabled(
void* address,
size_t size,
const char* type_name) {
if (AllocationObserverHook* hook =
allocation_observer_hook_.load(std::memory_order_relaxed)) {
hook(address, size, type_name);
}
}
bool PartitionAllocHooks::AllocationOverrideHookIfEnabled(
void** out,
int flags,
size_t size,
const char* type_name) {
if (AllocationOverrideHook* hook =
allocation_override_hook_.load(std::memory_order_relaxed)) {
return hook(out, flags, size, type_name);
}
return false;
}
void PartitionAllocHooks::FreeObserverHookIfEnabled(void* address) {
if (FreeObserverHook* hook =
free_observer_hook_.load(std::memory_order_relaxed)) {
hook(address);
}
}
bool PartitionAllocHooks::FreeOverrideHookIfEnabled(void* address) {
if (FreeOverrideHook* hook =
free_override_hook_.load(std::memory_order_relaxed)) {
return hook(address);
}
return false;
}
void PartitionAllocHooks::ReallocObserverHookIfEnabled(void* old_address,
void* new_address,
size_t size,
const char* type_name) {
// Report a reallocation as a free followed by an allocation.
AllocationObserverHook* allocation_hook =
allocation_observer_hook_.load(std::memory_order_relaxed);
FreeObserverHook* free_hook =
free_observer_hook_.load(std::memory_order_relaxed);
if (allocation_hook && free_hook) {
free_hook(old_address);
allocation_hook(new_address, size, type_name);
}
}
bool PartitionAllocHooks::ReallocOverrideHookIfEnabled(size_t* out,
void* address) {
if (ReallocOverrideHook* hook =
realloc_override_hook_.load(std::memory_order_relaxed)) {
return hook(out, address);
}
return false;
}
static void PartitionAllocBaseInit(internal::PartitionRootBase* root) {
DCHECK(!root->initialized);
{
AutoLock guard(GetLock());
if (!g_initialized) {
g_initialized = true;
// We mark the sentinel bucket/page as free to make sure it is skipped by
// our logic to find a new active page.
internal::PartitionBucket::get_sentinel_bucket()->active_pages_head =
internal::PartitionPage::get_sentinel_page();
}
}
root->initialized = true;
// This is a "magic" value so we can test if a root pointer is valid.
root->inverted_self = ~reinterpret_cast<uintptr_t>(root);
}
void PartitionAllocGlobalInit(OomFunction on_out_of_memory) {
DCHECK(on_out_of_memory);
internal::PartitionRootBase::g_oom_handling_function = on_out_of_memory;
}
void PartitionRoot::Init(size_t bucket_count, size_t maximum_allocation) {
PartitionAllocBaseInit(this);
num_buckets = bucket_count;
max_allocation = maximum_allocation;
for (size_t i = 0; i < num_buckets; ++i) {
internal::PartitionBucket& bucket = buckets()[i];
bucket.Init(i == 0 ? kAllocationGranularity : (i << kBucketShift));
}
}
void PartitionRootGeneric::Init() {
subtle::SpinLock::Guard guard(lock);
PartitionAllocBaseInit(this);
// Precalculate some shift and mask constants used in the hot path.
// Example: malloc(41) == 101001 binary.
// Order is 6 (1 << 6-1) == 32 is highest bit set.
// order_index is the next three MSB == 010 == 2.
// sub_order_index_mask is a mask for the remaining bits == 11 (masking to 01
// for
// the sub_order_index).
size_t order;
for (order = 0; order <= kBitsPerSizeT; ++order) {
size_t order_index_shift;
if (order < kGenericNumBucketsPerOrderBits + 1)
order_index_shift = 0;
else
order_index_shift = order - (kGenericNumBucketsPerOrderBits + 1);
order_index_shifts[order] = order_index_shift;
size_t sub_order_index_mask;
if (order == kBitsPerSizeT) {
// This avoids invoking undefined behavior for an excessive shift.
sub_order_index_mask =
static_cast<size_t>(-1) >> (kGenericNumBucketsPerOrderBits + 1);
} else {
sub_order_index_mask = ((static_cast<size_t>(1) << order) - 1) >>
(kGenericNumBucketsPerOrderBits + 1);
}
order_sub_index_masks[order] = sub_order_index_mask;
}
// Set up the actual usable buckets first.
// Note that typical values (i.e. min allocation size of 8) will result in
// pseudo buckets (size==9 etc. or more generally, size is not a multiple
// of the smallest allocation granularity).
// We avoid them in the bucket lookup map, but we tolerate them to keep the
// code simpler and the structures more generic.
size_t i, j;
size_t current_size = kGenericSmallestBucket;
size_t current_increment =
kGenericSmallestBucket >> kGenericNumBucketsPerOrderBits;
internal::PartitionBucket* bucket = &buckets[0];
for (i = 0; i < kGenericNumBucketedOrders; ++i) {
for (j = 0; j < kGenericNumBucketsPerOrder; ++j) {
bucket->Init(current_size);
// Disable psuedo buckets so that touching them faults.
if (current_size % kGenericSmallestBucket)
bucket->active_pages_head = nullptr;
current_size += current_increment;
++bucket;
}
current_increment <<= 1;
}
DCHECK(current_size == 1 << kGenericMaxBucketedOrder);
DCHECK(bucket == &buckets[0] + kGenericNumBuckets);
// Then set up the fast size -> bucket lookup table.
bucket = &buckets[0];
internal::PartitionBucket** bucket_ptr = &bucket_lookups[0];
for (order = 0; order <= kBitsPerSizeT; ++order) {
for (j = 0; j < kGenericNumBucketsPerOrder; ++j) {
if (order < kGenericMinBucketedOrder) {
// Use the bucket of the finest granularity for malloc(0) etc.
*bucket_ptr++ = &buckets[0];
} else if (order > kGenericMaxBucketedOrder) {
*bucket_ptr++ = internal::PartitionBucket::get_sentinel_bucket();
} else {
internal::PartitionBucket* valid_bucket = bucket;
// Skip over invalid buckets.
while (valid_bucket->slot_size % kGenericSmallestBucket)
valid_bucket++;
*bucket_ptr++ = valid_bucket;
bucket++;
}
}
}
DCHECK(bucket == &buckets[0] + kGenericNumBuckets);
DCHECK(bucket_ptr == &bucket_lookups[0] +
((kBitsPerSizeT + 1) * kGenericNumBucketsPerOrder));
// And there's one last bucket lookup that will be hit for e.g. malloc(-1),
// which tries to overflow to a non-existant order.
*bucket_ptr = internal::PartitionBucket::get_sentinel_bucket();
}
bool PartitionReallocDirectMappedInPlace(PartitionRootGeneric* root,
internal::PartitionPage* page,
size_t raw_size) {
DCHECK(page->bucket->is_direct_mapped());
raw_size = internal::PartitionCookieSizeAdjustAdd(raw_size);
// Note that the new size might be a bucketed size; this function is called
// whenever we're reallocating a direct mapped allocation.
size_t new_size = internal::PartitionBucket::get_direct_map_size(raw_size);
if (new_size < kGenericMinDirectMappedDownsize)
return false;
// bucket->slot_size is the current size of the allocation.
size_t current_size = page->bucket->slot_size;
char* char_ptr = static_cast<char*>(internal::PartitionPage::ToPointer(page));
if (new_size == current_size) {
// No need to move any memory around, but update size and cookie below.
} else if (new_size < current_size) {
size_t map_size =
internal::PartitionDirectMapExtent::FromPage(page)->map_size;
// Don't reallocate in-place if new size is less than 80 % of the full
// map size, to avoid holding on to too much unused address space.
if ((new_size / kSystemPageSize) * 5 < (map_size / kSystemPageSize) * 4)
return false;
// Shrink by decommitting unneeded pages and making them inaccessible.
size_t decommit_size = current_size - new_size;
root->DecommitSystemPages(char_ptr + new_size, decommit_size);
SetSystemPagesAccess(char_ptr + new_size, decommit_size, PageInaccessible);
} else if (new_size <=
internal::PartitionDirectMapExtent::FromPage(page)->map_size) {
// Grow within the actually allocated memory. Just need to make the
// pages accessible again.
size_t recommit_size = new_size - current_size;
SetSystemPagesAccess(char_ptr + current_size, recommit_size, PageReadWrite);
root->RecommitSystemPages(char_ptr + current_size, recommit_size);
#if DCHECK_IS_ON()
memset(char_ptr + current_size, kUninitializedByte, recommit_size);
#endif
} else {
// We can't perform the realloc in-place.
// TODO: support this too when possible.
return false;
}
#if DCHECK_IS_ON()
// Write a new trailing cookie.
internal::PartitionCookieWriteValue(char_ptr + raw_size -
internal::kCookieSize);
#endif
page->set_raw_size(raw_size);
DCHECK(page->get_raw_size() == raw_size);
page->bucket->slot_size = new_size;
return true;
}
void* PartitionReallocGenericFlags(PartitionRootGeneric* root,
int flags,
void* ptr,
size_t new_size,
const char* type_name) {
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
CHECK_MAX_SIZE_OR_RETURN_NULLPTR(new_size, flags);
void* result = realloc(ptr, new_size);
CHECK(result || flags & PartitionAllocReturnNull);
return result;
#else
if (UNLIKELY(!ptr))
return PartitionAllocGenericFlags(root, flags, new_size, type_name);
if (UNLIKELY(!new_size)) {
root->Free(ptr);
return nullptr;
}
if (new_size > kGenericMaxDirectMapped) {
if (flags & PartitionAllocReturnNull)
return nullptr;
internal::PartitionExcessiveAllocationSize(new_size);
}
const bool hooks_enabled = PartitionAllocHooks::AreHooksEnabled();
bool overridden = false;
size_t actual_old_size;
if (UNLIKELY(hooks_enabled)) {
overridden = PartitionAllocHooks::ReallocOverrideHookIfEnabled(
&actual_old_size, ptr);
}
if (LIKELY(!overridden)) {
internal::PartitionPage* page = internal::PartitionPage::FromPointer(
internal::PartitionCookieFreePointerAdjust(ptr));
// TODO(palmer): See if we can afford to make this a CHECK.
DCHECK(root->IsValidPage(page));
if (UNLIKELY(page->bucket->is_direct_mapped())) {
// We may be able to perform the realloc in place by changing the
// accessibility of memory pages and, if reducing the size, decommitting
// them.
if (PartitionReallocDirectMappedInPlace(root, page, new_size)) {
if (UNLIKELY(hooks_enabled)) {
PartitionAllocHooks::ReallocObserverHookIfEnabled(ptr, ptr, new_size,
type_name);
}
return ptr;
}
}
const size_t actual_new_size = root->ActualSize(new_size);
actual_old_size = PartitionAllocGetSize(ptr);
// TODO: note that tcmalloc will "ignore" a downsizing realloc() unless the
// new size is a significant percentage smaller. We could do the same if we
// determine it is a win.
if (actual_new_size == actual_old_size) {
// Trying to allocate a block of size |new_size| would give us a block of
// the same size as the one we've already got, so re-use the allocation
// after updating statistics (and cookies, if present).
page->set_raw_size(internal::PartitionCookieSizeAdjustAdd(new_size));
#if DCHECK_IS_ON()
// Write a new trailing cookie when it is possible to keep track of
// |new_size| via the raw size pointer.
if (page->get_raw_size_ptr())
internal::PartitionCookieWriteValue(static_cast<char*>(ptr) + new_size);
#endif
return ptr;
}
}
// This realloc cannot be resized in-place. Sadness.
void* ret = PartitionAllocGenericFlags(root, flags, new_size, type_name);
if (!ret) {
if (flags & PartitionAllocReturnNull)
return nullptr;
internal::PartitionExcessiveAllocationSize(new_size);
}
size_t copy_size = actual_old_size;
if (new_size < copy_size)
copy_size = new_size;
memcpy(ret, ptr, copy_size);
root->Free(ptr);
return ret;
#endif
}
void* PartitionRootGeneric::Realloc(void* ptr,
size_t new_size,
const char* type_name) {
return PartitionReallocGenericFlags(this, 0, ptr, new_size, type_name);
}
void* PartitionRootGeneric::TryRealloc(void* ptr,
size_t new_size,
const char* type_name) {
return PartitionReallocGenericFlags(this, PartitionAllocReturnNull, ptr,
new_size, type_name);
}
static size_t PartitionPurgePage(internal::PartitionPage* page, bool discard) {
const internal::PartitionBucket* bucket = page->bucket;
size_t slot_size = bucket->slot_size;
if (slot_size < kSystemPageSize || !page->num_allocated_slots)
return 0;
size_t bucket_num_slots = bucket->get_slots_per_span();
size_t discardable_bytes = 0;
size_t raw_size = page->get_raw_size();
if (raw_size) {
uint32_t used_bytes = static_cast<uint32_t>(RoundUpToSystemPage(raw_size));
discardable_bytes = bucket->slot_size - used_bytes;
if (discardable_bytes && discard) {
char* ptr =
reinterpret_cast<char*>(internal::PartitionPage::ToPointer(page));
ptr += used_bytes;
DiscardSystemPages(ptr, discardable_bytes);
}
return discardable_bytes;
}
constexpr size_t kMaxSlotCount =
(kPartitionPageSize * kMaxPartitionPagesPerSlotSpan) / kSystemPageSize;
DCHECK(bucket_num_slots <= kMaxSlotCount);
DCHECK(page->num_unprovisioned_slots < bucket_num_slots);
size_t num_slots = bucket_num_slots - page->num_unprovisioned_slots;
char slot_usage[kMaxSlotCount];
#if !defined(OS_WIN)
// The last freelist entry should not be discarded when using OS_WIN.
// DiscardVirtualMemory makes the contents of discarded memory undefined.
size_t last_slot = static_cast<size_t>(-1);
#endif
memset(slot_usage, 1, num_slots);
char* ptr = reinterpret_cast<char*>(internal::PartitionPage::ToPointer(page));
// First, walk the freelist for this page and make a bitmap of which slots
// are not in use.
for (internal::PartitionFreelistEntry* entry = page->freelist_head; entry;
/**/) {
size_t slot_index = (reinterpret_cast<char*>(entry) - ptr) / slot_size;
DCHECK(slot_index < num_slots);
slot_usage[slot_index] = 0;
entry = internal::EncodedPartitionFreelistEntry::Decode(entry->next);
#if !defined(OS_WIN)
// If we have a slot where the masked freelist entry is 0, we can actually
// discard that freelist entry because touching a discarded page is
// guaranteed to return original content or 0. (Note that this optimization
// won't fire on big-endian machines because the masking function is
// negation.)
if (!internal::PartitionFreelistEntry::Encode(entry))
last_slot = slot_index;
#endif
}
// If the slot(s) at the end of the slot span are not in used, we can truncate
// them entirely and rewrite the freelist.
size_t truncated_slots = 0;
while (!slot_usage[num_slots - 1]) {
truncated_slots++;
num_slots--;
DCHECK(num_slots);
}
// First, do the work of calculating the discardable bytes. Don't actually
// discard anything unless the discard flag was passed in.
if (truncated_slots) {
size_t unprovisioned_bytes = 0;
char* begin_ptr = ptr + (num_slots * slot_size);
char* end_ptr = begin_ptr + (slot_size * truncated_slots);
begin_ptr = reinterpret_cast<char*>(
RoundUpToSystemPage(reinterpret_cast<size_t>(begin_ptr)));
// We round the end pointer here up and not down because we're at the end of
// a slot span, so we "own" all the way up the page boundary.
end_ptr = reinterpret_cast<char*>(
RoundUpToSystemPage(reinterpret_cast<size_t>(end_ptr)));
DCHECK(end_ptr <= ptr + bucket->get_bytes_per_span());
if (begin_ptr < end_ptr) {
unprovisioned_bytes = end_ptr - begin_ptr;
discardable_bytes += unprovisioned_bytes;
}
if (unprovisioned_bytes && discard) {
DCHECK(truncated_slots > 0);
size_t num_new_entries = 0;
page->num_unprovisioned_slots += static_cast<uint16_t>(truncated_slots);
// Rewrite the freelist.
internal::PartitionFreelistEntry* head = nullptr;
internal::PartitionFreelistEntry* back = head;
for (size_t slot_index = 0; slot_index < num_slots; ++slot_index) {
if (slot_usage[slot_index])
continue;
auto* entry = reinterpret_cast<internal::PartitionFreelistEntry*>(
ptr + (slot_size * slot_index));
if (!head) {
head = entry;
back = entry;
} else {
back->next = internal::PartitionFreelistEntry::Encode(entry);
back = entry;
}
num_new_entries++;
#if !defined(OS_WIN)
last_slot = slot_index;
#endif
}
page->freelist_head = head;
if (back)
back->next = internal::PartitionFreelistEntry::Encode(nullptr);
DCHECK(num_new_entries == num_slots - page->num_allocated_slots);
// Discard the memory.
DiscardSystemPages(begin_ptr, unprovisioned_bytes);
}
}
// Next, walk the slots and for any not in use, consider where the system page
// boundaries occur. We can release any system pages back to the system as
// long as we don't interfere with a freelist pointer or an adjacent slot.
for (size_t i = 0; i < num_slots; ++i) {
if (slot_usage[i])
continue;
// The first address we can safely discard is just after the freelist
// pointer. There's one quirk: if the freelist pointer is actually NULL, we
// can discard that pointer value too.
char* begin_ptr = ptr + (i * slot_size);
char* end_ptr = begin_ptr + slot_size;
#if !defined(OS_WIN)
if (i != last_slot)
begin_ptr += sizeof(internal::PartitionFreelistEntry);
#else
begin_ptr += sizeof(internal::PartitionFreelistEntry);
#endif
begin_ptr = reinterpret_cast<char*>(
RoundUpToSystemPage(reinterpret_cast<size_t>(begin_ptr)));
end_ptr = reinterpret_cast<char*>(
RoundDownToSystemPage(reinterpret_cast<size_t>(end_ptr)));
if (begin_ptr < end_ptr) {
size_t partial_slot_bytes = end_ptr - begin_ptr;
discardable_bytes += partial_slot_bytes;
if (discard)
DiscardSystemPages(begin_ptr, partial_slot_bytes);
}
}
return discardable_bytes;
}
static void PartitionPurgeBucket(internal::PartitionBucket* bucket) {
if (bucket->active_pages_head !=
internal::PartitionPage::get_sentinel_page()) {
for (internal::PartitionPage* page = bucket->active_pages_head; page;
page = page->next_page) {
DCHECK(page != internal::PartitionPage::get_sentinel_page());
PartitionPurgePage(page, true);
}
}
}
void PartitionRoot::PurgeMemory(int flags) {
if (flags & PartitionPurgeDecommitEmptyPages)
DecommitEmptyPages();
// We don't currently do anything for PartitionPurgeDiscardUnusedSystemPages
// here because that flag is only useful for allocations >= system page size.
// We only have allocations that large inside generic partitions at the
// moment.
}
void PartitionRootGeneric::PurgeMemory(int flags) {
subtle::SpinLock::Guard guard(lock);
if (flags & PartitionPurgeDecommitEmptyPages)
DecommitEmptyPages();
if (flags & PartitionPurgeDiscardUnusedSystemPages) {
for (size_t i = 0; i < kGenericNumBuckets; ++i) {
internal::PartitionBucket* bucket = &buckets[i];
if (bucket->slot_size >= kSystemPageSize)
PartitionPurgeBucket(bucket);
}
}
}
static void PartitionDumpPageStats(PartitionBucketMemoryStats* stats_out,
internal::PartitionPage* page) {
uint16_t bucket_num_slots = page->bucket->get_slots_per_span();
if (page->is_decommitted()) {
++stats_out->num_decommitted_pages;
return;
}
stats_out->discardable_bytes += PartitionPurgePage(page, false);
size_t raw_size = page->get_raw_size();
if (raw_size) {
stats_out->active_bytes += static_cast<uint32_t>(raw_size);
} else {
stats_out->active_bytes +=
(page->num_allocated_slots * stats_out->bucket_slot_size);
}
size_t page_bytes_resident =
RoundUpToSystemPage((bucket_num_slots - page->num_unprovisioned_slots) *
stats_out->bucket_slot_size);
stats_out->resident_bytes += page_bytes_resident;
if (page->is_empty()) {
stats_out->decommittable_bytes += page_bytes_resident;
++stats_out->num_empty_pages;
} else if (page->is_full()) {
++stats_out->num_full_pages;
} else {
DCHECK(page->is_active());
++stats_out->num_active_pages;
}
}
static void PartitionDumpBucketStats(PartitionBucketMemoryStats* stats_out,
const internal::PartitionBucket* bucket) {
DCHECK(!bucket->is_direct_mapped());
stats_out->is_valid = false;
// If the active page list is empty (==
// internal::PartitionPage::get_sentinel_page()), the bucket might still need
// to be reported if it has a list of empty, decommitted or full pages.
if (bucket->active_pages_head ==
internal::PartitionPage::get_sentinel_page() &&
!bucket->empty_pages_head && !bucket->decommitted_pages_head &&
!bucket->num_full_pages)
return;
memset(stats_out, '\0', sizeof(*stats_out));
stats_out->is_valid = true;
stats_out->is_direct_map = false;
stats_out->num_full_pages = static_cast<size_t>(bucket->num_full_pages);
stats_out->bucket_slot_size = bucket->slot_size;
uint16_t bucket_num_slots = bucket->get_slots_per_span();
size_t bucket_useful_storage = stats_out->bucket_slot_size * bucket_num_slots;
stats_out->allocated_page_size = bucket->get_bytes_per_span();
stats_out->active_bytes = bucket->num_full_pages * bucket_useful_storage;
stats_out->resident_bytes =
bucket->num_full_pages * stats_out->allocated_page_size;
for (internal::PartitionPage* page = bucket->empty_pages_head; page;
page = page->next_page) {
DCHECK(page->is_empty() || page->is_decommitted());
PartitionDumpPageStats(stats_out, page);
}
for (internal::PartitionPage* page = bucket->decommitted_pages_head; page;
page = page->next_page) {
DCHECK(page->is_decommitted());
PartitionDumpPageStats(stats_out, page);
}
if (bucket->active_pages_head !=
internal::PartitionPage::get_sentinel_page()) {
for (internal::PartitionPage* page = bucket->active_pages_head; page;
page = page->next_page) {
DCHECK(page != internal::PartitionPage::get_sentinel_page());
PartitionDumpPageStats(stats_out, page);
}
}
}
void PartitionRootGeneric::DumpStats(const char* partition_name,
bool is_light_dump,
PartitionStatsDumper* dumper) {
PartitionMemoryStats stats = {0};
stats.total_mmapped_bytes =
total_size_of_super_pages + total_size_of_direct_mapped_pages;
stats.total_committed_bytes = total_size_of_committed_pages;
size_t direct_mapped_allocations_total_size = 0;
static const size_t kMaxReportableDirectMaps = 4096;
// Allocate on the heap rather than on the stack to avoid stack overflow
// skirmishes (on Windows, in particular).
std::unique_ptr<uint32_t[]> direct_map_lengths = nullptr;
if (!is_light_dump) {
direct_map_lengths =
std::unique_ptr<uint32_t[]>(new uint32_t[kMaxReportableDirectMaps]);
}
PartitionBucketMemoryStats bucket_stats[kGenericNumBuckets];
size_t num_direct_mapped_allocations = 0;
{
subtle::SpinLock::Guard guard(lock);
for (size_t i = 0; i < kGenericNumBuckets; ++i) {
const internal::PartitionBucket* bucket = &buckets[i];
// Don't report the pseudo buckets that the generic allocator sets up in
// order to preserve a fast size->bucket map (see
// PartitionRootGeneric::Init() for details).
if (!bucket->active_pages_head)
bucket_stats[i].is_valid = false;
else
PartitionDumpBucketStats(&bucket_stats[i], bucket);
if (bucket_stats[i].is_valid) {
stats.total_resident_bytes += bucket_stats[i].resident_bytes;
stats.total_active_bytes += bucket_stats[i].active_bytes;
stats.total_decommittable_bytes += bucket_stats[i].decommittable_bytes;
stats.total_discardable_bytes += bucket_stats[i].discardable_bytes;
}
}
for (internal::PartitionDirectMapExtent* extent = direct_map_list;
extent && num_direct_mapped_allocations < kMaxReportableDirectMaps;
extent = extent->next_extent, ++num_direct_mapped_allocations) {
DCHECK(!extent->next_extent ||
extent->next_extent->prev_extent == extent);
size_t slot_size = extent->bucket->slot_size;
direct_mapped_allocations_total_size += slot_size;
if (is_light_dump)
continue;
direct_map_lengths[num_direct_mapped_allocations] = slot_size;
}
}
if (!is_light_dump) {
// Call |PartitionsDumpBucketStats| after collecting stats because it can
// try to allocate using |PartitionRootGeneric::Alloc()| and it can't
// obtain the lock.
for (size_t i = 0; i < kGenericNumBuckets; ++i) {
if (bucket_stats[i].is_valid)
dumper->PartitionsDumpBucketStats(partition_name, &bucket_stats[i]);
}
for (size_t i = 0; i < num_direct_mapped_allocations; ++i) {
uint32_t size = direct_map_lengths[i];
PartitionBucketMemoryStats mapped_stats = {};
mapped_stats.is_valid = true;
mapped_stats.is_direct_map = true;
mapped_stats.num_full_pages = 1;
mapped_stats.allocated_page_size = size;
mapped_stats.bucket_slot_size = size;
mapped_stats.active_bytes = size;
mapped_stats.resident_bytes = size;
dumper->PartitionsDumpBucketStats(partition_name, &mapped_stats);
}
}
stats.total_resident_bytes += direct_mapped_allocations_total_size;
stats.total_active_bytes += direct_mapped_allocations_total_size;
dumper->PartitionDumpTotals(partition_name, &stats);
}
void PartitionRoot::DumpStats(const char* partition_name,
bool is_light_dump,
PartitionStatsDumper* dumper) {
PartitionMemoryStats stats = {0};
stats.total_mmapped_bytes = total_size_of_super_pages;
stats.total_committed_bytes = total_size_of_committed_pages;
DCHECK(!total_size_of_direct_mapped_pages);
static constexpr size_t kMaxReportableBuckets = 4096 / sizeof(void*);
std::unique_ptr<PartitionBucketMemoryStats[]> memory_stats;
if (!is_light_dump) {
memory_stats = std::unique_ptr<PartitionBucketMemoryStats[]>(
new PartitionBucketMemoryStats[kMaxReportableBuckets]);
}
const size_t partition_num_buckets = num_buckets;
DCHECK(partition_num_buckets <= kMaxReportableBuckets);
for (size_t i = 0; i < partition_num_buckets; ++i) {
PartitionBucketMemoryStats bucket_stats = {0};
PartitionDumpBucketStats(&bucket_stats, &buckets()[i]);
if (bucket_stats.is_valid) {
stats.total_resident_bytes += bucket_stats.resident_bytes;
stats.total_active_bytes += bucket_stats.active_bytes;
stats.total_decommittable_bytes += bucket_stats.decommittable_bytes;
stats.total_discardable_bytes += bucket_stats.discardable_bytes;
}
if (!is_light_dump) {
if (bucket_stats.is_valid)
memory_stats[i] = bucket_stats;
else
memory_stats[i].is_valid = false;
}
}
if (!is_light_dump) {
// PartitionsDumpBucketStats is called after collecting stats because it
// can use PartitionRoot::Alloc() to allocate and this can affect the
// statistics.
for (size_t i = 0; i < partition_num_buckets; ++i) {
if (memory_stats[i].is_valid)
dumper->PartitionsDumpBucketStats(partition_name, &memory_stats[i]);
}
}
dumper->PartitionDumpTotals(partition_name, &stats);
}
} // namespace base

View file

@ -0,0 +1,540 @@
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_H_
// DESCRIPTION
// PartitionRoot::Alloc() / PartitionRootGeneric::Alloc() and PartitionFree() /
// PartitionRootGeneric::Free() are approximately analagous to malloc() and
// free().
//
// The main difference is that a PartitionRoot / PartitionRootGeneric object
// must be supplied to these functions, representing a specific "heap partition"
// that will be used to satisfy the allocation. Different partitions are
// guaranteed to exist in separate address spaces, including being separate from
// the main system heap. If the contained objects are all freed, physical memory
// is returned to the system but the address space remains reserved.
// See PartitionAlloc.md for other security properties PartitionAlloc provides.
//
// THE ONLY LEGITIMATE WAY TO OBTAIN A PartitionRoot IS THROUGH THE
// SizeSpecificPartitionAllocator / PartitionAllocatorGeneric classes. To
// minimize the instruction count to the fullest extent possible, the
// PartitionRoot is really just a header adjacent to other data areas provided
// by the allocator class.
//
// The PartitionRoot::Alloc() variant of the API has the following caveats:
// - Allocations and frees against a single partition must be single threaded.
// - Allocations must not exceed a max size, chosen at compile-time via a
// templated parameter to PartitionAllocator.
// - Allocation sizes must be aligned to the system pointer size.
// - Allocations are bucketed exactly according to size.
//
// And for PartitionRootGeneric::Alloc():
// - Multi-threaded use against a single partition is ok; locking is handled.
// - Allocations of any arbitrary size can be handled (subject to a limit of
// INT_MAX bytes for security reasons).
// - Bucketing is by approximate size, for example an allocation of 4000 bytes
// might be placed into a 4096-byte bucket. Bucket sizes are chosen to try and
// keep worst-case waste to ~10%.
//
// The allocators are designed to be extremely fast, thanks to the following
// properties and design:
// - Just two single (reasonably predicatable) branches in the hot / fast path
// for both allocating and (significantly) freeing.
// - A minimal number of operations in the hot / fast path, with the slow paths
// in separate functions, leading to the possibility of inlining.
// - Each partition page (which is usually multiple physical pages) has a
// metadata structure which allows fast mapping of free() address to an
// underlying bucket.
// - Supports a lock-free API for fast performance in single-threaded cases.
// - The freelist for a given bucket is split across a number of partition
// pages, enabling various simple tricks to try and minimize fragmentation.
// - Fine-grained bucket sizes leading to less waste and better packing.
//
// The following security properties could be investigated in the future:
// - Per-object bucketing (instead of per-size) is mostly available at the API,
// but not used yet.
// - No randomness of freelist entries or bucket position.
// - Better checking for wild pointers in free().
// - Better freelist masking function to guarantee fault on 32-bit.
#include <limits.h>
#include <string.h>
#include "base/allocator/partition_allocator/memory_reclaimer.h"
#include "base/allocator/partition_allocator/page_allocator.h"
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
#include "base/allocator/partition_allocator/partition_bucket.h"
#include "base/allocator/partition_allocator/partition_cookie.h"
#include "base/allocator/partition_allocator/partition_page.h"
#include "base/allocator/partition_allocator/partition_root_base.h"
#include "base/allocator/partition_allocator/spin_lock.h"
#include "base/base_export.h"
#include "base/bits.h"
#include "base/compiler_specific.h"
#include "base/logging.h"
#include "base/stl_util.h"
#include "base/sys_byteorder.h"
#include "build/build_config.h"
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
#include <stdlib.h>
#endif
// We use this to make MEMORY_TOOL_REPLACES_ALLOCATOR behave the same for max
// size as other alloc code.
#define CHECK_MAX_SIZE_OR_RETURN_NULLPTR(size, flags) \
if (size > kGenericMaxDirectMapped) { \
if (flags & PartitionAllocReturnNull) { \
return nullptr; \
} \
CHECK(false); \
}
namespace base {
class PartitionStatsDumper;
enum PartitionPurgeFlags {
// Decommitting the ring list of empty pages is reasonably fast.
PartitionPurgeDecommitEmptyPages = 1 << 0,
// Discarding unused system pages is slower, because it involves walking all
// freelists in all active partition pages of all buckets >= system page
// size. It often frees a similar amount of memory to decommitting the empty
// pages, though.
PartitionPurgeDiscardUnusedSystemPages = 1 << 1,
};
// Never instantiate a PartitionRoot directly, instead use PartitionAlloc.
struct BASE_EXPORT PartitionRoot : public internal::PartitionRootBase {
PartitionRoot();
~PartitionRoot() override;
// This references the buckets OFF the edge of this struct. All uses of
// PartitionRoot must have the bucket array come right after.
//
// The PartitionAlloc templated class ensures the following is correct.
ALWAYS_INLINE internal::PartitionBucket* buckets() {
return reinterpret_cast<internal::PartitionBucket*>(this + 1);
}
ALWAYS_INLINE const internal::PartitionBucket* buckets() const {
return reinterpret_cast<const internal::PartitionBucket*>(this + 1);
}
void Init(size_t bucket_count, size_t maximum_allocation);
ALWAYS_INLINE void* Alloc(size_t size, const char* type_name);
ALWAYS_INLINE void* AllocFlags(int flags, size_t size, const char* type_name);
void PurgeMemory(int flags) override;
void DumpStats(const char* partition_name,
bool is_light_dump,
PartitionStatsDumper* dumper);
};
// Never instantiate a PartitionRootGeneric directly, instead use
// PartitionAllocatorGeneric.
struct BASE_EXPORT PartitionRootGeneric : public internal::PartitionRootBase {
PartitionRootGeneric();
~PartitionRootGeneric() override;
subtle::SpinLock lock;
// Some pre-computed constants.
size_t order_index_shifts[kBitsPerSizeT + 1] = {};
size_t order_sub_index_masks[kBitsPerSizeT + 1] = {};
// The bucket lookup table lets us map a size_t to a bucket quickly.
// The trailing +1 caters for the overflow case for very large allocation
// sizes. It is one flat array instead of a 2D array because in the 2D
// world, we'd need to index array[blah][max+1] which risks undefined
// behavior.
internal::PartitionBucket*
bucket_lookups[((kBitsPerSizeT + 1) * kGenericNumBucketsPerOrder) + 1] =
{};
internal::PartitionBucket buckets[kGenericNumBuckets] = {};
// Public API.
void Init();
ALWAYS_INLINE void* Alloc(size_t size, const char* type_name);
ALWAYS_INLINE void* AllocFlags(int flags, size_t size, const char* type_name);
ALWAYS_INLINE void Free(void* ptr);
NOINLINE void* Realloc(void* ptr, size_t new_size, const char* type_name);
// Overload that may return nullptr if reallocation isn't possible. In this
// case, |ptr| remains valid.
NOINLINE void* TryRealloc(void* ptr, size_t new_size, const char* type_name);
ALWAYS_INLINE size_t ActualSize(size_t size);
void PurgeMemory(int flags) override;
void DumpStats(const char* partition_name,
bool is_light_dump,
PartitionStatsDumper* partition_stats_dumper);
};
// Struct used to retrieve total memory usage of a partition. Used by
// PartitionStatsDumper implementation.
struct PartitionMemoryStats {
size_t total_mmapped_bytes; // Total bytes mmaped from the system.
size_t total_committed_bytes; // Total size of commmitted pages.
size_t total_resident_bytes; // Total bytes provisioned by the partition.
size_t total_active_bytes; // Total active bytes in the partition.
size_t total_decommittable_bytes; // Total bytes that could be decommitted.
size_t total_discardable_bytes; // Total bytes that could be discarded.
};
// Struct used to retrieve memory statistics about a partition bucket. Used by
// PartitionStatsDumper implementation.
struct PartitionBucketMemoryStats {
bool is_valid; // Used to check if the stats is valid.
bool is_direct_map; // True if this is a direct mapping; size will not be
// unique.
uint32_t bucket_slot_size; // The size of the slot in bytes.
uint32_t allocated_page_size; // Total size the partition page allocated from
// the system.
uint32_t active_bytes; // Total active bytes used in the bucket.
uint32_t resident_bytes; // Total bytes provisioned in the bucket.
uint32_t decommittable_bytes; // Total bytes that could be decommitted.
uint32_t discardable_bytes; // Total bytes that could be discarded.
uint32_t num_full_pages; // Number of pages with all slots allocated.
uint32_t num_active_pages; // Number of pages that have at least one
// provisioned slot.
uint32_t num_empty_pages; // Number of pages that are empty
// but not decommitted.
uint32_t num_decommitted_pages; // Number of pages that are empty
// and decommitted.
};
// Interface that is passed to PartitionDumpStats and
// PartitionDumpStatsGeneric for using the memory statistics.
class BASE_EXPORT PartitionStatsDumper {
public:
// Called to dump total memory used by partition, once per partition.
virtual void PartitionDumpTotals(const char* partition_name,
const PartitionMemoryStats*) = 0;
// Called to dump stats about buckets, for each bucket.
virtual void PartitionsDumpBucketStats(const char* partition_name,
const PartitionBucketMemoryStats*) = 0;
};
BASE_EXPORT void PartitionAllocGlobalInit(OomFunction on_out_of_memory);
// PartitionAlloc supports setting hooks to observe allocations/frees as they
// occur as well as 'override' hooks that allow overriding those operations.
class BASE_EXPORT PartitionAllocHooks {
public:
// Log allocation and free events.
typedef void AllocationObserverHook(void* address,
size_t size,
const char* type_name);
typedef void FreeObserverHook(void* address);
// If it returns true, the allocation has been overridden with the pointer in
// *out.
typedef bool AllocationOverrideHook(void** out,
int flags,
size_t size,
const char* type_name);
// If it returns true, then the allocation was overridden and has been freed.
typedef bool FreeOverrideHook(void* address);
// If it returns true, the underlying allocation is overridden and *out holds
// the size of the underlying allocation.
typedef bool ReallocOverrideHook(size_t* out, void* address);
// To unhook, call Set*Hooks with nullptrs.
static void SetObserverHooks(AllocationObserverHook* alloc_hook,
FreeObserverHook* free_hook);
static void SetOverrideHooks(AllocationOverrideHook* alloc_hook,
FreeOverrideHook* free_hook,
ReallocOverrideHook realloc_hook);
// Helper method to check whether hooks are enabled. This is an optimization
// so that if a function needs to call observer and override hooks in two
// different places this value can be cached and only loaded once.
static bool AreHooksEnabled() {
return hooks_enabled_.load(std::memory_order_relaxed);
}
static void AllocationObserverHookIfEnabled(void* address,
size_t size,
const char* type_name);
static bool AllocationOverrideHookIfEnabled(void** out,
int flags,
size_t size,
const char* type_name);
static void FreeObserverHookIfEnabled(void* address);
static bool FreeOverrideHookIfEnabled(void* address);
static void ReallocObserverHookIfEnabled(void* old_address,
void* new_address,
size_t size,
const char* type_name);
static bool ReallocOverrideHookIfEnabled(size_t* out, void* address);
private:
// Single bool that is used to indicate whether observer or allocation hooks
// are set to reduce the numbers of loads required to check whether hooking is
// enabled.
static std::atomic<bool> hooks_enabled_;
// Lock used to synchronize Set*Hooks calls.
static std::atomic<AllocationObserverHook*> allocation_observer_hook_;
static std::atomic<FreeObserverHook*> free_observer_hook_;
static std::atomic<AllocationOverrideHook*> allocation_override_hook_;
static std::atomic<FreeOverrideHook*> free_override_hook_;
static std::atomic<ReallocOverrideHook*> realloc_override_hook_;
};
ALWAYS_INLINE void* PartitionRoot::Alloc(size_t size, const char* type_name) {
return AllocFlags(0, size, type_name);
}
ALWAYS_INLINE void* PartitionRoot::AllocFlags(int flags,
size_t size,
const char* type_name) {
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
CHECK_MAX_SIZE_OR_RETURN_NULLPTR(size, flags);
void* result = malloc(size);
CHECK(result);
return result;
#else
DCHECK(max_allocation == 0 || size <= max_allocation);
void* result;
const bool hooks_enabled = PartitionAllocHooks::AreHooksEnabled();
if (UNLIKELY(hooks_enabled)) {
if (PartitionAllocHooks::AllocationOverrideHookIfEnabled(&result, flags,
size, type_name)) {
PartitionAllocHooks::AllocationObserverHookIfEnabled(result, size,
type_name);
return result;
}
}
size_t requested_size = size;
size = internal::PartitionCookieSizeAdjustAdd(size);
DCHECK(initialized);
size_t index = size >> kBucketShift;
DCHECK(index < num_buckets);
DCHECK(size == index << kBucketShift);
internal::PartitionBucket* bucket = &buckets()[index];
result = AllocFromBucket(bucket, flags, size);
if (UNLIKELY(hooks_enabled)) {
PartitionAllocHooks::AllocationObserverHookIfEnabled(result, requested_size,
type_name);
}
return result;
#endif // defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
}
ALWAYS_INLINE bool PartitionAllocSupportsGetSize() {
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
return false;
#else
return true;
#endif
}
ALWAYS_INLINE size_t PartitionAllocGetSize(void* ptr) {
// No need to lock here. Only |ptr| being freed by another thread could
// cause trouble, and the caller is responsible for that not happening.
DCHECK(PartitionAllocSupportsGetSize());
ptr = internal::PartitionCookieFreePointerAdjust(ptr);
internal::PartitionPage* page = internal::PartitionPage::FromPointer(ptr);
// TODO(palmer): See if we can afford to make this a CHECK.
DCHECK(internal::PartitionRootBase::IsValidPage(page));
size_t size = page->bucket->slot_size;
return internal::PartitionCookieSizeAdjustSubtract(size);
}
ALWAYS_INLINE void PartitionFree(void* ptr) {
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
free(ptr);
#else
// TODO(palmer): Check ptr alignment before continuing. Shall we do the check
// inside PartitionCookieFreePointerAdjust?
if (PartitionAllocHooks::AreHooksEnabled()) {
PartitionAllocHooks::FreeObserverHookIfEnabled(ptr);
if (PartitionAllocHooks::FreeOverrideHookIfEnabled(ptr))
return;
}
ptr = internal::PartitionCookieFreePointerAdjust(ptr);
internal::PartitionPage* page = internal::PartitionPage::FromPointer(ptr);
// TODO(palmer): See if we can afford to make this a CHECK.
DCHECK(internal::PartitionRootBase::IsValidPage(page));
page->Free(ptr);
#endif
}
ALWAYS_INLINE internal::PartitionBucket* PartitionGenericSizeToBucket(
PartitionRootGeneric* root,
size_t size) {
size_t order = kBitsPerSizeT - bits::CountLeadingZeroBitsSizeT(size);
// The order index is simply the next few bits after the most significant bit.
size_t order_index = (size >> root->order_index_shifts[order]) &
(kGenericNumBucketsPerOrder - 1);
// And if the remaining bits are non-zero we must bump the bucket up.
size_t sub_order_index = size & root->order_sub_index_masks[order];
internal::PartitionBucket* bucket =
root->bucket_lookups[(order << kGenericNumBucketsPerOrderBits) +
order_index + !!sub_order_index];
CHECK(bucket);
DCHECK(!bucket->slot_size || bucket->slot_size >= size);
DCHECK(!(bucket->slot_size % kGenericSmallestBucket));
return bucket;
}
ALWAYS_INLINE void* PartitionAllocGenericFlags(PartitionRootGeneric* root,
int flags,
size_t size,
const char* type_name) {
DCHECK_LT(flags, PartitionAllocLastFlag << 1);
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
CHECK_MAX_SIZE_OR_RETURN_NULLPTR(size, flags);
const bool zero_fill = flags & PartitionAllocZeroFill;
void* result = zero_fill ? calloc(1, size) : malloc(size);
CHECK(result || flags & PartitionAllocReturnNull);
return result;
#else
DCHECK(root->initialized);
// Only SizeSpecificPartitionAllocator should use max_allocation.
DCHECK(root->max_allocation == 0);
void* result;
const bool hooks_enabled = PartitionAllocHooks::AreHooksEnabled();
if (UNLIKELY(hooks_enabled)) {
if (PartitionAllocHooks::AllocationOverrideHookIfEnabled(&result, flags,
size, type_name)) {
PartitionAllocHooks::AllocationObserverHookIfEnabled(result, size,
type_name);
return result;
}
}
size_t requested_size = size;
size = internal::PartitionCookieSizeAdjustAdd(size);
internal::PartitionBucket* bucket = PartitionGenericSizeToBucket(root, size);
{
subtle::SpinLock::Guard guard(root->lock);
result = root->AllocFromBucket(bucket, flags, size);
}
if (UNLIKELY(hooks_enabled)) {
PartitionAllocHooks::AllocationObserverHookIfEnabled(result, requested_size,
type_name);
}
return result;
#endif
}
ALWAYS_INLINE void* PartitionRootGeneric::Alloc(size_t size,
const char* type_name) {
return PartitionAllocGenericFlags(this, 0, size, type_name);
}
ALWAYS_INLINE void* PartitionRootGeneric::AllocFlags(int flags,
size_t size,
const char* type_name) {
return PartitionAllocGenericFlags(this, flags, size, type_name);
}
ALWAYS_INLINE void PartitionRootGeneric::Free(void* ptr) {
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
free(ptr);
#else
DCHECK(initialized);
if (UNLIKELY(!ptr))
return;
if (PartitionAllocHooks::AreHooksEnabled()) {
PartitionAllocHooks::FreeObserverHookIfEnabled(ptr);
if (PartitionAllocHooks::FreeOverrideHookIfEnabled(ptr))
return;
}
ptr = internal::PartitionCookieFreePointerAdjust(ptr);
internal::PartitionPage* page = internal::PartitionPage::FromPointer(ptr);
// TODO(palmer): See if we can afford to make this a CHECK.
DCHECK(IsValidPage(page));
{
subtle::SpinLock::Guard guard(lock);
page->Free(ptr);
}
#endif
}
BASE_EXPORT void* PartitionReallocGenericFlags(PartitionRootGeneric* root,
int flags,
void* ptr,
size_t new_size,
const char* type_name);
ALWAYS_INLINE size_t PartitionRootGeneric::ActualSize(size_t size) {
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
return size;
#else
DCHECK(initialized);
size = internal::PartitionCookieSizeAdjustAdd(size);
internal::PartitionBucket* bucket = PartitionGenericSizeToBucket(this, size);
if (LIKELY(!bucket->is_direct_mapped())) {
size = bucket->slot_size;
} else if (size > kGenericMaxDirectMapped) {
// Too large to allocate => return the size unchanged.
} else {
size = internal::PartitionBucket::get_direct_map_size(size);
}
return internal::PartitionCookieSizeAdjustSubtract(size);
#endif
}
template <size_t N>
class SizeSpecificPartitionAllocator {
public:
SizeSpecificPartitionAllocator() {
memset(actual_buckets_, 0,
sizeof(internal::PartitionBucket) * base::size(actual_buckets_));
}
~SizeSpecificPartitionAllocator() {
PartitionAllocMemoryReclaimer::Instance()->UnregisterPartition(
&partition_root_);
}
static const size_t kMaxAllocation = N - kAllocationGranularity;
static const size_t kNumBuckets = N / kAllocationGranularity;
void init() {
partition_root_.Init(kNumBuckets, kMaxAllocation);
PartitionAllocMemoryReclaimer::Instance()->RegisterPartition(
&partition_root_);
}
ALWAYS_INLINE PartitionRoot* root() { return &partition_root_; }
private:
PartitionRoot partition_root_;
internal::PartitionBucket actual_buckets_[kNumBuckets];
};
class BASE_EXPORT PartitionAllocatorGeneric {
public:
PartitionAllocatorGeneric();
~PartitionAllocatorGeneric() {
PartitionAllocMemoryReclaimer::Instance()->UnregisterPartition(
&partition_root_);
}
void init() {
partition_root_.Init();
PartitionAllocMemoryReclaimer::Instance()->RegisterPartition(
&partition_root_);
}
ALWAYS_INLINE PartitionRootGeneric* root() { return &partition_root_; }
private:
PartitionRootGeneric partition_root_;
};
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_H_

View file

@ -0,0 +1,188 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_CONSTANTS_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_CONSTANTS_H_
#include <limits.h>
#include "base/allocator/partition_allocator/page_allocator_constants.h"
#include "base/logging.h"
#include "build/build_config.h"
namespace base {
// Allocation granularity of sizeof(void*) bytes.
static const size_t kAllocationGranularity = sizeof(void*);
static const size_t kAllocationGranularityMask = kAllocationGranularity - 1;
static const size_t kBucketShift = (kAllocationGranularity == 8) ? 3 : 2;
// Underlying partition storage pages (`PartitionPage`s) are a power-of-2 size.
// It is typical for a `PartitionPage` to be based on multiple system pages.
// Most references to "page" refer to `PartitionPage`s.
//
// *Super pages* are the underlying system allocations we make. Super pages
// contain multiple partition pages and include space for a small amount of
// metadata per partition page.
//
// Inside super pages, we store *slot spans*. A slot span is a continguous range
// of one or more `PartitionPage`s that stores allocations of the same size.
// Slot span sizes are adjusted depending on the allocation size, to make sure
// the packing does not lead to unused (wasted) space at the end of the last
// system page of the span. For our current maximum slot span size of 64 KiB and
// other constant values, we pack _all_ `PartitionRootGeneric::Alloc` sizes
// perfectly up against the end of a system page.
#if defined(_MIPS_ARCH_LOONGSON)
static const size_t kPartitionPageShift = 16; // 64 KiB
#elif defined(ARCH_CPU_PPC64)
static const size_t kPartitionPageShift = 18; // 256 KiB
#else
static const size_t kPartitionPageShift = 14; // 16 KiB
#endif
static const size_t kPartitionPageSize = 1 << kPartitionPageShift;
static const size_t kPartitionPageOffsetMask = kPartitionPageSize - 1;
static const size_t kPartitionPageBaseMask = ~kPartitionPageOffsetMask;
// TODO: Should this be 1 if defined(_MIPS_ARCH_LOONGSON)?
static const size_t kMaxPartitionPagesPerSlotSpan = 4;
// To avoid fragmentation via never-used freelist entries, we hand out partition
// freelist sections gradually, in units of the dominant system page size. What
// we're actually doing is avoiding filling the full `PartitionPage` (16 KiB)
// with freelist pointers right away. Writing freelist pointers will fault and
// dirty a private page, which is very wasteful if we never actually store
// objects there.
static const size_t kNumSystemPagesPerPartitionPage =
kPartitionPageSize / kSystemPageSize;
static const size_t kMaxSystemPagesPerSlotSpan =
kNumSystemPagesPerPartitionPage * kMaxPartitionPagesPerSlotSpan;
// We reserve virtual address space in 2 MiB chunks (aligned to 2 MiB as well).
// These chunks are called *super pages*. We do this so that we can store
// metadata in the first few pages of each 2 MiB-aligned section. This makes
// freeing memory very fast. We specifically choose 2 MiB because this virtual
// address block represents a full but single PTE allocation on ARM, ia32 and
// x64.
//
// The layout of the super page is as follows. The sizes below are the same for
// 32- and 64-bit platforms.
//
// +-----------------------+
// | Guard page (4 KiB) |
// | Metadata page (4 KiB) |
// | Guard pages (8 KiB) |
// | Slot span |
// | Slot span |
// | ... |
// | Slot span |
// | Guard page (4 KiB) |
// +-----------------------+
//
// Each slot span is a contiguous range of one or more `PartitionPage`s.
//
// The metadata page has the following format. Note that the `PartitionPage`
// that is not at the head of a slot span is "unused". In other words, the
// metadata for the slot span is stored only in the first `PartitionPage` of the
// slot span. Metadata accesses to other `PartitionPage`s are redirected to the
// first `PartitionPage`.
//
// +---------------------------------------------+
// | SuperPageExtentEntry (32 B) |
// | PartitionPage of slot span 1 (32 B, used) |
// | PartitionPage of slot span 1 (32 B, unused) |
// | PartitionPage of slot span 1 (32 B, unused) |
// | PartitionPage of slot span 2 (32 B, used) |
// | PartitionPage of slot span 3 (32 B, used) |
// | ... |
// | PartitionPage of slot span N (32 B, unused) |
// +---------------------------------------------+
//
// A direct-mapped page has a similar layout to fake it looking like a super
// page:
//
// +-----------------------+
// | Guard page (4 KiB) |
// | Metadata page (4 KiB) |
// | Guard pages (8 KiB) |
// | Direct mapped object |
// | Guard page (4 KiB) |
// +-----------------------+
//
// A direct-mapped page's metadata page has the following layout:
//
// +--------------------------------+
// | SuperPageExtentEntry (32 B) |
// | PartitionPage (32 B) |
// | PartitionBucket (32 B) |
// | PartitionDirectMapExtent (8 B) |
// +--------------------------------+
static const size_t kSuperPageShift = 21; // 2 MiB
static const size_t kSuperPageSize = 1 << kSuperPageShift;
static const size_t kSuperPageOffsetMask = kSuperPageSize - 1;
static const size_t kSuperPageBaseMask = ~kSuperPageOffsetMask;
static const size_t kNumPartitionPagesPerSuperPage =
kSuperPageSize / kPartitionPageSize;
// The following kGeneric* constants apply to the generic variants of the API.
// The "order" of an allocation is closely related to the power-of-1 size of the
// allocation. More precisely, the order is the bit index of the
// most-significant-bit in the allocation size, where the bit numbers starts at
// index 1 for the least-significant-bit.
//
// In terms of allocation sizes, order 0 covers 0, order 1 covers 1, order 2
// covers 2->3, order 3 covers 4->7, order 4 covers 8->15.
static const size_t kGenericMinBucketedOrder = 4; // 8 bytes.
// The largest bucketed order is 1 << (20 - 1), storing [512 KiB, 1 MiB):
static const size_t kGenericMaxBucketedOrder = 20;
static const size_t kGenericNumBucketedOrders =
(kGenericMaxBucketedOrder - kGenericMinBucketedOrder) + 1;
// Eight buckets per order (for the higher orders), e.g. order 8 is 128, 144,
// 160, ..., 240:
static const size_t kGenericNumBucketsPerOrderBits = 3;
static const size_t kGenericNumBucketsPerOrder =
1 << kGenericNumBucketsPerOrderBits;
static const size_t kGenericNumBuckets =
kGenericNumBucketedOrders * kGenericNumBucketsPerOrder;
static const size_t kGenericSmallestBucket = 1
<< (kGenericMinBucketedOrder - 1);
static const size_t kGenericMaxBucketSpacing =
1 << ((kGenericMaxBucketedOrder - 1) - kGenericNumBucketsPerOrderBits);
static const size_t kGenericMaxBucketed =
(1 << (kGenericMaxBucketedOrder - 1)) +
((kGenericNumBucketsPerOrder - 1) * kGenericMaxBucketSpacing);
// Limit when downsizing a direct mapping using `realloc`:
static const size_t kGenericMinDirectMappedDownsize = kGenericMaxBucketed + 1;
static const size_t kGenericMaxDirectMapped =
(1UL << 31) + kPageAllocationGranularity; // 2 GiB plus 1 more page.
static const size_t kBitsPerSizeT = sizeof(void*) * CHAR_BIT;
// Constant for the memory reclaim logic.
static const size_t kMaxFreeableSpans = 16;
// If the total size in bytes of allocated but not committed pages exceeds this
// value (probably it is a "out of virtual address space" crash), a special
// crash stack trace is generated at
// `PartitionOutOfMemoryWithLotsOfUncommitedPages`. This is to distinguish "out
// of virtual address space" from "out of physical memory" in crash reports.
static const size_t kReasonableSizeOfUnusedPages = 1024 * 1024 * 1024; // 1 GiB
// These byte values match tcmalloc.
static const unsigned char kUninitializedByte = 0xAB;
static const unsigned char kFreedByte = 0xCD;
// Flags for `PartitionAllocGenericFlags`.
enum PartitionAllocFlags {
PartitionAllocReturnNull = 1 << 0,
PartitionAllocZeroFill = 1 << 1,
PartitionAllocLastFlag = PartitionAllocZeroFill
};
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_CONSTANTS_H_

View file

@ -0,0 +1,565 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/partition_bucket.h"
#include "base/allocator/partition_allocator/oom.h"
#include "base/allocator/partition_allocator/page_allocator.h"
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
#include "base/allocator/partition_allocator/partition_direct_map_extent.h"
#include "base/allocator/partition_allocator/partition_oom.h"
#include "base/allocator/partition_allocator/partition_page.h"
#include "base/allocator/partition_allocator/partition_root_base.h"
#include "base/logging.h"
#include "build/build_config.h"
namespace base {
namespace internal {
namespace {
ALWAYS_INLINE PartitionPage* PartitionDirectMap(PartitionRootBase* root,
int flags,
size_t raw_size) {
size_t size = PartitionBucket::get_direct_map_size(raw_size);
// Because we need to fake looking like a super page, we need to allocate
// a bunch of system pages more than "size":
// - The first few system pages are the partition page in which the super
// page metadata is stored. We fault just one system page out of a partition
// page sized clump.
// - We add a trailing guard page on 32-bit (on 64-bit we rely on the
// massive address space plus randomization instead).
size_t map_size = size + kPartitionPageSize;
#if !defined(ARCH_CPU_64_BITS)
map_size += kSystemPageSize;
#endif
// Round up to the allocation granularity.
map_size += kPageAllocationGranularityOffsetMask;
map_size &= kPageAllocationGranularityBaseMask;
char* ptr = reinterpret_cast<char*>(AllocPages(nullptr, map_size,
kSuperPageSize, PageReadWrite,
PageTag::kPartitionAlloc));
if (UNLIKELY(!ptr))
return nullptr;
size_t committed_page_size = size + kSystemPageSize;
root->total_size_of_direct_mapped_pages += committed_page_size;
root->IncreaseCommittedPages(committed_page_size);
char* slot = ptr + kPartitionPageSize;
SetSystemPagesAccess(ptr + (kSystemPageSize * 2),
kPartitionPageSize - (kSystemPageSize * 2),
PageInaccessible);
#if !defined(ARCH_CPU_64_BITS)
SetSystemPagesAccess(ptr, kSystemPageSize, PageInaccessible);
SetSystemPagesAccess(slot + size, kSystemPageSize, PageInaccessible);
#endif
PartitionSuperPageExtentEntry* extent =
reinterpret_cast<PartitionSuperPageExtentEntry*>(
PartitionSuperPageToMetadataArea(ptr));
extent->root = root;
// The new structures are all located inside a fresh system page so they
// will all be zeroed out. These DCHECKs are for documentation.
DCHECK(!extent->super_page_base);
DCHECK(!extent->super_pages_end);
DCHECK(!extent->next);
PartitionPage* page = PartitionPage::FromPointerNoAlignmentCheck(slot);
PartitionBucket* bucket = reinterpret_cast<PartitionBucket*>(
reinterpret_cast<char*>(page) + (kPageMetadataSize * 2));
DCHECK(!page->next_page);
DCHECK(!page->num_allocated_slots);
DCHECK(!page->num_unprovisioned_slots);
DCHECK(!page->page_offset);
DCHECK(!page->empty_cache_index);
page->bucket = bucket;
page->freelist_head = reinterpret_cast<PartitionFreelistEntry*>(slot);
PartitionFreelistEntry* next_entry =
reinterpret_cast<PartitionFreelistEntry*>(slot);
next_entry->next = PartitionFreelistEntry::Encode(nullptr);
DCHECK(!bucket->active_pages_head);
DCHECK(!bucket->empty_pages_head);
DCHECK(!bucket->decommitted_pages_head);
DCHECK(!bucket->num_system_pages_per_slot_span);
DCHECK(!bucket->num_full_pages);
bucket->slot_size = size;
PartitionDirectMapExtent* map_extent =
PartitionDirectMapExtent::FromPage(page);
map_extent->map_size = map_size - kPartitionPageSize - kSystemPageSize;
map_extent->bucket = bucket;
// Maintain the doubly-linked list of all direct mappings.
map_extent->next_extent = root->direct_map_list;
if (map_extent->next_extent)
map_extent->next_extent->prev_extent = map_extent;
map_extent->prev_extent = nullptr;
root->direct_map_list = map_extent;
return page;
}
} // namespace
// static
PartitionBucket PartitionBucket::sentinel_bucket_;
PartitionBucket* PartitionBucket::get_sentinel_bucket() {
return &sentinel_bucket_;
}
// TODO(ajwong): This seems to interact badly with
// get_pages_per_slot_span() which rounds the value from this up to a
// multiple of kNumSystemPagesPerPartitionPage (aka 4) anyways.
// http://crbug.com/776537
//
// TODO(ajwong): The waste calculation seems wrong. The PTE usage should cover
// both used and unsed pages.
// http://crbug.com/776537
uint8_t PartitionBucket::get_system_pages_per_slot_span() {
// This works out reasonably for the current bucket sizes of the generic
// allocator, and the current values of partition page size and constants.
// Specifically, we have enough room to always pack the slots perfectly into
// some number of system pages. The only waste is the waste associated with
// unfaulted pages (i.e. wasted address space).
// TODO: we end up using a lot of system pages for very small sizes. For
// example, we'll use 12 system pages for slot size 24. The slot size is
// so small that the waste would be tiny with just 4, or 1, system pages.
// Later, we can investigate whether there are anti-fragmentation benefits
// to using fewer system pages.
double best_waste_ratio = 1.0f;
uint16_t best_pages = 0;
if (slot_size > kMaxSystemPagesPerSlotSpan * kSystemPageSize) {
// TODO(ajwong): Why is there a DCHECK here for this?
// http://crbug.com/776537
DCHECK(!(slot_size % kSystemPageSize));
best_pages = static_cast<uint16_t>(slot_size / kSystemPageSize);
// TODO(ajwong): Should this be checking against
// kMaxSystemPagesPerSlotSpan or numeric_limits<uint8_t>::max?
// http://crbug.com/776537
CHECK(best_pages < (1 << 8));
return static_cast<uint8_t>(best_pages);
}
DCHECK(slot_size <= kMaxSystemPagesPerSlotSpan * kSystemPageSize);
for (uint16_t i = kNumSystemPagesPerPartitionPage - 1;
i <= kMaxSystemPagesPerSlotSpan; ++i) {
size_t page_size = kSystemPageSize * i;
size_t num_slots = page_size / slot_size;
size_t waste = page_size - (num_slots * slot_size);
// Leaving a page unfaulted is not free; the page will occupy an empty page
// table entry. Make a simple attempt to account for that.
//
// TODO(ajwong): This looks wrong. PTEs are allocated for all pages
// regardless of whether or not they are wasted. Should it just
// be waste += i * sizeof(void*)?
// http://crbug.com/776537
size_t num_remainder_pages = i & (kNumSystemPagesPerPartitionPage - 1);
size_t num_unfaulted_pages =
num_remainder_pages
? (kNumSystemPagesPerPartitionPage - num_remainder_pages)
: 0;
waste += sizeof(void*) * num_unfaulted_pages;
double waste_ratio =
static_cast<double>(waste) / static_cast<double>(page_size);
if (waste_ratio < best_waste_ratio) {
best_waste_ratio = waste_ratio;
best_pages = i;
}
}
DCHECK(best_pages > 0);
CHECK(best_pages <= kMaxSystemPagesPerSlotSpan);
return static_cast<uint8_t>(best_pages);
}
void PartitionBucket::Init(uint32_t new_slot_size) {
slot_size = new_slot_size;
active_pages_head = PartitionPage::get_sentinel_page();
empty_pages_head = nullptr;
decommitted_pages_head = nullptr;
num_full_pages = 0;
num_system_pages_per_slot_span = get_system_pages_per_slot_span();
}
NOINLINE void PartitionBucket::OnFull() {
OOM_CRASH(0);
}
ALWAYS_INLINE void* PartitionBucket::AllocNewSlotSpan(
PartitionRootBase* root,
int flags,
uint16_t num_partition_pages) {
DCHECK(!(reinterpret_cast<uintptr_t>(root->next_partition_page) %
kPartitionPageSize));
DCHECK(!(reinterpret_cast<uintptr_t>(root->next_partition_page_end) %
kPartitionPageSize));
DCHECK(num_partition_pages <= kNumPartitionPagesPerSuperPage);
size_t total_size = kPartitionPageSize * num_partition_pages;
size_t num_partition_pages_left =
(root->next_partition_page_end - root->next_partition_page) >>
kPartitionPageShift;
if (LIKELY(num_partition_pages_left >= num_partition_pages)) {
// In this case, we can still hand out pages from the current super page
// allocation.
char* ret = root->next_partition_page;
// Fresh System Pages in the SuperPages are decommited. Commit them
// before vending them back.
SetSystemPagesAccess(ret, total_size, PageReadWrite);
root->next_partition_page += total_size;
root->IncreaseCommittedPages(total_size);
return ret;
}
// Need a new super page. We want to allocate super pages in a continguous
// address region as much as possible. This is important for not causing
// page table bloat and not fragmenting address spaces in 32 bit
// architectures.
char* requested_address = root->next_super_page;
char* super_page = reinterpret_cast<char*>(
AllocPages(requested_address, kSuperPageSize, kSuperPageSize,
PageReadWrite, PageTag::kPartitionAlloc));
if (UNLIKELY(!super_page))
return nullptr;
root->total_size_of_super_pages += kSuperPageSize;
root->IncreaseCommittedPages(total_size);
// |total_size| MUST be less than kSuperPageSize - (kPartitionPageSize*2).
// This is a trustworthy value because num_partition_pages is not user
// controlled.
//
// TODO(ajwong): Introduce a DCHECK.
root->next_super_page = super_page + kSuperPageSize;
char* ret = super_page + kPartitionPageSize;
root->next_partition_page = ret + total_size;
root->next_partition_page_end = root->next_super_page - kPartitionPageSize;
// Make the first partition page in the super page a guard page, but leave a
// hole in the middle.
// This is where we put page metadata and also a tiny amount of extent
// metadata.
SetSystemPagesAccess(super_page, kSystemPageSize, PageInaccessible);
SetSystemPagesAccess(super_page + (kSystemPageSize * 2),
kPartitionPageSize - (kSystemPageSize * 2),
PageInaccessible);
// SetSystemPagesAccess(super_page + (kSuperPageSize -
// kPartitionPageSize),
// kPartitionPageSize, PageInaccessible);
// All remaining slotspans for the unallocated PartitionPages inside the
// SuperPage are conceptually decommitted. Correctly set the state here
// so they do not occupy resources.
//
// TODO(ajwong): Refactor Page Allocator API so the SuperPage comes in
// decommited initially.
SetSystemPagesAccess(super_page + kPartitionPageSize + total_size,
(kSuperPageSize - kPartitionPageSize - total_size),
PageInaccessible);
// If we were after a specific address, but didn't get it, assume that
// the system chose a lousy address. Here most OS'es have a default
// algorithm that isn't randomized. For example, most Linux
// distributions will allocate the mapping directly before the last
// successful mapping, which is far from random. So we just get fresh
// randomness for the next mapping attempt.
if (requested_address && requested_address != super_page)
root->next_super_page = nullptr;
// We allocated a new super page so update super page metadata.
// First check if this is a new extent or not.
PartitionSuperPageExtentEntry* latest_extent =
reinterpret_cast<PartitionSuperPageExtentEntry*>(
PartitionSuperPageToMetadataArea(super_page));
// By storing the root in every extent metadata object, we have a fast way
// to go from a pointer within the partition to the root object.
latest_extent->root = root;
// Most new extents will be part of a larger extent, and these three fields
// are unused, but we initialize them to 0 so that we get a clear signal
// in case they are accidentally used.
latest_extent->super_page_base = nullptr;
latest_extent->super_pages_end = nullptr;
latest_extent->next = nullptr;
PartitionSuperPageExtentEntry* current_extent = root->current_extent;
bool is_new_extent = (super_page != requested_address);
if (UNLIKELY(is_new_extent)) {
if (UNLIKELY(!current_extent)) {
DCHECK(!root->first_extent);
root->first_extent = latest_extent;
} else {
DCHECK(current_extent->super_page_base);
current_extent->next = latest_extent;
}
root->current_extent = latest_extent;
latest_extent->super_page_base = super_page;
latest_extent->super_pages_end = super_page + kSuperPageSize;
} else {
// We allocated next to an existing extent so just nudge the size up a
// little.
DCHECK(current_extent->super_pages_end);
current_extent->super_pages_end += kSuperPageSize;
DCHECK(ret >= current_extent->super_page_base &&
ret < current_extent->super_pages_end);
}
return ret;
}
ALWAYS_INLINE uint16_t PartitionBucket::get_pages_per_slot_span() {
// Rounds up to nearest multiple of kNumSystemPagesPerPartitionPage.
return (num_system_pages_per_slot_span +
(kNumSystemPagesPerPartitionPage - 1)) /
kNumSystemPagesPerPartitionPage;
}
ALWAYS_INLINE void PartitionBucket::InitializeSlotSpan(PartitionPage* page) {
// The bucket never changes. We set it up once.
page->bucket = this;
page->empty_cache_index = -1;
page->Reset();
// If this page has just a single slot, do not set up page offsets for any
// page metadata other than the first one. This ensures that attempts to
// touch invalid page metadata fail.
if (page->num_unprovisioned_slots == 1)
return;
uint16_t num_partition_pages = get_pages_per_slot_span();
char* page_char_ptr = reinterpret_cast<char*>(page);
for (uint16_t i = 1; i < num_partition_pages; ++i) {
page_char_ptr += kPageMetadataSize;
PartitionPage* secondary_page =
reinterpret_cast<PartitionPage*>(page_char_ptr);
secondary_page->page_offset = i;
}
}
ALWAYS_INLINE char* PartitionBucket::AllocAndFillFreelist(PartitionPage* page) {
DCHECK(page != PartitionPage::get_sentinel_page());
uint16_t num_slots = page->num_unprovisioned_slots;
DCHECK(num_slots);
// We should only get here when _every_ slot is either used or unprovisioned.
// (The third state is "on the freelist". If we have a non-empty freelist, we
// should not get here.)
DCHECK(num_slots + page->num_allocated_slots == get_slots_per_span());
// Similarly, make explicitly sure that the freelist is empty.
DCHECK(!page->freelist_head);
DCHECK(page->num_allocated_slots >= 0);
size_t size = slot_size;
char* base = reinterpret_cast<char*>(PartitionPage::ToPointer(page));
char* return_object = base + (size * page->num_allocated_slots);
char* first_freelist_pointer = return_object + size;
char* first_freelist_pointer_extent =
first_freelist_pointer + sizeof(PartitionFreelistEntry*);
// Our goal is to fault as few system pages as possible. We calculate the
// page containing the "end" of the returned slot, and then allow freelist
// pointers to be written up to the end of that page.
char* sub_page_limit = reinterpret_cast<char*>(
RoundUpToSystemPage(reinterpret_cast<size_t>(first_freelist_pointer)));
char* slots_limit = return_object + (size * num_slots);
char* freelist_limit = sub_page_limit;
if (UNLIKELY(slots_limit < freelist_limit))
freelist_limit = slots_limit;
uint16_t num_new_freelist_entries = 0;
if (LIKELY(first_freelist_pointer_extent <= freelist_limit)) {
// Only consider used space in the slot span. If we consider wasted
// space, we may get an off-by-one when a freelist pointer fits in the
// wasted space, but a slot does not.
// We know we can fit at least one freelist pointer.
num_new_freelist_entries = 1;
// Any further entries require space for the whole slot span.
num_new_freelist_entries += static_cast<uint16_t>(
(freelist_limit - first_freelist_pointer_extent) / size);
}
// We always return an object slot -- that's the +1 below.
// We do not neccessarily create any new freelist entries, because we cross
// sub page boundaries frequently for large bucket sizes.
DCHECK(num_new_freelist_entries + 1 <= num_slots);
num_slots -= (num_new_freelist_entries + 1);
page->num_unprovisioned_slots = num_slots;
page->num_allocated_slots++;
if (LIKELY(num_new_freelist_entries)) {
char* freelist_pointer = first_freelist_pointer;
PartitionFreelistEntry* entry =
reinterpret_cast<PartitionFreelistEntry*>(freelist_pointer);
page->freelist_head = entry;
while (--num_new_freelist_entries) {
freelist_pointer += size;
PartitionFreelistEntry* next_entry =
reinterpret_cast<PartitionFreelistEntry*>(freelist_pointer);
entry->next = PartitionFreelistEntry::Encode(next_entry);
entry = next_entry;
}
entry->next = PartitionFreelistEntry::Encode(nullptr);
} else {
page->freelist_head = nullptr;
}
return return_object;
}
bool PartitionBucket::SetNewActivePage() {
PartitionPage* page = active_pages_head;
if (page == PartitionPage::get_sentinel_page())
return false;
PartitionPage* next_page;
for (; page; page = next_page) {
next_page = page->next_page;
DCHECK(page->bucket == this);
DCHECK(page != empty_pages_head);
DCHECK(page != decommitted_pages_head);
if (LIKELY(page->is_active())) {
// This page is usable because it has freelist entries, or has
// unprovisioned slots we can create freelist entries from.
active_pages_head = page;
return true;
}
// Deal with empty and decommitted pages.
if (LIKELY(page->is_empty())) {
page->next_page = empty_pages_head;
empty_pages_head = page;
} else if (LIKELY(page->is_decommitted())) {
page->next_page = decommitted_pages_head;
decommitted_pages_head = page;
} else {
DCHECK(page->is_full());
// If we get here, we found a full page. Skip over it too, and also
// tag it as full (via a negative value). We need it tagged so that
// free'ing can tell, and move it back into the active page list.
page->num_allocated_slots = -page->num_allocated_slots;
++num_full_pages;
// num_full_pages is a uint16_t for efficient packing so guard against
// overflow to be safe.
if (UNLIKELY(!num_full_pages))
OnFull();
// Not necessary but might help stop accidents.
page->next_page = nullptr;
}
}
active_pages_head = PartitionPage::get_sentinel_page();
return false;
}
void* PartitionBucket::SlowPathAlloc(PartitionRootBase* root,
int flags,
size_t size,
bool* is_already_zeroed) {
// The slow path is called when the freelist is empty.
DCHECK(!active_pages_head->freelist_head);
PartitionPage* new_page = nullptr;
*is_already_zeroed = false;
// For the PartitionRootGeneric::Alloc() API, we have a bunch of buckets
// marked as special cases. We bounce them through to the slow path so that
// we can still have a blazing fast hot path due to lack of corner-case
// branches.
//
// Note: The ordering of the conditionals matter! In particular,
// SetNewActivePage() has a side-effect even when returning
// false where it sweeps the active page list and may move things into
// the empty or decommitted lists which affects the subsequent conditional.
bool return_null = flags & PartitionAllocReturnNull;
if (UNLIKELY(is_direct_mapped())) {
DCHECK(size > kGenericMaxBucketed);
DCHECK(this == get_sentinel_bucket());
DCHECK(active_pages_head == PartitionPage::get_sentinel_page());
if (size > kGenericMaxDirectMapped) {
if (return_null)
return nullptr;
PartitionExcessiveAllocationSize(size);
}
new_page = PartitionDirectMap(root, flags, size);
*is_already_zeroed = true;
} else if (LIKELY(SetNewActivePage())) {
// First, did we find an active page in the active pages list?
new_page = active_pages_head;
DCHECK(new_page->is_active());
} else if (LIKELY(empty_pages_head != nullptr) ||
LIKELY(decommitted_pages_head != nullptr)) {
// Second, look in our lists of empty and decommitted pages.
// Check empty pages first, which are preferred, but beware that an
// empty page might have been decommitted.
while (LIKELY((new_page = empty_pages_head) != nullptr)) {
DCHECK(new_page->bucket == this);
DCHECK(new_page->is_empty() || new_page->is_decommitted());
empty_pages_head = new_page->next_page;
// Accept the empty page unless it got decommitted.
if (new_page->freelist_head) {
new_page->next_page = nullptr;
break;
}
DCHECK(new_page->is_decommitted());
new_page->next_page = decommitted_pages_head;
decommitted_pages_head = new_page;
}
if (UNLIKELY(!new_page) && LIKELY(decommitted_pages_head != nullptr)) {
new_page = decommitted_pages_head;
DCHECK(new_page->bucket == this);
DCHECK(new_page->is_decommitted());
decommitted_pages_head = new_page->next_page;
void* addr = PartitionPage::ToPointer(new_page);
root->RecommitSystemPages(addr, new_page->bucket->get_bytes_per_span());
new_page->Reset();
// TODO(https://crbug.com/890752): Optimizing here might cause pages to
// not be zeroed.
// *is_already_zeroed = true;
}
DCHECK(new_page);
} else {
// Third. If we get here, we need a brand new page.
uint16_t num_partition_pages = get_pages_per_slot_span();
void* raw_pages = AllocNewSlotSpan(root, flags, num_partition_pages);
if (LIKELY(raw_pages != nullptr)) {
new_page = PartitionPage::FromPointerNoAlignmentCheck(raw_pages);
InitializeSlotSpan(new_page);
// TODO(https://crbug.com/890752): Optimizing here causes pages to not be
// zeroed on at least macOS.
// *is_already_zeroed = true;
}
}
// Bail if we had a memory allocation failure.
if (UNLIKELY(!new_page)) {
DCHECK(active_pages_head == PartitionPage::get_sentinel_page());
if (return_null)
return nullptr;
root->OutOfMemory(size);
}
// TODO(ajwong): Is there a way to avoid the reading of bucket here?
// It seems like in many of the conditional branches above, |this| ==
// |new_page->bucket|. Maybe pull this into another function?
PartitionBucket* bucket = new_page->bucket;
DCHECK(bucket != get_sentinel_bucket());
bucket->active_pages_head = new_page;
new_page->set_raw_size(size);
// If we found an active page with free slots, or an empty page, we have a
// usable freelist head.
if (LIKELY(new_page->freelist_head != nullptr)) {
PartitionFreelistEntry* entry = new_page->freelist_head;
PartitionFreelistEntry* new_head =
EncodedPartitionFreelistEntry::Decode(entry->next);
new_page->freelist_head = new_head;
new_page->num_allocated_slots++;
return entry;
}
// Otherwise, we need to build the freelist.
DCHECK(new_page->num_unprovisioned_slots);
return AllocAndFillFreelist(new_page);
}
} // namespace internal
} // namespace base

View file

@ -0,0 +1,129 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_BUCKET_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_BUCKET_H_
#include <stddef.h>
#include <stdint.h>
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
#include "base/base_export.h"
#include "base/compiler_specific.h"
#include "base/logging.h"
namespace base {
namespace internal {
struct PartitionPage;
struct PartitionRootBase;
struct PartitionBucket {
// Accessed most in hot path => goes first.
PartitionPage* active_pages_head;
PartitionPage* empty_pages_head;
PartitionPage* decommitted_pages_head;
uint32_t slot_size;
uint32_t num_system_pages_per_slot_span : 8;
uint32_t num_full_pages : 24;
// Public API.
void Init(uint32_t new_slot_size);
// Sets |is_already_zeroed| to true if the allocation was satisfied by
// requesting (a) new page(s) from the operating system, or false otherwise.
// This enables an optimization for when callers use |PartitionAllocZeroFill|:
// there is no need to call memset on fresh pages; the OS has already zeroed
// them. (See |PartitionRootBase::AllocFromBucket|.)
//
// Note the matching Free() functions are in PartitionPage.
BASE_EXPORT NOINLINE void* SlowPathAlloc(PartitionRootBase* root,
int flags,
size_t size,
bool* is_already_zeroed);
ALWAYS_INLINE bool is_direct_mapped() const {
return !num_system_pages_per_slot_span;
}
ALWAYS_INLINE size_t get_bytes_per_span() const {
// TODO(ajwong): Change to CheckedMul. https://crbug.com/787153
// https://crbug.com/680657
return num_system_pages_per_slot_span * kSystemPageSize;
}
ALWAYS_INLINE uint16_t get_slots_per_span() const {
// TODO(ajwong): Change to CheckedMul. https://crbug.com/787153
// https://crbug.com/680657
return static_cast<uint16_t>(get_bytes_per_span() / slot_size);
}
static ALWAYS_INLINE size_t get_direct_map_size(size_t size) {
// Caller must check that the size is not above the kGenericMaxDirectMapped
// limit before calling. This also guards against integer overflow in the
// calculation here.
DCHECK(size <= kGenericMaxDirectMapped);
return (size + kSystemPageOffsetMask) & kSystemPageBaseMask;
}
// TODO(ajwong): Can this be made private? https://crbug.com/787153
static PartitionBucket* get_sentinel_bucket();
// This helper function scans a bucket's active page list for a suitable new
// active page. When it finds a suitable new active page (one that has
// free slots and is not empty), it is set as the new active page. If there
// is no suitable new active page, the current active page is set to
// PartitionPage::get_sentinel_page(). As potential pages are scanned, they
// are tidied up according to their state. Empty pages are swept on to the
// empty page list, decommitted pages on to the decommitted page list and full
// pages are unlinked from any list.
//
// This is where the guts of the bucket maintenance is done!
bool SetNewActivePage();
private:
static void OutOfMemory(const PartitionRootBase* root);
static void OutOfMemoryWithLotsOfUncommitedPages();
static NOINLINE void OnFull();
// Returns a natural number of PartitionPages (calculated by
// get_system_pages_per_slot_span()) to allocate from the current
// SuperPage when the bucket runs out of slots.
ALWAYS_INLINE uint16_t get_pages_per_slot_span();
// Returns the number of system pages in a slot span.
//
// The calculation attemps to find the best number of System Pages to
// allocate for the given slot_size to minimize wasted space. It uses a
// heuristic that looks at number of bytes wasted after the last slot and
// attempts to account for the PTE usage of each System Page.
uint8_t get_system_pages_per_slot_span();
// Allocates a new slot span with size |num_partition_pages| from the
// current extent. Metadata within this slot span will be uninitialized.
// Returns nullptr on error.
ALWAYS_INLINE void* AllocNewSlotSpan(PartitionRootBase* root,
int flags,
uint16_t num_partition_pages);
// Each bucket allocates a slot span when it runs out of slots.
// A slot span's size is equal to get_pages_per_slot_span() number of
// PartitionPages. This function initializes all PartitionPage within the
// span to point to the first PartitionPage which holds all the metadata
// for the span and registers this bucket as the owner of the span. It does
// NOT put the slots into the bucket's freelist.
ALWAYS_INLINE void InitializeSlotSpan(PartitionPage* page);
// Allocates one slot from the given |page| and then adds the remainder to
// the current bucket. If the |page| was freshly allocated, it must have been
// passed through InitializeSlotSpan() first.
ALWAYS_INLINE char* AllocAndFillFreelist(PartitionPage* page);
static PartitionBucket sentinel_bucket_;
};
} // namespace internal
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_BUCKET_H_

View file

@ -0,0 +1,70 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_COOKIE_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_COOKIE_H_
#include "base/compiler_specific.h"
#include "base/logging.h"
namespace base {
namespace internal {
#if DCHECK_IS_ON()
// Handles alignment up to XMM instructions on Intel.
static constexpr size_t kCookieSize = 16;
static constexpr unsigned char kCookieValue[kCookieSize] = {
0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xD0, 0x0D,
0x13, 0x37, 0xF0, 0x05, 0xBA, 0x11, 0xAB, 0x1E};
#endif
ALWAYS_INLINE void PartitionCookieCheckValue(void* ptr) {
#if DCHECK_IS_ON()
unsigned char* cookie_ptr = reinterpret_cast<unsigned char*>(ptr);
for (size_t i = 0; i < kCookieSize; ++i, ++cookie_ptr)
DCHECK(*cookie_ptr == kCookieValue[i]);
#endif
}
ALWAYS_INLINE size_t PartitionCookieSizeAdjustAdd(size_t size) {
#if DCHECK_IS_ON()
// Add space for cookies, checking for integer overflow. TODO(palmer):
// Investigate the performance and code size implications of using
// CheckedNumeric throughout PA.
DCHECK(size + (2 * kCookieSize) > size);
size += 2 * kCookieSize;
#endif
return size;
}
ALWAYS_INLINE void* PartitionCookieFreePointerAdjust(void* ptr) {
#if DCHECK_IS_ON()
// The value given to the application is actually just after the cookie.
ptr = static_cast<char*>(ptr) - kCookieSize;
#endif
return ptr;
}
ALWAYS_INLINE size_t PartitionCookieSizeAdjustSubtract(size_t size) {
#if DCHECK_IS_ON()
// Remove space for cookies.
DCHECK(size >= 2 * kCookieSize);
size -= 2 * kCookieSize;
#endif
return size;
}
ALWAYS_INLINE void PartitionCookieWriteValue(void* ptr) {
#if DCHECK_IS_ON()
unsigned char* cookie_ptr = reinterpret_cast<unsigned char*>(ptr);
for (size_t i = 0; i < kCookieSize; ++i, ++cookie_ptr)
*cookie_ptr = kCookieValue[i];
#endif
}
} // namespace internal
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_COOKIE_H_

View file

@ -0,0 +1,34 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_DIRECT_MAP_EXTENT_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_DIRECT_MAP_EXTENT_H_
#include "base/allocator/partition_allocator/partition_bucket.h"
#include "base/allocator/partition_allocator/partition_page.h"
#include "base/logging.h"
namespace base {
namespace internal {
struct PartitionDirectMapExtent {
PartitionDirectMapExtent* next_extent;
PartitionDirectMapExtent* prev_extent;
PartitionBucket* bucket;
size_t map_size; // Mapped size, not including guard pages and meta-data.
ALWAYS_INLINE static PartitionDirectMapExtent* FromPage(PartitionPage* page);
};
ALWAYS_INLINE PartitionDirectMapExtent* PartitionDirectMapExtent::FromPage(
PartitionPage* page) {
DCHECK(page->bucket->is_direct_mapped());
return reinterpret_cast<PartitionDirectMapExtent*>(
reinterpret_cast<char*>(page) + 3 * kPageMetadataSize);
}
} // namespace internal
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_DIRECT_MAP_EXTENT_H_

View file

@ -0,0 +1,71 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_FREELIST_ENTRY_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_FREELIST_ENTRY_H_
#include <stdint.h>
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
#include "base/compiler_specific.h"
#include "base/sys_byteorder.h"
#include "build/build_config.h"
namespace base {
namespace internal {
struct EncodedPartitionFreelistEntry;
struct PartitionFreelistEntry {
EncodedPartitionFreelistEntry* next;
PartitionFreelistEntry() = delete;
~PartitionFreelistEntry() = delete;
ALWAYS_INLINE static EncodedPartitionFreelistEntry* Encode(
PartitionFreelistEntry* ptr) {
return reinterpret_cast<EncodedPartitionFreelistEntry*>(Transform(ptr));
}
private:
friend struct EncodedPartitionFreelistEntry;
static ALWAYS_INLINE void* Transform(void* ptr) {
// We use bswap on little endian as a fast mask for two reasons:
// 1) If an object is freed and its vtable used where the attacker doesn't
// get the chance to run allocations between the free and use, the vtable
// dereference is likely to fault.
// 2) If the attacker has a linear buffer overflow and elects to try and
// corrupt a freelist pointer, partial pointer overwrite attacks are
// thwarted.
// For big endian, similar guarantees are arrived at with a negation.
#if defined(ARCH_CPU_BIG_ENDIAN)
uintptr_t masked = ~reinterpret_cast<uintptr_t>(ptr);
#else
uintptr_t masked = ByteSwapUintPtrT(reinterpret_cast<uintptr_t>(ptr));
#endif
return reinterpret_cast<void*>(masked);
}
};
struct EncodedPartitionFreelistEntry {
char scrambled[sizeof(PartitionFreelistEntry*)];
EncodedPartitionFreelistEntry() = delete;
~EncodedPartitionFreelistEntry() = delete;
ALWAYS_INLINE static PartitionFreelistEntry* Decode(
EncodedPartitionFreelistEntry* ptr) {
return reinterpret_cast<PartitionFreelistEntry*>(
PartitionFreelistEntry::Transform(ptr));
}
};
static_assert(sizeof(PartitionFreelistEntry) ==
sizeof(EncodedPartitionFreelistEntry),
"Should not have padding");
} // namespace internal
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_FREELIST_ENTRY_H_

View file

@ -0,0 +1,24 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/partition_oom.h"
#include "base/allocator/partition_allocator/oom.h"
#include "build/build_config.h"
namespace base {
namespace internal {
void NOINLINE PartitionExcessiveAllocationSize(size_t size) {
OOM_CRASH(size);
}
#if !defined(ARCH_CPU_64_BITS)
NOINLINE void PartitionOutOfMemoryWithLotsOfUncommitedPages(size_t size) {
OOM_CRASH(size);
}
#endif
} // namespace internal
} // namespace base

View file

@ -0,0 +1,28 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// Holds functions for generating OOM errors from PartitionAlloc. This is
// distinct from oom.h in that it is meant only for use in PartitionAlloc.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_OOM_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_OOM_H_
#include <stddef.h>
#include "base/compiler_specific.h"
#include "build/build_config.h"
namespace base {
namespace internal {
NOINLINE void PartitionExcessiveAllocationSize(size_t size);
#if !defined(ARCH_CPU_64_BITS)
NOINLINE void PartitionOutOfMemoryWithLotsOfUncommitedPages(size_t size);
#endif
} // namespace internal
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_OOM_H_

View file

@ -0,0 +1,164 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/partition_page.h"
#include "base/allocator/partition_allocator/partition_direct_map_extent.h"
#include "base/allocator/partition_allocator/partition_root_base.h"
#include "base/logging.h"
namespace base {
namespace internal {
namespace {
ALWAYS_INLINE void PartitionDirectUnmap(PartitionPage* page) {
PartitionRootBase* root = PartitionRootBase::FromPage(page);
const PartitionDirectMapExtent* extent =
PartitionDirectMapExtent::FromPage(page);
size_t unmap_size = extent->map_size;
// Maintain the doubly-linked list of all direct mappings.
if (extent->prev_extent) {
DCHECK(extent->prev_extent->next_extent == extent);
extent->prev_extent->next_extent = extent->next_extent;
} else {
root->direct_map_list = extent->next_extent;
}
if (extent->next_extent) {
DCHECK(extent->next_extent->prev_extent == extent);
extent->next_extent->prev_extent = extent->prev_extent;
}
// Add on the size of the trailing guard page and preceeding partition
// page.
unmap_size += kPartitionPageSize + kSystemPageSize;
size_t uncommitted_page_size = page->bucket->slot_size + kSystemPageSize;
root->DecreaseCommittedPages(uncommitted_page_size);
DCHECK(root->total_size_of_direct_mapped_pages >= uncommitted_page_size);
root->total_size_of_direct_mapped_pages -= uncommitted_page_size;
DCHECK(!(unmap_size & kPageAllocationGranularityOffsetMask));
char* ptr = reinterpret_cast<char*>(PartitionPage::ToPointer(page));
// Account for the mapping starting a partition page before the actual
// allocation address.
ptr -= kPartitionPageSize;
FreePages(ptr, unmap_size);
}
ALWAYS_INLINE void PartitionRegisterEmptyPage(PartitionPage* page) {
DCHECK(page->is_empty());
PartitionRootBase* root = PartitionRootBase::FromPage(page);
// If the page is already registered as empty, give it another life.
if (page->empty_cache_index != -1) {
DCHECK(page->empty_cache_index >= 0);
DCHECK(static_cast<unsigned>(page->empty_cache_index) < kMaxFreeableSpans);
DCHECK(root->global_empty_page_ring[page->empty_cache_index] == page);
root->global_empty_page_ring[page->empty_cache_index] = nullptr;
}
int16_t current_index = root->global_empty_page_ring_index;
PartitionPage* page_to_decommit = root->global_empty_page_ring[current_index];
// The page might well have been re-activated, filled up, etc. before we get
// around to looking at it here.
if (page_to_decommit)
page_to_decommit->DecommitIfPossible(root);
// We put the empty slot span on our global list of "pages that were once
// empty". thus providing it a bit of breathing room to get re-used before
// we really free it. This improves performance, particularly on Mac OS X
// which has subpar memory management performance.
root->global_empty_page_ring[current_index] = page;
page->empty_cache_index = current_index;
++current_index;
if (current_index == kMaxFreeableSpans)
current_index = 0;
root->global_empty_page_ring_index = current_index;
}
} // namespace
// static
PartitionPage PartitionPage::sentinel_page_;
PartitionPage* PartitionPage::get_sentinel_page() {
return &sentinel_page_;
}
void PartitionPage::FreeSlowPath() {
DCHECK(this != get_sentinel_page());
if (LIKELY(num_allocated_slots == 0)) {
// Page became fully unused.
if (UNLIKELY(bucket->is_direct_mapped())) {
PartitionDirectUnmap(this);
return;
}
// If it's the current active page, change it. We bounce the page to
// the empty list as a force towards defragmentation.
if (LIKELY(this == bucket->active_pages_head))
bucket->SetNewActivePage();
DCHECK(bucket->active_pages_head != this);
set_raw_size(0);
DCHECK(!get_raw_size());
PartitionRegisterEmptyPage(this);
} else {
DCHECK(!bucket->is_direct_mapped());
// Ensure that the page is full. That's the only valid case if we
// arrive here.
DCHECK(num_allocated_slots < 0);
// A transition of num_allocated_slots from 0 to -1 is not legal, and
// likely indicates a double-free.
CHECK(num_allocated_slots != -1);
num_allocated_slots = -num_allocated_slots - 2;
DCHECK(num_allocated_slots == bucket->get_slots_per_span() - 1);
// Fully used page became partially used. It must be put back on the
// non-full page list. Also make it the current page to increase the
// chances of it being filled up again. The old current page will be
// the next page.
DCHECK(!next_page);
if (LIKELY(bucket->active_pages_head != get_sentinel_page()))
next_page = bucket->active_pages_head;
bucket->active_pages_head = this;
--bucket->num_full_pages;
// Special case: for a partition page with just a single slot, it may
// now be empty and we want to run it through the empty logic.
if (UNLIKELY(num_allocated_slots == 0))
FreeSlowPath();
}
}
void PartitionPage::Decommit(PartitionRootBase* root) {
DCHECK(is_empty());
DCHECK(!bucket->is_direct_mapped());
void* addr = PartitionPage::ToPointer(this);
root->DecommitSystemPages(addr, bucket->get_bytes_per_span());
// We actually leave the decommitted page in the active list. We'll sweep
// it on to the decommitted page list when we next walk the active page
// list.
// Pulling this trick enables us to use a singly-linked page list for all
// cases, which is critical in keeping the page metadata structure down to
// 32 bytes in size.
freelist_head = nullptr;
num_unprovisioned_slots = 0;
DCHECK(is_decommitted());
}
void PartitionPage::DecommitIfPossible(PartitionRootBase* root) {
DCHECK(empty_cache_index >= 0);
DCHECK(static_cast<unsigned>(empty_cache_index) < kMaxFreeableSpans);
DCHECK(this == root->global_empty_page_ring[empty_cache_index]);
empty_cache_index = -1;
if (is_empty())
Decommit(root);
}
} // namespace internal
} // namespace base

View file

@ -0,0 +1,293 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_PAGE_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_PAGE_H_
#include <string.h>
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
#include "base/allocator/partition_allocator/partition_bucket.h"
#include "base/allocator/partition_allocator/partition_cookie.h"
#include "base/allocator/partition_allocator/partition_freelist_entry.h"
#include "base/allocator/partition_allocator/random.h"
#include "base/logging.h"
namespace base {
namespace internal {
struct PartitionRootBase;
// Some notes on page states. A page can be in one of four major states:
// 1) Active.
// 2) Full.
// 3) Empty.
// 4) Decommitted.
// An active page has available free slots. A full page has no free slots. An
// empty page has no free slots, and a decommitted page is an empty page that
// had its backing memory released back to the system.
// There are two linked lists tracking the pages. The "active page" list is an
// approximation of a list of active pages. It is an approximation because
// full, empty and decommitted pages may briefly be present in the list until
// we next do a scan over it.
// The "empty page" list is an accurate list of pages which are either empty
// or decommitted.
//
// The significant page transitions are:
// - free() will detect when a full page has a slot free()'d and immediately
// return the page to the head of the active list.
// - free() will detect when a page is fully emptied. It _may_ add it to the
// empty list or it _may_ leave it on the active list until a future list scan.
// - malloc() _may_ scan the active page list in order to fulfil the request.
// If it does this, full, empty and decommitted pages encountered will be
// booted out of the active list. If there are no suitable active pages found,
// an empty or decommitted page (if one exists) will be pulled from the empty
// list on to the active list.
//
// TODO(ajwong): Evaluate if this should be named PartitionSlotSpanMetadata or
// similar. If so, all uses of the term "page" in comments, member variables,
// local variables, and documentation that refer to this concept should be
// updated.
struct PartitionPage {
PartitionFreelistEntry* freelist_head;
PartitionPage* next_page;
PartitionBucket* bucket;
// Deliberately signed, 0 for empty or decommitted page, -n for full pages:
int16_t num_allocated_slots;
uint16_t num_unprovisioned_slots;
uint16_t page_offset;
int16_t empty_cache_index; // -1 if not in the empty cache.
// Public API
// Note the matching Alloc() functions are in PartitionPage.
BASE_EXPORT NOINLINE void FreeSlowPath();
ALWAYS_INLINE void Free(void* ptr);
void Decommit(PartitionRootBase* root);
void DecommitIfPossible(PartitionRootBase* root);
// Pointer manipulation functions. These must be static as the input |page|
// pointer may be the result of an offset calculation and therefore cannot
// be trusted. The objective of these functions is to sanitize this input.
ALWAYS_INLINE static void* ToPointer(const PartitionPage* page);
ALWAYS_INLINE static PartitionPage* FromPointerNoAlignmentCheck(void* ptr);
ALWAYS_INLINE static PartitionPage* FromPointer(void* ptr);
ALWAYS_INLINE const size_t* get_raw_size_ptr() const;
ALWAYS_INLINE size_t* get_raw_size_ptr() {
return const_cast<size_t*>(
const_cast<const PartitionPage*>(this)->get_raw_size_ptr());
}
ALWAYS_INLINE size_t get_raw_size() const;
ALWAYS_INLINE void set_raw_size(size_t size);
ALWAYS_INLINE void Reset();
// TODO(ajwong): Can this be made private? https://crbug.com/787153
BASE_EXPORT static PartitionPage* get_sentinel_page();
// Page State accessors.
// Note that it's only valid to call these functions on pages found on one of
// the page lists. Specifically, you can't call these functions on full pages
// that were detached from the active list.
//
// This restriction provides the flexibity for some of the status fields to
// be repurposed when a page is taken off a list. See the negation of
// |num_allocated_slots| when a full page is removed from the active list
// for an example of such repurposing.
ALWAYS_INLINE bool is_active() const;
ALWAYS_INLINE bool is_full() const;
ALWAYS_INLINE bool is_empty() const;
ALWAYS_INLINE bool is_decommitted() const;
private:
// g_sentinel_page is used as a sentinel to indicate that there is no page
// in the active page list. We can use nullptr, but in that case we need
// to add a null-check branch to the hot allocation path. We want to avoid
// that.
//
// Note, this declaration is kept in the header as opposed to an anonymous
// namespace so the getter can be fully inlined.
static PartitionPage sentinel_page_;
};
static_assert(sizeof(PartitionPage) <= kPageMetadataSize,
"PartitionPage must be able to fit in a metadata slot");
ALWAYS_INLINE char* PartitionSuperPageToMetadataArea(char* ptr) {
uintptr_t pointer_as_uint = reinterpret_cast<uintptr_t>(ptr);
DCHECK(!(pointer_as_uint & kSuperPageOffsetMask));
// The metadata area is exactly one system page (the guard page) into the
// super page.
return reinterpret_cast<char*>(pointer_as_uint + kSystemPageSize);
}
ALWAYS_INLINE PartitionPage* PartitionPage::FromPointerNoAlignmentCheck(
void* ptr) {
uintptr_t pointer_as_uint = reinterpret_cast<uintptr_t>(ptr);
char* super_page_ptr =
reinterpret_cast<char*>(pointer_as_uint & kSuperPageBaseMask);
uintptr_t partition_page_index =
(pointer_as_uint & kSuperPageOffsetMask) >> kPartitionPageShift;
// Index 0 is invalid because it is the metadata and guard area and
// the last index is invalid because it is a guard page.
DCHECK(partition_page_index);
DCHECK(partition_page_index < kNumPartitionPagesPerSuperPage - 1);
PartitionPage* page = reinterpret_cast<PartitionPage*>(
PartitionSuperPageToMetadataArea(super_page_ptr) +
(partition_page_index << kPageMetadataShift));
// Partition pages in the same slot span can share the same page object.
// Adjust for that.
size_t delta = page->page_offset << kPageMetadataShift;
page =
reinterpret_cast<PartitionPage*>(reinterpret_cast<char*>(page) - delta);
return page;
}
// Resturns start of the slot span for the PartitionPage.
ALWAYS_INLINE void* PartitionPage::ToPointer(const PartitionPage* page) {
uintptr_t pointer_as_uint = reinterpret_cast<uintptr_t>(page);
uintptr_t super_page_offset = (pointer_as_uint & kSuperPageOffsetMask);
// A valid |page| must be past the first guard System page and within
// the following metadata region.
DCHECK(super_page_offset > kSystemPageSize);
// Must be less than total metadata region.
DCHECK(super_page_offset < kSystemPageSize + (kNumPartitionPagesPerSuperPage *
kPageMetadataSize));
uintptr_t partition_page_index =
(super_page_offset - kSystemPageSize) >> kPageMetadataShift;
// Index 0 is invalid because it is the superpage extent metadata and the
// last index is invalid because the whole PartitionPage is set as guard
// pages for the metadata region.
DCHECK(partition_page_index);
DCHECK(partition_page_index < kNumPartitionPagesPerSuperPage - 1);
uintptr_t super_page_base = (pointer_as_uint & kSuperPageBaseMask);
void* ret = reinterpret_cast<void*>(
super_page_base + (partition_page_index << kPartitionPageShift));
return ret;
}
ALWAYS_INLINE PartitionPage* PartitionPage::FromPointer(void* ptr) {
PartitionPage* page = PartitionPage::FromPointerNoAlignmentCheck(ptr);
// Checks that the pointer is a multiple of bucket size.
DCHECK(!((reinterpret_cast<uintptr_t>(ptr) -
reinterpret_cast<uintptr_t>(PartitionPage::ToPointer(page))) %
page->bucket->slot_size));
return page;
}
ALWAYS_INLINE const size_t* PartitionPage::get_raw_size_ptr() const {
// For single-slot buckets which span more than one partition page, we
// have some spare metadata space to store the raw allocation size. We
// can use this to report better statistics.
if (bucket->slot_size <= kMaxSystemPagesPerSlotSpan * kSystemPageSize)
return nullptr;
DCHECK((bucket->slot_size % kSystemPageSize) == 0);
DCHECK(bucket->is_direct_mapped() || bucket->get_slots_per_span() == 1);
const PartitionPage* the_next_page = this + 1;
return reinterpret_cast<const size_t*>(&the_next_page->freelist_head);
}
ALWAYS_INLINE size_t PartitionPage::get_raw_size() const {
const size_t* ptr = get_raw_size_ptr();
if (UNLIKELY(ptr != nullptr))
return *ptr;
return 0;
}
ALWAYS_INLINE void PartitionPage::Free(void* ptr) {
#if DCHECK_IS_ON()
size_t slot_size = bucket->slot_size;
const size_t raw_size = get_raw_size();
if (raw_size) {
slot_size = raw_size;
}
// If these asserts fire, you probably corrupted memory.
PartitionCookieCheckValue(ptr);
PartitionCookieCheckValue(reinterpret_cast<char*>(ptr) + slot_size -
kCookieSize);
memset(ptr, kFreedByte, slot_size);
#endif
DCHECK(num_allocated_slots);
// Catches an immediate double free.
CHECK(ptr != freelist_head);
// Look for double free one level deeper in debug.
DCHECK(!freelist_head ||
ptr != EncodedPartitionFreelistEntry::Decode(freelist_head->next));
internal::PartitionFreelistEntry* entry =
static_cast<internal::PartitionFreelistEntry*>(ptr);
entry->next = internal::PartitionFreelistEntry::Encode(freelist_head);
freelist_head = entry;
--num_allocated_slots;
if (UNLIKELY(num_allocated_slots <= 0)) {
FreeSlowPath();
} else {
// All single-slot allocations must go through the slow path to
// correctly update the size metadata.
DCHECK(get_raw_size() == 0);
}
}
ALWAYS_INLINE bool PartitionPage::is_active() const {
DCHECK(this != get_sentinel_page());
DCHECK(!page_offset);
return (num_allocated_slots > 0 &&
(freelist_head || num_unprovisioned_slots));
}
ALWAYS_INLINE bool PartitionPage::is_full() const {
DCHECK(this != get_sentinel_page());
DCHECK(!page_offset);
bool ret = (num_allocated_slots == bucket->get_slots_per_span());
if (ret) {
DCHECK(!freelist_head);
DCHECK(!num_unprovisioned_slots);
}
return ret;
}
ALWAYS_INLINE bool PartitionPage::is_empty() const {
DCHECK(this != get_sentinel_page());
DCHECK(!page_offset);
return (!num_allocated_slots && freelist_head);
}
ALWAYS_INLINE bool PartitionPage::is_decommitted() const {
DCHECK(this != get_sentinel_page());
DCHECK(!page_offset);
bool ret = (!num_allocated_slots && !freelist_head);
if (ret) {
DCHECK(!num_unprovisioned_slots);
DCHECK(empty_cache_index == -1);
}
return ret;
}
ALWAYS_INLINE void PartitionPage::set_raw_size(size_t size) {
size_t* raw_size_ptr = get_raw_size_ptr();
if (UNLIKELY(raw_size_ptr != nullptr))
*raw_size_ptr = size;
}
ALWAYS_INLINE void PartitionPage::Reset() {
DCHECK(is_decommitted());
num_unprovisioned_slots = bucket->get_slots_per_span();
DCHECK(num_unprovisioned_slots);
next_page = nullptr;
}
} // namespace internal
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_PAGE_H_

View file

@ -0,0 +1,40 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/partition_root_base.h"
#include "base/allocator/partition_allocator/oom.h"
#include "base/allocator/partition_allocator/partition_oom.h"
#include "base/allocator/partition_allocator/partition_page.h"
#include "build/build_config.h"
namespace base {
namespace internal {
NOINLINE void PartitionRootBase::OutOfMemory(size_t size) {
#if !defined(ARCH_CPU_64_BITS)
// Check whether this OOM is due to a lot of super pages that are allocated
// but not committed, probably due to http://crbug.com/421387.
if (total_size_of_super_pages + total_size_of_direct_mapped_pages -
total_size_of_committed_pages >
kReasonableSizeOfUnusedPages) {
PartitionOutOfMemoryWithLotsOfUncommitedPages(size);
}
#endif
if (PartitionRootBase::g_oom_handling_function)
(*PartitionRootBase::g_oom_handling_function)(size);
OOM_CRASH(size);
}
void PartitionRootBase::DecommitEmptyPages() {
for (size_t i = 0; i < kMaxFreeableSpans; ++i) {
internal::PartitionPage* page = global_empty_page_ring[i];
if (page)
page->DecommitIfPossible(this);
global_empty_page_ring[i] = nullptr;
}
}
} // namespace internal
} // namespace base

View file

@ -0,0 +1,200 @@
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ROOT_BASE_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ROOT_BASE_H_
#include "base/allocator/partition_allocator/page_allocator.h"
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
#include "base/allocator/partition_allocator/partition_bucket.h"
#include "base/allocator/partition_allocator/partition_direct_map_extent.h"
#include "base/allocator/partition_allocator/partition_page.h"
#include "base/logging.h"
#include "build/build_config.h"
namespace base {
typedef void (*OomFunction)(size_t);
namespace internal {
struct PartitionPage;
struct PartitionRootBase;
// An "extent" is a span of consecutive superpages. We link to the partition's
// next extent (if there is one) to the very start of a superpage's metadata
// area.
struct PartitionSuperPageExtentEntry {
PartitionRootBase* root;
char* super_page_base;
char* super_pages_end;
PartitionSuperPageExtentEntry* next;
};
static_assert(
sizeof(PartitionSuperPageExtentEntry) <= kPageMetadataSize,
"PartitionSuperPageExtentEntry must be able to fit in a metadata slot");
struct BASE_EXPORT PartitionRootBase {
PartitionRootBase();
virtual ~PartitionRootBase();
size_t total_size_of_committed_pages = 0;
size_t total_size_of_super_pages = 0;
size_t total_size_of_direct_mapped_pages = 0;
// Invariant: total_size_of_committed_pages <=
// total_size_of_super_pages +
// total_size_of_direct_mapped_pages.
unsigned num_buckets = 0;
unsigned max_allocation = 0;
bool initialized = false;
char* next_super_page = nullptr;
char* next_partition_page = nullptr;
char* next_partition_page_end = nullptr;
PartitionSuperPageExtentEntry* current_extent = nullptr;
PartitionSuperPageExtentEntry* first_extent = nullptr;
PartitionDirectMapExtent* direct_map_list = nullptr;
PartitionPage* global_empty_page_ring[kMaxFreeableSpans] = {};
int16_t global_empty_page_ring_index = 0;
uintptr_t inverted_self = 0;
// Public API
// Allocates out of the given bucket. Properly, this function should probably
// be in PartitionBucket, but because the implementation needs to be inlined
// for performance, and because it needs to inspect PartitionPage,
// it becomes impossible to have it in PartitionBucket as this causes a
// cyclical dependency on PartitionPage function implementations.
//
// Moving it a layer lower couples PartitionRootBase and PartitionBucket, but
// preserves the layering of the includes.
//
// Note the matching Free() functions are in PartitionPage.
ALWAYS_INLINE void* AllocFromBucket(PartitionBucket* bucket,
int flags,
size_t size);
ALWAYS_INLINE static bool IsValidPage(PartitionPage* page);
ALWAYS_INLINE static PartitionRootBase* FromPage(PartitionPage* page);
// g_oom_handling_function is invoked when PartitionAlloc hits OutOfMemory.
static OomFunction g_oom_handling_function;
NOINLINE void OutOfMemory(size_t size);
ALWAYS_INLINE void IncreaseCommittedPages(size_t len);
ALWAYS_INLINE void DecreaseCommittedPages(size_t len);
ALWAYS_INLINE void DecommitSystemPages(void* address, size_t length);
ALWAYS_INLINE void RecommitSystemPages(void* address, size_t length);
// Frees memory from this partition, if possible, by decommitting pages.
// |flags| is an OR of base::PartitionPurgeFlags.
virtual void PurgeMemory(int flags) = 0;
void DecommitEmptyPages();
};
ALWAYS_INLINE void* PartitionRootBase::AllocFromBucket(PartitionBucket* bucket,
int flags,
size_t size) {
bool zero_fill = flags & PartitionAllocZeroFill;
bool is_already_zeroed = false;
PartitionPage* page = bucket->active_pages_head;
// Check that this page is neither full nor freed.
DCHECK(page->num_allocated_slots >= 0);
void* ret = page->freelist_head;
if (LIKELY(ret != 0)) {
// If these DCHECKs fire, you probably corrupted memory. TODO(palmer): See
// if we can afford to make these CHECKs.
DCHECK(PartitionRootBase::IsValidPage(page));
// All large allocations must go through the slow path to correctly update
// the size metadata.
DCHECK(page->get_raw_size() == 0);
internal::PartitionFreelistEntry* new_head =
internal::EncodedPartitionFreelistEntry::Decode(
page->freelist_head->next);
page->freelist_head = new_head;
page->num_allocated_slots++;
} else {
ret = bucket->SlowPathAlloc(this, flags, size, &is_already_zeroed);
// TODO(palmer): See if we can afford to make this a CHECK.
DCHECK(!ret ||
PartitionRootBase::IsValidPage(PartitionPage::FromPointer(ret)));
}
#if DCHECK_IS_ON()
if (!ret) {
return nullptr;
}
page = PartitionPage::FromPointer(ret);
// TODO(ajwong): Can |page->bucket| ever not be |this|? If not, can this just
// be bucket->slot_size?
size_t new_slot_size = page->bucket->slot_size;
size_t raw_size = page->get_raw_size();
if (raw_size) {
DCHECK(raw_size == size);
new_slot_size = raw_size;
}
size_t no_cookie_size = PartitionCookieSizeAdjustSubtract(new_slot_size);
char* char_ret = static_cast<char*>(ret);
// The value given to the application is actually just after the cookie.
ret = char_ret + kCookieSize;
// Fill the region kUninitializedByte or 0, and surround it with 2 cookies.
PartitionCookieWriteValue(char_ret);
if (!zero_fill) {
memset(ret, kUninitializedByte, no_cookie_size);
} else if (!is_already_zeroed) {
memset(ret, 0, no_cookie_size);
}
PartitionCookieWriteValue(char_ret + kCookieSize + no_cookie_size);
#else
if (ret && zero_fill && !is_already_zeroed) {
memset(ret, 0, size);
}
#endif
return ret;
}
ALWAYS_INLINE bool PartitionRootBase::IsValidPage(PartitionPage* page) {
PartitionRootBase* root = PartitionRootBase::FromPage(page);
return root->inverted_self == ~reinterpret_cast<uintptr_t>(root);
}
ALWAYS_INLINE PartitionRootBase* PartitionRootBase::FromPage(
PartitionPage* page) {
PartitionSuperPageExtentEntry* extent_entry =
reinterpret_cast<PartitionSuperPageExtentEntry*>(
reinterpret_cast<uintptr_t>(page) & kSystemPageBaseMask);
return extent_entry->root;
}
ALWAYS_INLINE void PartitionRootBase::IncreaseCommittedPages(size_t len) {
total_size_of_committed_pages += len;
DCHECK(total_size_of_committed_pages <=
total_size_of_super_pages + total_size_of_direct_mapped_pages);
}
ALWAYS_INLINE void PartitionRootBase::DecreaseCommittedPages(size_t len) {
total_size_of_committed_pages -= len;
DCHECK(total_size_of_committed_pages <=
total_size_of_super_pages + total_size_of_direct_mapped_pages);
}
ALWAYS_INLINE void PartitionRootBase::DecommitSystemPages(void* address,
size_t length) {
::base::DecommitSystemPages(address, length);
DecreaseCommittedPages(length);
}
ALWAYS_INLINE void PartitionRootBase::RecommitSystemPages(void* address,
size_t length) {
CHECK(::base::RecommitSystemPages(address, length, PageReadWrite));
IncreaseCommittedPages(length);
}
} // namespace internal
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ROOT_BASE_H_

View file

@ -0,0 +1,74 @@
// Copyright 2019 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/random.h"
#include "base/allocator/partition_allocator/spin_lock.h"
#include "base/logging.h"
#include "base/no_destructor.h"
#include "base/rand_util.h"
#include "base/synchronization/lock.h"
namespace base {
namespace {
Lock& GetLock() {
static NoDestructor<Lock> lock;
return *lock;
}
} // namespace
// This is the same PRNG as used by tcmalloc for mapping address randomness;
// see http://burtleburtle.net/bob/rand/smallprng.html.
struct RandomContext {
bool initialized;
uint32_t a;
uint32_t b;
uint32_t c;
uint32_t d;
};
static RandomContext g_context GUARDED_BY(GetLock());
namespace {
RandomContext& GetRandomContext() EXCLUSIVE_LOCKS_REQUIRED(GetLock()) {
if (UNLIKELY(!g_context.initialized)) {
const uint64_t r1 = RandUint64();
const uint64_t r2 = RandUint64();
g_context.a = static_cast<uint32_t>(r1);
g_context.b = static_cast<uint32_t>(r1 >> 32);
g_context.c = static_cast<uint32_t>(r2);
g_context.d = static_cast<uint32_t>(r2 >> 32);
g_context.initialized = true;
}
return g_context;
}
} // namespace
uint32_t RandomValue() {
AutoLock guard(GetLock());
RandomContext& x = GetRandomContext();
#define rot(x, k) (((x) << (k)) | ((x) >> (32 - (k))))
uint32_t e = x.a - rot(x.b, 27);
x.a = x.b ^ rot(x.c, 17);
x.b = x.c + x.d;
x.c = x.d + e;
x.d = e + x.a;
return x.d;
#undef rot
}
void SetMmapSeedForTesting(uint64_t seed) {
AutoLock guard(GetLock());
RandomContext& x = GetRandomContext();
x.a = x.b = static_cast<uint32_t>(seed);
x.c = x.d = static_cast<uint32_t>(seed >> 32);
x.initialized = true;
}
} // namespace base

View file

@ -0,0 +1,27 @@
// Copyright 2019 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_RANDOM_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_RANDOM_H_
#include <stdint.h>
#include "base/base_export.h"
namespace base {
// Returns a random value. The generator's internal state is initialized with
// `base::RandUint64` which is very unpredictable, but which is expensive due to
// the need to call into the kernel. Therefore this generator uses a fast,
// entirely user-space function after initialization.
BASE_EXPORT uint32_t RandomValue();
// Sets the seed for the random number generator to a known value, to cause the
// RNG to generate a predictable sequence of outputs. May be called multiple
// times.
BASE_EXPORT void SetMmapSeedForTesting(uint64_t seed);
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_RANDOM_H_

View file

@ -0,0 +1,104 @@
// Copyright 2015 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "base/allocator/partition_allocator/spin_lock.h"
#include "base/threading/platform_thread.h"
#include "build/build_config.h"
#if defined(OS_WIN)
#include <windows.h>
#elif defined(OS_POSIX) || defined(OS_FUCHSIA)
#include <sched.h>
#endif
// The YIELD_PROCESSOR macro wraps an architecture specific-instruction that
// informs the processor we're in a busy wait, so it can handle the branch more
// intelligently and e.g. reduce power to our core or give more resources to the
// other hyper-thread on this core. See the following for context:
// https://software.intel.com/en-us/articles/benefitting-power-and-performance-sleep-loops
//
// The YIELD_THREAD macro tells the OS to relinquish our quantum. This is
// basically a worst-case fallback, and if you're hitting it with any frequency
// you really should be using a proper lock (such as |base::Lock|)rather than
// these spinlocks.
#if defined(OS_WIN)
#define YIELD_PROCESSOR YieldProcessor()
#define YIELD_THREAD SwitchToThread()
#elif defined(OS_POSIX) || defined(OS_FUCHSIA)
#if defined(ARCH_CPU_X86_64) || defined(ARCH_CPU_X86)
#define YIELD_PROCESSOR __asm__ __volatile__("pause")
#elif (defined(ARCH_CPU_ARMEL) && __ARM_ARCH >= 6) || defined(ARCH_CPU_ARM64)
#define YIELD_PROCESSOR __asm__ __volatile__("yield")
#elif defined(ARCH_CPU_MIPSEL)
// The MIPS32 docs state that the PAUSE instruction is a no-op on older
// architectures (first added in MIPS32r2). To avoid assembler errors when
// targeting pre-r2, we must encode the instruction manually.
#define YIELD_PROCESSOR __asm__ __volatile__(".word 0x00000140")
#elif defined(ARCH_CPU_MIPS64EL) && __mips_isa_rev >= 2
// Don't bother doing using .word here since r2 is the lowest supported mips64
// that Chromium supports.
#define YIELD_PROCESSOR __asm__ __volatile__("pause")
#elif defined(ARCH_CPU_PPC64_FAMILY)
#define YIELD_PROCESSOR __asm__ __volatile__("or 31,31,31")
#elif defined(ARCH_CPU_S390_FAMILY)
// just do nothing
#define YIELD_PROCESSOR ((void)0)
#endif // ARCH
#ifndef YIELD_PROCESSOR
#warning "Processor yield not supported on this architecture."
#define YIELD_PROCESSOR ((void)0)
#endif
#define YIELD_THREAD sched_yield()
#else // Other OS
#warning "Thread yield not supported on this OS."
#define YIELD_THREAD ((void)0)
#endif // OS_WIN
namespace base {
namespace subtle {
void SpinLock::LockSlow() {
// The value of |kYieldProcessorTries| is cargo culted from TCMalloc, Windows
// critical section defaults, and various other recommendations.
// TODO(jschuh): Further tuning may be warranted.
static const int kYieldProcessorTries = 1000;
// The value of |kYieldThreadTries| is completely made up.
static const int kYieldThreadTries = 10;
int yield_thread_count = 0;
do {
do {
for (int count = 0; count < kYieldProcessorTries; ++count) {
// Let the processor know we're spinning.
YIELD_PROCESSOR;
if (!lock_.load(std::memory_order_relaxed) &&
LIKELY(!lock_.exchange(true, std::memory_order_acquire)))
return;
}
if (yield_thread_count < kYieldThreadTries) {
++yield_thread_count;
// Give the OS a chance to schedule something on this core.
YIELD_THREAD;
} else {
// At this point, it's likely that the lock is held by a lower priority
// thread that is unavailable to finish its work because of higher
// priority threads spinning here. Sleeping should ensure that they make
// progress.
PlatformThread::Sleep(TimeDelta::FromMilliseconds(1));
}
} while (lock_.load(std::memory_order_relaxed));
} while (UNLIKELY(lock_.exchange(true, std::memory_order_acquire)));
}
} // namespace subtle
} // namespace base

View file

@ -0,0 +1,50 @@
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_SPIN_LOCK_H_
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_SPIN_LOCK_H_
#include <atomic>
#include <memory>
#include <mutex>
#include "base/base_export.h"
#include "base/compiler_specific.h"
// Spinlock is a simple spinlock class based on the standard CPU primitive of
// atomic increment and decrement of an int at a given memory address. These are
// intended only for very short duration locks and assume a system with multiple
// cores. For any potentially longer wait you should use a real lock, such as
// |base::Lock|.
namespace base {
namespace subtle {
class BASE_EXPORT SpinLock {
public:
constexpr SpinLock() = default;
~SpinLock() = default;
using Guard = std::lock_guard<SpinLock>;
ALWAYS_INLINE void lock() {
static_assert(sizeof(lock_) == sizeof(int),
"int and lock_ are different sizes");
if (LIKELY(!lock_.exchange(true, std::memory_order_acquire)))
return;
LockSlow();
}
ALWAYS_INLINE void unlock() { lock_.store(false, std::memory_order_release); }
private:
// This is called if the initial attempt to acquire the lock fails. It's
// slower, but has a much better scheduling and power consumption behavior.
void LockSlow();
std::atomic_int lock_{0};
};
} // namespace subtle
} // namespace base
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_SPIN_LOCK_H_

View file

@ -0,0 +1,208 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// This code should move into the default Windows shim once the win-specific
// allocation shim has been removed, and the generic shim has becaome the
// default.
#include "winheap_stubs_win.h"
#include <limits.h>
#include <malloc.h>
#include <new.h>
#include <windows.h>
#include <algorithm>
#include <limits>
#include "base/bits.h"
#include "base/logging.h"
namespace base {
namespace allocator {
bool g_is_win_shim_layer_initialized = false;
namespace {
const size_t kWindowsPageSize = 4096;
const size_t kMaxWindowsAllocation = INT_MAX - kWindowsPageSize;
inline HANDLE get_heap_handle() {
return reinterpret_cast<HANDLE>(_get_heap_handle());
}
} // namespace
void* WinHeapMalloc(size_t size) {
if (size < kMaxWindowsAllocation)
return HeapAlloc(get_heap_handle(), 0, size);
return nullptr;
}
void WinHeapFree(void* ptr) {
if (!ptr)
return;
HeapFree(get_heap_handle(), 0, ptr);
}
void* WinHeapRealloc(void* ptr, size_t size) {
if (!ptr)
return WinHeapMalloc(size);
if (!size) {
WinHeapFree(ptr);
return nullptr;
}
if (size < kMaxWindowsAllocation)
return HeapReAlloc(get_heap_handle(), 0, ptr, size);
return nullptr;
}
size_t WinHeapGetSizeEstimate(void* ptr) {
if (!ptr)
return 0;
return HeapSize(get_heap_handle(), 0, ptr);
}
// Call the new handler, if one has been set.
// Returns true on successfully calling the handler, false otherwise.
bool WinCallNewHandler(size_t size) {
#ifdef _CPPUNWIND
#error "Exceptions in allocator shim are not supported!"
#endif // _CPPUNWIND
// Get the current new handler.
_PNH nh = _query_new_handler();
if (!nh)
return false;
// Since exceptions are disabled, we don't really know if new_handler
// failed. Assume it will abort if it fails.
return nh(size) ? true : false;
}
// The Windows _aligned_* functions are implemented by creating an allocation
// with enough space to create an aligned allocation internally. The offset to
// the original allocation is prefixed to the aligned allocation so that it can
// be correctly freed.
namespace {
struct AlignedPrefix {
// Offset to the original allocation point.
unsigned int original_allocation_offset;
// Make sure an unsigned int is enough to store the offset
static_assert(
kMaxWindowsAllocation < std::numeric_limits<unsigned int>::max(),
"original_allocation_offset must be able to fit into an unsigned int");
#if DCHECK_IS_ON()
// Magic value used to check that _aligned_free() and _aligned_realloc() are
// only ever called on an aligned allocated chunk.
static constexpr unsigned int kMagic = 0x12003400;
unsigned int magic;
#endif // DCHECK_IS_ON()
};
// Compute how large an allocation we need to fit an allocation with the given
// size and alignment and space for a prefix pointer.
size_t AdjustedSize(size_t size, size_t alignment) {
// Minimal alignment is the prefix size so the prefix is properly aligned.
alignment = std::max(alignment, alignof(AlignedPrefix));
return size + sizeof(AlignedPrefix) + alignment - 1;
}
// Align the allocation and write the prefix.
void* AlignAllocation(void* ptr, size_t alignment) {
// Minimal alignment is the prefix size so the prefix is properly aligned.
alignment = std::max(alignment, alignof(AlignedPrefix));
uintptr_t address = reinterpret_cast<uintptr_t>(ptr);
address = base::bits::Align(address + sizeof(AlignedPrefix), alignment);
// Write the prefix.
AlignedPrefix* prefix = reinterpret_cast<AlignedPrefix*>(address) - 1;
prefix->original_allocation_offset =
address - reinterpret_cast<uintptr_t>(ptr);
#if DCHECK_IS_ON()
prefix->magic = AlignedPrefix::kMagic;
#endif // DCHECK_IS_ON()
return reinterpret_cast<void*>(address);
}
// Return the original allocation from an aligned allocation.
void* UnalignAllocation(void* ptr) {
AlignedPrefix* prefix = reinterpret_cast<AlignedPrefix*>(ptr) - 1;
#if DCHECK_IS_ON()
DCHECK_EQ(prefix->magic, AlignedPrefix::kMagic);
#endif // DCHECK_IS_ON()
void* unaligned =
static_cast<uint8_t*>(ptr) - prefix->original_allocation_offset;
CHECK_LT(unaligned, ptr);
CHECK_LE(
reinterpret_cast<uintptr_t>(ptr) - reinterpret_cast<uintptr_t>(unaligned),
kMaxWindowsAllocation);
return unaligned;
}
} // namespace
void* WinHeapAlignedMalloc(size_t size, size_t alignment) {
CHECK(base::bits::IsPowerOfTwo(alignment));
size_t adjusted = AdjustedSize(size, alignment);
if (adjusted >= kMaxWindowsAllocation)
return nullptr;
void* ptr = WinHeapMalloc(adjusted);
if (!ptr)
return nullptr;
return AlignAllocation(ptr, alignment);
}
void* WinHeapAlignedRealloc(void* ptr, size_t size, size_t alignment) {
CHECK(base::bits::IsPowerOfTwo(alignment));
if (!ptr)
return WinHeapAlignedMalloc(size, alignment);
if (!size) {
WinHeapAlignedFree(ptr);
return nullptr;
}
size_t adjusted = AdjustedSize(size, alignment);
if (adjusted >= kMaxWindowsAllocation)
return nullptr;
// Try to resize the allocation in place first.
void* unaligned = UnalignAllocation(ptr);
if (HeapReAlloc(get_heap_handle(), HEAP_REALLOC_IN_PLACE_ONLY, unaligned,
adjusted)) {
return ptr;
}
// Otherwise manually perform an _aligned_malloc() and copy since an
// unaligned allocation from HeapReAlloc() would force us to copy the
// allocation twice.
void* new_ptr = WinHeapAlignedMalloc(size, alignment);
if (!new_ptr)
return nullptr;
size_t gap =
reinterpret_cast<uintptr_t>(ptr) - reinterpret_cast<uintptr_t>(unaligned);
size_t old_size = WinHeapGetSizeEstimate(unaligned) - gap;
memcpy(new_ptr, ptr, std::min(size, old_size));
WinHeapAlignedFree(ptr);
return new_ptr;
}
void WinHeapAlignedFree(void* ptr) {
if (!ptr)
return;
void* original_allocation = UnalignAllocation(ptr);
WinHeapFree(original_allocation);
}
} // namespace allocator
} // namespace base

View file

@ -0,0 +1,48 @@
// Copyright 2016 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// Thin allocation wrappers for the windows heap. This file should be deleted
// once the win-specific allocation shim has been removed, and the generic shim
// has becaome the default.
#ifndef BASE_ALLOCATOR_WINHEAP_STUBS_H_
#define BASE_ALLOCATOR_WINHEAP_STUBS_H_
#include <stdint.h>
#include "base/base_export.h"
namespace base {
namespace allocator {
// Set to true if the link-time magic has successfully hooked into the CRT's
// heap initialization.
extern bool g_is_win_shim_layer_initialized;
// Thin wrappers to implement the standard C allocation semantics on the
// CRT's Windows heap.
void* WinHeapMalloc(size_t size);
void WinHeapFree(void* ptr);
void* WinHeapRealloc(void* ptr, size_t size);
// Returns a lower-bound estimate for the full amount of memory consumed by the
// the allocation |ptr|.
size_t WinHeapGetSizeEstimate(void* ptr);
// Call the new handler, if one has been set.
// Returns true on successfully calling the handler, false otherwise.
bool WinCallNewHandler(size_t size);
// Wrappers to implement the interface for the _aligned_* functions on top of
// the CRT's Windows heap. Exported for tests.
BASE_EXPORT void* WinHeapAlignedMalloc(size_t size, size_t alignment);
BASE_EXPORT void* WinHeapAlignedRealloc(void* ptr,
size_t size,
size_t alignment);
BASE_EXPORT void WinHeapAlignedFree(void* ptr);
} // namespace allocator
} // namespace base
#endif // BASE_ALLOCATOR_WINHEAP_STUBS_H_