Repo created
This commit is contained in:
parent
81b91f4139
commit
f8c34fa5ee
22732 changed files with 4815320 additions and 2 deletions
4
TMessagesProj/jni/voip/webrtc/base/allocator/OWNERS
Normal file
4
TMessagesProj/jni/voip/webrtc/base/allocator/OWNERS
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
primiano@chromium.org
|
||||
wfh@chromium.org
|
||||
|
||||
# COMPONENT: Internals
|
||||
197
TMessagesProj/jni/voip/webrtc/base/allocator/README.md
Normal file
197
TMessagesProj/jni/voip/webrtc/base/allocator/README.md
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
This document describes how malloc / new calls are routed in the various Chrome
|
||||
platforms.
|
||||
|
||||
Bare in mind that the chromium codebase does not always just use `malloc()`.
|
||||
Some examples:
|
||||
- Large parts of the renderer (Blink) use two home-brewed allocators,
|
||||
PartitionAlloc and BlinkGC (Oilpan).
|
||||
- Some subsystems, such as the V8 JavaScript engine, handle memory management
|
||||
autonomously.
|
||||
- Various parts of the codebase use abstractions such as `SharedMemory` or
|
||||
`DiscardableMemory` which, similarly to the above, have their own page-level
|
||||
memory management.
|
||||
|
||||
Background
|
||||
----------
|
||||
The `allocator` target defines at compile-time the platform-specific choice of
|
||||
the allocator and extra-hooks which services calls to malloc/new. The relevant
|
||||
build-time flags involved are `use_allocator` and `use_allocator_shim`.
|
||||
|
||||
The default choices are as follows:
|
||||
|
||||
**Windows**
|
||||
`use_allocator: winheap`, the default Windows heap.
|
||||
Additionally, `static_library` (i.e. non-component) builds have a shim
|
||||
layer wrapping malloc/new, which is controlled by `use_allocator_shim`.
|
||||
The shim layer provides extra security features, such as preventing large
|
||||
allocations that can hit signed vs. unsigned bugs in third_party code.
|
||||
|
||||
**Linux Desktop / CrOS**
|
||||
`use_allocator: tcmalloc`, a forked copy of tcmalloc which resides in
|
||||
`third_party/tcmalloc/chromium`. Setting `use_allocator: none` causes the build
|
||||
to fall back to the system (Glibc) symbols.
|
||||
|
||||
**Android**
|
||||
`use_allocator: none`, always use the allocator symbols coming from Android's
|
||||
libc (Bionic). As it is developed as part of the OS, it is considered to be
|
||||
optimized for small devices and more memory-efficient than other choices.
|
||||
The actual implementation backing malloc symbols in Bionic is up to the board
|
||||
config and can vary (typically *dlmalloc* or *jemalloc* on most Nexus devices).
|
||||
|
||||
**Mac/iOS**
|
||||
`use_allocator: none`, we always use the system's allocator implementation.
|
||||
|
||||
In addition, when building for `asan` / `msan` both the allocator and the shim
|
||||
layer are disabled.
|
||||
|
||||
Layering and build deps
|
||||
-----------------------
|
||||
The `allocator` target provides both the source files for tcmalloc (where
|
||||
applicable) and the linker flags required for the Windows shim layer.
|
||||
The `base` target is (almost) the only one depending on `allocator`. No other
|
||||
targets should depend on it, with the exception of the very few executables /
|
||||
dynamic libraries that don't depend, either directly or indirectly, on `base`
|
||||
within the scope of a linker unit.
|
||||
|
||||
More importantly, **no other place outside of `/base` should depend on the
|
||||
specific allocator** (e.g., directly include `third_party/tcmalloc`).
|
||||
If such a functional dependency is required that should be achieved using
|
||||
abstractions in `base` (see `/base/allocator/allocator_extension.h` and
|
||||
`/base/memory/`)
|
||||
|
||||
**Why `base` depends on `allocator`?**
|
||||
Because it needs to provide services that depend on the actual allocator
|
||||
implementation. In the past `base` used to pretend to be allocator-agnostic
|
||||
and get the dependencies injected by other layers. This ended up being an
|
||||
inconsistent mess.
|
||||
See the [allocator cleanup doc][url-allocator-cleanup] for more context.
|
||||
|
||||
Linker unit targets (executables and shared libraries) that depend in some way
|
||||
on `base` (most of the targets in the codebase) get automatically the correct
|
||||
set of linker flags to pull in tcmalloc or the Windows shim-layer.
|
||||
|
||||
|
||||
Source code
|
||||
-----------
|
||||
This directory contains just the allocator (i.e. shim) layer that switches
|
||||
between the different underlying memory allocation implementations.
|
||||
|
||||
The tcmalloc library originates outside of Chromium and exists in
|
||||
`../../third_party/tcmalloc` (currently, the actual location is defined in the
|
||||
allocator.gyp file). The third party sources use a vendor-branch SCM pattern to
|
||||
track Chromium-specific changes independently from upstream changes.
|
||||
|
||||
The general intent is to push local changes upstream so that over
|
||||
time we no longer need any forked files.
|
||||
|
||||
|
||||
Unified allocator shim
|
||||
----------------------
|
||||
On most platforms, Chrome overrides the malloc / operator new symbols (and
|
||||
corresponding free / delete and other variants). This is to enforce security
|
||||
checks and lately to enable the
|
||||
[memory-infra heap profiler][url-memory-infra-heap-profiler].
|
||||
Historically each platform had its special logic for defining the allocator
|
||||
symbols in different places of the codebase. The unified allocator shim is
|
||||
a project aimed to unify the symbol definition and allocator routing logic in
|
||||
a central place.
|
||||
|
||||
- Full documentation: [Allocator shim design doc][url-allocator-shim].
|
||||
- Current state: Available and enabled by default on Android, CrOS, Linux,
|
||||
Mac OS and Windows.
|
||||
- Tracking bug: [https://crbug.com/550886][crbug.com/550886].
|
||||
- Build-time flag: `use_allocator_shim`.
|
||||
|
||||
**Overview of the unified allocator shim**
|
||||
The allocator shim consists of three stages:
|
||||
```
|
||||
+-------------------------+ +-----------------------+ +----------------+
|
||||
| malloc & friends | -> | shim layer | -> | Routing to |
|
||||
| symbols definition | | implementation | | allocator |
|
||||
+-------------------------+ +-----------------------+ +----------------+
|
||||
| - libc symbols (malloc, | | - Security checks | | - tcmalloc |
|
||||
| calloc, free, ...) | | - Chain of dispatchers| | - glibc |
|
||||
| - C++ symbols (operator | | that can intercept | | - Android |
|
||||
| new, delete, ...) | | and override | | bionic |
|
||||
| - glibc weak symbols | | allocations | | - WinHeap |
|
||||
| (__libc_malloc, ...) | +-----------------------+ +----------------+
|
||||
+-------------------------+
|
||||
```
|
||||
|
||||
**1. malloc symbols definition**
|
||||
This stage takes care of overriding the symbols `malloc`, `free`,
|
||||
`operator new`, `operator delete` and friends and routing those calls inside the
|
||||
allocator shim (next point).
|
||||
This is taken care of by the headers in `allocator_shim_override_*`.
|
||||
|
||||
*On Linux/CrOS*: the allocator symbols are defined as exported global symbols
|
||||
in `allocator_shim_override_libc_symbols.h` (for `malloc`, `free` and friends)
|
||||
and in `allocator_shim_override_cpp_symbols.h` (for `operator new`,
|
||||
`operator delete` and friends).
|
||||
This enables proper interposition of malloc symbols referenced by the main
|
||||
executable and any third party libraries. Symbol resolution on Linux is a breadth first search that starts from the root link unit, that is the executable
|
||||
(see EXECUTABLE AND LINKABLE FORMAT (ELF) - Portable Formats Specification).
|
||||
Additionally, when tcmalloc is the default allocator, some extra glibc symbols
|
||||
are also defined in `allocator_shim_override_glibc_weak_symbols.h`, for subtle
|
||||
reasons explained in that file.
|
||||
The Linux/CrOS shim was introduced by
|
||||
[crrev.com/1675143004](https://crrev.com/1675143004).
|
||||
|
||||
*On Android*: load-time symbol interposition (unlike the Linux/CrOS case) is not
|
||||
possible. This is because Android processes are `fork()`-ed from the Android
|
||||
zygote, which pre-loads libc.so and only later native code gets loaded via
|
||||
`dlopen()` (symbols from `dlopen()`-ed libraries get a different resolution
|
||||
scope).
|
||||
In this case, the approach instead of wrapping symbol resolution at link time
|
||||
(i.e. during the build), via the `--Wl,-wrap,malloc` linker flag.
|
||||
The use of this wrapping flag causes:
|
||||
- All references to allocator symbols in the Chrome codebase to be rewritten as
|
||||
references to `__wrap_malloc` and friends. The `__wrap_malloc` symbols are
|
||||
defined in the `allocator_shim_override_linker_wrapped_symbols.h` and
|
||||
route allocator calls inside the shim layer.
|
||||
- The reference to the original `malloc` symbols (which typically is defined by
|
||||
the system's libc.so) are accessible via the special `__real_malloc` and
|
||||
friends symbols (which will be relocated, at load time, against `malloc`).
|
||||
|
||||
In summary, this approach is transparent to the dynamic loader, which still sees
|
||||
undefined symbol references to malloc symbols.
|
||||
These symbols will be resolved against libc.so as usual.
|
||||
More details in [crrev.com/1719433002](https://crrev.com/1719433002).
|
||||
|
||||
**2. Shim layer implementation**
|
||||
This stage contains the actual shim implementation. This consists of:
|
||||
- A singly linked list of dispatchers (structs with function pointers to `malloc`-like functions). Dispatchers can be dynamically inserted at runtime
|
||||
(using the `InsertAllocatorDispatch` API). They can intercept and override
|
||||
allocator calls.
|
||||
- The security checks (suicide on malloc-failure via `std::new_handler`, etc).
|
||||
This happens inside `allocator_shim.cc`
|
||||
|
||||
**3. Final allocator routing**
|
||||
The final element of the aforementioned dispatcher chain is statically defined
|
||||
at build time and ultimately routes the allocator calls to the actual allocator
|
||||
(as described in the *Background* section above). This is taken care of by the
|
||||
headers in `allocator_shim_default_dispatch_to_*` files.
|
||||
|
||||
|
||||
Appendixes
|
||||
----------
|
||||
**How does the Windows shim layer replace the malloc symbols?**
|
||||
The mechanism for hooking LIBCMT in Windows is rather tricky. The core
|
||||
problem is that by default, the Windows library does not declare malloc and
|
||||
free as weak symbols. Because of this, they cannot be overridden. To work
|
||||
around this, we start with the LIBCMT.LIB, and manually remove all allocator
|
||||
related functions from it using the visual studio library tool. Once removed,
|
||||
we can now link against the library and provide custom versions of the
|
||||
allocator related functionality.
|
||||
See the script `preb_libc.py` in this folder.
|
||||
|
||||
Related links
|
||||
-------------
|
||||
- [Unified allocator shim doc - Feb 2016][url-allocator-shim]
|
||||
- [Allocator cleanup doc - Jan 2016][url-allocator-cleanup]
|
||||
- [Proposal to use PartitionAlloc as default allocator](https://crbug.com/339604)
|
||||
- [Memory-Infra: Tools to profile memory usage in Chrome](/docs/memory-infra/README.md)
|
||||
|
||||
[url-allocator-cleanup]: https://docs.google.com/document/d/1V77Kgp_4tfaaWPEZVxNevoD02wXiatnAv7Ssgr0hmjg/edit?usp=sharing
|
||||
[url-memory-infra-heap-profiler]: /docs/memory-infra/heap_profiler.md
|
||||
[url-allocator-shim]: https://docs.google.com/document/d/1yKlO1AO4XjpDad9rjcBOI15EKdAGsuGO_IeZy0g0kxo/edit?usp=sharing
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/allocator_check.h"
|
||||
|
||||
#include "base/allocator/buildflags.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
#if defined(OS_WIN)
|
||||
#include "base/allocator/winheap_stubs_win.h"
|
||||
#endif
|
||||
|
||||
#if defined(OS_LINUX)
|
||||
#include <malloc.h>
|
||||
#endif
|
||||
|
||||
#if defined(OS_MACOSX)
|
||||
#include "base/allocator/allocator_interception_mac.h"
|
||||
#endif
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
bool IsAllocatorInitialized() {
|
||||
#if defined(OS_WIN) && BUILDFLAG(USE_ALLOCATOR_SHIM)
|
||||
// Set by allocator_shim_override_ucrt_symbols_win.h when the
|
||||
// shimmed _set_new_mode() is called.
|
||||
return g_is_win_shim_layer_initialized;
|
||||
#elif defined(OS_LINUX) && BUILDFLAG(USE_TCMALLOC) && \
|
||||
!defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
// From third_party/tcmalloc/chromium/src/gperftools/tcmalloc.h.
|
||||
// TODO(primiano): replace with an include once base can depend on allocator.
|
||||
#define TC_MALLOPT_IS_OVERRIDDEN_BY_TCMALLOC 0xbeef42
|
||||
return (mallopt(TC_MALLOPT_IS_OVERRIDDEN_BY_TCMALLOC, 0) ==
|
||||
TC_MALLOPT_IS_OVERRIDDEN_BY_TCMALLOC);
|
||||
#elif defined(OS_MACOSX) && !defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
// From allocator_interception_mac.mm.
|
||||
return base::allocator::g_replaced_default_zone;
|
||||
#else
|
||||
return true;
|
||||
#endif
|
||||
}
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_ALLOCATOR_ALLOCATOR_CHECK_H_
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_ALLOCATOR_CHECK_H_
|
||||
|
||||
#include "base/base_export.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
BASE_EXPORT bool IsAllocatorInitialized();
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_ALLOCATOR_ALLOCATOR_CHECK_H_
|
||||
|
|
@ -0,0 +1,73 @@
|
|||
// Copyright (c) 2012 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/allocator_extension.h"
|
||||
#include "base/allocator/buildflags.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
#if BUILDFLAG(USE_TCMALLOC)
|
||||
#include "third_party/tcmalloc/chromium/src/gperftools/heap-profiler.h"
|
||||
#include "third_party/tcmalloc/chromium/src/gperftools/malloc_extension.h"
|
||||
#include "third_party/tcmalloc/chromium/src/gperftools/malloc_hook.h"
|
||||
#endif
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
void ReleaseFreeMemory() {
|
||||
#if BUILDFLAG(USE_TCMALLOC)
|
||||
::MallocExtension::instance()->ReleaseFreeMemory();
|
||||
#endif
|
||||
}
|
||||
|
||||
bool GetNumericProperty(const char* name, size_t* value) {
|
||||
#if BUILDFLAG(USE_TCMALLOC)
|
||||
return ::MallocExtension::instance()->GetNumericProperty(name, value);
|
||||
#endif
|
||||
return false;
|
||||
}
|
||||
|
||||
bool SetNumericProperty(const char* name, size_t value) {
|
||||
#if BUILDFLAG(USE_TCMALLOC)
|
||||
return ::MallocExtension::instance()->SetNumericProperty(name, value);
|
||||
#endif
|
||||
return false;
|
||||
}
|
||||
|
||||
void GetHeapSample(std::string* writer) {
|
||||
#if BUILDFLAG(USE_TCMALLOC)
|
||||
::MallocExtension::instance()->GetHeapSample(writer);
|
||||
#endif
|
||||
}
|
||||
|
||||
bool IsHeapProfilerRunning() {
|
||||
#if BUILDFLAG(USE_TCMALLOC) && defined(ENABLE_PROFILING)
|
||||
return ::IsHeapProfilerRunning();
|
||||
#endif
|
||||
return false;
|
||||
}
|
||||
|
||||
void SetHooks(AllocHookFunc alloc_hook, FreeHookFunc free_hook) {
|
||||
// TODO(sque): Use allocator shim layer instead.
|
||||
#if BUILDFLAG(USE_TCMALLOC)
|
||||
// Make sure no hooks get overwritten.
|
||||
auto prev_alloc_hook = MallocHook::SetNewHook(alloc_hook);
|
||||
if (alloc_hook)
|
||||
DCHECK(!prev_alloc_hook);
|
||||
|
||||
auto prev_free_hook = MallocHook::SetDeleteHook(free_hook);
|
||||
if (free_hook)
|
||||
DCHECK(!prev_free_hook);
|
||||
#endif
|
||||
}
|
||||
|
||||
int GetCallStack(void** stack, int max_stack_size) {
|
||||
#if BUILDFLAG(USE_TCMALLOC)
|
||||
return MallocHook::GetCallerStackTrace(stack, max_stack_size, 0);
|
||||
#endif
|
||||
return 0;
|
||||
}
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
// Copyright (c) 2012 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_ALLOCATOR_EXTENSION_H_
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_EXTENSION_H_
|
||||
|
||||
#include <stddef.h> // for size_t
|
||||
#include <string>
|
||||
|
||||
#include "base/base_export.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
// Callback types for alloc and free.
|
||||
using AllocHookFunc = void (*)(const void*, size_t);
|
||||
using FreeHookFunc = void (*)(const void*);
|
||||
|
||||
// Request that the allocator release any free memory it knows about to the
|
||||
// system.
|
||||
BASE_EXPORT void ReleaseFreeMemory();
|
||||
|
||||
// Get the named property's |value|. Returns true if the property is known.
|
||||
// Returns false if the property is not a valid property name for the current
|
||||
// allocator implementation.
|
||||
// |name| or |value| cannot be NULL
|
||||
BASE_EXPORT bool GetNumericProperty(const char* name, size_t* value);
|
||||
|
||||
// Set the named property's |value|. Returns true if the property is known and
|
||||
// writable. Returns false if the property is not a valid property name for the
|
||||
// current allocator implementation, or is not writable. |name| cannot be NULL.
|
||||
BASE_EXPORT bool SetNumericProperty(const char* name, size_t value);
|
||||
|
||||
// Outputs to |writer| a sample of live objects and the stack traces
|
||||
// that allocated these objects. The format of the returned output
|
||||
// is equivalent to the output of the heap profiler and can
|
||||
// therefore be passed to "pprof".
|
||||
// NOTE: by default, the allocator does not do any heap sampling, and this
|
||||
// function will always return an empty sample. To get useful
|
||||
// data from GetHeapSample, you must also set the numeric property
|
||||
// "tcmalloc.sampling_period_bytes" to a value such as 524288.
|
||||
BASE_EXPORT void GetHeapSample(std::string* writer);
|
||||
|
||||
BASE_EXPORT bool IsHeapProfilerRunning();
|
||||
|
||||
// Register callbacks for alloc and free. Can only store one callback at a time
|
||||
// for each of alloc and free.
|
||||
BASE_EXPORT void SetHooks(AllocHookFunc alloc_hook, FreeHookFunc free_hook);
|
||||
|
||||
// Attempts to unwind the call stack from the current location where this
|
||||
// function is being called from. Must be called from a hook function registered
|
||||
// by calling SetSingle{Alloc,Free}Hook, directly or indirectly.
|
||||
//
|
||||
// Arguments:
|
||||
// stack: pointer to a pre-allocated array of void*'s.
|
||||
// max_stack_size: indicates the size of the array in |stack|.
|
||||
//
|
||||
// Returns the number of call stack frames stored in |stack|, or 0 if no call
|
||||
// stack information is available.
|
||||
BASE_EXPORT int GetCallStack(void** stack, int max_stack_size);
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_ALLOCATOR_EXTENSION_H_
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
// Copyright 2017 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_ALLOCATOR_INTERCEPTION_MAC_H_
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_INTERCEPTION_MAC_H_
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
#include "base/base_export.h"
|
||||
#include "third_party/apple_apsl/malloc.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
struct MallocZoneFunctions;
|
||||
|
||||
// Saves the function pointers currently used by the default zone.
|
||||
void StoreFunctionsForDefaultZone();
|
||||
|
||||
// Same as StoreFunctionsForDefaultZone, but for all malloc zones.
|
||||
void StoreFunctionsForAllZones();
|
||||
|
||||
// For all malloc zones that have been stored, replace their functions with
|
||||
// |functions|.
|
||||
void ReplaceFunctionsForStoredZones(const MallocZoneFunctions* functions);
|
||||
|
||||
extern bool g_replaced_default_zone;
|
||||
|
||||
// Calls the original implementation of malloc/calloc prior to interception.
|
||||
bool UncheckedMallocMac(size_t size, void** result);
|
||||
bool UncheckedCallocMac(size_t num_items, size_t size, void** result);
|
||||
|
||||
// Intercepts calls to default and purgeable malloc zones. Intercepts Core
|
||||
// Foundation and Objective-C allocations.
|
||||
// Has no effect on the default malloc zone if the allocator shim already
|
||||
// performs that interception.
|
||||
BASE_EXPORT void InterceptAllocationsMac();
|
||||
|
||||
// Updates all malloc zones to use their original functions.
|
||||
// Also calls ClearAllMallocZonesForTesting.
|
||||
BASE_EXPORT void UninterceptMallocZonesForTesting();
|
||||
|
||||
// Periodically checks for, and shims new malloc zones. Stops checking after 1
|
||||
// minute.
|
||||
BASE_EXPORT void PeriodicallyShimNewMallocZones();
|
||||
|
||||
// Exposed for testing.
|
||||
BASE_EXPORT void ShimNewMallocZones();
|
||||
BASE_EXPORT void ReplaceZoneFunctions(ChromeMallocZone* zone,
|
||||
const MallocZoneFunctions* functions);
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_ALLOCATOR_INTERCEPTION_MAC_H_
|
||||
|
|
@ -0,0 +1,568 @@
|
|||
// Copyright 2017 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
// This file contains all the logic necessary to intercept allocations on
|
||||
// macOS. "malloc zones" are an abstraction that allows the process to intercept
|
||||
// all malloc-related functions. There is no good mechanism [short of
|
||||
// interposition] to determine new malloc zones are added, so there's no clean
|
||||
// mechanism to intercept all malloc zones. This file contains logic to
|
||||
// intercept the default and purgeable zones, which always exist. A cursory
|
||||
// review of Chrome seems to imply that non-default zones are almost never used.
|
||||
//
|
||||
// This file also contains logic to intercept Core Foundation and Objective-C
|
||||
// allocations. The implementations forward to the default malloc zone, so the
|
||||
// only reason to intercept these calls is to re-label OOM crashes with slightly
|
||||
// more details.
|
||||
|
||||
#include "base/allocator/allocator_interception_mac.h"
|
||||
|
||||
#include <CoreFoundation/CoreFoundation.h>
|
||||
#import <Foundation/Foundation.h>
|
||||
#include <errno.h>
|
||||
#include <mach/mach.h>
|
||||
#include <mach/mach_vm.h>
|
||||
#import <objc/runtime.h>
|
||||
#include <stddef.h>
|
||||
|
||||
#include <new>
|
||||
|
||||
#include "base/allocator/buildflags.h"
|
||||
#include "base/allocator/malloc_zone_functions_mac.h"
|
||||
#include "base/bind.h"
|
||||
#include "base/bits.h"
|
||||
#include "base/logging.h"
|
||||
#include "base/mac/mac_util.h"
|
||||
#include "base/mac/mach_logging.h"
|
||||
#include "base/process/memory.h"
|
||||
#include "base/threading/sequenced_task_runner_handle.h"
|
||||
#include "build/build_config.h"
|
||||
#include "third_party/apple_apsl/CFBase.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
bool g_replaced_default_zone = false;
|
||||
|
||||
namespace {
|
||||
|
||||
bool g_oom_killer_enabled;
|
||||
|
||||
// Starting with Mac OS X 10.7, the zone allocators set up by the system are
|
||||
// read-only, to prevent them from being overwritten in an attack. However,
|
||||
// blindly unprotecting and reprotecting the zone allocators fails with
|
||||
// GuardMalloc because GuardMalloc sets up its zone allocator using a block of
|
||||
// memory in its bss. Explicit saving/restoring of the protection is required.
|
||||
//
|
||||
// This function takes a pointer to a malloc zone, de-protects it if necessary,
|
||||
// and returns (in the out parameters) a region of memory (if any) to be
|
||||
// re-protected when modifications are complete. This approach assumes that
|
||||
// there is no contention for the protection of this memory.
|
||||
void DeprotectMallocZone(ChromeMallocZone* default_zone,
|
||||
mach_vm_address_t* reprotection_start,
|
||||
mach_vm_size_t* reprotection_length,
|
||||
vm_prot_t* reprotection_value) {
|
||||
mach_port_t unused;
|
||||
*reprotection_start = reinterpret_cast<mach_vm_address_t>(default_zone);
|
||||
struct vm_region_basic_info_64 info;
|
||||
mach_msg_type_number_t count = VM_REGION_BASIC_INFO_COUNT_64;
|
||||
kern_return_t result = mach_vm_region(
|
||||
mach_task_self(), reprotection_start, reprotection_length,
|
||||
VM_REGION_BASIC_INFO_64, reinterpret_cast<vm_region_info_t>(&info),
|
||||
&count, &unused);
|
||||
MACH_CHECK(result == KERN_SUCCESS, result) << "mach_vm_region";
|
||||
|
||||
// The kernel always returns a null object for VM_REGION_BASIC_INFO_64, but
|
||||
// balance it with a deallocate in case this ever changes. See 10.9.2
|
||||
// xnu-2422.90.20/osfmk/vm/vm_map.c vm_map_region.
|
||||
mach_port_deallocate(mach_task_self(), unused);
|
||||
|
||||
// Does the region fully enclose the zone pointers? Possibly unwarranted
|
||||
// simplification used: using the size of a full version 8 malloc zone rather
|
||||
// than the actual smaller size if the passed-in zone is not version 8.
|
||||
CHECK(*reprotection_start <=
|
||||
reinterpret_cast<mach_vm_address_t>(default_zone));
|
||||
mach_vm_size_t zone_offset =
|
||||
reinterpret_cast<mach_vm_size_t>(default_zone) -
|
||||
reinterpret_cast<mach_vm_size_t>(*reprotection_start);
|
||||
CHECK(zone_offset + sizeof(ChromeMallocZone) <= *reprotection_length);
|
||||
|
||||
if (info.protection & VM_PROT_WRITE) {
|
||||
// No change needed; the zone is already writable.
|
||||
*reprotection_start = 0;
|
||||
*reprotection_length = 0;
|
||||
*reprotection_value = VM_PROT_NONE;
|
||||
} else {
|
||||
*reprotection_value = info.protection;
|
||||
result = mach_vm_protect(mach_task_self(), *reprotection_start,
|
||||
*reprotection_length, false,
|
||||
info.protection | VM_PROT_WRITE);
|
||||
MACH_CHECK(result == KERN_SUCCESS, result) << "mach_vm_protect";
|
||||
}
|
||||
}
|
||||
|
||||
#if !defined(ADDRESS_SANITIZER)
|
||||
|
||||
MallocZoneFunctions g_old_zone;
|
||||
MallocZoneFunctions g_old_purgeable_zone;
|
||||
|
||||
void* oom_killer_malloc(struct _malloc_zone_t* zone, size_t size) {
|
||||
void* result = g_old_zone.malloc(zone, size);
|
||||
if (!result && size)
|
||||
TerminateBecauseOutOfMemory(size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void* oom_killer_calloc(struct _malloc_zone_t* zone,
|
||||
size_t num_items,
|
||||
size_t size) {
|
||||
void* result = g_old_zone.calloc(zone, num_items, size);
|
||||
if (!result && num_items && size)
|
||||
TerminateBecauseOutOfMemory(num_items * size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void* oom_killer_valloc(struct _malloc_zone_t* zone, size_t size) {
|
||||
void* result = g_old_zone.valloc(zone, size);
|
||||
if (!result && size)
|
||||
TerminateBecauseOutOfMemory(size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void oom_killer_free(struct _malloc_zone_t* zone, void* ptr) {
|
||||
g_old_zone.free(zone, ptr);
|
||||
}
|
||||
|
||||
void* oom_killer_realloc(struct _malloc_zone_t* zone, void* ptr, size_t size) {
|
||||
void* result = g_old_zone.realloc(zone, ptr, size);
|
||||
if (!result && size)
|
||||
TerminateBecauseOutOfMemory(size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void* oom_killer_memalign(struct _malloc_zone_t* zone,
|
||||
size_t alignment,
|
||||
size_t size) {
|
||||
void* result = g_old_zone.memalign(zone, alignment, size);
|
||||
// Only die if posix_memalign would have returned ENOMEM, since there are
|
||||
// other reasons why NULL might be returned (see
|
||||
// http://opensource.apple.com/source/Libc/Libc-583/gen/malloc.c ).
|
||||
if (!result && size && alignment >= sizeof(void*) &&
|
||||
base::bits::IsPowerOfTwo(alignment)) {
|
||||
TerminateBecauseOutOfMemory(size);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
void* oom_killer_malloc_purgeable(struct _malloc_zone_t* zone, size_t size) {
|
||||
void* result = g_old_purgeable_zone.malloc(zone, size);
|
||||
if (!result && size)
|
||||
TerminateBecauseOutOfMemory(size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void* oom_killer_calloc_purgeable(struct _malloc_zone_t* zone,
|
||||
size_t num_items,
|
||||
size_t size) {
|
||||
void* result = g_old_purgeable_zone.calloc(zone, num_items, size);
|
||||
if (!result && num_items && size)
|
||||
TerminateBecauseOutOfMemory(num_items * size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void* oom_killer_valloc_purgeable(struct _malloc_zone_t* zone, size_t size) {
|
||||
void* result = g_old_purgeable_zone.valloc(zone, size);
|
||||
if (!result && size)
|
||||
TerminateBecauseOutOfMemory(size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void oom_killer_free_purgeable(struct _malloc_zone_t* zone, void* ptr) {
|
||||
g_old_purgeable_zone.free(zone, ptr);
|
||||
}
|
||||
|
||||
void* oom_killer_realloc_purgeable(struct _malloc_zone_t* zone,
|
||||
void* ptr,
|
||||
size_t size) {
|
||||
void* result = g_old_purgeable_zone.realloc(zone, ptr, size);
|
||||
if (!result && size)
|
||||
TerminateBecauseOutOfMemory(size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void* oom_killer_memalign_purgeable(struct _malloc_zone_t* zone,
|
||||
size_t alignment,
|
||||
size_t size) {
|
||||
void* result = g_old_purgeable_zone.memalign(zone, alignment, size);
|
||||
// Only die if posix_memalign would have returned ENOMEM, since there are
|
||||
// other reasons why NULL might be returned (see
|
||||
// http://opensource.apple.com/source/Libc/Libc-583/gen/malloc.c ).
|
||||
if (!result && size && alignment >= sizeof(void*) &&
|
||||
base::bits::IsPowerOfTwo(alignment)) {
|
||||
TerminateBecauseOutOfMemory(size);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
#endif // !defined(ADDRESS_SANITIZER)
|
||||
|
||||
#if !defined(ADDRESS_SANITIZER)
|
||||
|
||||
// === Core Foundation CFAllocators ===
|
||||
|
||||
bool CanGetContextForCFAllocator() {
|
||||
return !base::mac::IsOSLaterThan10_15_DontCallThis();
|
||||
}
|
||||
|
||||
CFAllocatorContext* ContextForCFAllocator(CFAllocatorRef allocator) {
|
||||
ChromeCFAllocatorLions* our_allocator = const_cast<ChromeCFAllocatorLions*>(
|
||||
reinterpret_cast<const ChromeCFAllocatorLions*>(allocator));
|
||||
return &our_allocator->_context;
|
||||
}
|
||||
|
||||
CFAllocatorAllocateCallBack g_old_cfallocator_system_default;
|
||||
CFAllocatorAllocateCallBack g_old_cfallocator_malloc;
|
||||
CFAllocatorAllocateCallBack g_old_cfallocator_malloc_zone;
|
||||
|
||||
void* oom_killer_cfallocator_system_default(CFIndex alloc_size,
|
||||
CFOptionFlags hint,
|
||||
void* info) {
|
||||
void* result = g_old_cfallocator_system_default(alloc_size, hint, info);
|
||||
if (!result)
|
||||
TerminateBecauseOutOfMemory(alloc_size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void* oom_killer_cfallocator_malloc(CFIndex alloc_size,
|
||||
CFOptionFlags hint,
|
||||
void* info) {
|
||||
void* result = g_old_cfallocator_malloc(alloc_size, hint, info);
|
||||
if (!result)
|
||||
TerminateBecauseOutOfMemory(alloc_size);
|
||||
return result;
|
||||
}
|
||||
|
||||
void* oom_killer_cfallocator_malloc_zone(CFIndex alloc_size,
|
||||
CFOptionFlags hint,
|
||||
void* info) {
|
||||
void* result = g_old_cfallocator_malloc_zone(alloc_size, hint, info);
|
||||
if (!result)
|
||||
TerminateBecauseOutOfMemory(alloc_size);
|
||||
return result;
|
||||
}
|
||||
|
||||
#endif // !defined(ADDRESS_SANITIZER)
|
||||
|
||||
// === Cocoa NSObject allocation ===
|
||||
|
||||
typedef id (*allocWithZone_t)(id, SEL, NSZone*);
|
||||
allocWithZone_t g_old_allocWithZone;
|
||||
|
||||
id oom_killer_allocWithZone(id self, SEL _cmd, NSZone* zone) {
|
||||
id result = g_old_allocWithZone(self, _cmd, zone);
|
||||
if (!result)
|
||||
TerminateBecauseOutOfMemory(0);
|
||||
return result;
|
||||
}
|
||||
|
||||
void UninterceptMallocZoneForTesting(struct _malloc_zone_t* zone) {
|
||||
ChromeMallocZone* chrome_zone = reinterpret_cast<ChromeMallocZone*>(zone);
|
||||
if (!IsMallocZoneAlreadyStored(chrome_zone))
|
||||
return;
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(zone);
|
||||
ReplaceZoneFunctions(chrome_zone, &functions);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
bool UncheckedMallocMac(size_t size, void** result) {
|
||||
#if defined(ADDRESS_SANITIZER)
|
||||
*result = malloc(size);
|
||||
#else
|
||||
if (g_old_zone.malloc) {
|
||||
*result = g_old_zone.malloc(malloc_default_zone(), size);
|
||||
} else {
|
||||
*result = malloc(size);
|
||||
}
|
||||
#endif // defined(ADDRESS_SANITIZER)
|
||||
|
||||
return *result != NULL;
|
||||
}
|
||||
|
||||
bool UncheckedCallocMac(size_t num_items, size_t size, void** result) {
|
||||
#if defined(ADDRESS_SANITIZER)
|
||||
*result = calloc(num_items, size);
|
||||
#else
|
||||
if (g_old_zone.calloc) {
|
||||
*result = g_old_zone.calloc(malloc_default_zone(), num_items, size);
|
||||
} else {
|
||||
*result = calloc(num_items, size);
|
||||
}
|
||||
#endif // defined(ADDRESS_SANITIZER)
|
||||
|
||||
return *result != NULL;
|
||||
}
|
||||
|
||||
void StoreFunctionsForDefaultZone() {
|
||||
ChromeMallocZone* default_zone = reinterpret_cast<ChromeMallocZone*>(
|
||||
malloc_default_zone());
|
||||
StoreMallocZone(default_zone);
|
||||
}
|
||||
|
||||
void StoreFunctionsForAllZones() {
|
||||
// This ensures that the default zone is always at the front of the array,
|
||||
// which is important for performance.
|
||||
StoreFunctionsForDefaultZone();
|
||||
|
||||
vm_address_t* zones;
|
||||
unsigned int count;
|
||||
kern_return_t kr = malloc_get_all_zones(mach_task_self(), 0, &zones, &count);
|
||||
if (kr != KERN_SUCCESS)
|
||||
return;
|
||||
for (unsigned int i = 0; i < count; ++i) {
|
||||
ChromeMallocZone* zone = reinterpret_cast<ChromeMallocZone*>(zones[i]);
|
||||
StoreMallocZone(zone);
|
||||
}
|
||||
}
|
||||
|
||||
void ReplaceFunctionsForStoredZones(const MallocZoneFunctions* functions) {
|
||||
// The default zone does not get returned in malloc_get_all_zones().
|
||||
ChromeMallocZone* default_zone =
|
||||
reinterpret_cast<ChromeMallocZone*>(malloc_default_zone());
|
||||
if (DoesMallocZoneNeedReplacing(default_zone, functions)) {
|
||||
ReplaceZoneFunctions(default_zone, functions);
|
||||
}
|
||||
|
||||
vm_address_t* zones;
|
||||
unsigned int count;
|
||||
kern_return_t kr =
|
||||
malloc_get_all_zones(mach_task_self(), nullptr, &zones, &count);
|
||||
if (kr != KERN_SUCCESS)
|
||||
return;
|
||||
for (unsigned int i = 0; i < count; ++i) {
|
||||
ChromeMallocZone* zone = reinterpret_cast<ChromeMallocZone*>(zones[i]);
|
||||
if (DoesMallocZoneNeedReplacing(zone, functions)) {
|
||||
ReplaceZoneFunctions(zone, functions);
|
||||
}
|
||||
}
|
||||
g_replaced_default_zone = true;
|
||||
}
|
||||
|
||||
void InterceptAllocationsMac() {
|
||||
if (g_oom_killer_enabled)
|
||||
return;
|
||||
|
||||
g_oom_killer_enabled = true;
|
||||
|
||||
// === C malloc/calloc/valloc/realloc/posix_memalign ===
|
||||
|
||||
// This approach is not perfect, as requests for amounts of memory larger than
|
||||
// MALLOC_ABSOLUTE_MAX_SIZE (currently SIZE_T_MAX - (2 * PAGE_SIZE)) will
|
||||
// still fail with a NULL rather than dying (see
|
||||
// http://opensource.apple.com/source/Libc/Libc-583/gen/malloc.c for details).
|
||||
// Unfortunately, it's the best we can do. Also note that this does not affect
|
||||
// allocations from non-default zones.
|
||||
|
||||
#if !defined(ADDRESS_SANITIZER)
|
||||
// Don't do anything special on OOM for the malloc zones replaced by
|
||||
// AddressSanitizer, as modifying or protecting them may not work correctly.
|
||||
ChromeMallocZone* default_zone =
|
||||
reinterpret_cast<ChromeMallocZone*>(malloc_default_zone());
|
||||
if (!IsMallocZoneAlreadyStored(default_zone)) {
|
||||
StoreZoneFunctions(default_zone, &g_old_zone);
|
||||
MallocZoneFunctions new_functions = {};
|
||||
new_functions.malloc = oom_killer_malloc;
|
||||
new_functions.calloc = oom_killer_calloc;
|
||||
new_functions.valloc = oom_killer_valloc;
|
||||
new_functions.free = oom_killer_free;
|
||||
new_functions.realloc = oom_killer_realloc;
|
||||
new_functions.memalign = oom_killer_memalign;
|
||||
|
||||
ReplaceZoneFunctions(default_zone, &new_functions);
|
||||
g_replaced_default_zone = true;
|
||||
}
|
||||
|
||||
ChromeMallocZone* purgeable_zone =
|
||||
reinterpret_cast<ChromeMallocZone*>(malloc_default_purgeable_zone());
|
||||
if (purgeable_zone && !IsMallocZoneAlreadyStored(purgeable_zone)) {
|
||||
StoreZoneFunctions(purgeable_zone, &g_old_purgeable_zone);
|
||||
MallocZoneFunctions new_functions = {};
|
||||
new_functions.malloc = oom_killer_malloc_purgeable;
|
||||
new_functions.calloc = oom_killer_calloc_purgeable;
|
||||
new_functions.valloc = oom_killer_valloc_purgeable;
|
||||
new_functions.free = oom_killer_free_purgeable;
|
||||
new_functions.realloc = oom_killer_realloc_purgeable;
|
||||
new_functions.memalign = oom_killer_memalign_purgeable;
|
||||
ReplaceZoneFunctions(purgeable_zone, &new_functions);
|
||||
}
|
||||
#endif
|
||||
|
||||
// === C malloc_zone_batch_malloc ===
|
||||
|
||||
// batch_malloc is omitted because the default malloc zone's implementation
|
||||
// only supports batch_malloc for "tiny" allocations from the free list. It
|
||||
// will fail for allocations larger than "tiny", and will only allocate as
|
||||
// many blocks as it's able to from the free list. These factors mean that it
|
||||
// can return less than the requested memory even in a non-out-of-memory
|
||||
// situation. There's no good way to detect whether a batch_malloc failure is
|
||||
// due to these other factors, or due to genuine memory or address space
|
||||
// exhaustion. The fact that it only allocates space from the "tiny" free list
|
||||
// means that it's likely that a failure will not be due to memory exhaustion.
|
||||
// Similarly, these constraints on batch_malloc mean that callers must always
|
||||
// be expecting to receive less memory than was requested, even in situations
|
||||
// where memory pressure is not a concern. Finally, the only public interface
|
||||
// to batch_malloc is malloc_zone_batch_malloc, which is specific to the
|
||||
// system's malloc implementation. It's unlikely that anyone's even heard of
|
||||
// it.
|
||||
|
||||
#ifndef ADDRESS_SANITIZER
|
||||
// === Core Foundation CFAllocators ===
|
||||
|
||||
// This will not catch allocation done by custom allocators, but will catch
|
||||
// all allocation done by system-provided ones.
|
||||
|
||||
CHECK(!g_old_cfallocator_system_default && !g_old_cfallocator_malloc &&
|
||||
!g_old_cfallocator_malloc_zone)
|
||||
<< "Old allocators unexpectedly non-null";
|
||||
|
||||
bool cf_allocator_internals_known = CanGetContextForCFAllocator();
|
||||
|
||||
if (cf_allocator_internals_known) {
|
||||
CFAllocatorContext* context =
|
||||
ContextForCFAllocator(kCFAllocatorSystemDefault);
|
||||
CHECK(context) << "Failed to get context for kCFAllocatorSystemDefault.";
|
||||
g_old_cfallocator_system_default = context->allocate;
|
||||
CHECK(g_old_cfallocator_system_default)
|
||||
<< "Failed to get kCFAllocatorSystemDefault allocation function.";
|
||||
context->allocate = oom_killer_cfallocator_system_default;
|
||||
|
||||
context = ContextForCFAllocator(kCFAllocatorMalloc);
|
||||
CHECK(context) << "Failed to get context for kCFAllocatorMalloc.";
|
||||
g_old_cfallocator_malloc = context->allocate;
|
||||
CHECK(g_old_cfallocator_malloc)
|
||||
<< "Failed to get kCFAllocatorMalloc allocation function.";
|
||||
context->allocate = oom_killer_cfallocator_malloc;
|
||||
|
||||
context = ContextForCFAllocator(kCFAllocatorMallocZone);
|
||||
CHECK(context) << "Failed to get context for kCFAllocatorMallocZone.";
|
||||
g_old_cfallocator_malloc_zone = context->allocate;
|
||||
CHECK(g_old_cfallocator_malloc_zone)
|
||||
<< "Failed to get kCFAllocatorMallocZone allocation function.";
|
||||
context->allocate = oom_killer_cfallocator_malloc_zone;
|
||||
} else {
|
||||
DLOG(WARNING) << "Internals of CFAllocator not known; out-of-memory "
|
||||
"failures via CFAllocator will not result in termination. "
|
||||
"http://crbug.com/45650";
|
||||
}
|
||||
#endif
|
||||
|
||||
// === Cocoa NSObject allocation ===
|
||||
|
||||
// Note that both +[NSObject new] and +[NSObject alloc] call through to
|
||||
// +[NSObject allocWithZone:].
|
||||
|
||||
CHECK(!g_old_allocWithZone) << "Old allocator unexpectedly non-null";
|
||||
|
||||
Class nsobject_class = [NSObject class];
|
||||
Method orig_method =
|
||||
class_getClassMethod(nsobject_class, @selector(allocWithZone:));
|
||||
g_old_allocWithZone =
|
||||
reinterpret_cast<allocWithZone_t>(method_getImplementation(orig_method));
|
||||
CHECK(g_old_allocWithZone)
|
||||
<< "Failed to get allocWithZone allocation function.";
|
||||
method_setImplementation(orig_method,
|
||||
reinterpret_cast<IMP>(oom_killer_allocWithZone));
|
||||
}
|
||||
|
||||
void UninterceptMallocZonesForTesting() {
|
||||
UninterceptMallocZoneForTesting(malloc_default_zone());
|
||||
vm_address_t* zones;
|
||||
unsigned int count;
|
||||
kern_return_t kr = malloc_get_all_zones(mach_task_self(), 0, &zones, &count);
|
||||
CHECK(kr == KERN_SUCCESS);
|
||||
for (unsigned int i = 0; i < count; ++i) {
|
||||
UninterceptMallocZoneForTesting(
|
||||
reinterpret_cast<struct _malloc_zone_t*>(zones[i]));
|
||||
}
|
||||
|
||||
ClearAllMallocZonesForTesting();
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
void ShimNewMallocZonesAndReschedule(base::Time end_time,
|
||||
base::TimeDelta delay) {
|
||||
ShimNewMallocZones();
|
||||
|
||||
if (base::Time::Now() > end_time)
|
||||
return;
|
||||
|
||||
base::TimeDelta next_delay = delay * 2;
|
||||
SequencedTaskRunnerHandle::Get()->PostDelayedTask(
|
||||
FROM_HERE,
|
||||
base::BindOnce(&ShimNewMallocZonesAndReschedule, end_time, next_delay),
|
||||
delay);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void PeriodicallyShimNewMallocZones() {
|
||||
base::Time end_time = base::Time::Now() + base::TimeDelta::FromMinutes(1);
|
||||
base::TimeDelta initial_delay = base::TimeDelta::FromSeconds(1);
|
||||
ShimNewMallocZonesAndReschedule(end_time, initial_delay);
|
||||
}
|
||||
|
||||
void ShimNewMallocZones() {
|
||||
StoreFunctionsForAllZones();
|
||||
|
||||
// Use the functions for the default zone as a template to replace those
|
||||
// new zones.
|
||||
ChromeMallocZone* default_zone =
|
||||
reinterpret_cast<ChromeMallocZone*>(malloc_default_zone());
|
||||
DCHECK(IsMallocZoneAlreadyStored(default_zone));
|
||||
|
||||
MallocZoneFunctions new_functions;
|
||||
StoreZoneFunctions(default_zone, &new_functions);
|
||||
ReplaceFunctionsForStoredZones(&new_functions);
|
||||
}
|
||||
|
||||
void ReplaceZoneFunctions(ChromeMallocZone* zone,
|
||||
const MallocZoneFunctions* functions) {
|
||||
// Remove protection.
|
||||
mach_vm_address_t reprotection_start = 0;
|
||||
mach_vm_size_t reprotection_length = 0;
|
||||
vm_prot_t reprotection_value = VM_PROT_NONE;
|
||||
DeprotectMallocZone(zone, &reprotection_start, &reprotection_length,
|
||||
&reprotection_value);
|
||||
|
||||
CHECK(functions->malloc && functions->calloc && functions->valloc &&
|
||||
functions->free && functions->realloc);
|
||||
zone->malloc = functions->malloc;
|
||||
zone->calloc = functions->calloc;
|
||||
zone->valloc = functions->valloc;
|
||||
zone->free = functions->free;
|
||||
zone->realloc = functions->realloc;
|
||||
if (functions->batch_malloc)
|
||||
zone->batch_malloc = functions->batch_malloc;
|
||||
if (functions->batch_free)
|
||||
zone->batch_free = functions->batch_free;
|
||||
if (functions->size)
|
||||
zone->size = functions->size;
|
||||
if (zone->version >= 5 && functions->memalign) {
|
||||
zone->memalign = functions->memalign;
|
||||
}
|
||||
if (zone->version >= 6 && functions->free_definite_size) {
|
||||
zone->free_definite_size = functions->free_definite_size;
|
||||
}
|
||||
|
||||
// Restore protection if it was active.
|
||||
if (reprotection_start) {
|
||||
kern_return_t result =
|
||||
mach_vm_protect(mach_task_self(), reprotection_start,
|
||||
reprotection_length, false, reprotection_value);
|
||||
MACH_CHECK(result == KERN_SUCCESS, result) << "mach_vm_protect";
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
386
TMessagesProj/jni/voip/webrtc/base/allocator/allocator_shim.cc
Normal file
386
TMessagesProj/jni/voip/webrtc/base/allocator/allocator_shim.cc
Normal file
|
|
@ -0,0 +1,386 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/allocator_shim.h"
|
||||
|
||||
#include <errno.h>
|
||||
|
||||
#include <atomic>
|
||||
#include <new>
|
||||
|
||||
#include "base/allocator/buildflags.h"
|
||||
#include "base/atomicops.h"
|
||||
#include "base/bits.h"
|
||||
#include "base/logging.h"
|
||||
#include "base/macros.h"
|
||||
#include "base/process/process_metrics.h"
|
||||
#include "base/threading/platform_thread.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
#if !defined(OS_WIN)
|
||||
#include <unistd.h>
|
||||
#else
|
||||
#include "base/allocator/winheap_stubs_win.h"
|
||||
#endif
|
||||
|
||||
#if defined(OS_MACOSX)
|
||||
#include <malloc/malloc.h>
|
||||
|
||||
#include "base/allocator/allocator_interception_mac.h"
|
||||
#endif
|
||||
|
||||
// No calls to malloc / new in this file. They would would cause re-entrancy of
|
||||
// the shim, which is hard to deal with. Keep this code as simple as possible
|
||||
// and don't use any external C++ object here, not even //base ones. Even if
|
||||
// they are safe to use today, in future they might be refactored.
|
||||
|
||||
namespace {
|
||||
|
||||
base::subtle::AtomicWord g_chain_head =
|
||||
reinterpret_cast<base::subtle::AtomicWord>(
|
||||
&base::allocator::AllocatorDispatch::default_dispatch);
|
||||
|
||||
bool g_call_new_handler_on_malloc_failure = false;
|
||||
|
||||
inline size_t GetCachedPageSize() {
|
||||
static size_t pagesize = 0;
|
||||
if (!pagesize)
|
||||
pagesize = base::GetPageSize();
|
||||
return pagesize;
|
||||
}
|
||||
|
||||
// Calls the std::new handler thread-safely. Returns true if a new_handler was
|
||||
// set and called, false if no new_handler was set.
|
||||
bool CallNewHandler(size_t size) {
|
||||
#if defined(OS_WIN)
|
||||
return base::allocator::WinCallNewHandler(size);
|
||||
#else
|
||||
std::new_handler nh = std::get_new_handler();
|
||||
if (!nh)
|
||||
return false;
|
||||
(*nh)();
|
||||
// Assume the new_handler will abort if it fails. Exception are disabled and
|
||||
// we don't support the case of a new_handler throwing std::bad_balloc.
|
||||
return true;
|
||||
#endif
|
||||
}
|
||||
|
||||
inline const base::allocator::AllocatorDispatch* GetChainHead() {
|
||||
// TODO(primiano): Just use NoBarrier_Load once crbug.com/593344 is fixed.
|
||||
// Unfortunately due to that bug NoBarrier_Load() is mistakenly fully
|
||||
// barriered on Linux+Clang, and that causes visible perf regressons.
|
||||
return reinterpret_cast<const base::allocator::AllocatorDispatch*>(
|
||||
#if defined(OS_LINUX) && defined(__clang__)
|
||||
*static_cast<const volatile base::subtle::AtomicWord*>(&g_chain_head)
|
||||
#else
|
||||
base::subtle::NoBarrier_Load(&g_chain_head)
|
||||
#endif
|
||||
);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
void SetCallNewHandlerOnMallocFailure(bool value) {
|
||||
g_call_new_handler_on_malloc_failure = value;
|
||||
}
|
||||
|
||||
void* UncheckedAlloc(size_t size) {
|
||||
const allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
return chain_head->alloc_function(chain_head, size, nullptr);
|
||||
}
|
||||
|
||||
void InsertAllocatorDispatch(AllocatorDispatch* dispatch) {
|
||||
// Loop in case of (an unlikely) race on setting the list head.
|
||||
size_t kMaxRetries = 7;
|
||||
for (size_t i = 0; i < kMaxRetries; ++i) {
|
||||
const AllocatorDispatch* chain_head = GetChainHead();
|
||||
dispatch->next = chain_head;
|
||||
|
||||
// This function guarantees to be thread-safe w.r.t. concurrent
|
||||
// insertions. It also has to guarantee that all the threads always
|
||||
// see a consistent chain, hence the atomic_thread_fence() below.
|
||||
// InsertAllocatorDispatch() is NOT a fastpath, as opposite to malloc(), so
|
||||
// we don't really want this to be a release-store with a corresponding
|
||||
// acquire-load during malloc().
|
||||
std::atomic_thread_fence(std::memory_order_seq_cst);
|
||||
subtle::AtomicWord old_value =
|
||||
reinterpret_cast<subtle::AtomicWord>(chain_head);
|
||||
// Set the chain head to the new dispatch atomically. If we lose the race,
|
||||
// the comparison will fail, and the new head of chain will be returned.
|
||||
if (subtle::NoBarrier_CompareAndSwap(
|
||||
&g_chain_head, old_value,
|
||||
reinterpret_cast<subtle::AtomicWord>(dispatch)) == old_value) {
|
||||
// Success.
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
CHECK(false); // Too many retries, this shouldn't happen.
|
||||
}
|
||||
|
||||
void RemoveAllocatorDispatchForTesting(AllocatorDispatch* dispatch) {
|
||||
DCHECK_EQ(GetChainHead(), dispatch);
|
||||
subtle::NoBarrier_Store(&g_chain_head,
|
||||
reinterpret_cast<subtle::AtomicWord>(dispatch->next));
|
||||
}
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
||||
// The Shim* functions below are the entry-points into the shim-layer and
|
||||
// are supposed to be invoked by the allocator_shim_override_*
|
||||
// headers to route the malloc / new symbols through the shim layer.
|
||||
// They are defined as ALWAYS_INLINE in order to remove a level of indirection
|
||||
// between the system-defined entry points and the shim implementations.
|
||||
extern "C" {
|
||||
|
||||
// The general pattern for allocations is:
|
||||
// - Try to allocate, if succeded return the pointer.
|
||||
// - If the allocation failed:
|
||||
// - Call the std::new_handler if it was a C++ allocation.
|
||||
// - Call the std::new_handler if it was a malloc() (or calloc() or similar)
|
||||
// AND SetCallNewHandlerOnMallocFailure(true).
|
||||
// - If the std::new_handler is NOT set just return nullptr.
|
||||
// - If the std::new_handler is set:
|
||||
// - Assume it will abort() if it fails (very likely the new_handler will
|
||||
// just suicide priting a message).
|
||||
// - Assume it did succeed if it returns, in which case reattempt the alloc.
|
||||
|
||||
ALWAYS_INLINE void* ShimCppNew(size_t size) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
void* ptr;
|
||||
do {
|
||||
void* context = nullptr;
|
||||
#if defined(OS_MACOSX)
|
||||
context = malloc_default_zone();
|
||||
#endif
|
||||
ptr = chain_head->alloc_function(chain_head, size, context);
|
||||
} while (!ptr && CallNewHandler(size));
|
||||
return ptr;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* ShimCppAlignedNew(size_t size, size_t alignment) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
void* ptr;
|
||||
do {
|
||||
void* context = nullptr;
|
||||
#if defined(OS_MACOSX)
|
||||
context = malloc_default_zone();
|
||||
#endif
|
||||
ptr = chain_head->alloc_aligned_function(chain_head, alignment, size,
|
||||
context);
|
||||
} while (!ptr && CallNewHandler(size));
|
||||
return ptr;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void ShimCppDelete(void* address) {
|
||||
void* context = nullptr;
|
||||
#if defined(OS_MACOSX)
|
||||
context = malloc_default_zone();
|
||||
#endif
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
return chain_head->free_function(chain_head, address, context);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* ShimMalloc(size_t size, void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
void* ptr;
|
||||
do {
|
||||
ptr = chain_head->alloc_function(chain_head, size, context);
|
||||
} while (!ptr && g_call_new_handler_on_malloc_failure &&
|
||||
CallNewHandler(size));
|
||||
return ptr;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* ShimCalloc(size_t n, size_t size, void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
void* ptr;
|
||||
do {
|
||||
ptr = chain_head->alloc_zero_initialized_function(chain_head, n, size,
|
||||
context);
|
||||
} while (!ptr && g_call_new_handler_on_malloc_failure &&
|
||||
CallNewHandler(size));
|
||||
return ptr;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* ShimRealloc(void* address, size_t size, void* context) {
|
||||
// realloc(size == 0) means free() and might return a nullptr. We should
|
||||
// not call the std::new_handler in that case, though.
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
void* ptr;
|
||||
do {
|
||||
ptr = chain_head->realloc_function(chain_head, address, size, context);
|
||||
} while (!ptr && size && g_call_new_handler_on_malloc_failure &&
|
||||
CallNewHandler(size));
|
||||
return ptr;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* ShimMemalign(size_t alignment, size_t size, void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
void* ptr;
|
||||
do {
|
||||
ptr = chain_head->alloc_aligned_function(chain_head, alignment, size,
|
||||
context);
|
||||
} while (!ptr && g_call_new_handler_on_malloc_failure &&
|
||||
CallNewHandler(size));
|
||||
return ptr;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE int ShimPosixMemalign(void** res, size_t alignment, size_t size) {
|
||||
// posix_memalign is supposed to check the arguments. See tc_posix_memalign()
|
||||
// in tc_malloc.cc.
|
||||
if (((alignment % sizeof(void*)) != 0) ||
|
||||
!base::bits::IsPowerOfTwo(alignment)) {
|
||||
return EINVAL;
|
||||
}
|
||||
void* ptr = ShimMemalign(alignment, size, nullptr);
|
||||
*res = ptr;
|
||||
return ptr ? 0 : ENOMEM;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* ShimValloc(size_t size, void* context) {
|
||||
return ShimMemalign(GetCachedPageSize(), size, context);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* ShimPvalloc(size_t size) {
|
||||
// pvalloc(0) should allocate one page, according to its man page.
|
||||
if (size == 0) {
|
||||
size = GetCachedPageSize();
|
||||
} else {
|
||||
size = (size + GetCachedPageSize() - 1) & ~(GetCachedPageSize() - 1);
|
||||
}
|
||||
// The third argument is nullptr because pvalloc is glibc only and does not
|
||||
// exist on OSX/BSD systems.
|
||||
return ShimMemalign(GetCachedPageSize(), size, nullptr);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void ShimFree(void* address, void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
return chain_head->free_function(chain_head, address, context);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE size_t ShimGetSizeEstimate(const void* address, void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
return chain_head->get_size_estimate_function(
|
||||
chain_head, const_cast<void*>(address), context);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE unsigned ShimBatchMalloc(size_t size,
|
||||
void** results,
|
||||
unsigned num_requested,
|
||||
void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
return chain_head->batch_malloc_function(chain_head, size, results,
|
||||
num_requested, context);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void ShimBatchFree(void** to_be_freed,
|
||||
unsigned num_to_be_freed,
|
||||
void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
return chain_head->batch_free_function(chain_head, to_be_freed,
|
||||
num_to_be_freed, context);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void ShimFreeDefiniteSize(void* ptr, size_t size, void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
return chain_head->free_definite_size_function(chain_head, ptr, size,
|
||||
context);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* ShimAlignedMalloc(size_t size,
|
||||
size_t alignment,
|
||||
void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
void* ptr = nullptr;
|
||||
do {
|
||||
ptr = chain_head->aligned_malloc_function(chain_head, size, alignment,
|
||||
context);
|
||||
} while (!ptr && g_call_new_handler_on_malloc_failure &&
|
||||
CallNewHandler(size));
|
||||
return ptr;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* ShimAlignedRealloc(void* address,
|
||||
size_t size,
|
||||
size_t alignment,
|
||||
void* context) {
|
||||
// _aligned_realloc(size == 0) means _aligned_free() and might return a
|
||||
// nullptr. We should not call the std::new_handler in that case, though.
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
void* ptr = nullptr;
|
||||
do {
|
||||
ptr = chain_head->aligned_realloc_function(chain_head, address, size,
|
||||
alignment, context);
|
||||
} while (!ptr && size && g_call_new_handler_on_malloc_failure &&
|
||||
CallNewHandler(size));
|
||||
return ptr;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void ShimAlignedFree(void* address, void* context) {
|
||||
const base::allocator::AllocatorDispatch* const chain_head = GetChainHead();
|
||||
return chain_head->aligned_free_function(chain_head, address, context);
|
||||
}
|
||||
|
||||
} // extern "C"
|
||||
|
||||
#if !defined(OS_WIN) && !defined(OS_MACOSX)
|
||||
// Cpp symbols (new / delete) should always be routed through the shim layer
|
||||
// except on Windows and macOS where the malloc intercept is deep enough that it
|
||||
// also catches the cpp calls.
|
||||
#include "base/allocator/allocator_shim_override_cpp_symbols.h"
|
||||
#endif
|
||||
|
||||
#if defined(OS_ANDROID)
|
||||
// Android does not support symbol interposition. The way malloc symbols are
|
||||
// intercepted on Android is by using link-time -wrap flags.
|
||||
#include "base/allocator/allocator_shim_override_linker_wrapped_symbols.h"
|
||||
#elif defined(OS_WIN)
|
||||
// On Windows we use plain link-time overriding of the CRT symbols.
|
||||
#include "base/allocator/allocator_shim_override_ucrt_symbols_win.h"
|
||||
#elif defined(OS_MACOSX)
|
||||
#include "base/allocator/allocator_shim_default_dispatch_to_mac_zoned_malloc.h"
|
||||
#include "base/allocator/allocator_shim_override_mac_symbols.h"
|
||||
#else
|
||||
#include "base/allocator/allocator_shim_override_libc_symbols.h"
|
||||
#endif
|
||||
|
||||
// In the case of tcmalloc we also want to plumb into the glibc hooks
|
||||
// to avoid that allocations made in glibc itself (e.g., strdup()) get
|
||||
// accidentally performed on the glibc heap instead of the tcmalloc one.
|
||||
#if BUILDFLAG(USE_TCMALLOC)
|
||||
#include "base/allocator/allocator_shim_override_glibc_weak_symbols.h"
|
||||
#endif
|
||||
|
||||
#if defined(OS_MACOSX)
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
void InitializeAllocatorShim() {
|
||||
// Prepares the default dispatch. After the intercepted malloc calls have
|
||||
// traversed the shim this will route them to the default malloc zone.
|
||||
InitializeDefaultDispatchToMacAllocator();
|
||||
|
||||
MallocZoneFunctions functions = MallocZoneFunctionsToReplaceDefault();
|
||||
|
||||
// This replaces the default malloc zone, causing calls to malloc & friends
|
||||
// from the codebase to be routed to ShimMalloc() above.
|
||||
base::allocator::ReplaceFunctionsForStoredZones(&functions);
|
||||
}
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
#endif
|
||||
|
||||
// Cross-checks.
|
||||
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
#error The allocator shim should not be compiled when building for memory tools.
|
||||
#endif
|
||||
|
||||
#if (defined(__GNUC__) && defined(__EXCEPTIONS)) || \
|
||||
(defined(_MSC_VER) && defined(_CPPUNWIND))
|
||||
#error This code cannot be used when exceptions are turned on.
|
||||
#endif
|
||||
152
TMessagesProj/jni/voip/webrtc/base/allocator/allocator_shim.h
Normal file
152
TMessagesProj/jni/voip/webrtc/base/allocator/allocator_shim.h
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_ALLOCATOR_SHIM_H_
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_H_
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
#include "base/base_export.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
// Allocator Shim API. Allows to:
|
||||
// - Configure the behavior of the allocator (what to do on OOM failures).
|
||||
// - Install new hooks (AllocatorDispatch) in the allocator chain.
|
||||
|
||||
// When this shim layer is enabled, the route of an allocation is as-follows:
|
||||
//
|
||||
// [allocator_shim_override_*.h] Intercept malloc() / operator new calls:
|
||||
// The override_* headers define the symbols required to intercept calls to
|
||||
// malloc() and operator new (if not overridden by specific C++ classes).
|
||||
//
|
||||
// [allocator_shim.cc] Routing allocation calls to the shim:
|
||||
// The headers above route the calls to the internal ShimMalloc(), ShimFree(),
|
||||
// ShimCppNew() etc. methods defined in allocator_shim.cc.
|
||||
// These methods will: (1) forward the allocation call to the front of the
|
||||
// AllocatorDispatch chain. (2) perform security hardenings (e.g., might
|
||||
// call std::new_handler on OOM failure).
|
||||
//
|
||||
// [allocator_shim_default_dispatch_to_*.cc] The AllocatorDispatch chain:
|
||||
// It is a singly linked list where each element is a struct with function
|
||||
// pointers (|malloc_function|, |free_function|, etc). Normally the chain
|
||||
// consists of a single AllocatorDispatch element, herein called
|
||||
// the "default dispatch", which is statically defined at build time and
|
||||
// ultimately routes the calls to the actual allocator defined by the build
|
||||
// config (tcmalloc, glibc, ...).
|
||||
//
|
||||
// It is possible to dynamically insert further AllocatorDispatch stages
|
||||
// to the front of the chain, for debugging / profiling purposes.
|
||||
//
|
||||
// All the functions must be thread safe. The shim does not enforce any
|
||||
// serialization. This is to route to thread-aware allocators (e.g, tcmalloc)
|
||||
// wihout introducing unnecessary perf hits.
|
||||
|
||||
struct AllocatorDispatch {
|
||||
using AllocFn = void*(const AllocatorDispatch* self,
|
||||
size_t size,
|
||||
void* context);
|
||||
using AllocZeroInitializedFn = void*(const AllocatorDispatch* self,
|
||||
size_t n,
|
||||
size_t size,
|
||||
void* context);
|
||||
using AllocAlignedFn = void*(const AllocatorDispatch* self,
|
||||
size_t alignment,
|
||||
size_t size,
|
||||
void* context);
|
||||
using ReallocFn = void*(const AllocatorDispatch* self,
|
||||
void* address,
|
||||
size_t size,
|
||||
void* context);
|
||||
using FreeFn = void(const AllocatorDispatch* self,
|
||||
void* address,
|
||||
void* context);
|
||||
// Returns the best available estimate for the actual amount of memory
|
||||
// consumed by the allocation |address|. If possible, this should include
|
||||
// heap overhead or at least a decent estimate of the full cost of the
|
||||
// allocation. If no good estimate is possible, returns zero.
|
||||
using GetSizeEstimateFn = size_t(const AllocatorDispatch* self,
|
||||
void* address,
|
||||
void* context);
|
||||
using BatchMallocFn = unsigned(const AllocatorDispatch* self,
|
||||
size_t size,
|
||||
void** results,
|
||||
unsigned num_requested,
|
||||
void* context);
|
||||
using BatchFreeFn = void(const AllocatorDispatch* self,
|
||||
void** to_be_freed,
|
||||
unsigned num_to_be_freed,
|
||||
void* context);
|
||||
using FreeDefiniteSizeFn = void(const AllocatorDispatch* self,
|
||||
void* ptr,
|
||||
size_t size,
|
||||
void* context);
|
||||
using AlignedMallocFn = void*(const AllocatorDispatch* self,
|
||||
size_t size,
|
||||
size_t alignment,
|
||||
void* context);
|
||||
using AlignedReallocFn = void*(const AllocatorDispatch* self,
|
||||
void* address,
|
||||
size_t size,
|
||||
size_t alignment,
|
||||
void* context);
|
||||
using AlignedFreeFn = void(const AllocatorDispatch* self,
|
||||
void* address,
|
||||
void* context);
|
||||
|
||||
AllocFn* const alloc_function;
|
||||
AllocZeroInitializedFn* const alloc_zero_initialized_function;
|
||||
AllocAlignedFn* const alloc_aligned_function;
|
||||
ReallocFn* const realloc_function;
|
||||
FreeFn* const free_function;
|
||||
GetSizeEstimateFn* const get_size_estimate_function;
|
||||
// batch_malloc, batch_free, and free_definite_size are specific to the OSX
|
||||
// and iOS allocators.
|
||||
BatchMallocFn* const batch_malloc_function;
|
||||
BatchFreeFn* const batch_free_function;
|
||||
FreeDefiniteSizeFn* const free_definite_size_function;
|
||||
// _aligned_malloc, _aligned_realloc, and _aligned_free are specific to the
|
||||
// Windows allocator.
|
||||
AlignedMallocFn* const aligned_malloc_function;
|
||||
AlignedReallocFn* const aligned_realloc_function;
|
||||
AlignedFreeFn* const aligned_free_function;
|
||||
|
||||
const AllocatorDispatch* next;
|
||||
|
||||
// |default_dispatch| is statically defined by one (and only one) of the
|
||||
// allocator_shim_default_dispatch_to_*.cc files, depending on the build
|
||||
// configuration.
|
||||
static const AllocatorDispatch default_dispatch;
|
||||
};
|
||||
|
||||
// When true makes malloc behave like new, w.r.t calling the new_handler if
|
||||
// the allocation fails (see set_new_mode() in Windows).
|
||||
BASE_EXPORT void SetCallNewHandlerOnMallocFailure(bool value);
|
||||
|
||||
// Allocates |size| bytes or returns nullptr. It does NOT call the new_handler,
|
||||
// regardless of SetCallNewHandlerOnMallocFailure().
|
||||
BASE_EXPORT void* UncheckedAlloc(size_t size);
|
||||
|
||||
// Inserts |dispatch| in front of the allocator chain. This method is
|
||||
// thread-safe w.r.t concurrent invocations of InsertAllocatorDispatch().
|
||||
// The callers have responsibility for inserting a single dispatch no more
|
||||
// than once.
|
||||
BASE_EXPORT void InsertAllocatorDispatch(AllocatorDispatch* dispatch);
|
||||
|
||||
// Test-only. Rationale: (1) lack of use cases; (2) dealing safely with a
|
||||
// removal of arbitrary elements from a singly linked list would require a lock
|
||||
// in malloc(), which we really don't want.
|
||||
BASE_EXPORT void RemoveAllocatorDispatchForTesting(AllocatorDispatch* dispatch);
|
||||
|
||||
#if defined(OS_MACOSX)
|
||||
// On macOS, the allocator shim needs to be turned on during runtime.
|
||||
BASE_EXPORT void InitializeAllocatorShim();
|
||||
#endif // defined(OS_MACOSX)
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_ALLOCATOR_SHIM_H_
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/allocator_shim.h"
|
||||
#include "base/compiler_specific.h"
|
||||
|
||||
#include <dlfcn.h>
|
||||
#include <malloc.h>
|
||||
|
||||
// This translation unit defines a default dispatch for the allocator shim which
|
||||
// routes allocations to libc functions.
|
||||
// The code here is strongly inspired from tcmalloc's libc_override_glibc.h.
|
||||
|
||||
extern "C" {
|
||||
void* __libc_malloc(size_t size);
|
||||
void* __libc_calloc(size_t n, size_t size);
|
||||
void* __libc_realloc(void* address, size_t size);
|
||||
void* __libc_memalign(size_t alignment, size_t size);
|
||||
void __libc_free(void* ptr);
|
||||
} // extern "C"
|
||||
|
||||
namespace {
|
||||
|
||||
using base::allocator::AllocatorDispatch;
|
||||
|
||||
void* GlibcMalloc(const AllocatorDispatch*, size_t size, void* context) {
|
||||
return __libc_malloc(size);
|
||||
}
|
||||
|
||||
void* GlibcCalloc(const AllocatorDispatch*,
|
||||
size_t n,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return __libc_calloc(n, size);
|
||||
}
|
||||
|
||||
void* GlibcRealloc(const AllocatorDispatch*,
|
||||
void* address,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return __libc_realloc(address, size);
|
||||
}
|
||||
|
||||
void* GlibcMemalign(const AllocatorDispatch*,
|
||||
size_t alignment,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return __libc_memalign(alignment, size);
|
||||
}
|
||||
|
||||
void GlibcFree(const AllocatorDispatch*, void* address, void* context) {
|
||||
__libc_free(address);
|
||||
}
|
||||
|
||||
NO_SANITIZE("cfi-icall")
|
||||
size_t GlibcGetSizeEstimate(const AllocatorDispatch*,
|
||||
void* address,
|
||||
void* context) {
|
||||
// glibc does not expose an alias to resolve malloc_usable_size. Dynamically
|
||||
// resolve it instead. This should be safe because glibc (and hence dlfcn)
|
||||
// does not use malloc_size internally and so there should not be a risk of
|
||||
// recursion.
|
||||
using MallocUsableSizeFunction = decltype(malloc_usable_size)*;
|
||||
static MallocUsableSizeFunction fn_ptr =
|
||||
reinterpret_cast<MallocUsableSizeFunction>(
|
||||
dlsym(RTLD_NEXT, "malloc_usable_size"));
|
||||
|
||||
return fn_ptr(address);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const AllocatorDispatch AllocatorDispatch::default_dispatch = {
|
||||
&GlibcMalloc, /* alloc_function */
|
||||
&GlibcCalloc, /* alloc_zero_initialized_function */
|
||||
&GlibcMemalign, /* alloc_aligned_function */
|
||||
&GlibcRealloc, /* realloc_function */
|
||||
&GlibcFree, /* free_function */
|
||||
&GlibcGetSizeEstimate, /* get_size_estimate_function */
|
||||
nullptr, /* batch_malloc_function */
|
||||
nullptr, /* batch_free_function */
|
||||
nullptr, /* free_definite_size_function */
|
||||
nullptr, /* aligned_malloc_function */
|
||||
nullptr, /* aligned_realloc_function */
|
||||
nullptr, /* aligned_free_function */
|
||||
nullptr, /* next */
|
||||
};
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include <malloc.h>
|
||||
|
||||
#include "base/allocator/allocator_shim.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
// This translation unit defines a default dispatch for the allocator shim which
|
||||
// routes allocations to the original libc functions when using the link-time
|
||||
// -Wl,-wrap,malloc approach (see README.md).
|
||||
// The __real_X functions here are special symbols that the linker will relocate
|
||||
// against the real "X" undefined symbol, so that __real_malloc becomes the
|
||||
// equivalent of what an undefined malloc symbol reference would have been.
|
||||
// This is the counterpart of allocator_shim_override_linker_wrapped_symbols.h,
|
||||
// which routes the __wrap_X functions into the shim.
|
||||
|
||||
extern "C" {
|
||||
void* __real_malloc(size_t);
|
||||
void* __real_calloc(size_t, size_t);
|
||||
void* __real_realloc(void*, size_t);
|
||||
void* __real_memalign(size_t, size_t);
|
||||
void* __real_free(void*);
|
||||
} // extern "C"
|
||||
|
||||
namespace {
|
||||
|
||||
using base::allocator::AllocatorDispatch;
|
||||
|
||||
void* RealMalloc(const AllocatorDispatch*, size_t size, void* context) {
|
||||
return __real_malloc(size);
|
||||
}
|
||||
|
||||
void* RealCalloc(const AllocatorDispatch*,
|
||||
size_t n,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return __real_calloc(n, size);
|
||||
}
|
||||
|
||||
void* RealRealloc(const AllocatorDispatch*,
|
||||
void* address,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return __real_realloc(address, size);
|
||||
}
|
||||
|
||||
void* RealMemalign(const AllocatorDispatch*,
|
||||
size_t alignment,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return __real_memalign(alignment, size);
|
||||
}
|
||||
|
||||
void RealFree(const AllocatorDispatch*, void* address, void* context) {
|
||||
__real_free(address);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const AllocatorDispatch AllocatorDispatch::default_dispatch = {
|
||||
&RealMalloc, /* alloc_function */
|
||||
&RealCalloc, /* alloc_zero_initialized_function */
|
||||
&RealMemalign, /* alloc_aligned_function */
|
||||
&RealRealloc, /* realloc_function */
|
||||
&RealFree, /* free_function */
|
||||
nullptr, /* get_size_estimate_function */
|
||||
nullptr, /* batch_malloc_function */
|
||||
nullptr, /* batch_free_function */
|
||||
nullptr, /* free_definite_size_function */
|
||||
nullptr, /* aligned_malloc_function */
|
||||
nullptr, /* aligned_realloc_function */
|
||||
nullptr, /* aligned_free_function */
|
||||
nullptr, /* next */
|
||||
};
|
||||
|
|
@ -0,0 +1,112 @@
|
|||
// Copyright 2017 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/allocator_shim_default_dispatch_to_mac_zoned_malloc.h"
|
||||
|
||||
#include <utility>
|
||||
|
||||
#include "base/allocator/allocator_interception_mac.h"
|
||||
#include "base/allocator/allocator_shim.h"
|
||||
#include "base/allocator/malloc_zone_functions_mac.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
namespace {
|
||||
|
||||
void* MallocImpl(const AllocatorDispatch*, size_t size, void* context) {
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(context);
|
||||
return functions.malloc(reinterpret_cast<struct _malloc_zone_t*>(context),
|
||||
size);
|
||||
}
|
||||
|
||||
void* CallocImpl(const AllocatorDispatch*,
|
||||
size_t n,
|
||||
size_t size,
|
||||
void* context) {
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(context);
|
||||
return functions.calloc(reinterpret_cast<struct _malloc_zone_t*>(context), n,
|
||||
size);
|
||||
}
|
||||
|
||||
void* MemalignImpl(const AllocatorDispatch*,
|
||||
size_t alignment,
|
||||
size_t size,
|
||||
void* context) {
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(context);
|
||||
return functions.memalign(reinterpret_cast<struct _malloc_zone_t*>(context),
|
||||
alignment, size);
|
||||
}
|
||||
|
||||
void* ReallocImpl(const AllocatorDispatch*,
|
||||
void* ptr,
|
||||
size_t size,
|
||||
void* context) {
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(context);
|
||||
return functions.realloc(reinterpret_cast<struct _malloc_zone_t*>(context),
|
||||
ptr, size);
|
||||
}
|
||||
|
||||
void FreeImpl(const AllocatorDispatch*, void* ptr, void* context) {
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(context);
|
||||
functions.free(reinterpret_cast<struct _malloc_zone_t*>(context), ptr);
|
||||
}
|
||||
|
||||
size_t GetSizeEstimateImpl(const AllocatorDispatch*, void* ptr, void* context) {
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(context);
|
||||
return functions.size(reinterpret_cast<struct _malloc_zone_t*>(context), ptr);
|
||||
}
|
||||
|
||||
unsigned BatchMallocImpl(const AllocatorDispatch* self,
|
||||
size_t size,
|
||||
void** results,
|
||||
unsigned num_requested,
|
||||
void* context) {
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(context);
|
||||
return functions.batch_malloc(
|
||||
reinterpret_cast<struct _malloc_zone_t*>(context), size, results,
|
||||
num_requested);
|
||||
}
|
||||
|
||||
void BatchFreeImpl(const AllocatorDispatch* self,
|
||||
void** to_be_freed,
|
||||
unsigned num_to_be_freed,
|
||||
void* context) {
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(context);
|
||||
functions.batch_free(reinterpret_cast<struct _malloc_zone_t*>(context),
|
||||
to_be_freed, num_to_be_freed);
|
||||
}
|
||||
|
||||
void FreeDefiniteSizeImpl(const AllocatorDispatch* self,
|
||||
void* ptr,
|
||||
size_t size,
|
||||
void* context) {
|
||||
MallocZoneFunctions& functions = GetFunctionsForZone(context);
|
||||
functions.free_definite_size(
|
||||
reinterpret_cast<struct _malloc_zone_t*>(context), ptr, size);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void InitializeDefaultDispatchToMacAllocator() {
|
||||
StoreFunctionsForAllZones();
|
||||
}
|
||||
|
||||
const AllocatorDispatch AllocatorDispatch::default_dispatch = {
|
||||
&MallocImpl, /* alloc_function */
|
||||
&CallocImpl, /* alloc_zero_initialized_function */
|
||||
&MemalignImpl, /* alloc_aligned_function */
|
||||
&ReallocImpl, /* realloc_function */
|
||||
&FreeImpl, /* free_function */
|
||||
&GetSizeEstimateImpl, /* get_size_estimate_function */
|
||||
&BatchMallocImpl, /* batch_malloc_function */
|
||||
&BatchFreeImpl, /* batch_free_function */
|
||||
&FreeDefiniteSizeImpl, /* free_definite_size_function */
|
||||
nullptr, /* aligned_malloc_function */
|
||||
nullptr, /* aligned_realloc_function */
|
||||
nullptr, /* aligned_free_function */
|
||||
nullptr, /* next */
|
||||
};
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
// Copyright 2017 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_ALLOCATOR_SHIM_DEFAULT_DISPATCH_TO_ZONED_MALLOC_H_
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_DEFAULT_DISPATCH_TO_ZONED_MALLOC_H_
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
// This initializes AllocatorDispatch::default_dispatch by saving pointers to
|
||||
// the functions in the current default malloc zone. This must be called before
|
||||
// the default malloc zone is changed to have its intended effect.
|
||||
void InitializeDefaultDispatchToMacAllocator();
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_ALLOCATOR_SHIM_DEFAULT_DISPATCH_TO_ZONED_MALLOC_H_
|
||||
|
|
@ -0,0 +1,84 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/allocator_shim.h"
|
||||
#include "base/allocator/allocator_shim_internals.h"
|
||||
|
||||
#include "third_party/tcmalloc/chromium/src/config.h"
|
||||
#include "third_party/tcmalloc/chromium/src/gperftools/tcmalloc.h"
|
||||
|
||||
namespace {
|
||||
|
||||
using base::allocator::AllocatorDispatch;
|
||||
|
||||
void* TCMalloc(const AllocatorDispatch*, size_t size, void* context) {
|
||||
return tc_malloc(size);
|
||||
}
|
||||
|
||||
void* TCCalloc(const AllocatorDispatch*, size_t n, size_t size, void* context) {
|
||||
return tc_calloc(n, size);
|
||||
}
|
||||
|
||||
void* TCMemalign(const AllocatorDispatch*,
|
||||
size_t alignment,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return tc_memalign(alignment, size);
|
||||
}
|
||||
|
||||
void* TCRealloc(const AllocatorDispatch*,
|
||||
void* address,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return tc_realloc(address, size);
|
||||
}
|
||||
|
||||
void TCFree(const AllocatorDispatch*, void* address, void* context) {
|
||||
tc_free(address);
|
||||
}
|
||||
|
||||
size_t TCGetSizeEstimate(const AllocatorDispatch*,
|
||||
void* address,
|
||||
void* context) {
|
||||
return tc_malloc_size(address);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
const AllocatorDispatch AllocatorDispatch::default_dispatch = {
|
||||
&TCMalloc, /* alloc_function */
|
||||
&TCCalloc, /* alloc_zero_initialized_function */
|
||||
&TCMemalign, /* alloc_aligned_function */
|
||||
&TCRealloc, /* realloc_function */
|
||||
&TCFree, /* free_function */
|
||||
&TCGetSizeEstimate, /* get_size_estimate_function */
|
||||
nullptr, /* batch_malloc_function */
|
||||
nullptr, /* batch_free_function */
|
||||
nullptr, /* free_definite_size_function */
|
||||
nullptr, /* aligned_malloc_function */
|
||||
nullptr, /* aligned_realloc_function */
|
||||
nullptr, /* aligned_free_function */
|
||||
nullptr, /* next */
|
||||
};
|
||||
|
||||
// In the case of tcmalloc we have also to route the diagnostic symbols,
|
||||
// which are not part of the unified shim layer, to tcmalloc for consistency.
|
||||
|
||||
extern "C" {
|
||||
|
||||
SHIM_ALWAYS_EXPORT void malloc_stats(void) __THROW {
|
||||
return tc_malloc_stats();
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT int mallopt(int cmd, int value) __THROW {
|
||||
return tc_mallopt(cmd, value);
|
||||
}
|
||||
|
||||
#ifdef HAVE_STRUCT_MALLINFO
|
||||
SHIM_ALWAYS_EXPORT struct mallinfo mallinfo(void) __THROW {
|
||||
return tc_mallinfo();
|
||||
}
|
||||
#endif
|
||||
|
||||
} // extern "C"
|
||||
|
|
@ -0,0 +1,103 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/allocator_shim.h"
|
||||
|
||||
#include "base/allocator/winheap_stubs_win.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace {
|
||||
|
||||
using base::allocator::AllocatorDispatch;
|
||||
|
||||
void* DefaultWinHeapMallocImpl(const AllocatorDispatch*,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return base::allocator::WinHeapMalloc(size);
|
||||
}
|
||||
|
||||
void* DefaultWinHeapCallocImpl(const AllocatorDispatch* self,
|
||||
size_t n,
|
||||
size_t elem_size,
|
||||
void* context) {
|
||||
// Overflow check.
|
||||
const size_t size = n * elem_size;
|
||||
if (elem_size != 0 && size / elem_size != n)
|
||||
return nullptr;
|
||||
|
||||
void* result = DefaultWinHeapMallocImpl(self, size, context);
|
||||
if (result) {
|
||||
memset(result, 0, size);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
void* DefaultWinHeapMemalignImpl(const AllocatorDispatch* self,
|
||||
size_t alignment,
|
||||
size_t size,
|
||||
void* context) {
|
||||
CHECK(false) << "The windows heap does not support memalign.";
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
void* DefaultWinHeapReallocImpl(const AllocatorDispatch* self,
|
||||
void* address,
|
||||
size_t size,
|
||||
void* context) {
|
||||
return base::allocator::WinHeapRealloc(address, size);
|
||||
}
|
||||
|
||||
void DefaultWinHeapFreeImpl(const AllocatorDispatch*,
|
||||
void* address,
|
||||
void* context) {
|
||||
base::allocator::WinHeapFree(address);
|
||||
}
|
||||
|
||||
size_t DefaultWinHeapGetSizeEstimateImpl(const AllocatorDispatch*,
|
||||
void* address,
|
||||
void* context) {
|
||||
return base::allocator::WinHeapGetSizeEstimate(address);
|
||||
}
|
||||
|
||||
void* DefaultWinHeapAlignedMallocImpl(const AllocatorDispatch*,
|
||||
size_t size,
|
||||
size_t alignment,
|
||||
void* context) {
|
||||
return base::allocator::WinHeapAlignedMalloc(size, alignment);
|
||||
}
|
||||
|
||||
void* DefaultWinHeapAlignedReallocImpl(const AllocatorDispatch*,
|
||||
void* ptr,
|
||||
size_t size,
|
||||
size_t alignment,
|
||||
void* context) {
|
||||
return base::allocator::WinHeapAlignedRealloc(ptr, size, alignment);
|
||||
}
|
||||
|
||||
void DefaultWinHeapAlignedFreeImpl(const AllocatorDispatch*,
|
||||
void* ptr,
|
||||
void* context) {
|
||||
base::allocator::WinHeapAlignedFree(ptr);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// Guarantee that default_dispatch is compile-time initialized to avoid using
|
||||
// it before initialization (allocations before main in release builds with
|
||||
// optimizations disabled).
|
||||
constexpr AllocatorDispatch AllocatorDispatch::default_dispatch = {
|
||||
&DefaultWinHeapMallocImpl,
|
||||
&DefaultWinHeapCallocImpl,
|
||||
&DefaultWinHeapMemalignImpl,
|
||||
&DefaultWinHeapReallocImpl,
|
||||
&DefaultWinHeapFreeImpl,
|
||||
&DefaultWinHeapGetSizeEstimateImpl,
|
||||
nullptr, /* batch_malloc_function */
|
||||
nullptr, /* batch_free_function */
|
||||
nullptr, /* free_definite_size_function */
|
||||
&DefaultWinHeapAlignedMallocImpl,
|
||||
&DefaultWinHeapAlignedReallocImpl,
|
||||
&DefaultWinHeapAlignedFreeImpl,
|
||||
nullptr, /* next */
|
||||
};
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_ALLOCATOR_SHIM_INTERNALS_H_
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_INTERNALS_H_
|
||||
|
||||
#if defined(__GNUC__)
|
||||
|
||||
#include <sys/cdefs.h> // for __THROW
|
||||
|
||||
#ifndef __THROW // Not a glibc system
|
||||
#ifdef _NOEXCEPT // LLVM libc++ uses noexcept instead
|
||||
#define __THROW _NOEXCEPT
|
||||
#else
|
||||
#define __THROW
|
||||
#endif // !_NOEXCEPT
|
||||
#endif
|
||||
|
||||
// Shim layer symbols need to be ALWAYS exported, regardless of component build.
|
||||
//
|
||||
// If an exported symbol is linked into a DSO, it may be preempted by a
|
||||
// definition in the main executable. If this happens to an allocator symbol, it
|
||||
// will mean that the DSO will use the main executable's allocator. This is
|
||||
// normally relatively harmless -- regular allocations should all use the same
|
||||
// allocator, but if the DSO tries to hook the allocator it will not see any
|
||||
// allocations.
|
||||
//
|
||||
// However, if LLVM LTO is enabled, the compiler may inline the shim layer
|
||||
// symbols into callers. The end result is that allocator calls in DSOs may use
|
||||
// either the main executable's allocator or the DSO's allocator, depending on
|
||||
// whether the call was inlined. This is arguably a bug in LLVM caused by its
|
||||
// somewhat irregular handling of symbol interposition (see llvm.org/PR23501).
|
||||
// To work around the bug we use noinline to prevent the symbols from being
|
||||
// inlined.
|
||||
//
|
||||
// In the long run we probably want to avoid linking the allocator bits into
|
||||
// DSOs altogether. This will save a little space and stop giving DSOs the false
|
||||
// impression that they can hook the allocator.
|
||||
#define SHIM_ALWAYS_EXPORT __attribute__((visibility("default"), noinline))
|
||||
|
||||
#endif // __GNUC__
|
||||
|
||||
#endif // BASE_ALLOCATOR_ALLOCATOR_SHIM_INTERNALS_H_
|
||||
|
|
@ -0,0 +1,152 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_CPP_SYMBOLS_H_
|
||||
#error This header is meant to be included only once by allocator_shim.cc
|
||||
#endif
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_CPP_SYMBOLS_H_
|
||||
|
||||
// Preempt the default new/delete C++ symbols so they call the shim entry
|
||||
// points. This file is strongly inspired by tcmalloc's
|
||||
// libc_override_redefine.h.
|
||||
|
||||
#include <new>
|
||||
|
||||
#include "base/allocator/allocator_shim_internals.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
// std::align_val_t isn't available until C++17, but we want to override aligned
|
||||
// new/delete anyway to prevent a possible situation where a library gets loaded
|
||||
// in that uses the aligned operators. We want to avoid a situation where
|
||||
// separate heaps are used.
|
||||
// TODO(thomasanderson): Remove this once building with C++17 or later.
|
||||
#if defined(__cpp_aligned_new) && __cpp_aligned_new >= 201606
|
||||
#define ALIGN_VAL_T std::align_val_t
|
||||
#define ALIGN_LINKAGE
|
||||
#define ALIGN_NEW operator new
|
||||
#define ALIGN_NEW_NOTHROW operator new
|
||||
#define ALIGN_DEL operator delete
|
||||
#define ALIGN_DEL_SIZED operator delete
|
||||
#define ALIGN_DEL_NOTHROW operator delete
|
||||
#define ALIGN_NEW_ARR operator new[]
|
||||
#define ALIGN_NEW_ARR_NOTHROW operator new[]
|
||||
#define ALIGN_DEL_ARR operator delete[]
|
||||
#define ALIGN_DEL_ARR_SIZED operator delete[]
|
||||
#define ALIGN_DEL_ARR_NOTHROW operator delete[]
|
||||
#else
|
||||
#define ALIGN_VAL_T size_t
|
||||
#define ALIGN_LINKAGE extern "C"
|
||||
#if defined(OS_MACOSX) || defined(OS_WIN)
|
||||
#error "Mangling is different on these platforms."
|
||||
#else
|
||||
#define ALIGN_NEW _ZnwmSt11align_val_t
|
||||
#define ALIGN_NEW_NOTHROW _ZnwmSt11align_val_tRKSt9nothrow_t
|
||||
#define ALIGN_DEL _ZdlPvSt11align_val_t
|
||||
#define ALIGN_DEL_SIZED _ZdlPvmSt11align_val_t
|
||||
#define ALIGN_DEL_NOTHROW _ZdlPvSt11align_val_tRKSt9nothrow_t
|
||||
#define ALIGN_NEW_ARR _ZnamSt11align_val_t
|
||||
#define ALIGN_NEW_ARR_NOTHROW _ZnamSt11align_val_tRKSt9nothrow_t
|
||||
#define ALIGN_DEL_ARR _ZdaPvSt11align_val_t
|
||||
#define ALIGN_DEL_ARR_SIZED _ZdaPvmSt11align_val_t
|
||||
#define ALIGN_DEL_ARR_NOTHROW _ZdaPvSt11align_val_tRKSt9nothrow_t
|
||||
#endif
|
||||
#endif
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* operator new(size_t size) {
|
||||
return ShimCppNew(size);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void operator delete(void* p) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* operator new[](size_t size) {
|
||||
return ShimCppNew(size);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void operator delete[](void* p) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* operator new(size_t size,
|
||||
const std::nothrow_t&) __THROW {
|
||||
return ShimCppNew(size);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* operator new[](size_t size,
|
||||
const std::nothrow_t&) __THROW {
|
||||
return ShimCppNew(size);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void operator delete(void* p, const std::nothrow_t&) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void operator delete[](void* p,
|
||||
const std::nothrow_t&) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void operator delete(void* p, size_t) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void operator delete[](void* p, size_t) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void* ALIGN_NEW(std::size_t size,
|
||||
ALIGN_VAL_T alignment) {
|
||||
return ShimCppAlignedNew(size, static_cast<size_t>(alignment));
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void* ALIGN_NEW_NOTHROW(
|
||||
std::size_t size,
|
||||
ALIGN_VAL_T alignment,
|
||||
const std::nothrow_t&) __THROW {
|
||||
return ShimCppAlignedNew(size, static_cast<size_t>(alignment));
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void ALIGN_DEL(void* p, ALIGN_VAL_T) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void ALIGN_DEL_SIZED(void* p,
|
||||
std::size_t size,
|
||||
ALIGN_VAL_T) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void
|
||||
ALIGN_DEL_NOTHROW(void* p, ALIGN_VAL_T, const std::nothrow_t&) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void* ALIGN_NEW_ARR(std::size_t size,
|
||||
ALIGN_VAL_T alignment) {
|
||||
return ShimCppAlignedNew(size, static_cast<size_t>(alignment));
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void* ALIGN_NEW_ARR_NOTHROW(
|
||||
std::size_t size,
|
||||
ALIGN_VAL_T alignment,
|
||||
const std::nothrow_t&) __THROW {
|
||||
return ShimCppAlignedNew(size, static_cast<size_t>(alignment));
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void ALIGN_DEL_ARR(void* p,
|
||||
ALIGN_VAL_T) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void ALIGN_DEL_ARR_SIZED(void* p,
|
||||
std::size_t size,
|
||||
ALIGN_VAL_T) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
||||
ALIGN_LINKAGE SHIM_ALWAYS_EXPORT void
|
||||
ALIGN_DEL_ARR_NOTHROW(void* p, ALIGN_VAL_T, const std::nothrow_t&) __THROW {
|
||||
ShimCppDelete(p);
|
||||
}
|
||||
|
|
@ -0,0 +1,119 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_GLIBC_WEAK_SYMBOLS_H_
|
||||
#error This header is meant to be included only once by allocator_shim.cc
|
||||
#endif
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_GLIBC_WEAK_SYMBOLS_H_
|
||||
|
||||
// Alias the internal Glibc symbols to the shim entry points.
|
||||
// This file is strongly inspired by tcmalloc's libc_override_glibc.h.
|
||||
// Effectively this file does two things:
|
||||
// 1) Re-define the __malloc_hook & co symbols. Those symbols are defined as
|
||||
// weak in glibc and are meant to be defined strongly by client processes
|
||||
// to hook calls initiated from within glibc.
|
||||
// 2) Re-define Glibc-specific symbols (__libc_malloc). The historical reason
|
||||
// is that in the past (in RedHat 9) we had instances of libraries that were
|
||||
// allocating via malloc() and freeing using __libc_free().
|
||||
// See tcmalloc's libc_override_glibc.h for more context.
|
||||
|
||||
#include <features.h> // for __GLIBC__
|
||||
#include <malloc.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <new>
|
||||
|
||||
#include "base/allocator/allocator_shim_internals.h"
|
||||
|
||||
// __MALLOC_HOOK_VOLATILE not defined in all Glibc headers.
|
||||
#if !defined(__MALLOC_HOOK_VOLATILE)
|
||||
#define MALLOC_HOOK_MAYBE_VOLATILE /**/
|
||||
#else
|
||||
#define MALLOC_HOOK_MAYBE_VOLATILE __MALLOC_HOOK_VOLATILE
|
||||
#endif
|
||||
|
||||
extern "C" {
|
||||
|
||||
// 1) Re-define malloc_hook weak symbols.
|
||||
namespace {
|
||||
|
||||
void* GlibcMallocHook(size_t size, const void* caller) {
|
||||
return ShimMalloc(size, nullptr);
|
||||
}
|
||||
|
||||
void* GlibcReallocHook(void* ptr, size_t size, const void* caller) {
|
||||
return ShimRealloc(ptr, size, nullptr);
|
||||
}
|
||||
|
||||
void GlibcFreeHook(void* ptr, const void* caller) {
|
||||
return ShimFree(ptr, nullptr);
|
||||
}
|
||||
|
||||
void* GlibcMemalignHook(size_t align, size_t size, const void* caller) {
|
||||
return ShimMemalign(align, size, nullptr);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
__attribute__((visibility("default"))) void* (
|
||||
*MALLOC_HOOK_MAYBE_VOLATILE __malloc_hook)(size_t,
|
||||
const void*) = &GlibcMallocHook;
|
||||
|
||||
__attribute__((visibility("default"))) void* (
|
||||
*MALLOC_HOOK_MAYBE_VOLATILE __realloc_hook)(void*, size_t, const void*) =
|
||||
&GlibcReallocHook;
|
||||
|
||||
__attribute__((visibility("default"))) void (
|
||||
*MALLOC_HOOK_MAYBE_VOLATILE __free_hook)(void*,
|
||||
const void*) = &GlibcFreeHook;
|
||||
|
||||
__attribute__((visibility("default"))) void* (
|
||||
*MALLOC_HOOK_MAYBE_VOLATILE __memalign_hook)(size_t, size_t, const void*) =
|
||||
&GlibcMemalignHook;
|
||||
|
||||
// 2) Redefine libc symbols themselves.
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __libc_malloc(size_t size) {
|
||||
return ShimMalloc(size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void __libc_free(void* ptr) {
|
||||
ShimFree(ptr, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __libc_realloc(void* ptr, size_t size) {
|
||||
return ShimRealloc(ptr, size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __libc_calloc(size_t n, size_t size) {
|
||||
return ShimCalloc(n, size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void __libc_cfree(void* ptr) {
|
||||
return ShimFree(ptr, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __libc_memalign(size_t align, size_t s) {
|
||||
return ShimMemalign(align, s, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __libc_valloc(size_t size) {
|
||||
return ShimValloc(size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __libc_pvalloc(size_t size) {
|
||||
return ShimPvalloc(size);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT int __posix_memalign(void** r, size_t a, size_t s) {
|
||||
return ShimPosixMemalign(r, a, s);
|
||||
}
|
||||
|
||||
} // extern "C"
|
||||
|
||||
// Safety check.
|
||||
#if !defined(__GLIBC__)
|
||||
#error The target platform does not seem to use Glibc. Disable the allocator \
|
||||
shim by setting use_allocator_shim=false in GN args.
|
||||
#endif
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
// Its purpose is to preempt the Libc symbols for malloc/new so they call the
|
||||
// shim layer entry points.
|
||||
|
||||
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_LIBC_SYMBOLS_H_
|
||||
#error This header is meant to be included only once by allocator_shim.cc
|
||||
#endif
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_LIBC_SYMBOLS_H_
|
||||
|
||||
#include <malloc.h>
|
||||
|
||||
#include "base/allocator/allocator_shim_internals.h"
|
||||
|
||||
extern "C" {
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* malloc(size_t size) __THROW {
|
||||
return ShimMalloc(size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void free(void* ptr) __THROW {
|
||||
ShimFree(ptr, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* realloc(void* ptr, size_t size) __THROW {
|
||||
return ShimRealloc(ptr, size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* calloc(size_t n, size_t size) __THROW {
|
||||
return ShimCalloc(n, size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void cfree(void* ptr) __THROW {
|
||||
ShimFree(ptr, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* memalign(size_t align, size_t s) __THROW {
|
||||
return ShimMemalign(align, s, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* valloc(size_t size) __THROW {
|
||||
return ShimValloc(size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* pvalloc(size_t size) __THROW {
|
||||
return ShimPvalloc(size);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT int posix_memalign(void** r, size_t a, size_t s) __THROW {
|
||||
return ShimPosixMemalign(r, a, s);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT size_t malloc_size(void* address) __THROW {
|
||||
return ShimGetSizeEstimate(address, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT size_t malloc_usable_size(void* address) __THROW {
|
||||
return ShimGetSizeEstimate(address, nullptr);
|
||||
}
|
||||
|
||||
// The default dispatch translation unit has to define also the following
|
||||
// symbols (unless they are ultimately routed to the system symbols):
|
||||
// void malloc_stats(void);
|
||||
// int mallopt(int, int);
|
||||
// struct mallinfo mallinfo(void);
|
||||
|
||||
} // extern "C"
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_LINKER_WRAPPED_SYMBOLS_H_
|
||||
#error This header is meant to be included only once by allocator_shim.cc
|
||||
#endif
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_LINKER_WRAPPED_SYMBOLS_H_
|
||||
|
||||
// This header overrides the __wrap_X symbols when using the link-time
|
||||
// -Wl,-wrap,malloc shim-layer approach (see README.md).
|
||||
// All references to malloc, free, etc. within the linker unit that gets the
|
||||
// -wrap linker flags (e.g., libchrome.so) will be rewritten to the
|
||||
// linker as references to __wrap_malloc, __wrap_free, which are defined here.
|
||||
|
||||
#include "base/allocator/allocator_shim_internals.h"
|
||||
|
||||
extern "C" {
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __wrap_calloc(size_t n, size_t size) {
|
||||
return ShimCalloc(n, size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void __wrap_free(void* ptr) {
|
||||
ShimFree(ptr, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __wrap_malloc(size_t size) {
|
||||
return ShimMalloc(size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __wrap_memalign(size_t align, size_t size) {
|
||||
return ShimMemalign(align, size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT int __wrap_posix_memalign(void** res,
|
||||
size_t align,
|
||||
size_t size) {
|
||||
return ShimPosixMemalign(res, align, size);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __wrap_pvalloc(size_t size) {
|
||||
return ShimPvalloc(size);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __wrap_realloc(void* address, size_t size) {
|
||||
return ShimRealloc(address, size, nullptr);
|
||||
}
|
||||
|
||||
SHIM_ALWAYS_EXPORT void* __wrap_valloc(size_t size) {
|
||||
return ShimValloc(size, nullptr);
|
||||
}
|
||||
|
||||
} // extern "C"
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
// Copyright 2017 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_MAC_SYMBOLS_H_
|
||||
#error This header is meant to be included only once by allocator_shim.cc
|
||||
#endif
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_MAC_SYMBOLS_H_
|
||||
|
||||
#include "base/allocator/malloc_zone_functions_mac.h"
|
||||
#include "third_party/apple_apsl/malloc.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
MallocZoneFunctions MallocZoneFunctionsToReplaceDefault() {
|
||||
MallocZoneFunctions new_functions;
|
||||
memset(&new_functions, 0, sizeof(MallocZoneFunctions));
|
||||
new_functions.size = [](malloc_zone_t* zone, const void* ptr) -> size_t {
|
||||
return ShimGetSizeEstimate(ptr, zone);
|
||||
};
|
||||
new_functions.malloc = [](malloc_zone_t* zone, size_t size) -> void* {
|
||||
return ShimMalloc(size, zone);
|
||||
};
|
||||
new_functions.calloc = [](malloc_zone_t* zone, size_t n,
|
||||
size_t size) -> void* {
|
||||
return ShimCalloc(n, size, zone);
|
||||
};
|
||||
new_functions.valloc = [](malloc_zone_t* zone, size_t size) -> void* {
|
||||
return ShimValloc(size, zone);
|
||||
};
|
||||
new_functions.free = [](malloc_zone_t* zone, void* ptr) {
|
||||
ShimFree(ptr, zone);
|
||||
};
|
||||
new_functions.realloc = [](malloc_zone_t* zone, void* ptr,
|
||||
size_t size) -> void* {
|
||||
return ShimRealloc(ptr, size, zone);
|
||||
};
|
||||
new_functions.batch_malloc = [](struct _malloc_zone_t* zone, size_t size,
|
||||
void** results,
|
||||
unsigned num_requested) -> unsigned {
|
||||
return ShimBatchMalloc(size, results, num_requested, zone);
|
||||
};
|
||||
new_functions.batch_free = [](struct _malloc_zone_t* zone, void** to_be_freed,
|
||||
unsigned num_to_be_freed) -> void {
|
||||
ShimBatchFree(to_be_freed, num_to_be_freed, zone);
|
||||
};
|
||||
new_functions.memalign = [](malloc_zone_t* zone, size_t alignment,
|
||||
size_t size) -> void* {
|
||||
return ShimMemalign(alignment, size, zone);
|
||||
};
|
||||
new_functions.free_definite_size = [](malloc_zone_t* zone, void* ptr,
|
||||
size_t size) {
|
||||
ShimFreeDefiniteSize(ptr, size, zone);
|
||||
};
|
||||
return new_functions;
|
||||
}
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,174 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
// This header defines symbols to override the same functions in the Visual C++
|
||||
// CRT implementation.
|
||||
|
||||
#ifdef BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_UCRT_SYMBOLS_WIN_H_
|
||||
#error This header is meant to be included only once by allocator_shim.cc
|
||||
#endif
|
||||
#define BASE_ALLOCATOR_ALLOCATOR_SHIM_OVERRIDE_UCRT_SYMBOLS_WIN_H_
|
||||
|
||||
#include <malloc.h>
|
||||
|
||||
#include <windows.h>
|
||||
|
||||
extern "C" {
|
||||
|
||||
void* (*malloc_unchecked)(size_t) = &base::allocator::UncheckedAlloc;
|
||||
|
||||
namespace {
|
||||
|
||||
int win_new_mode = 0;
|
||||
|
||||
} // namespace
|
||||
|
||||
// This function behaves similarly to MSVC's _set_new_mode.
|
||||
// If flag is 0 (default), calls to malloc will behave normally.
|
||||
// If flag is 1, calls to malloc will behave like calls to new,
|
||||
// and the std_new_handler will be invoked on failure.
|
||||
// Returns the previous mode.
|
||||
//
|
||||
// Replaces _set_new_mode in ucrt\heap\new_mode.cpp
|
||||
int _set_new_mode(int flag) {
|
||||
// The MS CRT calls this function early on in startup, so this serves as a low
|
||||
// overhead proof that the allocator shim is in place for this process.
|
||||
base::allocator::g_is_win_shim_layer_initialized = true;
|
||||
int old_mode = win_new_mode;
|
||||
win_new_mode = flag;
|
||||
|
||||
base::allocator::SetCallNewHandlerOnMallocFailure(win_new_mode != 0);
|
||||
|
||||
return old_mode;
|
||||
}
|
||||
|
||||
// Replaces _query_new_mode in ucrt\heap\new_mode.cpp
|
||||
int _query_new_mode() {
|
||||
return win_new_mode;
|
||||
}
|
||||
|
||||
// These symbols override the CRT's implementation of the same functions.
|
||||
__declspec(restrict) void* malloc(size_t size) {
|
||||
return ShimMalloc(size, nullptr);
|
||||
}
|
||||
|
||||
void free(void* ptr) {
|
||||
ShimFree(ptr, nullptr);
|
||||
}
|
||||
|
||||
__declspec(restrict) void* realloc(void* ptr, size_t size) {
|
||||
return ShimRealloc(ptr, size, nullptr);
|
||||
}
|
||||
|
||||
__declspec(restrict) void* calloc(size_t n, size_t size) {
|
||||
return ShimCalloc(n, size, nullptr);
|
||||
}
|
||||
|
||||
// _msize() is the Windows equivalent of malloc_size().
|
||||
size_t _msize(void* memblock) {
|
||||
return ShimGetSizeEstimate(memblock, nullptr);
|
||||
}
|
||||
|
||||
__declspec(restrict) void* _aligned_malloc(size_t size, size_t alignment) {
|
||||
return ShimAlignedMalloc(size, alignment, nullptr);
|
||||
}
|
||||
|
||||
__declspec(restrict) void* _aligned_realloc(void* address,
|
||||
size_t size,
|
||||
size_t alignment) {
|
||||
return ShimAlignedRealloc(address, size, alignment, nullptr);
|
||||
}
|
||||
|
||||
void _aligned_free(void* address) {
|
||||
ShimAlignedFree(address, nullptr);
|
||||
}
|
||||
|
||||
// _recalloc_base is called by CRT internally.
|
||||
__declspec(restrict) void* _recalloc_base(void* block,
|
||||
size_t count,
|
||||
size_t size) {
|
||||
const size_t old_block_size = (block != nullptr) ? _msize(block) : 0;
|
||||
base::CheckedNumeric<size_t> new_block_size_checked = count;
|
||||
new_block_size_checked *= size;
|
||||
const size_t new_block_size = new_block_size_checked.ValueOrDie();
|
||||
|
||||
void* const new_block = realloc(block, new_block_size);
|
||||
|
||||
if (new_block != nullptr && old_block_size < new_block_size) {
|
||||
memset(static_cast<char*>(new_block) + old_block_size, 0,
|
||||
new_block_size - old_block_size);
|
||||
}
|
||||
|
||||
return new_block;
|
||||
}
|
||||
|
||||
__declspec(restrict) void* _recalloc(void* block, size_t count, size_t size) {
|
||||
return _recalloc_base(block, count, size);
|
||||
}
|
||||
|
||||
// The following uncommon _aligned_* routines are not used in Chromium and have
|
||||
// been shimmed to immediately crash to ensure that implementations are added if
|
||||
// uses are introduced.
|
||||
__declspec(restrict) void* _aligned_recalloc(void* address,
|
||||
size_t num,
|
||||
size_t size,
|
||||
size_t alignment) {
|
||||
CHECK(false) << "This routine has not been implemented";
|
||||
__builtin_unreachable();
|
||||
}
|
||||
|
||||
size_t _aligned_msize(void* address, size_t alignment, size_t offset) {
|
||||
CHECK(false) << "This routine has not been implemented";
|
||||
__builtin_unreachable();
|
||||
}
|
||||
|
||||
__declspec(restrict) void* _aligned_offset_malloc(size_t size,
|
||||
size_t alignment,
|
||||
size_t offset) {
|
||||
CHECK(false) << "This routine has not been implemented";
|
||||
__builtin_unreachable();
|
||||
}
|
||||
|
||||
__declspec(restrict) void* _aligned_offset_realloc(void* address,
|
||||
size_t size,
|
||||
size_t alignment,
|
||||
size_t offset) {
|
||||
CHECK(false) << "This routine has not been implemented";
|
||||
__builtin_unreachable();
|
||||
}
|
||||
|
||||
__declspec(restrict) void* _aligned_offset_recalloc(void* address,
|
||||
size_t num,
|
||||
size_t size,
|
||||
size_t alignment,
|
||||
size_t offset) {
|
||||
CHECK(false) << "This routine has not been implemented";
|
||||
__builtin_unreachable();
|
||||
}
|
||||
|
||||
// The symbols
|
||||
// * __acrt_heap
|
||||
// * __acrt_initialize_heap
|
||||
// * __acrt_uninitialize_heap
|
||||
// * _get_heap_handle
|
||||
// must be overridden all or none, as they are otherwise supplied
|
||||
// by heap_handle.obj in the ucrt.lib file.
|
||||
HANDLE __acrt_heap = nullptr;
|
||||
|
||||
bool __acrt_initialize_heap() {
|
||||
__acrt_heap = ::HeapCreate(0, 0, 0);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool __acrt_uninitialize_heap() {
|
||||
::HeapDestroy(__acrt_heap);
|
||||
__acrt_heap = nullptr;
|
||||
return true;
|
||||
}
|
||||
|
||||
intptr_t _get_heap_handle(void) {
|
||||
return reinterpret_cast<intptr_t>(__acrt_heap);
|
||||
}
|
||||
|
||||
} // extern "C"
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
// Copyright (c) 2012 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
// Workaround for crosbug:629593. Using AFDO on the tcmalloc files is
|
||||
// causing problems. The tcmalloc files depend on stack layouts and
|
||||
// AFDO can mess with them. Better not to use AFDO there. This is a
|
||||
// temporary hack. We will add a mechanism in the build system to
|
||||
// avoid using -fauto-profile for tcmalloc files.
|
||||
#if !defined(__clang__) && \
|
||||
(defined(OS_CHROMEOS) || (__GNUC__ > 5 && __GNUC__ < 7))
|
||||
// Note that this option only seems to be available in the chromeos GCC 4.9
|
||||
// toolchain, and stock GCC 5 upto 7.
|
||||
#pragma GCC optimize ("no-auto-profile")
|
||||
#endif
|
||||
|
||||
#if defined(TCMALLOC_FOR_DEBUGALLOCATION)
|
||||
#include "third_party/tcmalloc/chromium/src/debugallocation.cc"
|
||||
#else
|
||||
#include "third_party/tcmalloc/chromium/src/tcmalloc.cc"
|
||||
#endif
|
||||
|
|
@ -0,0 +1,115 @@
|
|||
// Copyright 2017 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/malloc_zone_functions_mac.h"
|
||||
|
||||
#include <atomic>
|
||||
|
||||
#include "base/synchronization/lock.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
MallocZoneFunctions g_malloc_zones[kMaxZoneCount];
|
||||
static_assert(std::is_pod<MallocZoneFunctions>::value,
|
||||
"MallocZoneFunctions must be POD");
|
||||
|
||||
void StoreZoneFunctions(const ChromeMallocZone* zone,
|
||||
MallocZoneFunctions* functions) {
|
||||
memset(functions, 0, sizeof(MallocZoneFunctions));
|
||||
functions->malloc = zone->malloc;
|
||||
functions->calloc = zone->calloc;
|
||||
functions->valloc = zone->valloc;
|
||||
functions->free = zone->free;
|
||||
functions->realloc = zone->realloc;
|
||||
functions->size = zone->size;
|
||||
CHECK(functions->malloc && functions->calloc && functions->valloc &&
|
||||
functions->free && functions->realloc && functions->size);
|
||||
|
||||
// These functions might be nullptr.
|
||||
functions->batch_malloc = zone->batch_malloc;
|
||||
functions->batch_free = zone->batch_free;
|
||||
|
||||
if (zone->version >= 5) {
|
||||
// Not all custom malloc zones have a memalign.
|
||||
functions->memalign = zone->memalign;
|
||||
}
|
||||
if (zone->version >= 6) {
|
||||
// This may be nullptr.
|
||||
functions->free_definite_size = zone->free_definite_size;
|
||||
}
|
||||
|
||||
functions->context = zone;
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
// All modifications to g_malloc_zones are gated behind this lock.
|
||||
// Dispatch to a malloc zone does not need to acquire this lock.
|
||||
base::Lock& GetLock() {
|
||||
static base::Lock* g_lock = new base::Lock;
|
||||
return *g_lock;
|
||||
}
|
||||
|
||||
void EnsureMallocZonesInitializedLocked() {
|
||||
GetLock().AssertAcquired();
|
||||
}
|
||||
|
||||
int g_zone_count = 0;
|
||||
|
||||
bool IsMallocZoneAlreadyStoredLocked(ChromeMallocZone* zone) {
|
||||
EnsureMallocZonesInitializedLocked();
|
||||
GetLock().AssertAcquired();
|
||||
for (int i = 0; i < g_zone_count; ++i) {
|
||||
if (g_malloc_zones[i].context == reinterpret_cast<void*>(zone))
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
bool StoreMallocZone(ChromeMallocZone* zone) {
|
||||
base::AutoLock l(GetLock());
|
||||
EnsureMallocZonesInitializedLocked();
|
||||
if (IsMallocZoneAlreadyStoredLocked(zone))
|
||||
return false;
|
||||
|
||||
if (g_zone_count == kMaxZoneCount)
|
||||
return false;
|
||||
|
||||
StoreZoneFunctions(zone, &g_malloc_zones[g_zone_count]);
|
||||
++g_zone_count;
|
||||
|
||||
// No other thread can possibly see these stores at this point. The code that
|
||||
// reads these values is triggered after this function returns. so we want to
|
||||
// guarantee that they are committed at this stage"
|
||||
std::atomic_thread_fence(std::memory_order_seq_cst);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool IsMallocZoneAlreadyStored(ChromeMallocZone* zone) {
|
||||
base::AutoLock l(GetLock());
|
||||
return IsMallocZoneAlreadyStoredLocked(zone);
|
||||
}
|
||||
|
||||
bool DoesMallocZoneNeedReplacing(ChromeMallocZone* zone,
|
||||
const MallocZoneFunctions* functions) {
|
||||
return IsMallocZoneAlreadyStored(zone) && zone->malloc != functions->malloc;
|
||||
}
|
||||
|
||||
int GetMallocZoneCountForTesting() {
|
||||
base::AutoLock l(GetLock());
|
||||
return g_zone_count;
|
||||
}
|
||||
|
||||
void ClearAllMallocZonesForTesting() {
|
||||
base::AutoLock l(GetLock());
|
||||
EnsureMallocZonesInitializedLocked();
|
||||
memset(g_malloc_zones, 0, kMaxZoneCount * sizeof(MallocZoneFunctions));
|
||||
g_zone_count = 0;
|
||||
}
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,103 @@
|
|||
// Copyright 2017 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_MALLOC_ZONE_FUNCTIONS_MAC_H_
|
||||
#define BASE_ALLOCATOR_MALLOC_ZONE_FUNCTIONS_MAC_H_
|
||||
|
||||
#include <malloc/malloc.h>
|
||||
#include <stddef.h>
|
||||
|
||||
#include "base/base_export.h"
|
||||
#include "base/logging.h"
|
||||
#include "third_party/apple_apsl/malloc.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
typedef void* (*malloc_type)(struct _malloc_zone_t* zone, size_t size);
|
||||
typedef void* (*calloc_type)(struct _malloc_zone_t* zone,
|
||||
size_t num_items,
|
||||
size_t size);
|
||||
typedef void* (*valloc_type)(struct _malloc_zone_t* zone, size_t size);
|
||||
typedef void (*free_type)(struct _malloc_zone_t* zone, void* ptr);
|
||||
typedef void* (*realloc_type)(struct _malloc_zone_t* zone,
|
||||
void* ptr,
|
||||
size_t size);
|
||||
typedef void* (*memalign_type)(struct _malloc_zone_t* zone,
|
||||
size_t alignment,
|
||||
size_t size);
|
||||
typedef unsigned (*batch_malloc_type)(struct _malloc_zone_t* zone,
|
||||
size_t size,
|
||||
void** results,
|
||||
unsigned num_requested);
|
||||
typedef void (*batch_free_type)(struct _malloc_zone_t* zone,
|
||||
void** to_be_freed,
|
||||
unsigned num_to_be_freed);
|
||||
typedef void (*free_definite_size_type)(struct _malloc_zone_t* zone,
|
||||
void* ptr,
|
||||
size_t size);
|
||||
typedef size_t (*size_fn_type)(struct _malloc_zone_t* zone, const void* ptr);
|
||||
|
||||
struct MallocZoneFunctions {
|
||||
malloc_type malloc;
|
||||
calloc_type calloc;
|
||||
valloc_type valloc;
|
||||
free_type free;
|
||||
realloc_type realloc;
|
||||
memalign_type memalign;
|
||||
batch_malloc_type batch_malloc;
|
||||
batch_free_type batch_free;
|
||||
free_definite_size_type free_definite_size;
|
||||
size_fn_type size;
|
||||
const ChromeMallocZone* context;
|
||||
};
|
||||
|
||||
BASE_EXPORT void StoreZoneFunctions(const ChromeMallocZone* zone,
|
||||
MallocZoneFunctions* functions);
|
||||
static constexpr int kMaxZoneCount = 30;
|
||||
BASE_EXPORT extern MallocZoneFunctions g_malloc_zones[kMaxZoneCount];
|
||||
|
||||
// The array g_malloc_zones stores all information about malloc zones before
|
||||
// they are shimmed. This information needs to be accessed during dispatch back
|
||||
// into the zone, and additional zones may be added later in the execution fo
|
||||
// the program, so the array needs to be both thread-safe and high-performance.
|
||||
//
|
||||
// We begin by creating an array of MallocZoneFunctions of fixed size. We will
|
||||
// never modify the container, which provides thread-safety to iterators. When
|
||||
// we want to add a MallocZoneFunctions to the container, we:
|
||||
// 1. Fill in all the fields.
|
||||
// 2. Update the total zone count.
|
||||
// 3. Insert a memory barrier.
|
||||
// 4. Insert our shim.
|
||||
//
|
||||
// Each MallocZoneFunctions is uniquely identified by |context|, which is a
|
||||
// pointer to the original malloc zone. When we wish to dispatch back to the
|
||||
// original malloc zones, we iterate through the array, looking for a matching
|
||||
// |context|.
|
||||
//
|
||||
// Most allocations go through the default allocator. We will ensure that the
|
||||
// default allocator is stored as the first MallocZoneFunctions.
|
||||
//
|
||||
// Returns whether the zone was successfully stored.
|
||||
BASE_EXPORT bool StoreMallocZone(ChromeMallocZone* zone);
|
||||
BASE_EXPORT bool IsMallocZoneAlreadyStored(ChromeMallocZone* zone);
|
||||
BASE_EXPORT bool DoesMallocZoneNeedReplacing(
|
||||
ChromeMallocZone* zone,
|
||||
const MallocZoneFunctions* functions);
|
||||
|
||||
BASE_EXPORT int GetMallocZoneCountForTesting();
|
||||
BASE_EXPORT void ClearAllMallocZonesForTesting();
|
||||
|
||||
inline MallocZoneFunctions& GetFunctionsForZone(void* zone) {
|
||||
for (unsigned int i = 0; i < kMaxZoneCount; ++i) {
|
||||
if (g_malloc_zones[i].context == zone)
|
||||
return g_malloc_zones[i];
|
||||
}
|
||||
IMMEDIATE_CRASH();
|
||||
}
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_MALLOC_ZONE_FUNCTIONS_MAC_H_
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
ajwong@chromium.org
|
||||
haraken@chromium.org
|
||||
palmer@chromium.org
|
||||
tsepez@chromium.org
|
||||
|
||||
# TEAM: platform-architecture-dev@chromium.org
|
||||
# Also: security-dev@chromium.org
|
||||
# COMPONENT: Blink>MemoryAllocator>Partition
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
# PartitionAlloc Design
|
||||
|
||||
This document describes PartitionAlloc at a high level. For documentation about
|
||||
its implementation, see the comments in `partition_alloc.h`.
|
||||
|
||||
[TOC]
|
||||
|
||||
## Overview
|
||||
|
||||
PartitionAlloc is a memory allocator optimized for security, low allocation
|
||||
latency (when called appropriately), and good space efficiency (when called
|
||||
appropriately). This document aims to help you understand how PartitionAlloc
|
||||
works so that you can use it effectively.
|
||||
|
||||
## Partitions And Buckets
|
||||
|
||||
A *partition* is a heap that contains certain object types, objects of certain
|
||||
sizes, or objects of a certain lifetime (as the caller prefers). Callers can
|
||||
create as many partitions as they need. Each partition is separate and protected
|
||||
from any other partitions.
|
||||
|
||||
Each partition holds multiple buckets. A *bucket* is a region in a partition
|
||||
that contains similar-sized objects.
|
||||
|
||||
PartitionAlloc aligns each object allocation with the closest bucket size. For
|
||||
example, if a partition has 3 buckets for 64 bytes, 256 bytes, and 1024 bytes,
|
||||
then PartitionAlloc will satisfy an allocation request for 128 bytes by rounding
|
||||
it up to 256 bytes and allocating from the second bucket.
|
||||
|
||||
The special allocator class `template <size_t N> class
|
||||
SizeSpecificPartitionAllocator` will satisfy allocations only of size
|
||||
`kMaxAllocation = N - kAllocationGranularity` or less, and contains buckets for
|
||||
all `n * kAllocationGranularity` (n = 1, 2, ..., `kMaxAllocation`). Attempts to
|
||||
allocate more than `kMaxAllocation` will fail.
|
||||
|
||||
## Performance
|
||||
|
||||
The current implementation is optimized for the main thread use-case. For
|
||||
example, PartitionAlloc doesn't have threaded caches.
|
||||
|
||||
PartitionAlloc is designed to be extremely fast in its fast paths. The fast
|
||||
paths of allocation and deallocation require just 2 (reasonably predictable)
|
||||
branches. The number of operations in the fast paths is minimal, leading to the
|
||||
possibility of inlining.
|
||||
|
||||
For an example of how to use partitions to get good performance and good safety,
|
||||
see Blink's usage, as described in `wtf/allocator/Allocator.md`.
|
||||
|
||||
Large allocations (> kGenericMaxBucketed == 960KB) are realized by direct
|
||||
memory mmapping. This size makes sense because 960KB = 0xF0000. The next larger
|
||||
bucket size is 1MB = 0x100000 which is greater than 1/2 the available space in
|
||||
a SuperPage meaning it would not be possible to pack even 2 sequential
|
||||
allocations in a SuperPage.
|
||||
|
||||
`PartitionRootGeneric::Alloc()` acquires a lock for thread safety. (The current
|
||||
implementation uses a spin lock on the assumption that thread contention will be
|
||||
rare in its callers. The original caller was Blink, where this is generally
|
||||
true. Spin locks also have the benefit of simplicity.)
|
||||
|
||||
Callers can get thread-unsafe performance using a
|
||||
`SizeSpecificPartitionAllocator` or otherwise using `PartitionAlloc` (instead of
|
||||
`PartitionRootGeneric::Alloc()`). Callers can also arrange for low contention,
|
||||
such as by using a dedicated partition for single-threaded, latency-critical
|
||||
allocations.
|
||||
|
||||
Because PartitionAlloc guarantees that address space regions used for one
|
||||
partition are never reused for other partitions, partitions can eat a large
|
||||
amount of virtual address space (even if not of actual memory).
|
||||
|
||||
Mixing various random objects in the same partition will generally lead to lower
|
||||
efficiency. For good performance, group similar objects into the same partition.
|
||||
|
||||
## Security
|
||||
|
||||
Security is one of the most important goals of PartitionAlloc.
|
||||
|
||||
PartitionAlloc guarantees that different partitions exist in different regions
|
||||
of the process' address space. When the caller has freed all objects contained
|
||||
in a page in a partition, PartitionAlloc returns the physical memory to the
|
||||
operating system, but continues to reserve the region of address space.
|
||||
PartitionAlloc will only reuse an address space region for the same partition.
|
||||
|
||||
PartitionAlloc also guarantees that:
|
||||
|
||||
* Linear overflows cannot corrupt into the partition. (There is a guard page at
|
||||
the beginning of each partition.)
|
||||
|
||||
* Linear overflows cannot corrupt out of the partition. (There is a guard page
|
||||
at the end of each partition.)
|
||||
|
||||
* Linear overflow or underflow cannot corrupt the allocation metadata.
|
||||
PartitionAlloc records metadata in a dedicated region out-of-line (not adjacent
|
||||
to objects).
|
||||
|
||||
* Objects of different sizes will likely be allocated in different buckets, and
|
||||
hence at different addresses. One page can contain only similar-sized objects.
|
||||
|
||||
* Dereference of a freelist pointer should fault.
|
||||
|
||||
* Partial pointer overwrite of freelist pointer should fault.
|
||||
|
||||
* Large allocations have guard pages at the beginning and end.
|
||||
|
|
@ -0,0 +1,68 @@
|
|||
// Copyright 2014 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/address_space_randomization.h"
|
||||
|
||||
#include "base/allocator/partition_allocator/page_allocator.h"
|
||||
#include "base/allocator/partition_allocator/random.h"
|
||||
#include "base/allocator/partition_allocator/spin_lock.h"
|
||||
#include "base/logging.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
#if defined(OS_WIN)
|
||||
#include <windows.h> // Must be in front of other Windows header files.
|
||||
|
||||
#include <VersionHelpers.h>
|
||||
#endif
|
||||
|
||||
namespace base {
|
||||
|
||||
void* GetRandomPageBase() {
|
||||
uintptr_t random = static_cast<uintptr_t>(RandomValue());
|
||||
|
||||
#if defined(ARCH_CPU_64_BITS)
|
||||
random <<= 32ULL;
|
||||
random |= static_cast<uintptr_t>(RandomValue());
|
||||
|
||||
// The kASLRMask and kASLROffset constants will be suitable for the
|
||||
// OS and build configuration.
|
||||
#if defined(OS_WIN) && !defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
// Windows >= 8.1 has the full 47 bits. Use them where available.
|
||||
static bool windows_81 = false;
|
||||
static bool windows_81_initialized = false;
|
||||
if (!windows_81_initialized) {
|
||||
windows_81 = IsWindows8Point1OrGreater();
|
||||
windows_81_initialized = true;
|
||||
}
|
||||
if (!windows_81) {
|
||||
random &= internal::kASLRMaskBefore8_10;
|
||||
} else {
|
||||
random &= internal::kASLRMask;
|
||||
}
|
||||
random += internal::kASLROffset;
|
||||
#else
|
||||
random &= internal::kASLRMask;
|
||||
random += internal::kASLROffset;
|
||||
#endif // defined(OS_WIN) && !defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
#else // defined(ARCH_CPU_32_BITS)
|
||||
#if defined(OS_WIN)
|
||||
// On win32 host systems the randomization plus huge alignment causes
|
||||
// excessive fragmentation. Plus most of these systems lack ASLR, so the
|
||||
// randomization isn't buying anything. In that case we just skip it.
|
||||
// TODO(palmer): Just dump the randomization when HE-ASLR is present.
|
||||
static BOOL is_wow64 = -1;
|
||||
if (is_wow64 == -1 && !IsWow64Process(GetCurrentProcess(), &is_wow64))
|
||||
is_wow64 = FALSE;
|
||||
if (!is_wow64)
|
||||
return nullptr;
|
||||
#endif // defined(OS_WIN)
|
||||
random &= internal::kASLRMask;
|
||||
random += internal::kASLROffset;
|
||||
#endif // defined(ARCH_CPU_32_BITS)
|
||||
|
||||
DCHECK_EQ(0ULL, (random & kPageAllocationGranularityOffsetMask));
|
||||
return reinterpret_cast<void*>(random);
|
||||
}
|
||||
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,201 @@
|
|||
// Copyright 2014 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_ADDRESS_SPACE_RANDOMIZATION_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_ADDRESS_SPACE_RANDOMIZATION_H_
|
||||
|
||||
#include "base/allocator/partition_allocator/page_allocator.h"
|
||||
#include "base/base_export.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
// Calculates a random preferred mapping address. In calculating an address, we
|
||||
// balance good ASLR against not fragmenting the address space too badly.
|
||||
BASE_EXPORT void* GetRandomPageBase();
|
||||
|
||||
namespace internal {
|
||||
|
||||
constexpr uintptr_t AslrAddress(uintptr_t mask) {
|
||||
return mask & kPageAllocationGranularityBaseMask;
|
||||
}
|
||||
constexpr uintptr_t AslrMask(uintptr_t bits) {
|
||||
return AslrAddress((1ULL << bits) - 1ULL);
|
||||
}
|
||||
|
||||
// Turn off formatting, because the thicket of nested ifdefs below is
|
||||
// incomprehensible without indentation. It is also incomprehensible with
|
||||
// indentation, but the only other option is a combinatorial explosion of
|
||||
// *_{win,linux,mac,foo}_{32,64}.h files.
|
||||
//
|
||||
// clang-format off
|
||||
|
||||
#if defined(ARCH_CPU_64_BITS)
|
||||
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
|
||||
// We shouldn't allocate system pages at all for sanitizer builds. However,
|
||||
// we do, and if random hint addresses interfere with address ranges
|
||||
// hard-coded in those tools, bad things happen. This address range is
|
||||
// copied from TSAN source but works with all tools. See
|
||||
// https://crbug.com/539863.
|
||||
constexpr uintptr_t kASLRMask = AslrAddress(0x007fffffffffULL);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0x7e8000000000ULL);
|
||||
|
||||
#elif defined(OS_WIN)
|
||||
|
||||
// Windows 8.10 and newer support the full 48 bit address range. Older
|
||||
// versions of Windows only support 44 bits. Since kASLROffset is non-zero
|
||||
// and may cause a carry, use 47 and 43 bit masks. See
|
||||
// http://www.alex-ionescu.com/?p=246
|
||||
constexpr uintptr_t kASLRMask = AslrMask(47);
|
||||
constexpr uintptr_t kASLRMaskBefore8_10 = AslrMask(43);
|
||||
// Try not to map pages into the range where Windows loads DLLs by default.
|
||||
constexpr uintptr_t kASLROffset = 0x80000000ULL;
|
||||
|
||||
#elif defined(OS_MACOSX)
|
||||
|
||||
// macOS as of 10.12.5 does not clean up entries in page map levels 3/4
|
||||
// [PDP/PML4] created from mmap or mach_vm_allocate, even after the region
|
||||
// is destroyed. Using a virtual address space that is too large causes a
|
||||
// leak of about 1 wired [can never be paged out] page per call to mmap. The
|
||||
// page is only reclaimed when the process is killed. Confine the hint to a
|
||||
// 39-bit section of the virtual address space.
|
||||
//
|
||||
// This implementation adapted from
|
||||
// https://chromium-review.googlesource.com/c/v8/v8/+/557958. The difference
|
||||
// is that here we clamp to 39 bits, not 32.
|
||||
//
|
||||
// TODO(crbug.com/738925): Remove this limitation if/when the macOS behavior
|
||||
// changes.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(38);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0x1000000000ULL);
|
||||
|
||||
#elif defined(OS_POSIX) || defined(OS_FUCHSIA)
|
||||
|
||||
#if defined(ARCH_CPU_X86_64)
|
||||
|
||||
// Linux (and macOS) support the full 47-bit user space of x64 processors.
|
||||
// Use only 46 to allow the kernel a chance to fulfill the request.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(46);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0);
|
||||
|
||||
#elif defined(ARCH_CPU_ARM64)
|
||||
|
||||
#if defined(OS_ANDROID)
|
||||
|
||||
// Restrict the address range on Android to avoid a large performance
|
||||
// regression in single-process WebViews. See https://crbug.com/837640.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(30);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0x20000000ULL);
|
||||
|
||||
#else
|
||||
|
||||
// ARM64 on Linux has 39-bit user space. Use 38 bits since kASLROffset
|
||||
// could cause a carry.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(38);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0x1000000000ULL);
|
||||
|
||||
#endif
|
||||
|
||||
#elif defined(ARCH_CPU_PPC64)
|
||||
|
||||
#if defined(OS_AIX)
|
||||
|
||||
// AIX has 64 bits of virtual addressing, but we limit the address range
|
||||
// to (a) minimize segment lookaside buffer (SLB) misses; and (b) use
|
||||
// extra address space to isolate the mmap regions.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(30);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0x400000000000ULL);
|
||||
|
||||
#elif defined(ARCH_CPU_BIG_ENDIAN)
|
||||
|
||||
// Big-endian Linux PPC has 44 bits of virtual addressing. Use 42.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(42);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0);
|
||||
|
||||
#else // !defined(OS_AIX) && !defined(ARCH_CPU_BIG_ENDIAN)
|
||||
|
||||
// Little-endian Linux PPC has 48 bits of virtual addressing. Use 46.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(46);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0);
|
||||
|
||||
#endif // !defined(OS_AIX) && !defined(ARCH_CPU_BIG_ENDIAN)
|
||||
|
||||
#elif defined(ARCH_CPU_S390X)
|
||||
|
||||
// Linux on Z uses bits 22 - 32 for Region Indexing, which translates to
|
||||
// 42 bits of virtual addressing. Truncate to 40 bits to allow kernel a
|
||||
// chance to fulfill the request.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(40);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0);
|
||||
|
||||
#elif defined(ARCH_CPU_S390)
|
||||
|
||||
// 31 bits of virtual addressing. Truncate to 29 bits to allow the kernel
|
||||
// a chance to fulfill the request.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(29);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0);
|
||||
|
||||
#else // !defined(ARCH_CPU_X86_64) && !defined(ARCH_CPU_PPC64) &&
|
||||
// !defined(ARCH_CPU_S390X) && !defined(ARCH_CPU_S390)
|
||||
|
||||
// For all other POSIX variants, use 30 bits.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(30);
|
||||
|
||||
#if defined(OS_SOLARIS)
|
||||
|
||||
// For our Solaris/illumos mmap hint, we pick a random address in the
|
||||
// bottom half of the top half of the address space (that is, the third
|
||||
// quarter). Because we do not MAP_FIXED, this will be treated only as a
|
||||
// hint -- the system will not fail to mmap because something else
|
||||
// happens to already be mapped at our random address. We deliberately
|
||||
// set the hint high enough to get well above the system's break (that
|
||||
// is, the heap); Solaris and illumos will try the hint and if that
|
||||
// fails allocate as if there were no hint at all. The high hint
|
||||
// prevents the break from getting hemmed in at low values, ceding half
|
||||
// of the address space to the system heap.
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0x80000000ULL);
|
||||
|
||||
#elif defined(OS_AIX)
|
||||
|
||||
// The range 0x30000000 - 0xD0000000 is available on AIX; choose the
|
||||
// upper range.
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0x90000000ULL);
|
||||
|
||||
#else // !defined(OS_SOLARIS) && !defined(OS_AIX)
|
||||
|
||||
// The range 0x20000000 - 0x60000000 is relatively unpopulated across a
|
||||
// variety of ASLR modes (PAE kernel, NX compat mode, etc) and on macOS
|
||||
// 10.6 and 10.7.
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0x20000000ULL);
|
||||
|
||||
#endif // !defined(OS_SOLARIS) && !defined(OS_AIX)
|
||||
|
||||
#endif // !defined(ARCH_CPU_X86_64) && !defined(ARCH_CPU_PPC64) &&
|
||||
// !defined(ARCH_CPU_S390X) && !defined(ARCH_CPU_S390)
|
||||
|
||||
#endif // defined(OS_POSIX)
|
||||
|
||||
#elif defined(ARCH_CPU_32_BITS)
|
||||
|
||||
// This is a good range on 32-bit Windows and Android (the only platforms on
|
||||
// which we support 32-bitness). Allocates in the 0.5 - 1.5 GiB region. There
|
||||
// is no issue with carries here.
|
||||
constexpr uintptr_t kASLRMask = AslrMask(30);
|
||||
constexpr uintptr_t kASLROffset = AslrAddress(0x20000000ULL);
|
||||
|
||||
#else
|
||||
|
||||
#error Please tell us about your exotic hardware! Sounds interesting.
|
||||
|
||||
#endif // defined(ARCH_CPU_32_BITS)
|
||||
|
||||
// clang-format on
|
||||
|
||||
} // namespace internal
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_ADDRESS_SPACE_RANDOMIZATION_H_
|
||||
|
|
@ -0,0 +1,127 @@
|
|||
// Copyright 2019 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/memory_reclaimer.h"
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_alloc.h"
|
||||
#include "base/bind.h"
|
||||
#include "base/location.h"
|
||||
#include "base/metrics/histogram_functions.h"
|
||||
#include "base/timer/elapsed_timer.h"
|
||||
#include "base/trace_event/trace_event.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
constexpr TimeDelta PartitionAllocMemoryReclaimer::kStatsRecordingTimeDelta;
|
||||
|
||||
// static
|
||||
PartitionAllocMemoryReclaimer* PartitionAllocMemoryReclaimer::Instance() {
|
||||
static NoDestructor<PartitionAllocMemoryReclaimer> instance;
|
||||
return instance.get();
|
||||
}
|
||||
|
||||
void PartitionAllocMemoryReclaimer::RegisterPartition(
|
||||
internal::PartitionRootBase* partition) {
|
||||
AutoLock lock(lock_);
|
||||
DCHECK(partition);
|
||||
auto it_and_whether_inserted = partitions_.insert(partition);
|
||||
DCHECK(it_and_whether_inserted.second);
|
||||
}
|
||||
|
||||
void PartitionAllocMemoryReclaimer::UnregisterPartition(
|
||||
internal::PartitionRootBase* partition) {
|
||||
AutoLock lock(lock_);
|
||||
DCHECK(partition);
|
||||
size_t erased_count = partitions_.erase(partition);
|
||||
DCHECK_EQ(1u, erased_count);
|
||||
}
|
||||
|
||||
void PartitionAllocMemoryReclaimer::Start(
|
||||
scoped_refptr<SequencedTaskRunner> task_runner) {
|
||||
DCHECK(!timer_);
|
||||
DCHECK(task_runner);
|
||||
|
||||
{
|
||||
AutoLock lock(lock_);
|
||||
DCHECK(!partitions_.empty());
|
||||
}
|
||||
|
||||
// This does not need to run on the main thread, however there are a few
|
||||
// reasons to do it there:
|
||||
// - Most of PartitionAlloc's usage is on the main thread, hence PA's metadata
|
||||
// is more likely in cache when executing on the main thread.
|
||||
// - Memory reclaim takes the partition lock for each partition. As a
|
||||
// consequence, while reclaim is running, the main thread is unlikely to be
|
||||
// able to make progress, as it would be waiting on the lock.
|
||||
// - Finally, this runs in idle time only, so there should be no visible
|
||||
// impact.
|
||||
//
|
||||
// From local testing, time to reclaim is 100us-1ms, and reclaiming every few
|
||||
// seconds is useful. Since this is meant to run during idle time only, it is
|
||||
// a reasonable starting point balancing effectivenes vs cost. See
|
||||
// crbug.com/942512 for details and experimental results.
|
||||
constexpr TimeDelta kInterval = TimeDelta::FromSeconds(4);
|
||||
|
||||
timer_ = std::make_unique<RepeatingTimer>();
|
||||
timer_->SetTaskRunner(task_runner);
|
||||
// Here and below, |Unretained(this)| is fine as |this| lives forever, as a
|
||||
// singleton.
|
||||
timer_->Start(
|
||||
FROM_HERE, kInterval,
|
||||
BindRepeating(&PartitionAllocMemoryReclaimer::Reclaim, Unretained(this)));
|
||||
|
||||
task_runner->PostDelayedTask(
|
||||
FROM_HERE,
|
||||
BindOnce(&PartitionAllocMemoryReclaimer::RecordStatistics,
|
||||
Unretained(this)),
|
||||
kStatsRecordingTimeDelta);
|
||||
}
|
||||
|
||||
PartitionAllocMemoryReclaimer::PartitionAllocMemoryReclaimer() = default;
|
||||
PartitionAllocMemoryReclaimer::~PartitionAllocMemoryReclaimer() = default;
|
||||
|
||||
void PartitionAllocMemoryReclaimer::Reclaim() {
|
||||
TRACE_EVENT0("base", "PartitionAllocMemoryReclaimer::Reclaim()");
|
||||
// Reclaim will almost always call into the kernel, so tail latency of this
|
||||
// task would likely be affected by descheduling.
|
||||
//
|
||||
// On Linux (and Android) at least, ThreadTicks also includes kernel time, so
|
||||
// this is a good measure of the true cost of decommit.
|
||||
ElapsedThreadTimer timer;
|
||||
constexpr int kFlags =
|
||||
PartitionPurgeDecommitEmptyPages | PartitionPurgeDiscardUnusedSystemPages;
|
||||
|
||||
{
|
||||
AutoLock lock(lock_); // Has to protect from concurrent (Un)Register calls.
|
||||
for (auto* partition : partitions_)
|
||||
partition->PurgeMemory(kFlags);
|
||||
}
|
||||
|
||||
has_called_reclaim_ = true;
|
||||
if (timer.is_supported())
|
||||
total_reclaim_thread_time_ += timer.Elapsed();
|
||||
}
|
||||
|
||||
void PartitionAllocMemoryReclaimer::RecordStatistics() {
|
||||
if (!ElapsedThreadTimer().is_supported())
|
||||
return;
|
||||
if (!has_called_reclaim_)
|
||||
return;
|
||||
|
||||
UmaHistogramTimes("Memory.PartitionAlloc.MainThreadTime.5min",
|
||||
total_reclaim_thread_time_);
|
||||
has_called_reclaim_ = false;
|
||||
total_reclaim_thread_time_ = TimeDelta();
|
||||
}
|
||||
|
||||
void PartitionAllocMemoryReclaimer::ResetForTesting() {
|
||||
AutoLock lock(lock_);
|
||||
|
||||
has_called_reclaim_ = false;
|
||||
total_reclaim_thread_time_ = TimeDelta();
|
||||
timer_ = nullptr;
|
||||
partitions_.clear();
|
||||
}
|
||||
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,79 @@
|
|||
// Copyright 2019 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_MEMORY_RECLAIMER_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_MEMORY_RECLAIMER_H_
|
||||
|
||||
#include <memory>
|
||||
#include <set>
|
||||
|
||||
#include "base/bind.h"
|
||||
#include "base/callback.h"
|
||||
#include "base/location.h"
|
||||
#include "base/no_destructor.h"
|
||||
#include "base/single_thread_task_runner.h"
|
||||
#include "base/thread_annotations.h"
|
||||
#include "base/time/time.h"
|
||||
#include "base/timer/elapsed_timer.h"
|
||||
#include "base/timer/timer.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
namespace internal {
|
||||
|
||||
struct PartitionRootBase;
|
||||
|
||||
} // namespace internal
|
||||
|
||||
// Posts and handles memory reclaim tasks for PartitionAlloc.
|
||||
//
|
||||
// Thread safety: |RegisterPartition()| and |UnregisterPartition()| can be
|
||||
// called from any thread, concurrently with reclaim. Reclaim itself runs in the
|
||||
// context of the provided |SequencedTaskRunner|, meaning that the caller must
|
||||
// take care of this runner being compatible with the various partitions.
|
||||
//
|
||||
// Singleton as this runs as long as the process is alive, and
|
||||
// having multiple instances would be wasteful.
|
||||
class BASE_EXPORT PartitionAllocMemoryReclaimer {
|
||||
public:
|
||||
static PartitionAllocMemoryReclaimer* Instance();
|
||||
|
||||
// Internal. Do not use.
|
||||
// Registers a partition to be tracked by the reclaimer.
|
||||
void RegisterPartition(internal::PartitionRootBase* partition);
|
||||
// Internal. Do not use.
|
||||
// Unregisters a partition to be tracked by the reclaimer.
|
||||
void UnregisterPartition(internal::PartitionRootBase* partition);
|
||||
// Starts the periodic reclaim. Should be called once.
|
||||
void Start(scoped_refptr<SequencedTaskRunner> task_runner);
|
||||
// Triggers an explicit reclaim now.
|
||||
void Reclaim();
|
||||
|
||||
static constexpr TimeDelta kStatsRecordingTimeDelta =
|
||||
TimeDelta::FromMinutes(5);
|
||||
|
||||
private:
|
||||
PartitionAllocMemoryReclaimer();
|
||||
~PartitionAllocMemoryReclaimer();
|
||||
void ReclaimAndReschedule();
|
||||
void RecordStatistics();
|
||||
void ResetForTesting();
|
||||
|
||||
// Total time spent in |Reclaim()|.
|
||||
bool has_called_reclaim_ = false;
|
||||
TimeDelta total_reclaim_thread_time_;
|
||||
// Schedules periodic |Reclaim()|.
|
||||
std::unique_ptr<RepeatingTimer> timer_;
|
||||
|
||||
Lock lock_;
|
||||
std::set<internal::PartitionRootBase*> partitions_ GUARDED_BY(lock_);
|
||||
|
||||
friend class NoDestructor<PartitionAllocMemoryReclaimer>;
|
||||
friend class PartitionAllocMemoryReclaimerTest;
|
||||
DISALLOW_COPY_AND_ASSIGN(PartitionAllocMemoryReclaimer);
|
||||
};
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_MEMORY_RECLAIMER_H_
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
// Copyright (c) 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_H_
|
||||
|
||||
#include "base/allocator/partition_allocator/oom_callback.h"
|
||||
#include "base/logging.h"
|
||||
#include "base/process/memory.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
#if defined(OS_WIN)
|
||||
#include <windows.h>
|
||||
#endif
|
||||
|
||||
namespace {
|
||||
// The crash is generated in a NOINLINE function so that we can classify the
|
||||
// crash as an OOM solely by analyzing the stack trace.
|
||||
NOINLINE void OnNoMemory(size_t size) {
|
||||
base::internal::RunPartitionAllocOomCallback();
|
||||
base::internal::OnNoMemoryInternal(size);
|
||||
IMMEDIATE_CRASH();
|
||||
}
|
||||
} // namespace
|
||||
|
||||
// OOM_CRASH(size) - Specialization of IMMEDIATE_CRASH which will raise a custom
|
||||
// exception on Windows to signal this is OOM and not a normal assert.
|
||||
// OOM_CRASH(size) is called by users of PageAllocator (including
|
||||
// PartitionAlloc) to signify an allocation failure from the platform.
|
||||
#define OOM_CRASH(size) \
|
||||
do { \
|
||||
OnNoMemory(size); \
|
||||
} while (0)
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_H_
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/oom_callback.h"
|
||||
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
namespace {
|
||||
PartitionAllocOomCallback g_oom_callback;
|
||||
} // namespace
|
||||
|
||||
void SetPartitionAllocOomCallback(PartitionAllocOomCallback callback) {
|
||||
DCHECK(!g_oom_callback);
|
||||
g_oom_callback = callback;
|
||||
}
|
||||
|
||||
namespace internal {
|
||||
void RunPartitionAllocOomCallback() {
|
||||
if (g_oom_callback)
|
||||
g_oom_callback();
|
||||
}
|
||||
} // namespace internal
|
||||
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_CALLBACK_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_CALLBACK_H_
|
||||
|
||||
#include "base/base_export.h"
|
||||
|
||||
namespace base {
|
||||
typedef void (*PartitionAllocOomCallback)();
|
||||
// Registers a callback to be invoked during an OOM_CRASH(). OOM_CRASH is
|
||||
// invoked by users of PageAllocator (including PartitionAlloc) to signify an
|
||||
// allocation failure from the platform.
|
||||
BASE_EXPORT void SetPartitionAllocOomCallback(
|
||||
PartitionAllocOomCallback callback);
|
||||
|
||||
namespace internal {
|
||||
BASE_EXPORT void RunPartitionAllocOomCallback();
|
||||
} // namespace internal
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_OOM_CALLBACK_H_
|
||||
|
|
@ -0,0 +1,263 @@
|
|||
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/page_allocator.h"
|
||||
|
||||
#include <limits.h>
|
||||
|
||||
#include <atomic>
|
||||
|
||||
#include "base/allocator/partition_allocator/address_space_randomization.h"
|
||||
#include "base/allocator/partition_allocator/page_allocator_internal.h"
|
||||
#include "base/allocator/partition_allocator/spin_lock.h"
|
||||
#include "base/bits.h"
|
||||
#include "base/logging.h"
|
||||
#include "base/no_destructor.h"
|
||||
#include "base/numerics/checked_math.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
#if defined(OS_WIN)
|
||||
#include <windows.h>
|
||||
#endif
|
||||
|
||||
#if defined(OS_WIN)
|
||||
#include "base/allocator/partition_allocator/page_allocator_internals_win.h"
|
||||
#elif defined(OS_POSIX)
|
||||
#include "base/allocator/partition_allocator/page_allocator_internals_posix.h"
|
||||
#elif defined(OS_FUCHSIA)
|
||||
#include "base/allocator/partition_allocator/page_allocator_internals_fuchsia.h"
|
||||
#else
|
||||
#error Platform not supported.
|
||||
#endif
|
||||
|
||||
namespace base {
|
||||
|
||||
namespace {
|
||||
|
||||
// We may reserve/release address space on different threads.
|
||||
subtle::SpinLock& GetReserveLock() {
|
||||
static NoDestructor<subtle::SpinLock> s_reserveLock;
|
||||
return *s_reserveLock;
|
||||
}
|
||||
|
||||
// We only support a single block of reserved address space.
|
||||
void* s_reservation_address = nullptr;
|
||||
size_t s_reservation_size = 0;
|
||||
|
||||
void* AllocPagesIncludingReserved(void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
PageTag page_tag,
|
||||
bool commit) {
|
||||
void* ret =
|
||||
SystemAllocPages(address, length, accessibility, page_tag, commit);
|
||||
if (ret == nullptr) {
|
||||
const bool cant_alloc_length = kHintIsAdvisory || address == nullptr;
|
||||
if (cant_alloc_length) {
|
||||
// The system cannot allocate |length| bytes. Release any reserved address
|
||||
// space and try once more.
|
||||
ReleaseReservation();
|
||||
ret = SystemAllocPages(address, length, accessibility, page_tag, commit);
|
||||
}
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
// Trims |base| to given |trim_length| and |alignment|.
|
||||
//
|
||||
// On failure, on Windows, this function returns nullptr and frees |base|.
|
||||
void* TrimMapping(void* base,
|
||||
size_t base_length,
|
||||
size_t trim_length,
|
||||
uintptr_t alignment,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
bool commit) {
|
||||
size_t pre_slack = reinterpret_cast<uintptr_t>(base) & (alignment - 1);
|
||||
if (pre_slack) {
|
||||
pre_slack = alignment - pre_slack;
|
||||
}
|
||||
size_t post_slack = base_length - pre_slack - trim_length;
|
||||
DCHECK(base_length >= trim_length || pre_slack || post_slack);
|
||||
DCHECK(pre_slack < base_length);
|
||||
DCHECK(post_slack < base_length);
|
||||
return TrimMappingInternal(base, base_length, trim_length, accessibility,
|
||||
commit, pre_slack, post_slack);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void* SystemAllocPages(void* hint,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
PageTag page_tag,
|
||||
bool commit) {
|
||||
DCHECK(!(length & kPageAllocationGranularityOffsetMask));
|
||||
DCHECK(!(reinterpret_cast<uintptr_t>(hint) &
|
||||
kPageAllocationGranularityOffsetMask));
|
||||
DCHECK(commit || accessibility == PageInaccessible);
|
||||
return SystemAllocPagesInternal(hint, length, accessibility, page_tag,
|
||||
commit);
|
||||
}
|
||||
|
||||
void* AllocPages(void* address,
|
||||
size_t length,
|
||||
size_t align,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
PageTag page_tag,
|
||||
bool commit) {
|
||||
DCHECK(length >= kPageAllocationGranularity);
|
||||
DCHECK(!(length & kPageAllocationGranularityOffsetMask));
|
||||
DCHECK(align >= kPageAllocationGranularity);
|
||||
// Alignment must be power of 2 for masking math to work.
|
||||
DCHECK(base::bits::IsPowerOfTwo(align));
|
||||
DCHECK(!(reinterpret_cast<uintptr_t>(address) &
|
||||
kPageAllocationGranularityOffsetMask));
|
||||
uintptr_t align_offset_mask = align - 1;
|
||||
uintptr_t align_base_mask = ~align_offset_mask;
|
||||
DCHECK(!(reinterpret_cast<uintptr_t>(address) & align_offset_mask));
|
||||
|
||||
// If the client passed null as the address, choose a good one.
|
||||
if (address == nullptr) {
|
||||
address = GetRandomPageBase();
|
||||
address = reinterpret_cast<void*>(reinterpret_cast<uintptr_t>(address) &
|
||||
align_base_mask);
|
||||
}
|
||||
|
||||
// First try to force an exact-size, aligned allocation from our random base.
|
||||
#if defined(ARCH_CPU_32_BITS)
|
||||
// On 32 bit systems, first try one random aligned address, and then try an
|
||||
// aligned address derived from the value of |ret|.
|
||||
constexpr int kExactSizeTries = 2;
|
||||
#else
|
||||
// On 64 bit systems, try 3 random aligned addresses.
|
||||
constexpr int kExactSizeTries = 3;
|
||||
#endif
|
||||
|
||||
for (int i = 0; i < kExactSizeTries; ++i) {
|
||||
void* ret = AllocPagesIncludingReserved(address, length, accessibility,
|
||||
page_tag, commit);
|
||||
if (ret != nullptr) {
|
||||
// If the alignment is to our liking, we're done.
|
||||
if (!(reinterpret_cast<uintptr_t>(ret) & align_offset_mask))
|
||||
return ret;
|
||||
// Free the memory and try again.
|
||||
FreePages(ret, length);
|
||||
} else {
|
||||
// |ret| is null; if this try was unhinted, we're OOM.
|
||||
if (kHintIsAdvisory || address == nullptr)
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
#if defined(ARCH_CPU_32_BITS)
|
||||
// For small address spaces, try the first aligned address >= |ret|. Note
|
||||
// |ret| may be null, in which case |address| becomes null.
|
||||
address = reinterpret_cast<void*>(
|
||||
(reinterpret_cast<uintptr_t>(ret) + align_offset_mask) &
|
||||
align_base_mask);
|
||||
#else // defined(ARCH_CPU_64_BITS)
|
||||
// Keep trying random addresses on systems that have a large address space.
|
||||
address = GetRandomPageBase();
|
||||
address = reinterpret_cast<void*>(reinterpret_cast<uintptr_t>(address) &
|
||||
align_base_mask);
|
||||
#endif
|
||||
}
|
||||
|
||||
// Make a larger allocation so we can force alignment.
|
||||
size_t try_length = length + (align - kPageAllocationGranularity);
|
||||
CHECK(try_length >= length);
|
||||
void* ret;
|
||||
|
||||
do {
|
||||
// Continue randomizing only on POSIX.
|
||||
address = kHintIsAdvisory ? GetRandomPageBase() : nullptr;
|
||||
ret = AllocPagesIncludingReserved(address, try_length, accessibility,
|
||||
page_tag, commit);
|
||||
// The retries are for Windows, where a race can steal our mapping on
|
||||
// resize.
|
||||
} while (ret != nullptr &&
|
||||
(ret = TrimMapping(ret, try_length, length, align, accessibility,
|
||||
commit)) == nullptr);
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
void FreePages(void* address, size_t length) {
|
||||
DCHECK(!(reinterpret_cast<uintptr_t>(address) &
|
||||
kPageAllocationGranularityOffsetMask));
|
||||
DCHECK(!(length & kPageAllocationGranularityOffsetMask));
|
||||
FreePagesInternal(address, length);
|
||||
}
|
||||
|
||||
bool TrySetSystemPagesAccess(void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
DCHECK(!(length & kSystemPageOffsetMask));
|
||||
return TrySetSystemPagesAccessInternal(address, length, accessibility);
|
||||
}
|
||||
|
||||
void SetSystemPagesAccess(void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
DCHECK(!(length & kSystemPageOffsetMask));
|
||||
SetSystemPagesAccessInternal(address, length, accessibility);
|
||||
}
|
||||
|
||||
void DecommitSystemPages(void* address, size_t length) {
|
||||
DCHECK_EQ(0UL, length & kSystemPageOffsetMask);
|
||||
DecommitSystemPagesInternal(address, length);
|
||||
}
|
||||
|
||||
bool RecommitSystemPages(void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
DCHECK_EQ(0UL, length & kSystemPageOffsetMask);
|
||||
DCHECK_NE(PageInaccessible, accessibility);
|
||||
return RecommitSystemPagesInternal(address, length, accessibility);
|
||||
}
|
||||
|
||||
void DiscardSystemPages(void* address, size_t length) {
|
||||
DCHECK_EQ(0UL, length & kSystemPageOffsetMask);
|
||||
DiscardSystemPagesInternal(address, length);
|
||||
}
|
||||
|
||||
bool ReserveAddressSpace(size_t size) {
|
||||
// To avoid deadlock, call only SystemAllocPages.
|
||||
subtle::SpinLock::Guard guard(GetReserveLock());
|
||||
if (s_reservation_address == nullptr) {
|
||||
void* mem = SystemAllocPages(nullptr, size, PageInaccessible,
|
||||
PageTag::kChromium, false);
|
||||
if (mem != nullptr) {
|
||||
// We guarantee this alignment when reserving address space.
|
||||
DCHECK(!(reinterpret_cast<uintptr_t>(mem) &
|
||||
kPageAllocationGranularityOffsetMask));
|
||||
s_reservation_address = mem;
|
||||
s_reservation_size = size;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
bool ReleaseReservation() {
|
||||
// To avoid deadlock, call only FreePages.
|
||||
subtle::SpinLock::Guard guard(GetReserveLock());
|
||||
if (!s_reservation_address)
|
||||
return false;
|
||||
|
||||
FreePages(s_reservation_address, s_reservation_size);
|
||||
s_reservation_address = nullptr;
|
||||
s_reservation_size = 0;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool HasReservationForTesting() {
|
||||
subtle::SpinLock::Guard guard(GetReserveLock());
|
||||
return s_reservation_address != nullptr;
|
||||
}
|
||||
|
||||
uint32_t GetAllocPageErrorCode() {
|
||||
return s_allocPageErrorCode;
|
||||
}
|
||||
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,198 @@
|
|||
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_H_
|
||||
|
||||
#include <stdint.h>
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
#include "base/allocator/partition_allocator/page_allocator_constants.h"
|
||||
#include "base/base_export.h"
|
||||
#include "base/compiler_specific.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
enum PageAccessibilityConfiguration {
|
||||
PageInaccessible,
|
||||
PageRead,
|
||||
PageReadWrite,
|
||||
PageReadExecute,
|
||||
// This flag is deprecated and will go away soon.
|
||||
// TODO(bbudge) Remove this as soon as V8 doesn't need RWX pages.
|
||||
PageReadWriteExecute,
|
||||
};
|
||||
|
||||
// macOS supports tagged memory regions, to help in debugging. On Android,
|
||||
// these tags are used to name anonymous mappings.
|
||||
enum class PageTag {
|
||||
kFirst = 240, // Minimum tag value.
|
||||
kBlinkGC = 252, // Blink GC pages.
|
||||
kPartitionAlloc = 253, // PartitionAlloc, no matter the partition.
|
||||
kChromium = 254, // Chromium page.
|
||||
kV8 = 255, // V8 heap pages.
|
||||
kLast = kV8 // Maximum tag value.
|
||||
};
|
||||
|
||||
// Allocate one or more pages.
|
||||
//
|
||||
// The requested |address| is just a hint; the actual address returned may
|
||||
// differ. The returned address will be aligned at least to |align| bytes.
|
||||
// |length| is in bytes, and must be a multiple of |kPageAllocationGranularity|.
|
||||
// |align| is in bytes, and must be a power-of-two multiple of
|
||||
// |kPageAllocationGranularity|.
|
||||
//
|
||||
// If |address| is null, then a suitable and randomized address will be chosen
|
||||
// automatically.
|
||||
//
|
||||
// |page_accessibility| controls the permission of the allocated pages.
|
||||
// |page_tag| is used on some platforms to identify the source of the
|
||||
// allocation. Use PageTag::kChromium as a catch-all category.
|
||||
//
|
||||
// This call will return null if the allocation cannot be satisfied.
|
||||
BASE_EXPORT void* AllocPages(void* address,
|
||||
size_t length,
|
||||
size_t align,
|
||||
PageAccessibilityConfiguration page_accessibility,
|
||||
PageTag tag,
|
||||
bool commit = true);
|
||||
|
||||
// Free one or more pages starting at |address| and continuing for |length|
|
||||
// bytes.
|
||||
//
|
||||
// |address| and |length| must match a previous call to |AllocPages|. Therefore,
|
||||
// |address| must be aligned to |kPageAllocationGranularity| bytes, and |length|
|
||||
// must be a multiple of |kPageAllocationGranularity|.
|
||||
BASE_EXPORT void FreePages(void* address, size_t length);
|
||||
|
||||
// Mark one or more system pages, starting at |address| with the given
|
||||
// |page_accessibility|. |length| must be a multiple of |kSystemPageSize| bytes.
|
||||
//
|
||||
// Returns true if the permission change succeeded. In most cases you must
|
||||
// |CHECK| the result.
|
||||
BASE_EXPORT WARN_UNUSED_RESULT bool TrySetSystemPagesAccess(
|
||||
void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration page_accessibility);
|
||||
|
||||
// Mark one or more system pages, starting at |address| with the given
|
||||
// |page_accessibility|. |length| must be a multiple of |kSystemPageSize| bytes.
|
||||
//
|
||||
// Performs a CHECK that the operation succeeds.
|
||||
BASE_EXPORT void SetSystemPagesAccess(
|
||||
void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration page_accessibility);
|
||||
|
||||
// Decommit one or more system pages starting at |address| and continuing for
|
||||
// |length| bytes. |length| must be a multiple of |kSystemPageSize|.
|
||||
//
|
||||
// Decommitted means that physical resources (RAM or swap) backing the allocated
|
||||
// virtual address range are released back to the system, but the address space
|
||||
// is still allocated to the process (possibly using up page table entries or
|
||||
// other accounting resources). Any access to a decommitted region of memory
|
||||
// is an error and will generate a fault.
|
||||
//
|
||||
// This operation is not atomic on all platforms.
|
||||
//
|
||||
// Note: "Committed memory" is a Windows Memory Subsystem concept that ensures
|
||||
// processes will not fault when touching a committed memory region. There is
|
||||
// no analogue in the POSIX memory API where virtual memory pages are
|
||||
// best-effort allocated resources on the first touch. To create a
|
||||
// platform-agnostic abstraction, this API simulates the Windows "decommit"
|
||||
// state by both discarding the region (allowing the OS to avoid swap
|
||||
// operations) and changing the page protections so accesses fault.
|
||||
//
|
||||
// TODO(ajwong): This currently does not change page protections on POSIX
|
||||
// systems due to a perf regression. Tracked at http://crbug.com/766882.
|
||||
BASE_EXPORT void DecommitSystemPages(void* address, size_t length);
|
||||
|
||||
// Recommit one or more system pages, starting at |address| and continuing for
|
||||
// |length| bytes with the given |page_accessibility|. |length| must be a
|
||||
// multiple of |kSystemPageSize|.
|
||||
//
|
||||
// Decommitted system pages must be recommitted with their original permissions
|
||||
// before they are used again.
|
||||
//
|
||||
// Returns true if the recommit change succeeded. In most cases you must |CHECK|
|
||||
// the result.
|
||||
BASE_EXPORT WARN_UNUSED_RESULT bool RecommitSystemPages(
|
||||
void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration page_accessibility);
|
||||
|
||||
// Discard one or more system pages starting at |address| and continuing for
|
||||
// |length| bytes. |length| must be a multiple of |kSystemPageSize|.
|
||||
//
|
||||
// Discarding is a hint to the system that the page is no longer required. The
|
||||
// hint may:
|
||||
// - Do nothing.
|
||||
// - Discard the page immediately, freeing up physical pages.
|
||||
// - Discard the page at some time in the future in response to memory
|
||||
// pressure.
|
||||
//
|
||||
// Only committed pages should be discarded. Discarding a page does not decommit
|
||||
// it, and it is valid to discard an already-discarded page. A read or write to
|
||||
// a discarded page will not fault.
|
||||
//
|
||||
// Reading from a discarded page may return the original page content, or a page
|
||||
// full of zeroes.
|
||||
//
|
||||
// Writing to a discarded page is the only guaranteed way to tell the system
|
||||
// that the page is required again. Once written to, the content of the page is
|
||||
// guaranteed stable once more. After being written to, the page content may be
|
||||
// based on the original page content, or a page of zeroes.
|
||||
BASE_EXPORT void DiscardSystemPages(void* address, size_t length);
|
||||
|
||||
// Rounds up |address| to the next multiple of |kSystemPageSize|. Returns
|
||||
// 0 for an |address| of 0.
|
||||
constexpr ALWAYS_INLINE uintptr_t RoundUpToSystemPage(uintptr_t address) {
|
||||
return (address + kSystemPageOffsetMask) & kSystemPageBaseMask;
|
||||
}
|
||||
|
||||
// Rounds down |address| to the previous multiple of |kSystemPageSize|. Returns
|
||||
// 0 for an |address| of 0.
|
||||
constexpr ALWAYS_INLINE uintptr_t RoundDownToSystemPage(uintptr_t address) {
|
||||
return address & kSystemPageBaseMask;
|
||||
}
|
||||
|
||||
// Rounds up |address| to the next multiple of |kPageAllocationGranularity|.
|
||||
// Returns 0 for an |address| of 0.
|
||||
constexpr ALWAYS_INLINE uintptr_t
|
||||
RoundUpToPageAllocationGranularity(uintptr_t address) {
|
||||
return (address + kPageAllocationGranularityOffsetMask) &
|
||||
kPageAllocationGranularityBaseMask;
|
||||
}
|
||||
|
||||
// Rounds down |address| to the previous multiple of
|
||||
// |kPageAllocationGranularity|. Returns 0 for an |address| of 0.
|
||||
constexpr ALWAYS_INLINE uintptr_t
|
||||
RoundDownToPageAllocationGranularity(uintptr_t address) {
|
||||
return address & kPageAllocationGranularityBaseMask;
|
||||
}
|
||||
|
||||
// Reserves (at least) |size| bytes of address space, aligned to
|
||||
// |kPageAllocationGranularity|. This can be called early on to make it more
|
||||
// likely that large allocations will succeed. Returns true if the reservation
|
||||
// succeeded, false if the reservation failed or a reservation was already made.
|
||||
BASE_EXPORT bool ReserveAddressSpace(size_t size);
|
||||
|
||||
// Releases any reserved address space. |AllocPages| calls this automatically on
|
||||
// an allocation failure. External allocators may also call this on failure.
|
||||
//
|
||||
// Returns true when an existing reservation was released.
|
||||
BASE_EXPORT bool ReleaseReservation();
|
||||
|
||||
// Returns true if there is currently an address space reservation.
|
||||
BASE_EXPORT bool HasReservationForTesting();
|
||||
|
||||
// Returns |errno| (POSIX) or the result of |GetLastError| (Windows) when |mmap|
|
||||
// (POSIX) or |VirtualAlloc| (Windows) fails.
|
||||
BASE_EXPORT uint32_t GetAllocPageErrorCode();
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_H_
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_CONSTANTS_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_CONSTANTS_H_
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
#if defined(OS_WIN) || defined(ARCH_CPU_PPC64)
|
||||
static constexpr size_t kPageAllocationGranularityShift = 16; // 64KB
|
||||
#elif defined(_MIPS_ARCH_LOONGSON)
|
||||
static constexpr size_t kPageAllocationGranularityShift = 14; // 16KB
|
||||
#else
|
||||
static constexpr size_t kPageAllocationGranularityShift = 12; // 4KB
|
||||
#endif
|
||||
static constexpr size_t kPageAllocationGranularity =
|
||||
1 << kPageAllocationGranularityShift;
|
||||
static constexpr size_t kPageAllocationGranularityOffsetMask =
|
||||
kPageAllocationGranularity - 1;
|
||||
static constexpr size_t kPageAllocationGranularityBaseMask =
|
||||
~kPageAllocationGranularityOffsetMask;
|
||||
|
||||
#if defined(_MIPS_ARCH_LOONGSON)
|
||||
static constexpr size_t kSystemPageSize = 16384;
|
||||
#elif defined(ARCH_CPU_PPC64)
|
||||
// Modern ppc64 systems support 4KB and 64KB page sizes.
|
||||
// Since 64KB is the de-facto standard on the platform
|
||||
// and binaries compiled for 64KB are likely to work on 4KB systems,
|
||||
// 64KB is a good choice here.
|
||||
static constexpr size_t kSystemPageSize = 65536;
|
||||
#else
|
||||
static constexpr size_t kSystemPageSize = 4096;
|
||||
#endif
|
||||
static constexpr size_t kSystemPageOffsetMask = kSystemPageSize - 1;
|
||||
static_assert((kSystemPageSize & (kSystemPageSize - 1)) == 0,
|
||||
"kSystemPageSize must be power of 2");
|
||||
static constexpr size_t kSystemPageBaseMask = ~kSystemPageOffsetMask;
|
||||
|
||||
static constexpr size_t kPageMetadataShift = 5; // 32 bytes per partition page.
|
||||
static constexpr size_t kPageMetadataSize = 1 << kPageMetadataShift;
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_CONSTANTS_H_
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNAL_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNAL_H_
|
||||
|
||||
namespace base {
|
||||
|
||||
void* SystemAllocPages(void* hint,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
PageTag page_tag,
|
||||
bool commit);
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNAL_H_
|
||||
|
|
@ -0,0 +1,202 @@
|
|||
// Copyright 2019 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
//
|
||||
// This file implements memory allocation primitives for PageAllocator using
|
||||
// Fuchsia's VMOs (Virtual Memory Objects). VMO API is documented in
|
||||
// https://fuchsia.dev/fuchsia-src/zircon/objects/vm_object . A VMO is a kernel
|
||||
// object that corresponds to a set of memory pages. VMO pages may be mapped
|
||||
// to an address space. The code below creates VMOs for each memory allocations
|
||||
// and maps them to the default address space of the current process.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_FUCHSIA_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_FUCHSIA_H_
|
||||
|
||||
#include <lib/zx/vmar.h>
|
||||
#include <lib/zx/vmo.h>
|
||||
|
||||
#include "base/allocator/partition_allocator/page_allocator.h"
|
||||
#include "base/fuchsia/fuchsia_logging.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
namespace {
|
||||
|
||||
// Returns VMO name for a PageTag.
|
||||
const char* PageTagToName(PageTag tag) {
|
||||
switch (tag) {
|
||||
case PageTag::kBlinkGC:
|
||||
return "cr_blink_gc";
|
||||
case PageTag::kPartitionAlloc:
|
||||
return "cr_partition_alloc";
|
||||
case PageTag::kChromium:
|
||||
return "cr_chromium";
|
||||
case PageTag::kV8:
|
||||
return "cr_v8";
|
||||
default:
|
||||
DCHECK(false);
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
zx_vm_option_t PageAccessibilityToZxVmOptions(
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
switch (accessibility) {
|
||||
case PageRead:
|
||||
return ZX_VM_PERM_READ;
|
||||
case PageReadWrite:
|
||||
return ZX_VM_PERM_READ | ZX_VM_PERM_WRITE;
|
||||
case PageReadExecute:
|
||||
return ZX_VM_PERM_READ | ZX_VM_PERM_EXECUTE;
|
||||
case PageReadWriteExecute:
|
||||
return ZX_VM_PERM_READ | ZX_VM_PERM_WRITE | ZX_VM_PERM_EXECUTE;
|
||||
default:
|
||||
NOTREACHED();
|
||||
FALLTHROUGH;
|
||||
case PageInaccessible:
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// zx_vmar_map() will fail if the VMO cannot be mapped at |vmar_offset|, i.e.
|
||||
// |hint| is not advisory.
|
||||
constexpr bool kHintIsAdvisory = false;
|
||||
|
||||
std::atomic<int32_t> s_allocPageErrorCode{0};
|
||||
|
||||
void* SystemAllocPagesInternal(void* hint,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
PageTag page_tag,
|
||||
bool commit) {
|
||||
zx::vmo vmo;
|
||||
zx_status_t status = zx::vmo::create(length, 0, &vmo);
|
||||
if (status != ZX_OK) {
|
||||
ZX_DLOG(INFO, status) << "zx_vmo_create";
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
const char* vmo_name = PageTagToName(page_tag);
|
||||
status = vmo.set_property(ZX_PROP_NAME, vmo_name, strlen(vmo_name));
|
||||
|
||||
// VMO names are used only for debugging, so failure to set a name is not
|
||||
// fatal.
|
||||
ZX_DCHECK(status == ZX_OK, status);
|
||||
|
||||
if (page_tag == PageTag::kV8) {
|
||||
// V8 uses JIT. Call zx_vmo_replace_as_executable() to allow code execution
|
||||
// in the new VMO.
|
||||
status = vmo.replace_as_executable(zx::resource(), &vmo);
|
||||
if (status != ZX_OK) {
|
||||
ZX_DLOG(INFO, status) << "zx_vmo_replace_as_executable";
|
||||
return nullptr;
|
||||
}
|
||||
}
|
||||
|
||||
zx_vm_option_t options = PageAccessibilityToZxVmOptions(accessibility);
|
||||
|
||||
uint64_t vmar_offset = 0;
|
||||
if (hint) {
|
||||
vmar_offset = reinterpret_cast<uint64_t>(hint);
|
||||
options |= ZX_VM_SPECIFIC;
|
||||
}
|
||||
|
||||
uint64_t address;
|
||||
status =
|
||||
zx::vmar::root_self()->map(vmar_offset, vmo,
|
||||
/*vmo_offset=*/0, length, options, &address);
|
||||
if (status != ZX_OK) {
|
||||
// map() is expected to fail if |hint| is set to an already-in-use location.
|
||||
if (!hint) {
|
||||
ZX_DLOG(ERROR, status) << "zx_vmar_map";
|
||||
}
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
return reinterpret_cast<void*>(address);
|
||||
}
|
||||
|
||||
void* TrimMappingInternal(void* base,
|
||||
size_t base_length,
|
||||
size_t trim_length,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
bool commit,
|
||||
size_t pre_slack,
|
||||
size_t post_slack) {
|
||||
DCHECK_EQ(base_length, trim_length + pre_slack + post_slack);
|
||||
|
||||
uint64_t base_address = reinterpret_cast<uint64_t>(base);
|
||||
|
||||
// Unmap head if necessary.
|
||||
if (pre_slack) {
|
||||
zx_status_t status = zx::vmar::root_self()->unmap(base_address, pre_slack);
|
||||
ZX_CHECK(status == ZX_OK, status);
|
||||
}
|
||||
|
||||
// Unmap tail if necessary.
|
||||
if (post_slack) {
|
||||
zx_status_t status = zx::vmar::root_self()->unmap(
|
||||
base_address + pre_slack + trim_length, post_slack);
|
||||
ZX_CHECK(status == ZX_OK, status);
|
||||
}
|
||||
|
||||
return reinterpret_cast<void*>(base_address + pre_slack);
|
||||
}
|
||||
|
||||
bool TrySetSystemPagesAccessInternal(
|
||||
void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
zx_status_t status = zx::vmar::root_self()->protect(
|
||||
reinterpret_cast<uint64_t>(address), length,
|
||||
PageAccessibilityToZxVmOptions(accessibility));
|
||||
return status == ZX_OK;
|
||||
}
|
||||
|
||||
void SetSystemPagesAccessInternal(
|
||||
void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
zx_status_t status = zx::vmar::root_self()->protect(
|
||||
reinterpret_cast<uint64_t>(address), length,
|
||||
PageAccessibilityToZxVmOptions(accessibility));
|
||||
ZX_CHECK(status == ZX_OK, status);
|
||||
}
|
||||
|
||||
void FreePagesInternal(void* address, size_t length) {
|
||||
uint64_t address_int = reinterpret_cast<uint64_t>(address);
|
||||
zx_status_t status = zx::vmar::root_self()->unmap(address_int, length);
|
||||
ZX_CHECK(status == ZX_OK, status);
|
||||
}
|
||||
|
||||
void DiscardSystemPagesInternal(void* address, size_t length) {
|
||||
// TODO(https://crbug.com/1022062): Mark pages as discardable, rather than
|
||||
// forcibly de-committing them immediately, when Fuchsia supports it.
|
||||
uint64_t address_int = reinterpret_cast<uint64_t>(address);
|
||||
zx_status_t status = zx::vmar::root_self()->op_range(
|
||||
ZX_VMO_OP_DECOMMIT, address_int, length, nullptr, 0);
|
||||
ZX_CHECK(status == ZX_OK, status);
|
||||
}
|
||||
|
||||
void DecommitSystemPagesInternal(void* address, size_t length) {
|
||||
// TODO(https://crbug.com/1022062): Review whether this implementation is
|
||||
// still appropriate once DiscardSystemPagesInternal() migrates to a "lazy"
|
||||
// discardable API.
|
||||
DiscardSystemPagesInternal(address, length);
|
||||
|
||||
SetSystemPagesAccessInternal(address, length, PageInaccessible);
|
||||
}
|
||||
|
||||
bool RecommitSystemPagesInternal(void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
SetSystemPagesAccessInternal(address, length, accessibility);
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_FUCHSIA_H_
|
||||
|
|
@ -0,0 +1,244 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_POSIX_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_POSIX_H_
|
||||
|
||||
#include <errno.h>
|
||||
#include <sys/mman.h>
|
||||
|
||||
#include "base/logging.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
#if defined(OS_MACOSX)
|
||||
#include "base/mac/foundation_util.h"
|
||||
#include "base/mac/mac_util.h"
|
||||
#include "base/mac/scoped_cftyperef.h"
|
||||
|
||||
#include <Security/Security.h>
|
||||
#include <mach/mach.h>
|
||||
#endif
|
||||
#if defined(OS_ANDROID)
|
||||
#include <sys/prctl.h>
|
||||
#endif
|
||||
#if defined(OS_LINUX)
|
||||
#include <sys/resource.h>
|
||||
|
||||
#include <algorithm>
|
||||
#endif
|
||||
|
||||
#include "base/allocator/partition_allocator/page_allocator.h"
|
||||
|
||||
#ifndef MAP_ANONYMOUS
|
||||
#define MAP_ANONYMOUS MAP_ANON
|
||||
#endif
|
||||
|
||||
namespace base {
|
||||
|
||||
namespace {
|
||||
|
||||
#if defined(OS_ANDROID)
|
||||
const char* PageTagToName(PageTag tag) {
|
||||
// Important: All the names should be string literals. As per prctl.h in
|
||||
// //third_party/android_ndk the kernel keeps a pointer to the name instead
|
||||
// of copying it.
|
||||
//
|
||||
// Having the name in .rodata ensures that the pointer remains valid as
|
||||
// long as the mapping is alive.
|
||||
switch (tag) {
|
||||
case PageTag::kBlinkGC:
|
||||
return "blink_gc";
|
||||
case PageTag::kPartitionAlloc:
|
||||
return "partition_alloc";
|
||||
case PageTag::kChromium:
|
||||
return "chromium";
|
||||
case PageTag::kV8:
|
||||
return "v8";
|
||||
default:
|
||||
DCHECK(false);
|
||||
return "";
|
||||
}
|
||||
}
|
||||
#endif // defined(OS_ANDROID)
|
||||
|
||||
#if defined(OS_MACOSX)
|
||||
// Tests whether the version of macOS supports the MAP_JIT flag and if the
|
||||
// current process is signed with the allow-jit entitlement.
|
||||
bool UseMapJit() {
|
||||
if (!mac::IsAtLeastOS10_14())
|
||||
return false;
|
||||
|
||||
ScopedCFTypeRef<SecTaskRef> task(SecTaskCreateFromSelf(kCFAllocatorDefault));
|
||||
ScopedCFTypeRef<CFErrorRef> error;
|
||||
ScopedCFTypeRef<CFTypeRef> value(SecTaskCopyValueForEntitlement(
|
||||
task.get(), CFSTR("com.apple.security.cs.allow-jit"),
|
||||
error.InitializeInto()));
|
||||
if (error)
|
||||
return false;
|
||||
return mac::CFCast<CFBooleanRef>(value.get()) == kCFBooleanTrue;
|
||||
}
|
||||
#endif // defined(OS_MACOSX)
|
||||
|
||||
} // namespace
|
||||
|
||||
// |mmap| uses a nearby address if the hint address is blocked.
|
||||
constexpr bool kHintIsAdvisory = true;
|
||||
std::atomic<int32_t> s_allocPageErrorCode{0};
|
||||
|
||||
int GetAccessFlags(PageAccessibilityConfiguration accessibility) {
|
||||
switch (accessibility) {
|
||||
case PageRead:
|
||||
return PROT_READ;
|
||||
case PageReadWrite:
|
||||
return PROT_READ | PROT_WRITE;
|
||||
case PageReadExecute:
|
||||
return PROT_READ | PROT_EXEC;
|
||||
case PageReadWriteExecute:
|
||||
return PROT_READ | PROT_WRITE | PROT_EXEC;
|
||||
default:
|
||||
NOTREACHED();
|
||||
FALLTHROUGH;
|
||||
case PageInaccessible:
|
||||
return PROT_NONE;
|
||||
}
|
||||
}
|
||||
|
||||
void* SystemAllocPagesInternal(void* hint,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
PageTag page_tag,
|
||||
bool commit) {
|
||||
#if defined(OS_MACOSX)
|
||||
// Use a custom tag to make it easier to distinguish Partition Alloc regions
|
||||
// in vmmap(1). Tags between 240-255 are supported.
|
||||
DCHECK_LE(PageTag::kFirst, page_tag);
|
||||
DCHECK_GE(PageTag::kLast, page_tag);
|
||||
int fd = VM_MAKE_TAG(static_cast<int>(page_tag));
|
||||
#else
|
||||
int fd = -1;
|
||||
#endif
|
||||
|
||||
int access_flag = GetAccessFlags(accessibility);
|
||||
int map_flags = MAP_ANONYMOUS | MAP_PRIVATE;
|
||||
|
||||
#if defined(OS_MACOSX)
|
||||
// On macOS 10.14 and higher, executables that are code signed with the
|
||||
// "runtime" option cannot execute writable memory by default. They can opt
|
||||
// into this capability by specifying the "com.apple.security.cs.allow-jit"
|
||||
// code signing entitlement and allocating the region with the MAP_JIT flag.
|
||||
static const bool kUseMapJit = UseMapJit();
|
||||
if (page_tag == PageTag::kV8 && kUseMapJit) {
|
||||
map_flags |= MAP_JIT;
|
||||
}
|
||||
#endif
|
||||
|
||||
void* ret =
|
||||
mmap(hint, length, access_flag, map_flags, fd, 0);
|
||||
if (ret == MAP_FAILED) {
|
||||
s_allocPageErrorCode = errno;
|
||||
ret = nullptr;
|
||||
}
|
||||
|
||||
#if defined(OS_ANDROID)
|
||||
// On Android, anonymous mappings can have a name attached to them. This is
|
||||
// useful for debugging, and double-checking memory attribution.
|
||||
if (ret) {
|
||||
// No error checking on purpose, testing only.
|
||||
prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, ret, length,
|
||||
PageTagToName(page_tag));
|
||||
}
|
||||
#endif
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
void* TrimMappingInternal(void* base,
|
||||
size_t base_length,
|
||||
size_t trim_length,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
bool commit,
|
||||
size_t pre_slack,
|
||||
size_t post_slack) {
|
||||
void* ret = base;
|
||||
// We can resize the allocation run. Release unneeded memory before and after
|
||||
// the aligned range.
|
||||
if (pre_slack) {
|
||||
int res = munmap(base, pre_slack);
|
||||
CHECK(!res);
|
||||
ret = reinterpret_cast<char*>(base) + pre_slack;
|
||||
}
|
||||
if (post_slack) {
|
||||
int res = munmap(reinterpret_cast<char*>(ret) + trim_length, post_slack);
|
||||
CHECK(!res);
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
bool TrySetSystemPagesAccessInternal(
|
||||
void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
return 0 == mprotect(address, length, GetAccessFlags(accessibility));
|
||||
}
|
||||
|
||||
void SetSystemPagesAccessInternal(
|
||||
void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
CHECK_EQ(0, mprotect(address, length, GetAccessFlags(accessibility)));
|
||||
}
|
||||
|
||||
void FreePagesInternal(void* address, size_t length) {
|
||||
CHECK(!munmap(address, length));
|
||||
}
|
||||
|
||||
void DecommitSystemPagesInternal(void* address, size_t length) {
|
||||
// In POSIX, there is no decommit concept. Discarding is an effective way of
|
||||
// implementing the Windows semantics where the OS is allowed to not swap the
|
||||
// pages in the region.
|
||||
//
|
||||
// TODO(ajwong): Also explore setting PageInaccessible to make the protection
|
||||
// semantics consistent between Windows and POSIX. This might have a perf cost
|
||||
// though as both decommit and recommit would incur an extra syscall.
|
||||
// http://crbug.com/766882
|
||||
DiscardSystemPages(address, length);
|
||||
}
|
||||
|
||||
bool RecommitSystemPagesInternal(void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
#if defined(OS_MACOSX)
|
||||
// On macOS, to update accounting, we need to make another syscall. For more
|
||||
// details, see https://crbug.com/823915.
|
||||
madvise(address, length, MADV_FREE_REUSE);
|
||||
#endif
|
||||
|
||||
// On POSIX systems, the caller need simply read the memory to recommit it.
|
||||
// This has the correct behavior because the API requires the permissions to
|
||||
// be the same as before decommitting and all configurations can read.
|
||||
return true;
|
||||
}
|
||||
|
||||
void DiscardSystemPagesInternal(void* address, size_t length) {
|
||||
#if defined(OS_MACOSX)
|
||||
int ret = madvise(address, length, MADV_FREE_REUSABLE);
|
||||
if (ret) {
|
||||
// MADV_FREE_REUSABLE sometimes fails, so fall back to MADV_DONTNEED.
|
||||
ret = madvise(address, length, MADV_DONTNEED);
|
||||
}
|
||||
CHECK(0 == ret);
|
||||
#else
|
||||
// We have experimented with other flags, but with suboptimal results.
|
||||
//
|
||||
// MADV_FREE (Linux): Makes our memory measurements less predictable;
|
||||
// performance benefits unclear.
|
||||
//
|
||||
// Therefore, we just do the simple thing: MADV_DONTNEED.
|
||||
CHECK(!madvise(address, length, MADV_DONTNEED));
|
||||
#endif
|
||||
}
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_POSIX_H_
|
||||
|
|
@ -0,0 +1,144 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_WIN_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_WIN_H_
|
||||
|
||||
#include "base/allocator/partition_allocator/oom.h"
|
||||
#include "base/allocator/partition_allocator/page_allocator_internal.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
// |VirtualAlloc| will fail if allocation at the hint address is blocked.
|
||||
constexpr bool kHintIsAdvisory = false;
|
||||
std::atomic<int32_t> s_allocPageErrorCode{ERROR_SUCCESS};
|
||||
|
||||
int GetAccessFlags(PageAccessibilityConfiguration accessibility) {
|
||||
switch (accessibility) {
|
||||
case PageRead:
|
||||
return PAGE_READONLY;
|
||||
case PageReadWrite:
|
||||
return PAGE_READWRITE;
|
||||
case PageReadExecute:
|
||||
return PAGE_EXECUTE_READ;
|
||||
case PageReadWriteExecute:
|
||||
return PAGE_EXECUTE_READWRITE;
|
||||
default:
|
||||
NOTREACHED();
|
||||
FALLTHROUGH;
|
||||
case PageInaccessible:
|
||||
return PAGE_NOACCESS;
|
||||
}
|
||||
}
|
||||
|
||||
void* SystemAllocPagesInternal(void* hint,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
PageTag page_tag,
|
||||
bool commit) {
|
||||
DWORD access_flag = GetAccessFlags(accessibility);
|
||||
const DWORD type_flags = commit ? (MEM_RESERVE | MEM_COMMIT) : MEM_RESERVE;
|
||||
void* ret = VirtualAlloc(hint, length, type_flags, access_flag);
|
||||
if (ret == nullptr) {
|
||||
s_allocPageErrorCode = GetLastError();
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
void* TrimMappingInternal(void* base,
|
||||
size_t base_length,
|
||||
size_t trim_length,
|
||||
PageAccessibilityConfiguration accessibility,
|
||||
bool commit,
|
||||
size_t pre_slack,
|
||||
size_t post_slack) {
|
||||
void* ret = base;
|
||||
if (pre_slack || post_slack) {
|
||||
// We cannot resize the allocation run. Free it and retry at the aligned
|
||||
// address within the freed range.
|
||||
ret = reinterpret_cast<char*>(base) + pre_slack;
|
||||
FreePages(base, base_length);
|
||||
ret = SystemAllocPages(ret, trim_length, accessibility, PageTag::kChromium,
|
||||
commit);
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
bool TrySetSystemPagesAccessInternal(
|
||||
void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
if (accessibility == PageInaccessible)
|
||||
return VirtualFree(address, length, MEM_DECOMMIT) != 0;
|
||||
return nullptr != VirtualAlloc(address, length, MEM_COMMIT,
|
||||
GetAccessFlags(accessibility));
|
||||
}
|
||||
|
||||
void SetSystemPagesAccessInternal(
|
||||
void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
if (accessibility == PageInaccessible) {
|
||||
if (!VirtualFree(address, length, MEM_DECOMMIT)) {
|
||||
// We check `GetLastError` for `ERROR_SUCCESS` here so that in a crash
|
||||
// report we get the error number.
|
||||
CHECK_EQ(static_cast<uint32_t>(ERROR_SUCCESS), GetLastError());
|
||||
}
|
||||
} else {
|
||||
if (!VirtualAlloc(address, length, MEM_COMMIT,
|
||||
GetAccessFlags(accessibility))) {
|
||||
int32_t error = GetLastError();
|
||||
if (error == ERROR_COMMITMENT_LIMIT)
|
||||
OOM_CRASH(length);
|
||||
// We check `GetLastError` for `ERROR_SUCCESS` here so that in a crash
|
||||
// report we get the error number.
|
||||
CHECK_EQ(ERROR_SUCCESS, error);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void FreePagesInternal(void* address, size_t length) {
|
||||
CHECK(VirtualFree(address, 0, MEM_RELEASE));
|
||||
}
|
||||
|
||||
void DecommitSystemPagesInternal(void* address, size_t length) {
|
||||
SetSystemPagesAccess(address, length, PageInaccessible);
|
||||
}
|
||||
|
||||
bool RecommitSystemPagesInternal(void* address,
|
||||
size_t length,
|
||||
PageAccessibilityConfiguration accessibility) {
|
||||
return TrySetSystemPagesAccess(address, length, accessibility);
|
||||
}
|
||||
|
||||
void DiscardSystemPagesInternal(void* address, size_t length) {
|
||||
// On Windows, discarded pages are not returned to the system immediately and
|
||||
// not guaranteed to be zeroed when returned to the application.
|
||||
using DiscardVirtualMemoryFunction =
|
||||
DWORD(WINAPI*)(PVOID virtualAddress, SIZE_T size);
|
||||
static DiscardVirtualMemoryFunction discard_virtual_memory =
|
||||
reinterpret_cast<DiscardVirtualMemoryFunction>(-1);
|
||||
if (discard_virtual_memory ==
|
||||
reinterpret_cast<DiscardVirtualMemoryFunction>(-1))
|
||||
discard_virtual_memory =
|
||||
reinterpret_cast<DiscardVirtualMemoryFunction>(GetProcAddress(
|
||||
GetModuleHandle(L"Kernel32.dll"), "DiscardVirtualMemory"));
|
||||
// Use DiscardVirtualMemory when available because it releases faster than
|
||||
// MEM_RESET.
|
||||
DWORD ret = 1;
|
||||
if (discard_virtual_memory) {
|
||||
ret = discard_virtual_memory(address, length);
|
||||
}
|
||||
// DiscardVirtualMemory is buggy in Win10 SP0, so fall back to MEM_RESET on
|
||||
// failure.
|
||||
if (ret) {
|
||||
void* ptr = VirtualAlloc(address, length, MEM_RESET, PAGE_READWRITE);
|
||||
CHECK(ptr);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PAGE_ALLOCATOR_INTERNALS_WIN_H_
|
||||
|
|
@ -0,0 +1,849 @@
|
|||
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_alloc.h"
|
||||
|
||||
#include <string.h>
|
||||
|
||||
#include <memory>
|
||||
#include <type_traits>
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_direct_map_extent.h"
|
||||
#include "base/allocator/partition_allocator/partition_oom.h"
|
||||
#include "base/allocator/partition_allocator/partition_page.h"
|
||||
#include "base/allocator/partition_allocator/spin_lock.h"
|
||||
#include "base/logging.h"
|
||||
#include "base/no_destructor.h"
|
||||
#include "base/synchronization/lock.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
// Two partition pages are used as guard / metadata page so make sure the super
|
||||
// page size is bigger.
|
||||
static_assert(kPartitionPageSize * 4 <= kSuperPageSize, "ok super page size");
|
||||
static_assert(!(kSuperPageSize % kPartitionPageSize), "ok super page multiple");
|
||||
// Four system pages gives us room to hack out a still-guard-paged piece
|
||||
// of metadata in the middle of a guard partition page.
|
||||
static_assert(kSystemPageSize * 4 <= kPartitionPageSize,
|
||||
"ok partition page size");
|
||||
static_assert(!(kPartitionPageSize % kSystemPageSize),
|
||||
"ok partition page multiple");
|
||||
static_assert(sizeof(internal::PartitionPage) <= kPageMetadataSize,
|
||||
"PartitionPage should not be too big");
|
||||
static_assert(sizeof(internal::PartitionBucket) <= kPageMetadataSize,
|
||||
"PartitionBucket should not be too big");
|
||||
static_assert(sizeof(internal::PartitionSuperPageExtentEntry) <=
|
||||
kPageMetadataSize,
|
||||
"PartitionSuperPageExtentEntry should not be too big");
|
||||
static_assert(kPageMetadataSize * kNumPartitionPagesPerSuperPage <=
|
||||
kSystemPageSize,
|
||||
"page metadata fits in hole");
|
||||
// Limit to prevent callers accidentally overflowing an int size.
|
||||
static_assert(kGenericMaxDirectMapped <=
|
||||
(1UL << 31) + kPageAllocationGranularity,
|
||||
"maximum direct mapped allocation");
|
||||
// Check that some of our zanier calculations worked out as expected.
|
||||
static_assert(kGenericSmallestBucket == 8, "generic smallest bucket");
|
||||
static_assert(kGenericMaxBucketed == 983040, "generic max bucketed");
|
||||
static_assert(kMaxSystemPagesPerSlotSpan < (1 << 8),
|
||||
"System pages per slot span must be less than 128.");
|
||||
|
||||
internal::PartitionRootBase::PartitionRootBase() = default;
|
||||
internal::PartitionRootBase::~PartitionRootBase() = default;
|
||||
PartitionRoot::PartitionRoot() = default;
|
||||
PartitionRoot::~PartitionRoot() = default;
|
||||
PartitionRootGeneric::PartitionRootGeneric() = default;
|
||||
PartitionRootGeneric::~PartitionRootGeneric() = default;
|
||||
PartitionAllocatorGeneric::PartitionAllocatorGeneric() = default;
|
||||
|
||||
Lock& GetLock() {
|
||||
static NoDestructor<Lock> s_initialized_lock;
|
||||
return *s_initialized_lock;
|
||||
}
|
||||
static bool g_initialized = false;
|
||||
|
||||
Lock& GetHooksLock() {
|
||||
static NoDestructor<Lock> lock;
|
||||
return *lock;
|
||||
}
|
||||
|
||||
OomFunction internal::PartitionRootBase::g_oom_handling_function = nullptr;
|
||||
std::atomic<bool> PartitionAllocHooks::hooks_enabled_(false);
|
||||
std::atomic<PartitionAllocHooks::AllocationObserverHook*>
|
||||
PartitionAllocHooks::allocation_observer_hook_(nullptr);
|
||||
std::atomic<PartitionAllocHooks::FreeObserverHook*>
|
||||
PartitionAllocHooks::free_observer_hook_(nullptr);
|
||||
std::atomic<PartitionAllocHooks::AllocationOverrideHook*>
|
||||
PartitionAllocHooks::allocation_override_hook_(nullptr);
|
||||
std::atomic<PartitionAllocHooks::FreeOverrideHook*>
|
||||
PartitionAllocHooks::free_override_hook_(nullptr);
|
||||
std::atomic<PartitionAllocHooks::ReallocOverrideHook*>
|
||||
PartitionAllocHooks::realloc_override_hook_(nullptr);
|
||||
|
||||
void PartitionAllocHooks::SetObserverHooks(AllocationObserverHook* alloc_hook,
|
||||
FreeObserverHook* free_hook) {
|
||||
AutoLock guard(GetHooksLock());
|
||||
|
||||
// Chained hooks are not supported. Registering a non-null hook when a
|
||||
// non-null hook is already registered indicates somebody is trying to
|
||||
// overwrite a hook.
|
||||
CHECK((!allocation_observer_hook_ && !free_observer_hook_) ||
|
||||
(!alloc_hook && !free_hook))
|
||||
<< "Overwriting already set observer hooks";
|
||||
allocation_observer_hook_ = alloc_hook;
|
||||
free_observer_hook_ = free_hook;
|
||||
|
||||
hooks_enabled_ = allocation_observer_hook_ || allocation_override_hook_;
|
||||
}
|
||||
|
||||
void PartitionAllocHooks::SetOverrideHooks(AllocationOverrideHook* alloc_hook,
|
||||
FreeOverrideHook* free_hook,
|
||||
ReallocOverrideHook realloc_hook) {
|
||||
AutoLock guard(GetHooksLock());
|
||||
|
||||
CHECK((!allocation_override_hook_ && !free_override_hook_ &&
|
||||
!realloc_override_hook_) ||
|
||||
(!alloc_hook && !free_hook && !realloc_hook))
|
||||
<< "Overwriting already set override hooks";
|
||||
allocation_override_hook_ = alloc_hook;
|
||||
free_override_hook_ = free_hook;
|
||||
realloc_override_hook_ = realloc_hook;
|
||||
|
||||
hooks_enabled_ = allocation_observer_hook_ || allocation_override_hook_;
|
||||
}
|
||||
|
||||
void PartitionAllocHooks::AllocationObserverHookIfEnabled(
|
||||
void* address,
|
||||
size_t size,
|
||||
const char* type_name) {
|
||||
if (AllocationObserverHook* hook =
|
||||
allocation_observer_hook_.load(std::memory_order_relaxed)) {
|
||||
hook(address, size, type_name);
|
||||
}
|
||||
}
|
||||
|
||||
bool PartitionAllocHooks::AllocationOverrideHookIfEnabled(
|
||||
void** out,
|
||||
int flags,
|
||||
size_t size,
|
||||
const char* type_name) {
|
||||
if (AllocationOverrideHook* hook =
|
||||
allocation_override_hook_.load(std::memory_order_relaxed)) {
|
||||
return hook(out, flags, size, type_name);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
void PartitionAllocHooks::FreeObserverHookIfEnabled(void* address) {
|
||||
if (FreeObserverHook* hook =
|
||||
free_observer_hook_.load(std::memory_order_relaxed)) {
|
||||
hook(address);
|
||||
}
|
||||
}
|
||||
|
||||
bool PartitionAllocHooks::FreeOverrideHookIfEnabled(void* address) {
|
||||
if (FreeOverrideHook* hook =
|
||||
free_override_hook_.load(std::memory_order_relaxed)) {
|
||||
return hook(address);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
void PartitionAllocHooks::ReallocObserverHookIfEnabled(void* old_address,
|
||||
void* new_address,
|
||||
size_t size,
|
||||
const char* type_name) {
|
||||
// Report a reallocation as a free followed by an allocation.
|
||||
AllocationObserverHook* allocation_hook =
|
||||
allocation_observer_hook_.load(std::memory_order_relaxed);
|
||||
FreeObserverHook* free_hook =
|
||||
free_observer_hook_.load(std::memory_order_relaxed);
|
||||
if (allocation_hook && free_hook) {
|
||||
free_hook(old_address);
|
||||
allocation_hook(new_address, size, type_name);
|
||||
}
|
||||
}
|
||||
bool PartitionAllocHooks::ReallocOverrideHookIfEnabled(size_t* out,
|
||||
void* address) {
|
||||
if (ReallocOverrideHook* hook =
|
||||
realloc_override_hook_.load(std::memory_order_relaxed)) {
|
||||
return hook(out, address);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
static void PartitionAllocBaseInit(internal::PartitionRootBase* root) {
|
||||
DCHECK(!root->initialized);
|
||||
{
|
||||
AutoLock guard(GetLock());
|
||||
if (!g_initialized) {
|
||||
g_initialized = true;
|
||||
// We mark the sentinel bucket/page as free to make sure it is skipped by
|
||||
// our logic to find a new active page.
|
||||
internal::PartitionBucket::get_sentinel_bucket()->active_pages_head =
|
||||
internal::PartitionPage::get_sentinel_page();
|
||||
}
|
||||
}
|
||||
|
||||
root->initialized = true;
|
||||
|
||||
// This is a "magic" value so we can test if a root pointer is valid.
|
||||
root->inverted_self = ~reinterpret_cast<uintptr_t>(root);
|
||||
}
|
||||
|
||||
void PartitionAllocGlobalInit(OomFunction on_out_of_memory) {
|
||||
DCHECK(on_out_of_memory);
|
||||
internal::PartitionRootBase::g_oom_handling_function = on_out_of_memory;
|
||||
}
|
||||
|
||||
void PartitionRoot::Init(size_t bucket_count, size_t maximum_allocation) {
|
||||
PartitionAllocBaseInit(this);
|
||||
|
||||
num_buckets = bucket_count;
|
||||
max_allocation = maximum_allocation;
|
||||
for (size_t i = 0; i < num_buckets; ++i) {
|
||||
internal::PartitionBucket& bucket = buckets()[i];
|
||||
bucket.Init(i == 0 ? kAllocationGranularity : (i << kBucketShift));
|
||||
}
|
||||
}
|
||||
|
||||
void PartitionRootGeneric::Init() {
|
||||
subtle::SpinLock::Guard guard(lock);
|
||||
|
||||
PartitionAllocBaseInit(this);
|
||||
|
||||
// Precalculate some shift and mask constants used in the hot path.
|
||||
// Example: malloc(41) == 101001 binary.
|
||||
// Order is 6 (1 << 6-1) == 32 is highest bit set.
|
||||
// order_index is the next three MSB == 010 == 2.
|
||||
// sub_order_index_mask is a mask for the remaining bits == 11 (masking to 01
|
||||
// for
|
||||
// the sub_order_index).
|
||||
size_t order;
|
||||
for (order = 0; order <= kBitsPerSizeT; ++order) {
|
||||
size_t order_index_shift;
|
||||
if (order < kGenericNumBucketsPerOrderBits + 1)
|
||||
order_index_shift = 0;
|
||||
else
|
||||
order_index_shift = order - (kGenericNumBucketsPerOrderBits + 1);
|
||||
order_index_shifts[order] = order_index_shift;
|
||||
size_t sub_order_index_mask;
|
||||
if (order == kBitsPerSizeT) {
|
||||
// This avoids invoking undefined behavior for an excessive shift.
|
||||
sub_order_index_mask =
|
||||
static_cast<size_t>(-1) >> (kGenericNumBucketsPerOrderBits + 1);
|
||||
} else {
|
||||
sub_order_index_mask = ((static_cast<size_t>(1) << order) - 1) >>
|
||||
(kGenericNumBucketsPerOrderBits + 1);
|
||||
}
|
||||
order_sub_index_masks[order] = sub_order_index_mask;
|
||||
}
|
||||
|
||||
// Set up the actual usable buckets first.
|
||||
// Note that typical values (i.e. min allocation size of 8) will result in
|
||||
// pseudo buckets (size==9 etc. or more generally, size is not a multiple
|
||||
// of the smallest allocation granularity).
|
||||
// We avoid them in the bucket lookup map, but we tolerate them to keep the
|
||||
// code simpler and the structures more generic.
|
||||
size_t i, j;
|
||||
size_t current_size = kGenericSmallestBucket;
|
||||
size_t current_increment =
|
||||
kGenericSmallestBucket >> kGenericNumBucketsPerOrderBits;
|
||||
internal::PartitionBucket* bucket = &buckets[0];
|
||||
for (i = 0; i < kGenericNumBucketedOrders; ++i) {
|
||||
for (j = 0; j < kGenericNumBucketsPerOrder; ++j) {
|
||||
bucket->Init(current_size);
|
||||
// Disable psuedo buckets so that touching them faults.
|
||||
if (current_size % kGenericSmallestBucket)
|
||||
bucket->active_pages_head = nullptr;
|
||||
current_size += current_increment;
|
||||
++bucket;
|
||||
}
|
||||
current_increment <<= 1;
|
||||
}
|
||||
DCHECK(current_size == 1 << kGenericMaxBucketedOrder);
|
||||
DCHECK(bucket == &buckets[0] + kGenericNumBuckets);
|
||||
|
||||
// Then set up the fast size -> bucket lookup table.
|
||||
bucket = &buckets[0];
|
||||
internal::PartitionBucket** bucket_ptr = &bucket_lookups[0];
|
||||
for (order = 0; order <= kBitsPerSizeT; ++order) {
|
||||
for (j = 0; j < kGenericNumBucketsPerOrder; ++j) {
|
||||
if (order < kGenericMinBucketedOrder) {
|
||||
// Use the bucket of the finest granularity for malloc(0) etc.
|
||||
*bucket_ptr++ = &buckets[0];
|
||||
} else if (order > kGenericMaxBucketedOrder) {
|
||||
*bucket_ptr++ = internal::PartitionBucket::get_sentinel_bucket();
|
||||
} else {
|
||||
internal::PartitionBucket* valid_bucket = bucket;
|
||||
// Skip over invalid buckets.
|
||||
while (valid_bucket->slot_size % kGenericSmallestBucket)
|
||||
valid_bucket++;
|
||||
*bucket_ptr++ = valid_bucket;
|
||||
bucket++;
|
||||
}
|
||||
}
|
||||
}
|
||||
DCHECK(bucket == &buckets[0] + kGenericNumBuckets);
|
||||
DCHECK(bucket_ptr == &bucket_lookups[0] +
|
||||
((kBitsPerSizeT + 1) * kGenericNumBucketsPerOrder));
|
||||
// And there's one last bucket lookup that will be hit for e.g. malloc(-1),
|
||||
// which tries to overflow to a non-existant order.
|
||||
*bucket_ptr = internal::PartitionBucket::get_sentinel_bucket();
|
||||
}
|
||||
|
||||
bool PartitionReallocDirectMappedInPlace(PartitionRootGeneric* root,
|
||||
internal::PartitionPage* page,
|
||||
size_t raw_size) {
|
||||
DCHECK(page->bucket->is_direct_mapped());
|
||||
|
||||
raw_size = internal::PartitionCookieSizeAdjustAdd(raw_size);
|
||||
|
||||
// Note that the new size might be a bucketed size; this function is called
|
||||
// whenever we're reallocating a direct mapped allocation.
|
||||
size_t new_size = internal::PartitionBucket::get_direct_map_size(raw_size);
|
||||
if (new_size < kGenericMinDirectMappedDownsize)
|
||||
return false;
|
||||
|
||||
// bucket->slot_size is the current size of the allocation.
|
||||
size_t current_size = page->bucket->slot_size;
|
||||
char* char_ptr = static_cast<char*>(internal::PartitionPage::ToPointer(page));
|
||||
if (new_size == current_size) {
|
||||
// No need to move any memory around, but update size and cookie below.
|
||||
} else if (new_size < current_size) {
|
||||
size_t map_size =
|
||||
internal::PartitionDirectMapExtent::FromPage(page)->map_size;
|
||||
|
||||
// Don't reallocate in-place if new size is less than 80 % of the full
|
||||
// map size, to avoid holding on to too much unused address space.
|
||||
if ((new_size / kSystemPageSize) * 5 < (map_size / kSystemPageSize) * 4)
|
||||
return false;
|
||||
|
||||
// Shrink by decommitting unneeded pages and making them inaccessible.
|
||||
size_t decommit_size = current_size - new_size;
|
||||
root->DecommitSystemPages(char_ptr + new_size, decommit_size);
|
||||
SetSystemPagesAccess(char_ptr + new_size, decommit_size, PageInaccessible);
|
||||
} else if (new_size <=
|
||||
internal::PartitionDirectMapExtent::FromPage(page)->map_size) {
|
||||
// Grow within the actually allocated memory. Just need to make the
|
||||
// pages accessible again.
|
||||
size_t recommit_size = new_size - current_size;
|
||||
SetSystemPagesAccess(char_ptr + current_size, recommit_size, PageReadWrite);
|
||||
root->RecommitSystemPages(char_ptr + current_size, recommit_size);
|
||||
|
||||
#if DCHECK_IS_ON()
|
||||
memset(char_ptr + current_size, kUninitializedByte, recommit_size);
|
||||
#endif
|
||||
} else {
|
||||
// We can't perform the realloc in-place.
|
||||
// TODO: support this too when possible.
|
||||
return false;
|
||||
}
|
||||
|
||||
#if DCHECK_IS_ON()
|
||||
// Write a new trailing cookie.
|
||||
internal::PartitionCookieWriteValue(char_ptr + raw_size -
|
||||
internal::kCookieSize);
|
||||
#endif
|
||||
|
||||
page->set_raw_size(raw_size);
|
||||
DCHECK(page->get_raw_size() == raw_size);
|
||||
|
||||
page->bucket->slot_size = new_size;
|
||||
return true;
|
||||
}
|
||||
|
||||
void* PartitionReallocGenericFlags(PartitionRootGeneric* root,
|
||||
int flags,
|
||||
void* ptr,
|
||||
size_t new_size,
|
||||
const char* type_name) {
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
CHECK_MAX_SIZE_OR_RETURN_NULLPTR(new_size, flags);
|
||||
void* result = realloc(ptr, new_size);
|
||||
CHECK(result || flags & PartitionAllocReturnNull);
|
||||
return result;
|
||||
#else
|
||||
if (UNLIKELY(!ptr))
|
||||
return PartitionAllocGenericFlags(root, flags, new_size, type_name);
|
||||
if (UNLIKELY(!new_size)) {
|
||||
root->Free(ptr);
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
if (new_size > kGenericMaxDirectMapped) {
|
||||
if (flags & PartitionAllocReturnNull)
|
||||
return nullptr;
|
||||
internal::PartitionExcessiveAllocationSize(new_size);
|
||||
}
|
||||
|
||||
const bool hooks_enabled = PartitionAllocHooks::AreHooksEnabled();
|
||||
bool overridden = false;
|
||||
size_t actual_old_size;
|
||||
if (UNLIKELY(hooks_enabled)) {
|
||||
overridden = PartitionAllocHooks::ReallocOverrideHookIfEnabled(
|
||||
&actual_old_size, ptr);
|
||||
}
|
||||
if (LIKELY(!overridden)) {
|
||||
internal::PartitionPage* page = internal::PartitionPage::FromPointer(
|
||||
internal::PartitionCookieFreePointerAdjust(ptr));
|
||||
// TODO(palmer): See if we can afford to make this a CHECK.
|
||||
DCHECK(root->IsValidPage(page));
|
||||
|
||||
if (UNLIKELY(page->bucket->is_direct_mapped())) {
|
||||
// We may be able to perform the realloc in place by changing the
|
||||
// accessibility of memory pages and, if reducing the size, decommitting
|
||||
// them.
|
||||
if (PartitionReallocDirectMappedInPlace(root, page, new_size)) {
|
||||
if (UNLIKELY(hooks_enabled)) {
|
||||
PartitionAllocHooks::ReallocObserverHookIfEnabled(ptr, ptr, new_size,
|
||||
type_name);
|
||||
}
|
||||
return ptr;
|
||||
}
|
||||
}
|
||||
|
||||
const size_t actual_new_size = root->ActualSize(new_size);
|
||||
actual_old_size = PartitionAllocGetSize(ptr);
|
||||
|
||||
// TODO: note that tcmalloc will "ignore" a downsizing realloc() unless the
|
||||
// new size is a significant percentage smaller. We could do the same if we
|
||||
// determine it is a win.
|
||||
if (actual_new_size == actual_old_size) {
|
||||
// Trying to allocate a block of size |new_size| would give us a block of
|
||||
// the same size as the one we've already got, so re-use the allocation
|
||||
// after updating statistics (and cookies, if present).
|
||||
page->set_raw_size(internal::PartitionCookieSizeAdjustAdd(new_size));
|
||||
#if DCHECK_IS_ON()
|
||||
// Write a new trailing cookie when it is possible to keep track of
|
||||
// |new_size| via the raw size pointer.
|
||||
if (page->get_raw_size_ptr())
|
||||
internal::PartitionCookieWriteValue(static_cast<char*>(ptr) + new_size);
|
||||
#endif
|
||||
return ptr;
|
||||
}
|
||||
}
|
||||
|
||||
// This realloc cannot be resized in-place. Sadness.
|
||||
void* ret = PartitionAllocGenericFlags(root, flags, new_size, type_name);
|
||||
if (!ret) {
|
||||
if (flags & PartitionAllocReturnNull)
|
||||
return nullptr;
|
||||
internal::PartitionExcessiveAllocationSize(new_size);
|
||||
}
|
||||
|
||||
size_t copy_size = actual_old_size;
|
||||
if (new_size < copy_size)
|
||||
copy_size = new_size;
|
||||
|
||||
memcpy(ret, ptr, copy_size);
|
||||
root->Free(ptr);
|
||||
return ret;
|
||||
#endif
|
||||
}
|
||||
|
||||
void* PartitionRootGeneric::Realloc(void* ptr,
|
||||
size_t new_size,
|
||||
const char* type_name) {
|
||||
return PartitionReallocGenericFlags(this, 0, ptr, new_size, type_name);
|
||||
}
|
||||
|
||||
void* PartitionRootGeneric::TryRealloc(void* ptr,
|
||||
size_t new_size,
|
||||
const char* type_name) {
|
||||
return PartitionReallocGenericFlags(this, PartitionAllocReturnNull, ptr,
|
||||
new_size, type_name);
|
||||
}
|
||||
|
||||
static size_t PartitionPurgePage(internal::PartitionPage* page, bool discard) {
|
||||
const internal::PartitionBucket* bucket = page->bucket;
|
||||
size_t slot_size = bucket->slot_size;
|
||||
if (slot_size < kSystemPageSize || !page->num_allocated_slots)
|
||||
return 0;
|
||||
|
||||
size_t bucket_num_slots = bucket->get_slots_per_span();
|
||||
size_t discardable_bytes = 0;
|
||||
|
||||
size_t raw_size = page->get_raw_size();
|
||||
if (raw_size) {
|
||||
uint32_t used_bytes = static_cast<uint32_t>(RoundUpToSystemPage(raw_size));
|
||||
discardable_bytes = bucket->slot_size - used_bytes;
|
||||
if (discardable_bytes && discard) {
|
||||
char* ptr =
|
||||
reinterpret_cast<char*>(internal::PartitionPage::ToPointer(page));
|
||||
ptr += used_bytes;
|
||||
DiscardSystemPages(ptr, discardable_bytes);
|
||||
}
|
||||
return discardable_bytes;
|
||||
}
|
||||
|
||||
constexpr size_t kMaxSlotCount =
|
||||
(kPartitionPageSize * kMaxPartitionPagesPerSlotSpan) / kSystemPageSize;
|
||||
DCHECK(bucket_num_slots <= kMaxSlotCount);
|
||||
DCHECK(page->num_unprovisioned_slots < bucket_num_slots);
|
||||
size_t num_slots = bucket_num_slots - page->num_unprovisioned_slots;
|
||||
char slot_usage[kMaxSlotCount];
|
||||
#if !defined(OS_WIN)
|
||||
// The last freelist entry should not be discarded when using OS_WIN.
|
||||
// DiscardVirtualMemory makes the contents of discarded memory undefined.
|
||||
size_t last_slot = static_cast<size_t>(-1);
|
||||
#endif
|
||||
memset(slot_usage, 1, num_slots);
|
||||
char* ptr = reinterpret_cast<char*>(internal::PartitionPage::ToPointer(page));
|
||||
// First, walk the freelist for this page and make a bitmap of which slots
|
||||
// are not in use.
|
||||
for (internal::PartitionFreelistEntry* entry = page->freelist_head; entry;
|
||||
/**/) {
|
||||
size_t slot_index = (reinterpret_cast<char*>(entry) - ptr) / slot_size;
|
||||
DCHECK(slot_index < num_slots);
|
||||
slot_usage[slot_index] = 0;
|
||||
entry = internal::EncodedPartitionFreelistEntry::Decode(entry->next);
|
||||
#if !defined(OS_WIN)
|
||||
// If we have a slot where the masked freelist entry is 0, we can actually
|
||||
// discard that freelist entry because touching a discarded page is
|
||||
// guaranteed to return original content or 0. (Note that this optimization
|
||||
// won't fire on big-endian machines because the masking function is
|
||||
// negation.)
|
||||
if (!internal::PartitionFreelistEntry::Encode(entry))
|
||||
last_slot = slot_index;
|
||||
#endif
|
||||
}
|
||||
|
||||
// If the slot(s) at the end of the slot span are not in used, we can truncate
|
||||
// them entirely and rewrite the freelist.
|
||||
size_t truncated_slots = 0;
|
||||
while (!slot_usage[num_slots - 1]) {
|
||||
truncated_slots++;
|
||||
num_slots--;
|
||||
DCHECK(num_slots);
|
||||
}
|
||||
// First, do the work of calculating the discardable bytes. Don't actually
|
||||
// discard anything unless the discard flag was passed in.
|
||||
if (truncated_slots) {
|
||||
size_t unprovisioned_bytes = 0;
|
||||
char* begin_ptr = ptr + (num_slots * slot_size);
|
||||
char* end_ptr = begin_ptr + (slot_size * truncated_slots);
|
||||
begin_ptr = reinterpret_cast<char*>(
|
||||
RoundUpToSystemPage(reinterpret_cast<size_t>(begin_ptr)));
|
||||
// We round the end pointer here up and not down because we're at the end of
|
||||
// a slot span, so we "own" all the way up the page boundary.
|
||||
end_ptr = reinterpret_cast<char*>(
|
||||
RoundUpToSystemPage(reinterpret_cast<size_t>(end_ptr)));
|
||||
DCHECK(end_ptr <= ptr + bucket->get_bytes_per_span());
|
||||
if (begin_ptr < end_ptr) {
|
||||
unprovisioned_bytes = end_ptr - begin_ptr;
|
||||
discardable_bytes += unprovisioned_bytes;
|
||||
}
|
||||
if (unprovisioned_bytes && discard) {
|
||||
DCHECK(truncated_slots > 0);
|
||||
size_t num_new_entries = 0;
|
||||
page->num_unprovisioned_slots += static_cast<uint16_t>(truncated_slots);
|
||||
|
||||
// Rewrite the freelist.
|
||||
internal::PartitionFreelistEntry* head = nullptr;
|
||||
internal::PartitionFreelistEntry* back = head;
|
||||
for (size_t slot_index = 0; slot_index < num_slots; ++slot_index) {
|
||||
if (slot_usage[slot_index])
|
||||
continue;
|
||||
|
||||
auto* entry = reinterpret_cast<internal::PartitionFreelistEntry*>(
|
||||
ptr + (slot_size * slot_index));
|
||||
if (!head) {
|
||||
head = entry;
|
||||
back = entry;
|
||||
} else {
|
||||
back->next = internal::PartitionFreelistEntry::Encode(entry);
|
||||
back = entry;
|
||||
}
|
||||
num_new_entries++;
|
||||
#if !defined(OS_WIN)
|
||||
last_slot = slot_index;
|
||||
#endif
|
||||
}
|
||||
|
||||
page->freelist_head = head;
|
||||
if (back)
|
||||
back->next = internal::PartitionFreelistEntry::Encode(nullptr);
|
||||
|
||||
DCHECK(num_new_entries == num_slots - page->num_allocated_slots);
|
||||
// Discard the memory.
|
||||
DiscardSystemPages(begin_ptr, unprovisioned_bytes);
|
||||
}
|
||||
}
|
||||
|
||||
// Next, walk the slots and for any not in use, consider where the system page
|
||||
// boundaries occur. We can release any system pages back to the system as
|
||||
// long as we don't interfere with a freelist pointer or an adjacent slot.
|
||||
for (size_t i = 0; i < num_slots; ++i) {
|
||||
if (slot_usage[i])
|
||||
continue;
|
||||
// The first address we can safely discard is just after the freelist
|
||||
// pointer. There's one quirk: if the freelist pointer is actually NULL, we
|
||||
// can discard that pointer value too.
|
||||
char* begin_ptr = ptr + (i * slot_size);
|
||||
char* end_ptr = begin_ptr + slot_size;
|
||||
#if !defined(OS_WIN)
|
||||
if (i != last_slot)
|
||||
begin_ptr += sizeof(internal::PartitionFreelistEntry);
|
||||
#else
|
||||
begin_ptr += sizeof(internal::PartitionFreelistEntry);
|
||||
#endif
|
||||
begin_ptr = reinterpret_cast<char*>(
|
||||
RoundUpToSystemPage(reinterpret_cast<size_t>(begin_ptr)));
|
||||
end_ptr = reinterpret_cast<char*>(
|
||||
RoundDownToSystemPage(reinterpret_cast<size_t>(end_ptr)));
|
||||
if (begin_ptr < end_ptr) {
|
||||
size_t partial_slot_bytes = end_ptr - begin_ptr;
|
||||
discardable_bytes += partial_slot_bytes;
|
||||
if (discard)
|
||||
DiscardSystemPages(begin_ptr, partial_slot_bytes);
|
||||
}
|
||||
}
|
||||
return discardable_bytes;
|
||||
}
|
||||
|
||||
static void PartitionPurgeBucket(internal::PartitionBucket* bucket) {
|
||||
if (bucket->active_pages_head !=
|
||||
internal::PartitionPage::get_sentinel_page()) {
|
||||
for (internal::PartitionPage* page = bucket->active_pages_head; page;
|
||||
page = page->next_page) {
|
||||
DCHECK(page != internal::PartitionPage::get_sentinel_page());
|
||||
PartitionPurgePage(page, true);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void PartitionRoot::PurgeMemory(int flags) {
|
||||
if (flags & PartitionPurgeDecommitEmptyPages)
|
||||
DecommitEmptyPages();
|
||||
// We don't currently do anything for PartitionPurgeDiscardUnusedSystemPages
|
||||
// here because that flag is only useful for allocations >= system page size.
|
||||
// We only have allocations that large inside generic partitions at the
|
||||
// moment.
|
||||
}
|
||||
|
||||
void PartitionRootGeneric::PurgeMemory(int flags) {
|
||||
subtle::SpinLock::Guard guard(lock);
|
||||
if (flags & PartitionPurgeDecommitEmptyPages)
|
||||
DecommitEmptyPages();
|
||||
if (flags & PartitionPurgeDiscardUnusedSystemPages) {
|
||||
for (size_t i = 0; i < kGenericNumBuckets; ++i) {
|
||||
internal::PartitionBucket* bucket = &buckets[i];
|
||||
if (bucket->slot_size >= kSystemPageSize)
|
||||
PartitionPurgeBucket(bucket);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static void PartitionDumpPageStats(PartitionBucketMemoryStats* stats_out,
|
||||
internal::PartitionPage* page) {
|
||||
uint16_t bucket_num_slots = page->bucket->get_slots_per_span();
|
||||
|
||||
if (page->is_decommitted()) {
|
||||
++stats_out->num_decommitted_pages;
|
||||
return;
|
||||
}
|
||||
|
||||
stats_out->discardable_bytes += PartitionPurgePage(page, false);
|
||||
|
||||
size_t raw_size = page->get_raw_size();
|
||||
if (raw_size) {
|
||||
stats_out->active_bytes += static_cast<uint32_t>(raw_size);
|
||||
} else {
|
||||
stats_out->active_bytes +=
|
||||
(page->num_allocated_slots * stats_out->bucket_slot_size);
|
||||
}
|
||||
|
||||
size_t page_bytes_resident =
|
||||
RoundUpToSystemPage((bucket_num_slots - page->num_unprovisioned_slots) *
|
||||
stats_out->bucket_slot_size);
|
||||
stats_out->resident_bytes += page_bytes_resident;
|
||||
if (page->is_empty()) {
|
||||
stats_out->decommittable_bytes += page_bytes_resident;
|
||||
++stats_out->num_empty_pages;
|
||||
} else if (page->is_full()) {
|
||||
++stats_out->num_full_pages;
|
||||
} else {
|
||||
DCHECK(page->is_active());
|
||||
++stats_out->num_active_pages;
|
||||
}
|
||||
}
|
||||
|
||||
static void PartitionDumpBucketStats(PartitionBucketMemoryStats* stats_out,
|
||||
const internal::PartitionBucket* bucket) {
|
||||
DCHECK(!bucket->is_direct_mapped());
|
||||
stats_out->is_valid = false;
|
||||
// If the active page list is empty (==
|
||||
// internal::PartitionPage::get_sentinel_page()), the bucket might still need
|
||||
// to be reported if it has a list of empty, decommitted or full pages.
|
||||
if (bucket->active_pages_head ==
|
||||
internal::PartitionPage::get_sentinel_page() &&
|
||||
!bucket->empty_pages_head && !bucket->decommitted_pages_head &&
|
||||
!bucket->num_full_pages)
|
||||
return;
|
||||
|
||||
memset(stats_out, '\0', sizeof(*stats_out));
|
||||
stats_out->is_valid = true;
|
||||
stats_out->is_direct_map = false;
|
||||
stats_out->num_full_pages = static_cast<size_t>(bucket->num_full_pages);
|
||||
stats_out->bucket_slot_size = bucket->slot_size;
|
||||
uint16_t bucket_num_slots = bucket->get_slots_per_span();
|
||||
size_t bucket_useful_storage = stats_out->bucket_slot_size * bucket_num_slots;
|
||||
stats_out->allocated_page_size = bucket->get_bytes_per_span();
|
||||
stats_out->active_bytes = bucket->num_full_pages * bucket_useful_storage;
|
||||
stats_out->resident_bytes =
|
||||
bucket->num_full_pages * stats_out->allocated_page_size;
|
||||
|
||||
for (internal::PartitionPage* page = bucket->empty_pages_head; page;
|
||||
page = page->next_page) {
|
||||
DCHECK(page->is_empty() || page->is_decommitted());
|
||||
PartitionDumpPageStats(stats_out, page);
|
||||
}
|
||||
for (internal::PartitionPage* page = bucket->decommitted_pages_head; page;
|
||||
page = page->next_page) {
|
||||
DCHECK(page->is_decommitted());
|
||||
PartitionDumpPageStats(stats_out, page);
|
||||
}
|
||||
|
||||
if (bucket->active_pages_head !=
|
||||
internal::PartitionPage::get_sentinel_page()) {
|
||||
for (internal::PartitionPage* page = bucket->active_pages_head; page;
|
||||
page = page->next_page) {
|
||||
DCHECK(page != internal::PartitionPage::get_sentinel_page());
|
||||
PartitionDumpPageStats(stats_out, page);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void PartitionRootGeneric::DumpStats(const char* partition_name,
|
||||
bool is_light_dump,
|
||||
PartitionStatsDumper* dumper) {
|
||||
PartitionMemoryStats stats = {0};
|
||||
stats.total_mmapped_bytes =
|
||||
total_size_of_super_pages + total_size_of_direct_mapped_pages;
|
||||
stats.total_committed_bytes = total_size_of_committed_pages;
|
||||
|
||||
size_t direct_mapped_allocations_total_size = 0;
|
||||
|
||||
static const size_t kMaxReportableDirectMaps = 4096;
|
||||
|
||||
// Allocate on the heap rather than on the stack to avoid stack overflow
|
||||
// skirmishes (on Windows, in particular).
|
||||
std::unique_ptr<uint32_t[]> direct_map_lengths = nullptr;
|
||||
if (!is_light_dump) {
|
||||
direct_map_lengths =
|
||||
std::unique_ptr<uint32_t[]>(new uint32_t[kMaxReportableDirectMaps]);
|
||||
}
|
||||
|
||||
PartitionBucketMemoryStats bucket_stats[kGenericNumBuckets];
|
||||
size_t num_direct_mapped_allocations = 0;
|
||||
{
|
||||
subtle::SpinLock::Guard guard(lock);
|
||||
|
||||
for (size_t i = 0; i < kGenericNumBuckets; ++i) {
|
||||
const internal::PartitionBucket* bucket = &buckets[i];
|
||||
// Don't report the pseudo buckets that the generic allocator sets up in
|
||||
// order to preserve a fast size->bucket map (see
|
||||
// PartitionRootGeneric::Init() for details).
|
||||
if (!bucket->active_pages_head)
|
||||
bucket_stats[i].is_valid = false;
|
||||
else
|
||||
PartitionDumpBucketStats(&bucket_stats[i], bucket);
|
||||
if (bucket_stats[i].is_valid) {
|
||||
stats.total_resident_bytes += bucket_stats[i].resident_bytes;
|
||||
stats.total_active_bytes += bucket_stats[i].active_bytes;
|
||||
stats.total_decommittable_bytes += bucket_stats[i].decommittable_bytes;
|
||||
stats.total_discardable_bytes += bucket_stats[i].discardable_bytes;
|
||||
}
|
||||
}
|
||||
|
||||
for (internal::PartitionDirectMapExtent* extent = direct_map_list;
|
||||
extent && num_direct_mapped_allocations < kMaxReportableDirectMaps;
|
||||
extent = extent->next_extent, ++num_direct_mapped_allocations) {
|
||||
DCHECK(!extent->next_extent ||
|
||||
extent->next_extent->prev_extent == extent);
|
||||
size_t slot_size = extent->bucket->slot_size;
|
||||
direct_mapped_allocations_total_size += slot_size;
|
||||
if (is_light_dump)
|
||||
continue;
|
||||
direct_map_lengths[num_direct_mapped_allocations] = slot_size;
|
||||
}
|
||||
}
|
||||
|
||||
if (!is_light_dump) {
|
||||
// Call |PartitionsDumpBucketStats| after collecting stats because it can
|
||||
// try to allocate using |PartitionRootGeneric::Alloc()| and it can't
|
||||
// obtain the lock.
|
||||
for (size_t i = 0; i < kGenericNumBuckets; ++i) {
|
||||
if (bucket_stats[i].is_valid)
|
||||
dumper->PartitionsDumpBucketStats(partition_name, &bucket_stats[i]);
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < num_direct_mapped_allocations; ++i) {
|
||||
uint32_t size = direct_map_lengths[i];
|
||||
|
||||
PartitionBucketMemoryStats mapped_stats = {};
|
||||
mapped_stats.is_valid = true;
|
||||
mapped_stats.is_direct_map = true;
|
||||
mapped_stats.num_full_pages = 1;
|
||||
mapped_stats.allocated_page_size = size;
|
||||
mapped_stats.bucket_slot_size = size;
|
||||
mapped_stats.active_bytes = size;
|
||||
mapped_stats.resident_bytes = size;
|
||||
dumper->PartitionsDumpBucketStats(partition_name, &mapped_stats);
|
||||
}
|
||||
}
|
||||
|
||||
stats.total_resident_bytes += direct_mapped_allocations_total_size;
|
||||
stats.total_active_bytes += direct_mapped_allocations_total_size;
|
||||
dumper->PartitionDumpTotals(partition_name, &stats);
|
||||
}
|
||||
|
||||
void PartitionRoot::DumpStats(const char* partition_name,
|
||||
bool is_light_dump,
|
||||
PartitionStatsDumper* dumper) {
|
||||
PartitionMemoryStats stats = {0};
|
||||
stats.total_mmapped_bytes = total_size_of_super_pages;
|
||||
stats.total_committed_bytes = total_size_of_committed_pages;
|
||||
DCHECK(!total_size_of_direct_mapped_pages);
|
||||
|
||||
static constexpr size_t kMaxReportableBuckets = 4096 / sizeof(void*);
|
||||
std::unique_ptr<PartitionBucketMemoryStats[]> memory_stats;
|
||||
if (!is_light_dump) {
|
||||
memory_stats = std::unique_ptr<PartitionBucketMemoryStats[]>(
|
||||
new PartitionBucketMemoryStats[kMaxReportableBuckets]);
|
||||
}
|
||||
|
||||
const size_t partition_num_buckets = num_buckets;
|
||||
DCHECK(partition_num_buckets <= kMaxReportableBuckets);
|
||||
|
||||
for (size_t i = 0; i < partition_num_buckets; ++i) {
|
||||
PartitionBucketMemoryStats bucket_stats = {0};
|
||||
PartitionDumpBucketStats(&bucket_stats, &buckets()[i]);
|
||||
if (bucket_stats.is_valid) {
|
||||
stats.total_resident_bytes += bucket_stats.resident_bytes;
|
||||
stats.total_active_bytes += bucket_stats.active_bytes;
|
||||
stats.total_decommittable_bytes += bucket_stats.decommittable_bytes;
|
||||
stats.total_discardable_bytes += bucket_stats.discardable_bytes;
|
||||
}
|
||||
if (!is_light_dump) {
|
||||
if (bucket_stats.is_valid)
|
||||
memory_stats[i] = bucket_stats;
|
||||
else
|
||||
memory_stats[i].is_valid = false;
|
||||
}
|
||||
}
|
||||
if (!is_light_dump) {
|
||||
// PartitionsDumpBucketStats is called after collecting stats because it
|
||||
// can use PartitionRoot::Alloc() to allocate and this can affect the
|
||||
// statistics.
|
||||
for (size_t i = 0; i < partition_num_buckets; ++i) {
|
||||
if (memory_stats[i].is_valid)
|
||||
dumper->PartitionsDumpBucketStats(partition_name, &memory_stats[i]);
|
||||
}
|
||||
}
|
||||
dumper->PartitionDumpTotals(partition_name, &stats);
|
||||
}
|
||||
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,540 @@
|
|||
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_H_
|
||||
|
||||
// DESCRIPTION
|
||||
// PartitionRoot::Alloc() / PartitionRootGeneric::Alloc() and PartitionFree() /
|
||||
// PartitionRootGeneric::Free() are approximately analagous to malloc() and
|
||||
// free().
|
||||
//
|
||||
// The main difference is that a PartitionRoot / PartitionRootGeneric object
|
||||
// must be supplied to these functions, representing a specific "heap partition"
|
||||
// that will be used to satisfy the allocation. Different partitions are
|
||||
// guaranteed to exist in separate address spaces, including being separate from
|
||||
// the main system heap. If the contained objects are all freed, physical memory
|
||||
// is returned to the system but the address space remains reserved.
|
||||
// See PartitionAlloc.md for other security properties PartitionAlloc provides.
|
||||
//
|
||||
// THE ONLY LEGITIMATE WAY TO OBTAIN A PartitionRoot IS THROUGH THE
|
||||
// SizeSpecificPartitionAllocator / PartitionAllocatorGeneric classes. To
|
||||
// minimize the instruction count to the fullest extent possible, the
|
||||
// PartitionRoot is really just a header adjacent to other data areas provided
|
||||
// by the allocator class.
|
||||
//
|
||||
// The PartitionRoot::Alloc() variant of the API has the following caveats:
|
||||
// - Allocations and frees against a single partition must be single threaded.
|
||||
// - Allocations must not exceed a max size, chosen at compile-time via a
|
||||
// templated parameter to PartitionAllocator.
|
||||
// - Allocation sizes must be aligned to the system pointer size.
|
||||
// - Allocations are bucketed exactly according to size.
|
||||
//
|
||||
// And for PartitionRootGeneric::Alloc():
|
||||
// - Multi-threaded use against a single partition is ok; locking is handled.
|
||||
// - Allocations of any arbitrary size can be handled (subject to a limit of
|
||||
// INT_MAX bytes for security reasons).
|
||||
// - Bucketing is by approximate size, for example an allocation of 4000 bytes
|
||||
// might be placed into a 4096-byte bucket. Bucket sizes are chosen to try and
|
||||
// keep worst-case waste to ~10%.
|
||||
//
|
||||
// The allocators are designed to be extremely fast, thanks to the following
|
||||
// properties and design:
|
||||
// - Just two single (reasonably predicatable) branches in the hot / fast path
|
||||
// for both allocating and (significantly) freeing.
|
||||
// - A minimal number of operations in the hot / fast path, with the slow paths
|
||||
// in separate functions, leading to the possibility of inlining.
|
||||
// - Each partition page (which is usually multiple physical pages) has a
|
||||
// metadata structure which allows fast mapping of free() address to an
|
||||
// underlying bucket.
|
||||
// - Supports a lock-free API for fast performance in single-threaded cases.
|
||||
// - The freelist for a given bucket is split across a number of partition
|
||||
// pages, enabling various simple tricks to try and minimize fragmentation.
|
||||
// - Fine-grained bucket sizes leading to less waste and better packing.
|
||||
//
|
||||
// The following security properties could be investigated in the future:
|
||||
// - Per-object bucketing (instead of per-size) is mostly available at the API,
|
||||
// but not used yet.
|
||||
// - No randomness of freelist entries or bucket position.
|
||||
// - Better checking for wild pointers in free().
|
||||
// - Better freelist masking function to guarantee fault on 32-bit.
|
||||
|
||||
#include <limits.h>
|
||||
#include <string.h>
|
||||
|
||||
#include "base/allocator/partition_allocator/memory_reclaimer.h"
|
||||
#include "base/allocator/partition_allocator/page_allocator.h"
|
||||
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
|
||||
#include "base/allocator/partition_allocator/partition_bucket.h"
|
||||
#include "base/allocator/partition_allocator/partition_cookie.h"
|
||||
#include "base/allocator/partition_allocator/partition_page.h"
|
||||
#include "base/allocator/partition_allocator/partition_root_base.h"
|
||||
#include "base/allocator/partition_allocator/spin_lock.h"
|
||||
#include "base/base_export.h"
|
||||
#include "base/bits.h"
|
||||
#include "base/compiler_specific.h"
|
||||
#include "base/logging.h"
|
||||
#include "base/stl_util.h"
|
||||
#include "base/sys_byteorder.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
#include <stdlib.h>
|
||||
#endif
|
||||
|
||||
// We use this to make MEMORY_TOOL_REPLACES_ALLOCATOR behave the same for max
|
||||
// size as other alloc code.
|
||||
#define CHECK_MAX_SIZE_OR_RETURN_NULLPTR(size, flags) \
|
||||
if (size > kGenericMaxDirectMapped) { \
|
||||
if (flags & PartitionAllocReturnNull) { \
|
||||
return nullptr; \
|
||||
} \
|
||||
CHECK(false); \
|
||||
}
|
||||
|
||||
namespace base {
|
||||
|
||||
class PartitionStatsDumper;
|
||||
|
||||
enum PartitionPurgeFlags {
|
||||
// Decommitting the ring list of empty pages is reasonably fast.
|
||||
PartitionPurgeDecommitEmptyPages = 1 << 0,
|
||||
// Discarding unused system pages is slower, because it involves walking all
|
||||
// freelists in all active partition pages of all buckets >= system page
|
||||
// size. It often frees a similar amount of memory to decommitting the empty
|
||||
// pages, though.
|
||||
PartitionPurgeDiscardUnusedSystemPages = 1 << 1,
|
||||
};
|
||||
|
||||
// Never instantiate a PartitionRoot directly, instead use PartitionAlloc.
|
||||
struct BASE_EXPORT PartitionRoot : public internal::PartitionRootBase {
|
||||
PartitionRoot();
|
||||
~PartitionRoot() override;
|
||||
// This references the buckets OFF the edge of this struct. All uses of
|
||||
// PartitionRoot must have the bucket array come right after.
|
||||
//
|
||||
// The PartitionAlloc templated class ensures the following is correct.
|
||||
ALWAYS_INLINE internal::PartitionBucket* buckets() {
|
||||
return reinterpret_cast<internal::PartitionBucket*>(this + 1);
|
||||
}
|
||||
ALWAYS_INLINE const internal::PartitionBucket* buckets() const {
|
||||
return reinterpret_cast<const internal::PartitionBucket*>(this + 1);
|
||||
}
|
||||
|
||||
void Init(size_t bucket_count, size_t maximum_allocation);
|
||||
|
||||
ALWAYS_INLINE void* Alloc(size_t size, const char* type_name);
|
||||
ALWAYS_INLINE void* AllocFlags(int flags, size_t size, const char* type_name);
|
||||
|
||||
void PurgeMemory(int flags) override;
|
||||
|
||||
void DumpStats(const char* partition_name,
|
||||
bool is_light_dump,
|
||||
PartitionStatsDumper* dumper);
|
||||
};
|
||||
|
||||
// Never instantiate a PartitionRootGeneric directly, instead use
|
||||
// PartitionAllocatorGeneric.
|
||||
struct BASE_EXPORT PartitionRootGeneric : public internal::PartitionRootBase {
|
||||
PartitionRootGeneric();
|
||||
~PartitionRootGeneric() override;
|
||||
subtle::SpinLock lock;
|
||||
// Some pre-computed constants.
|
||||
size_t order_index_shifts[kBitsPerSizeT + 1] = {};
|
||||
size_t order_sub_index_masks[kBitsPerSizeT + 1] = {};
|
||||
// The bucket lookup table lets us map a size_t to a bucket quickly.
|
||||
// The trailing +1 caters for the overflow case for very large allocation
|
||||
// sizes. It is one flat array instead of a 2D array because in the 2D
|
||||
// world, we'd need to index array[blah][max+1] which risks undefined
|
||||
// behavior.
|
||||
internal::PartitionBucket*
|
||||
bucket_lookups[((kBitsPerSizeT + 1) * kGenericNumBucketsPerOrder) + 1] =
|
||||
{};
|
||||
internal::PartitionBucket buckets[kGenericNumBuckets] = {};
|
||||
|
||||
// Public API.
|
||||
void Init();
|
||||
|
||||
ALWAYS_INLINE void* Alloc(size_t size, const char* type_name);
|
||||
ALWAYS_INLINE void* AllocFlags(int flags, size_t size, const char* type_name);
|
||||
ALWAYS_INLINE void Free(void* ptr);
|
||||
|
||||
NOINLINE void* Realloc(void* ptr, size_t new_size, const char* type_name);
|
||||
// Overload that may return nullptr if reallocation isn't possible. In this
|
||||
// case, |ptr| remains valid.
|
||||
NOINLINE void* TryRealloc(void* ptr, size_t new_size, const char* type_name);
|
||||
|
||||
ALWAYS_INLINE size_t ActualSize(size_t size);
|
||||
|
||||
void PurgeMemory(int flags) override;
|
||||
|
||||
void DumpStats(const char* partition_name,
|
||||
bool is_light_dump,
|
||||
PartitionStatsDumper* partition_stats_dumper);
|
||||
};
|
||||
|
||||
// Struct used to retrieve total memory usage of a partition. Used by
|
||||
// PartitionStatsDumper implementation.
|
||||
struct PartitionMemoryStats {
|
||||
size_t total_mmapped_bytes; // Total bytes mmaped from the system.
|
||||
size_t total_committed_bytes; // Total size of commmitted pages.
|
||||
size_t total_resident_bytes; // Total bytes provisioned by the partition.
|
||||
size_t total_active_bytes; // Total active bytes in the partition.
|
||||
size_t total_decommittable_bytes; // Total bytes that could be decommitted.
|
||||
size_t total_discardable_bytes; // Total bytes that could be discarded.
|
||||
};
|
||||
|
||||
// Struct used to retrieve memory statistics about a partition bucket. Used by
|
||||
// PartitionStatsDumper implementation.
|
||||
struct PartitionBucketMemoryStats {
|
||||
bool is_valid; // Used to check if the stats is valid.
|
||||
bool is_direct_map; // True if this is a direct mapping; size will not be
|
||||
// unique.
|
||||
uint32_t bucket_slot_size; // The size of the slot in bytes.
|
||||
uint32_t allocated_page_size; // Total size the partition page allocated from
|
||||
// the system.
|
||||
uint32_t active_bytes; // Total active bytes used in the bucket.
|
||||
uint32_t resident_bytes; // Total bytes provisioned in the bucket.
|
||||
uint32_t decommittable_bytes; // Total bytes that could be decommitted.
|
||||
uint32_t discardable_bytes; // Total bytes that could be discarded.
|
||||
uint32_t num_full_pages; // Number of pages with all slots allocated.
|
||||
uint32_t num_active_pages; // Number of pages that have at least one
|
||||
// provisioned slot.
|
||||
uint32_t num_empty_pages; // Number of pages that are empty
|
||||
// but not decommitted.
|
||||
uint32_t num_decommitted_pages; // Number of pages that are empty
|
||||
// and decommitted.
|
||||
};
|
||||
|
||||
// Interface that is passed to PartitionDumpStats and
|
||||
// PartitionDumpStatsGeneric for using the memory statistics.
|
||||
class BASE_EXPORT PartitionStatsDumper {
|
||||
public:
|
||||
// Called to dump total memory used by partition, once per partition.
|
||||
virtual void PartitionDumpTotals(const char* partition_name,
|
||||
const PartitionMemoryStats*) = 0;
|
||||
|
||||
// Called to dump stats about buckets, for each bucket.
|
||||
virtual void PartitionsDumpBucketStats(const char* partition_name,
|
||||
const PartitionBucketMemoryStats*) = 0;
|
||||
};
|
||||
|
||||
BASE_EXPORT void PartitionAllocGlobalInit(OomFunction on_out_of_memory);
|
||||
|
||||
// PartitionAlloc supports setting hooks to observe allocations/frees as they
|
||||
// occur as well as 'override' hooks that allow overriding those operations.
|
||||
class BASE_EXPORT PartitionAllocHooks {
|
||||
public:
|
||||
// Log allocation and free events.
|
||||
typedef void AllocationObserverHook(void* address,
|
||||
size_t size,
|
||||
const char* type_name);
|
||||
typedef void FreeObserverHook(void* address);
|
||||
|
||||
// If it returns true, the allocation has been overridden with the pointer in
|
||||
// *out.
|
||||
typedef bool AllocationOverrideHook(void** out,
|
||||
int flags,
|
||||
size_t size,
|
||||
const char* type_name);
|
||||
// If it returns true, then the allocation was overridden and has been freed.
|
||||
typedef bool FreeOverrideHook(void* address);
|
||||
// If it returns true, the underlying allocation is overridden and *out holds
|
||||
// the size of the underlying allocation.
|
||||
typedef bool ReallocOverrideHook(size_t* out, void* address);
|
||||
|
||||
// To unhook, call Set*Hooks with nullptrs.
|
||||
static void SetObserverHooks(AllocationObserverHook* alloc_hook,
|
||||
FreeObserverHook* free_hook);
|
||||
static void SetOverrideHooks(AllocationOverrideHook* alloc_hook,
|
||||
FreeOverrideHook* free_hook,
|
||||
ReallocOverrideHook realloc_hook);
|
||||
|
||||
// Helper method to check whether hooks are enabled. This is an optimization
|
||||
// so that if a function needs to call observer and override hooks in two
|
||||
// different places this value can be cached and only loaded once.
|
||||
static bool AreHooksEnabled() {
|
||||
return hooks_enabled_.load(std::memory_order_relaxed);
|
||||
}
|
||||
|
||||
static void AllocationObserverHookIfEnabled(void* address,
|
||||
size_t size,
|
||||
const char* type_name);
|
||||
static bool AllocationOverrideHookIfEnabled(void** out,
|
||||
int flags,
|
||||
size_t size,
|
||||
const char* type_name);
|
||||
|
||||
static void FreeObserverHookIfEnabled(void* address);
|
||||
static bool FreeOverrideHookIfEnabled(void* address);
|
||||
|
||||
static void ReallocObserverHookIfEnabled(void* old_address,
|
||||
void* new_address,
|
||||
size_t size,
|
||||
const char* type_name);
|
||||
static bool ReallocOverrideHookIfEnabled(size_t* out, void* address);
|
||||
|
||||
private:
|
||||
// Single bool that is used to indicate whether observer or allocation hooks
|
||||
// are set to reduce the numbers of loads required to check whether hooking is
|
||||
// enabled.
|
||||
static std::atomic<bool> hooks_enabled_;
|
||||
|
||||
// Lock used to synchronize Set*Hooks calls.
|
||||
static std::atomic<AllocationObserverHook*> allocation_observer_hook_;
|
||||
static std::atomic<FreeObserverHook*> free_observer_hook_;
|
||||
|
||||
static std::atomic<AllocationOverrideHook*> allocation_override_hook_;
|
||||
static std::atomic<FreeOverrideHook*> free_override_hook_;
|
||||
static std::atomic<ReallocOverrideHook*> realloc_override_hook_;
|
||||
};
|
||||
|
||||
ALWAYS_INLINE void* PartitionRoot::Alloc(size_t size, const char* type_name) {
|
||||
return AllocFlags(0, size, type_name);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* PartitionRoot::AllocFlags(int flags,
|
||||
size_t size,
|
||||
const char* type_name) {
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
CHECK_MAX_SIZE_OR_RETURN_NULLPTR(size, flags);
|
||||
void* result = malloc(size);
|
||||
CHECK(result);
|
||||
return result;
|
||||
#else
|
||||
DCHECK(max_allocation == 0 || size <= max_allocation);
|
||||
void* result;
|
||||
const bool hooks_enabled = PartitionAllocHooks::AreHooksEnabled();
|
||||
if (UNLIKELY(hooks_enabled)) {
|
||||
if (PartitionAllocHooks::AllocationOverrideHookIfEnabled(&result, flags,
|
||||
size, type_name)) {
|
||||
PartitionAllocHooks::AllocationObserverHookIfEnabled(result, size,
|
||||
type_name);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
size_t requested_size = size;
|
||||
size = internal::PartitionCookieSizeAdjustAdd(size);
|
||||
DCHECK(initialized);
|
||||
size_t index = size >> kBucketShift;
|
||||
DCHECK(index < num_buckets);
|
||||
DCHECK(size == index << kBucketShift);
|
||||
internal::PartitionBucket* bucket = &buckets()[index];
|
||||
result = AllocFromBucket(bucket, flags, size);
|
||||
if (UNLIKELY(hooks_enabled)) {
|
||||
PartitionAllocHooks::AllocationObserverHookIfEnabled(result, requested_size,
|
||||
type_name);
|
||||
}
|
||||
return result;
|
||||
#endif // defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
}
|
||||
|
||||
ALWAYS_INLINE bool PartitionAllocSupportsGetSize() {
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
return false;
|
||||
#else
|
||||
return true;
|
||||
#endif
|
||||
}
|
||||
|
||||
ALWAYS_INLINE size_t PartitionAllocGetSize(void* ptr) {
|
||||
// No need to lock here. Only |ptr| being freed by another thread could
|
||||
// cause trouble, and the caller is responsible for that not happening.
|
||||
DCHECK(PartitionAllocSupportsGetSize());
|
||||
ptr = internal::PartitionCookieFreePointerAdjust(ptr);
|
||||
internal::PartitionPage* page = internal::PartitionPage::FromPointer(ptr);
|
||||
// TODO(palmer): See if we can afford to make this a CHECK.
|
||||
DCHECK(internal::PartitionRootBase::IsValidPage(page));
|
||||
size_t size = page->bucket->slot_size;
|
||||
return internal::PartitionCookieSizeAdjustSubtract(size);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionFree(void* ptr) {
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
free(ptr);
|
||||
#else
|
||||
// TODO(palmer): Check ptr alignment before continuing. Shall we do the check
|
||||
// inside PartitionCookieFreePointerAdjust?
|
||||
if (PartitionAllocHooks::AreHooksEnabled()) {
|
||||
PartitionAllocHooks::FreeObserverHookIfEnabled(ptr);
|
||||
if (PartitionAllocHooks::FreeOverrideHookIfEnabled(ptr))
|
||||
return;
|
||||
}
|
||||
|
||||
ptr = internal::PartitionCookieFreePointerAdjust(ptr);
|
||||
internal::PartitionPage* page = internal::PartitionPage::FromPointer(ptr);
|
||||
// TODO(palmer): See if we can afford to make this a CHECK.
|
||||
DCHECK(internal::PartitionRootBase::IsValidPage(page));
|
||||
page->Free(ptr);
|
||||
#endif
|
||||
}
|
||||
|
||||
ALWAYS_INLINE internal::PartitionBucket* PartitionGenericSizeToBucket(
|
||||
PartitionRootGeneric* root,
|
||||
size_t size) {
|
||||
size_t order = kBitsPerSizeT - bits::CountLeadingZeroBitsSizeT(size);
|
||||
// The order index is simply the next few bits after the most significant bit.
|
||||
size_t order_index = (size >> root->order_index_shifts[order]) &
|
||||
(kGenericNumBucketsPerOrder - 1);
|
||||
// And if the remaining bits are non-zero we must bump the bucket up.
|
||||
size_t sub_order_index = size & root->order_sub_index_masks[order];
|
||||
internal::PartitionBucket* bucket =
|
||||
root->bucket_lookups[(order << kGenericNumBucketsPerOrderBits) +
|
||||
order_index + !!sub_order_index];
|
||||
CHECK(bucket);
|
||||
DCHECK(!bucket->slot_size || bucket->slot_size >= size);
|
||||
DCHECK(!(bucket->slot_size % kGenericSmallestBucket));
|
||||
return bucket;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* PartitionAllocGenericFlags(PartitionRootGeneric* root,
|
||||
int flags,
|
||||
size_t size,
|
||||
const char* type_name) {
|
||||
DCHECK_LT(flags, PartitionAllocLastFlag << 1);
|
||||
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
CHECK_MAX_SIZE_OR_RETURN_NULLPTR(size, flags);
|
||||
const bool zero_fill = flags & PartitionAllocZeroFill;
|
||||
void* result = zero_fill ? calloc(1, size) : malloc(size);
|
||||
CHECK(result || flags & PartitionAllocReturnNull);
|
||||
return result;
|
||||
#else
|
||||
DCHECK(root->initialized);
|
||||
// Only SizeSpecificPartitionAllocator should use max_allocation.
|
||||
DCHECK(root->max_allocation == 0);
|
||||
void* result;
|
||||
const bool hooks_enabled = PartitionAllocHooks::AreHooksEnabled();
|
||||
if (UNLIKELY(hooks_enabled)) {
|
||||
if (PartitionAllocHooks::AllocationOverrideHookIfEnabled(&result, flags,
|
||||
size, type_name)) {
|
||||
PartitionAllocHooks::AllocationObserverHookIfEnabled(result, size,
|
||||
type_name);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
size_t requested_size = size;
|
||||
size = internal::PartitionCookieSizeAdjustAdd(size);
|
||||
internal::PartitionBucket* bucket = PartitionGenericSizeToBucket(root, size);
|
||||
{
|
||||
subtle::SpinLock::Guard guard(root->lock);
|
||||
result = root->AllocFromBucket(bucket, flags, size);
|
||||
}
|
||||
if (UNLIKELY(hooks_enabled)) {
|
||||
PartitionAllocHooks::AllocationObserverHookIfEnabled(result, requested_size,
|
||||
type_name);
|
||||
}
|
||||
|
||||
return result;
|
||||
#endif
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* PartitionRootGeneric::Alloc(size_t size,
|
||||
const char* type_name) {
|
||||
return PartitionAllocGenericFlags(this, 0, size, type_name);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* PartitionRootGeneric::AllocFlags(int flags,
|
||||
size_t size,
|
||||
const char* type_name) {
|
||||
return PartitionAllocGenericFlags(this, flags, size, type_name);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionRootGeneric::Free(void* ptr) {
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
free(ptr);
|
||||
#else
|
||||
DCHECK(initialized);
|
||||
|
||||
if (UNLIKELY(!ptr))
|
||||
return;
|
||||
|
||||
if (PartitionAllocHooks::AreHooksEnabled()) {
|
||||
PartitionAllocHooks::FreeObserverHookIfEnabled(ptr);
|
||||
if (PartitionAllocHooks::FreeOverrideHookIfEnabled(ptr))
|
||||
return;
|
||||
}
|
||||
|
||||
ptr = internal::PartitionCookieFreePointerAdjust(ptr);
|
||||
internal::PartitionPage* page = internal::PartitionPage::FromPointer(ptr);
|
||||
// TODO(palmer): See if we can afford to make this a CHECK.
|
||||
DCHECK(IsValidPage(page));
|
||||
{
|
||||
subtle::SpinLock::Guard guard(lock);
|
||||
page->Free(ptr);
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
BASE_EXPORT void* PartitionReallocGenericFlags(PartitionRootGeneric* root,
|
||||
int flags,
|
||||
void* ptr,
|
||||
size_t new_size,
|
||||
const char* type_name);
|
||||
|
||||
ALWAYS_INLINE size_t PartitionRootGeneric::ActualSize(size_t size) {
|
||||
#if defined(MEMORY_TOOL_REPLACES_ALLOCATOR)
|
||||
return size;
|
||||
#else
|
||||
DCHECK(initialized);
|
||||
size = internal::PartitionCookieSizeAdjustAdd(size);
|
||||
internal::PartitionBucket* bucket = PartitionGenericSizeToBucket(this, size);
|
||||
if (LIKELY(!bucket->is_direct_mapped())) {
|
||||
size = bucket->slot_size;
|
||||
} else if (size > kGenericMaxDirectMapped) {
|
||||
// Too large to allocate => return the size unchanged.
|
||||
} else {
|
||||
size = internal::PartitionBucket::get_direct_map_size(size);
|
||||
}
|
||||
return internal::PartitionCookieSizeAdjustSubtract(size);
|
||||
#endif
|
||||
}
|
||||
|
||||
template <size_t N>
|
||||
class SizeSpecificPartitionAllocator {
|
||||
public:
|
||||
SizeSpecificPartitionAllocator() {
|
||||
memset(actual_buckets_, 0,
|
||||
sizeof(internal::PartitionBucket) * base::size(actual_buckets_));
|
||||
}
|
||||
~SizeSpecificPartitionAllocator() {
|
||||
PartitionAllocMemoryReclaimer::Instance()->UnregisterPartition(
|
||||
&partition_root_);
|
||||
}
|
||||
static const size_t kMaxAllocation = N - kAllocationGranularity;
|
||||
static const size_t kNumBuckets = N / kAllocationGranularity;
|
||||
void init() {
|
||||
partition_root_.Init(kNumBuckets, kMaxAllocation);
|
||||
PartitionAllocMemoryReclaimer::Instance()->RegisterPartition(
|
||||
&partition_root_);
|
||||
}
|
||||
ALWAYS_INLINE PartitionRoot* root() { return &partition_root_; }
|
||||
|
||||
private:
|
||||
PartitionRoot partition_root_;
|
||||
internal::PartitionBucket actual_buckets_[kNumBuckets];
|
||||
};
|
||||
|
||||
class BASE_EXPORT PartitionAllocatorGeneric {
|
||||
public:
|
||||
PartitionAllocatorGeneric();
|
||||
~PartitionAllocatorGeneric() {
|
||||
PartitionAllocMemoryReclaimer::Instance()->UnregisterPartition(
|
||||
&partition_root_);
|
||||
}
|
||||
|
||||
void init() {
|
||||
partition_root_.Init();
|
||||
PartitionAllocMemoryReclaimer::Instance()->RegisterPartition(
|
||||
&partition_root_);
|
||||
}
|
||||
ALWAYS_INLINE PartitionRootGeneric* root() { return &partition_root_; }
|
||||
|
||||
private:
|
||||
PartitionRootGeneric partition_root_;
|
||||
};
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_H_
|
||||
|
|
@ -0,0 +1,188 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_CONSTANTS_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_CONSTANTS_H_
|
||||
|
||||
#include <limits.h>
|
||||
|
||||
#include "base/allocator/partition_allocator/page_allocator_constants.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
// Allocation granularity of sizeof(void*) bytes.
|
||||
static const size_t kAllocationGranularity = sizeof(void*);
|
||||
static const size_t kAllocationGranularityMask = kAllocationGranularity - 1;
|
||||
static const size_t kBucketShift = (kAllocationGranularity == 8) ? 3 : 2;
|
||||
|
||||
// Underlying partition storage pages (`PartitionPage`s) are a power-of-2 size.
|
||||
// It is typical for a `PartitionPage` to be based on multiple system pages.
|
||||
// Most references to "page" refer to `PartitionPage`s.
|
||||
//
|
||||
// *Super pages* are the underlying system allocations we make. Super pages
|
||||
// contain multiple partition pages and include space for a small amount of
|
||||
// metadata per partition page.
|
||||
//
|
||||
// Inside super pages, we store *slot spans*. A slot span is a continguous range
|
||||
// of one or more `PartitionPage`s that stores allocations of the same size.
|
||||
// Slot span sizes are adjusted depending on the allocation size, to make sure
|
||||
// the packing does not lead to unused (wasted) space at the end of the last
|
||||
// system page of the span. For our current maximum slot span size of 64 KiB and
|
||||
// other constant values, we pack _all_ `PartitionRootGeneric::Alloc` sizes
|
||||
// perfectly up against the end of a system page.
|
||||
|
||||
#if defined(_MIPS_ARCH_LOONGSON)
|
||||
static const size_t kPartitionPageShift = 16; // 64 KiB
|
||||
#elif defined(ARCH_CPU_PPC64)
|
||||
static const size_t kPartitionPageShift = 18; // 256 KiB
|
||||
#else
|
||||
static const size_t kPartitionPageShift = 14; // 16 KiB
|
||||
#endif
|
||||
static const size_t kPartitionPageSize = 1 << kPartitionPageShift;
|
||||
static const size_t kPartitionPageOffsetMask = kPartitionPageSize - 1;
|
||||
static const size_t kPartitionPageBaseMask = ~kPartitionPageOffsetMask;
|
||||
// TODO: Should this be 1 if defined(_MIPS_ARCH_LOONGSON)?
|
||||
static const size_t kMaxPartitionPagesPerSlotSpan = 4;
|
||||
|
||||
// To avoid fragmentation via never-used freelist entries, we hand out partition
|
||||
// freelist sections gradually, in units of the dominant system page size. What
|
||||
// we're actually doing is avoiding filling the full `PartitionPage` (16 KiB)
|
||||
// with freelist pointers right away. Writing freelist pointers will fault and
|
||||
// dirty a private page, which is very wasteful if we never actually store
|
||||
// objects there.
|
||||
|
||||
static const size_t kNumSystemPagesPerPartitionPage =
|
||||
kPartitionPageSize / kSystemPageSize;
|
||||
static const size_t kMaxSystemPagesPerSlotSpan =
|
||||
kNumSystemPagesPerPartitionPage * kMaxPartitionPagesPerSlotSpan;
|
||||
|
||||
// We reserve virtual address space in 2 MiB chunks (aligned to 2 MiB as well).
|
||||
// These chunks are called *super pages*. We do this so that we can store
|
||||
// metadata in the first few pages of each 2 MiB-aligned section. This makes
|
||||
// freeing memory very fast. We specifically choose 2 MiB because this virtual
|
||||
// address block represents a full but single PTE allocation on ARM, ia32 and
|
||||
// x64.
|
||||
//
|
||||
// The layout of the super page is as follows. The sizes below are the same for
|
||||
// 32- and 64-bit platforms.
|
||||
//
|
||||
// +-----------------------+
|
||||
// | Guard page (4 KiB) |
|
||||
// | Metadata page (4 KiB) |
|
||||
// | Guard pages (8 KiB) |
|
||||
// | Slot span |
|
||||
// | Slot span |
|
||||
// | ... |
|
||||
// | Slot span |
|
||||
// | Guard page (4 KiB) |
|
||||
// +-----------------------+
|
||||
//
|
||||
// Each slot span is a contiguous range of one or more `PartitionPage`s.
|
||||
//
|
||||
// The metadata page has the following format. Note that the `PartitionPage`
|
||||
// that is not at the head of a slot span is "unused". In other words, the
|
||||
// metadata for the slot span is stored only in the first `PartitionPage` of the
|
||||
// slot span. Metadata accesses to other `PartitionPage`s are redirected to the
|
||||
// first `PartitionPage`.
|
||||
//
|
||||
// +---------------------------------------------+
|
||||
// | SuperPageExtentEntry (32 B) |
|
||||
// | PartitionPage of slot span 1 (32 B, used) |
|
||||
// | PartitionPage of slot span 1 (32 B, unused) |
|
||||
// | PartitionPage of slot span 1 (32 B, unused) |
|
||||
// | PartitionPage of slot span 2 (32 B, used) |
|
||||
// | PartitionPage of slot span 3 (32 B, used) |
|
||||
// | ... |
|
||||
// | PartitionPage of slot span N (32 B, unused) |
|
||||
// +---------------------------------------------+
|
||||
//
|
||||
// A direct-mapped page has a similar layout to fake it looking like a super
|
||||
// page:
|
||||
//
|
||||
// +-----------------------+
|
||||
// | Guard page (4 KiB) |
|
||||
// | Metadata page (4 KiB) |
|
||||
// | Guard pages (8 KiB) |
|
||||
// | Direct mapped object |
|
||||
// | Guard page (4 KiB) |
|
||||
// +-----------------------+
|
||||
//
|
||||
// A direct-mapped page's metadata page has the following layout:
|
||||
//
|
||||
// +--------------------------------+
|
||||
// | SuperPageExtentEntry (32 B) |
|
||||
// | PartitionPage (32 B) |
|
||||
// | PartitionBucket (32 B) |
|
||||
// | PartitionDirectMapExtent (8 B) |
|
||||
// +--------------------------------+
|
||||
|
||||
static const size_t kSuperPageShift = 21; // 2 MiB
|
||||
static const size_t kSuperPageSize = 1 << kSuperPageShift;
|
||||
static const size_t kSuperPageOffsetMask = kSuperPageSize - 1;
|
||||
static const size_t kSuperPageBaseMask = ~kSuperPageOffsetMask;
|
||||
static const size_t kNumPartitionPagesPerSuperPage =
|
||||
kSuperPageSize / kPartitionPageSize;
|
||||
|
||||
// The following kGeneric* constants apply to the generic variants of the API.
|
||||
// The "order" of an allocation is closely related to the power-of-1 size of the
|
||||
// allocation. More precisely, the order is the bit index of the
|
||||
// most-significant-bit in the allocation size, where the bit numbers starts at
|
||||
// index 1 for the least-significant-bit.
|
||||
//
|
||||
// In terms of allocation sizes, order 0 covers 0, order 1 covers 1, order 2
|
||||
// covers 2->3, order 3 covers 4->7, order 4 covers 8->15.
|
||||
|
||||
static const size_t kGenericMinBucketedOrder = 4; // 8 bytes.
|
||||
// The largest bucketed order is 1 << (20 - 1), storing [512 KiB, 1 MiB):
|
||||
static const size_t kGenericMaxBucketedOrder = 20;
|
||||
static const size_t kGenericNumBucketedOrders =
|
||||
(kGenericMaxBucketedOrder - kGenericMinBucketedOrder) + 1;
|
||||
// Eight buckets per order (for the higher orders), e.g. order 8 is 128, 144,
|
||||
// 160, ..., 240:
|
||||
static const size_t kGenericNumBucketsPerOrderBits = 3;
|
||||
static const size_t kGenericNumBucketsPerOrder =
|
||||
1 << kGenericNumBucketsPerOrderBits;
|
||||
static const size_t kGenericNumBuckets =
|
||||
kGenericNumBucketedOrders * kGenericNumBucketsPerOrder;
|
||||
static const size_t kGenericSmallestBucket = 1
|
||||
<< (kGenericMinBucketedOrder - 1);
|
||||
static const size_t kGenericMaxBucketSpacing =
|
||||
1 << ((kGenericMaxBucketedOrder - 1) - kGenericNumBucketsPerOrderBits);
|
||||
static const size_t kGenericMaxBucketed =
|
||||
(1 << (kGenericMaxBucketedOrder - 1)) +
|
||||
((kGenericNumBucketsPerOrder - 1) * kGenericMaxBucketSpacing);
|
||||
// Limit when downsizing a direct mapping using `realloc`:
|
||||
static const size_t kGenericMinDirectMappedDownsize = kGenericMaxBucketed + 1;
|
||||
static const size_t kGenericMaxDirectMapped =
|
||||
(1UL << 31) + kPageAllocationGranularity; // 2 GiB plus 1 more page.
|
||||
static const size_t kBitsPerSizeT = sizeof(void*) * CHAR_BIT;
|
||||
|
||||
// Constant for the memory reclaim logic.
|
||||
static const size_t kMaxFreeableSpans = 16;
|
||||
|
||||
// If the total size in bytes of allocated but not committed pages exceeds this
|
||||
// value (probably it is a "out of virtual address space" crash), a special
|
||||
// crash stack trace is generated at
|
||||
// `PartitionOutOfMemoryWithLotsOfUncommitedPages`. This is to distinguish "out
|
||||
// of virtual address space" from "out of physical memory" in crash reports.
|
||||
static const size_t kReasonableSizeOfUnusedPages = 1024 * 1024 * 1024; // 1 GiB
|
||||
|
||||
// These byte values match tcmalloc.
|
||||
static const unsigned char kUninitializedByte = 0xAB;
|
||||
static const unsigned char kFreedByte = 0xCD;
|
||||
|
||||
// Flags for `PartitionAllocGenericFlags`.
|
||||
enum PartitionAllocFlags {
|
||||
PartitionAllocReturnNull = 1 << 0,
|
||||
PartitionAllocZeroFill = 1 << 1,
|
||||
|
||||
PartitionAllocLastFlag = PartitionAllocZeroFill
|
||||
};
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ALLOC_CONSTANTS_H_
|
||||
|
|
@ -0,0 +1,565 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_bucket.h"
|
||||
|
||||
#include "base/allocator/partition_allocator/oom.h"
|
||||
#include "base/allocator/partition_allocator/page_allocator.h"
|
||||
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
|
||||
#include "base/allocator/partition_allocator/partition_direct_map_extent.h"
|
||||
#include "base/allocator/partition_allocator/partition_oom.h"
|
||||
#include "base/allocator/partition_allocator/partition_page.h"
|
||||
#include "base/allocator/partition_allocator/partition_root_base.h"
|
||||
#include "base/logging.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
namespace {
|
||||
|
||||
ALWAYS_INLINE PartitionPage* PartitionDirectMap(PartitionRootBase* root,
|
||||
int flags,
|
||||
size_t raw_size) {
|
||||
size_t size = PartitionBucket::get_direct_map_size(raw_size);
|
||||
|
||||
// Because we need to fake looking like a super page, we need to allocate
|
||||
// a bunch of system pages more than "size":
|
||||
// - The first few system pages are the partition page in which the super
|
||||
// page metadata is stored. We fault just one system page out of a partition
|
||||
// page sized clump.
|
||||
// - We add a trailing guard page on 32-bit (on 64-bit we rely on the
|
||||
// massive address space plus randomization instead).
|
||||
size_t map_size = size + kPartitionPageSize;
|
||||
#if !defined(ARCH_CPU_64_BITS)
|
||||
map_size += kSystemPageSize;
|
||||
#endif
|
||||
// Round up to the allocation granularity.
|
||||
map_size += kPageAllocationGranularityOffsetMask;
|
||||
map_size &= kPageAllocationGranularityBaseMask;
|
||||
|
||||
char* ptr = reinterpret_cast<char*>(AllocPages(nullptr, map_size,
|
||||
kSuperPageSize, PageReadWrite,
|
||||
PageTag::kPartitionAlloc));
|
||||
if (UNLIKELY(!ptr))
|
||||
return nullptr;
|
||||
|
||||
size_t committed_page_size = size + kSystemPageSize;
|
||||
root->total_size_of_direct_mapped_pages += committed_page_size;
|
||||
root->IncreaseCommittedPages(committed_page_size);
|
||||
|
||||
char* slot = ptr + kPartitionPageSize;
|
||||
SetSystemPagesAccess(ptr + (kSystemPageSize * 2),
|
||||
kPartitionPageSize - (kSystemPageSize * 2),
|
||||
PageInaccessible);
|
||||
#if !defined(ARCH_CPU_64_BITS)
|
||||
SetSystemPagesAccess(ptr, kSystemPageSize, PageInaccessible);
|
||||
SetSystemPagesAccess(slot + size, kSystemPageSize, PageInaccessible);
|
||||
#endif
|
||||
|
||||
PartitionSuperPageExtentEntry* extent =
|
||||
reinterpret_cast<PartitionSuperPageExtentEntry*>(
|
||||
PartitionSuperPageToMetadataArea(ptr));
|
||||
extent->root = root;
|
||||
// The new structures are all located inside a fresh system page so they
|
||||
// will all be zeroed out. These DCHECKs are for documentation.
|
||||
DCHECK(!extent->super_page_base);
|
||||
DCHECK(!extent->super_pages_end);
|
||||
DCHECK(!extent->next);
|
||||
PartitionPage* page = PartitionPage::FromPointerNoAlignmentCheck(slot);
|
||||
PartitionBucket* bucket = reinterpret_cast<PartitionBucket*>(
|
||||
reinterpret_cast<char*>(page) + (kPageMetadataSize * 2));
|
||||
DCHECK(!page->next_page);
|
||||
DCHECK(!page->num_allocated_slots);
|
||||
DCHECK(!page->num_unprovisioned_slots);
|
||||
DCHECK(!page->page_offset);
|
||||
DCHECK(!page->empty_cache_index);
|
||||
page->bucket = bucket;
|
||||
page->freelist_head = reinterpret_cast<PartitionFreelistEntry*>(slot);
|
||||
PartitionFreelistEntry* next_entry =
|
||||
reinterpret_cast<PartitionFreelistEntry*>(slot);
|
||||
next_entry->next = PartitionFreelistEntry::Encode(nullptr);
|
||||
|
||||
DCHECK(!bucket->active_pages_head);
|
||||
DCHECK(!bucket->empty_pages_head);
|
||||
DCHECK(!bucket->decommitted_pages_head);
|
||||
DCHECK(!bucket->num_system_pages_per_slot_span);
|
||||
DCHECK(!bucket->num_full_pages);
|
||||
bucket->slot_size = size;
|
||||
|
||||
PartitionDirectMapExtent* map_extent =
|
||||
PartitionDirectMapExtent::FromPage(page);
|
||||
map_extent->map_size = map_size - kPartitionPageSize - kSystemPageSize;
|
||||
map_extent->bucket = bucket;
|
||||
|
||||
// Maintain the doubly-linked list of all direct mappings.
|
||||
map_extent->next_extent = root->direct_map_list;
|
||||
if (map_extent->next_extent)
|
||||
map_extent->next_extent->prev_extent = map_extent;
|
||||
map_extent->prev_extent = nullptr;
|
||||
root->direct_map_list = map_extent;
|
||||
|
||||
return page;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// static
|
||||
PartitionBucket PartitionBucket::sentinel_bucket_;
|
||||
|
||||
PartitionBucket* PartitionBucket::get_sentinel_bucket() {
|
||||
return &sentinel_bucket_;
|
||||
}
|
||||
|
||||
// TODO(ajwong): This seems to interact badly with
|
||||
// get_pages_per_slot_span() which rounds the value from this up to a
|
||||
// multiple of kNumSystemPagesPerPartitionPage (aka 4) anyways.
|
||||
// http://crbug.com/776537
|
||||
//
|
||||
// TODO(ajwong): The waste calculation seems wrong. The PTE usage should cover
|
||||
// both used and unsed pages.
|
||||
// http://crbug.com/776537
|
||||
uint8_t PartitionBucket::get_system_pages_per_slot_span() {
|
||||
// This works out reasonably for the current bucket sizes of the generic
|
||||
// allocator, and the current values of partition page size and constants.
|
||||
// Specifically, we have enough room to always pack the slots perfectly into
|
||||
// some number of system pages. The only waste is the waste associated with
|
||||
// unfaulted pages (i.e. wasted address space).
|
||||
// TODO: we end up using a lot of system pages for very small sizes. For
|
||||
// example, we'll use 12 system pages for slot size 24. The slot size is
|
||||
// so small that the waste would be tiny with just 4, or 1, system pages.
|
||||
// Later, we can investigate whether there are anti-fragmentation benefits
|
||||
// to using fewer system pages.
|
||||
double best_waste_ratio = 1.0f;
|
||||
uint16_t best_pages = 0;
|
||||
if (slot_size > kMaxSystemPagesPerSlotSpan * kSystemPageSize) {
|
||||
// TODO(ajwong): Why is there a DCHECK here for this?
|
||||
// http://crbug.com/776537
|
||||
DCHECK(!(slot_size % kSystemPageSize));
|
||||
best_pages = static_cast<uint16_t>(slot_size / kSystemPageSize);
|
||||
// TODO(ajwong): Should this be checking against
|
||||
// kMaxSystemPagesPerSlotSpan or numeric_limits<uint8_t>::max?
|
||||
// http://crbug.com/776537
|
||||
CHECK(best_pages < (1 << 8));
|
||||
return static_cast<uint8_t>(best_pages);
|
||||
}
|
||||
DCHECK(slot_size <= kMaxSystemPagesPerSlotSpan * kSystemPageSize);
|
||||
for (uint16_t i = kNumSystemPagesPerPartitionPage - 1;
|
||||
i <= kMaxSystemPagesPerSlotSpan; ++i) {
|
||||
size_t page_size = kSystemPageSize * i;
|
||||
size_t num_slots = page_size / slot_size;
|
||||
size_t waste = page_size - (num_slots * slot_size);
|
||||
// Leaving a page unfaulted is not free; the page will occupy an empty page
|
||||
// table entry. Make a simple attempt to account for that.
|
||||
//
|
||||
// TODO(ajwong): This looks wrong. PTEs are allocated for all pages
|
||||
// regardless of whether or not they are wasted. Should it just
|
||||
// be waste += i * sizeof(void*)?
|
||||
// http://crbug.com/776537
|
||||
size_t num_remainder_pages = i & (kNumSystemPagesPerPartitionPage - 1);
|
||||
size_t num_unfaulted_pages =
|
||||
num_remainder_pages
|
||||
? (kNumSystemPagesPerPartitionPage - num_remainder_pages)
|
||||
: 0;
|
||||
waste += sizeof(void*) * num_unfaulted_pages;
|
||||
double waste_ratio =
|
||||
static_cast<double>(waste) / static_cast<double>(page_size);
|
||||
if (waste_ratio < best_waste_ratio) {
|
||||
best_waste_ratio = waste_ratio;
|
||||
best_pages = i;
|
||||
}
|
||||
}
|
||||
DCHECK(best_pages > 0);
|
||||
CHECK(best_pages <= kMaxSystemPagesPerSlotSpan);
|
||||
return static_cast<uint8_t>(best_pages);
|
||||
}
|
||||
|
||||
void PartitionBucket::Init(uint32_t new_slot_size) {
|
||||
slot_size = new_slot_size;
|
||||
active_pages_head = PartitionPage::get_sentinel_page();
|
||||
empty_pages_head = nullptr;
|
||||
decommitted_pages_head = nullptr;
|
||||
num_full_pages = 0;
|
||||
num_system_pages_per_slot_span = get_system_pages_per_slot_span();
|
||||
}
|
||||
|
||||
NOINLINE void PartitionBucket::OnFull() {
|
||||
OOM_CRASH(0);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* PartitionBucket::AllocNewSlotSpan(
|
||||
PartitionRootBase* root,
|
||||
int flags,
|
||||
uint16_t num_partition_pages) {
|
||||
DCHECK(!(reinterpret_cast<uintptr_t>(root->next_partition_page) %
|
||||
kPartitionPageSize));
|
||||
DCHECK(!(reinterpret_cast<uintptr_t>(root->next_partition_page_end) %
|
||||
kPartitionPageSize));
|
||||
DCHECK(num_partition_pages <= kNumPartitionPagesPerSuperPage);
|
||||
size_t total_size = kPartitionPageSize * num_partition_pages;
|
||||
size_t num_partition_pages_left =
|
||||
(root->next_partition_page_end - root->next_partition_page) >>
|
||||
kPartitionPageShift;
|
||||
if (LIKELY(num_partition_pages_left >= num_partition_pages)) {
|
||||
// In this case, we can still hand out pages from the current super page
|
||||
// allocation.
|
||||
char* ret = root->next_partition_page;
|
||||
|
||||
// Fresh System Pages in the SuperPages are decommited. Commit them
|
||||
// before vending them back.
|
||||
SetSystemPagesAccess(ret, total_size, PageReadWrite);
|
||||
|
||||
root->next_partition_page += total_size;
|
||||
root->IncreaseCommittedPages(total_size);
|
||||
return ret;
|
||||
}
|
||||
|
||||
// Need a new super page. We want to allocate super pages in a continguous
|
||||
// address region as much as possible. This is important for not causing
|
||||
// page table bloat and not fragmenting address spaces in 32 bit
|
||||
// architectures.
|
||||
char* requested_address = root->next_super_page;
|
||||
char* super_page = reinterpret_cast<char*>(
|
||||
AllocPages(requested_address, kSuperPageSize, kSuperPageSize,
|
||||
PageReadWrite, PageTag::kPartitionAlloc));
|
||||
if (UNLIKELY(!super_page))
|
||||
return nullptr;
|
||||
|
||||
root->total_size_of_super_pages += kSuperPageSize;
|
||||
root->IncreaseCommittedPages(total_size);
|
||||
|
||||
// |total_size| MUST be less than kSuperPageSize - (kPartitionPageSize*2).
|
||||
// This is a trustworthy value because num_partition_pages is not user
|
||||
// controlled.
|
||||
//
|
||||
// TODO(ajwong): Introduce a DCHECK.
|
||||
root->next_super_page = super_page + kSuperPageSize;
|
||||
char* ret = super_page + kPartitionPageSize;
|
||||
root->next_partition_page = ret + total_size;
|
||||
root->next_partition_page_end = root->next_super_page - kPartitionPageSize;
|
||||
// Make the first partition page in the super page a guard page, but leave a
|
||||
// hole in the middle.
|
||||
// This is where we put page metadata and also a tiny amount of extent
|
||||
// metadata.
|
||||
SetSystemPagesAccess(super_page, kSystemPageSize, PageInaccessible);
|
||||
SetSystemPagesAccess(super_page + (kSystemPageSize * 2),
|
||||
kPartitionPageSize - (kSystemPageSize * 2),
|
||||
PageInaccessible);
|
||||
// SetSystemPagesAccess(super_page + (kSuperPageSize -
|
||||
// kPartitionPageSize),
|
||||
// kPartitionPageSize, PageInaccessible);
|
||||
// All remaining slotspans for the unallocated PartitionPages inside the
|
||||
// SuperPage are conceptually decommitted. Correctly set the state here
|
||||
// so they do not occupy resources.
|
||||
//
|
||||
// TODO(ajwong): Refactor Page Allocator API so the SuperPage comes in
|
||||
// decommited initially.
|
||||
SetSystemPagesAccess(super_page + kPartitionPageSize + total_size,
|
||||
(kSuperPageSize - kPartitionPageSize - total_size),
|
||||
PageInaccessible);
|
||||
|
||||
// If we were after a specific address, but didn't get it, assume that
|
||||
// the system chose a lousy address. Here most OS'es have a default
|
||||
// algorithm that isn't randomized. For example, most Linux
|
||||
// distributions will allocate the mapping directly before the last
|
||||
// successful mapping, which is far from random. So we just get fresh
|
||||
// randomness for the next mapping attempt.
|
||||
if (requested_address && requested_address != super_page)
|
||||
root->next_super_page = nullptr;
|
||||
|
||||
// We allocated a new super page so update super page metadata.
|
||||
// First check if this is a new extent or not.
|
||||
PartitionSuperPageExtentEntry* latest_extent =
|
||||
reinterpret_cast<PartitionSuperPageExtentEntry*>(
|
||||
PartitionSuperPageToMetadataArea(super_page));
|
||||
// By storing the root in every extent metadata object, we have a fast way
|
||||
// to go from a pointer within the partition to the root object.
|
||||
latest_extent->root = root;
|
||||
// Most new extents will be part of a larger extent, and these three fields
|
||||
// are unused, but we initialize them to 0 so that we get a clear signal
|
||||
// in case they are accidentally used.
|
||||
latest_extent->super_page_base = nullptr;
|
||||
latest_extent->super_pages_end = nullptr;
|
||||
latest_extent->next = nullptr;
|
||||
|
||||
PartitionSuperPageExtentEntry* current_extent = root->current_extent;
|
||||
bool is_new_extent = (super_page != requested_address);
|
||||
if (UNLIKELY(is_new_extent)) {
|
||||
if (UNLIKELY(!current_extent)) {
|
||||
DCHECK(!root->first_extent);
|
||||
root->first_extent = latest_extent;
|
||||
} else {
|
||||
DCHECK(current_extent->super_page_base);
|
||||
current_extent->next = latest_extent;
|
||||
}
|
||||
root->current_extent = latest_extent;
|
||||
latest_extent->super_page_base = super_page;
|
||||
latest_extent->super_pages_end = super_page + kSuperPageSize;
|
||||
} else {
|
||||
// We allocated next to an existing extent so just nudge the size up a
|
||||
// little.
|
||||
DCHECK(current_extent->super_pages_end);
|
||||
current_extent->super_pages_end += kSuperPageSize;
|
||||
DCHECK(ret >= current_extent->super_page_base &&
|
||||
ret < current_extent->super_pages_end);
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE uint16_t PartitionBucket::get_pages_per_slot_span() {
|
||||
// Rounds up to nearest multiple of kNumSystemPagesPerPartitionPage.
|
||||
return (num_system_pages_per_slot_span +
|
||||
(kNumSystemPagesPerPartitionPage - 1)) /
|
||||
kNumSystemPagesPerPartitionPage;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionBucket::InitializeSlotSpan(PartitionPage* page) {
|
||||
// The bucket never changes. We set it up once.
|
||||
page->bucket = this;
|
||||
page->empty_cache_index = -1;
|
||||
|
||||
page->Reset();
|
||||
|
||||
// If this page has just a single slot, do not set up page offsets for any
|
||||
// page metadata other than the first one. This ensures that attempts to
|
||||
// touch invalid page metadata fail.
|
||||
if (page->num_unprovisioned_slots == 1)
|
||||
return;
|
||||
|
||||
uint16_t num_partition_pages = get_pages_per_slot_span();
|
||||
char* page_char_ptr = reinterpret_cast<char*>(page);
|
||||
for (uint16_t i = 1; i < num_partition_pages; ++i) {
|
||||
page_char_ptr += kPageMetadataSize;
|
||||
PartitionPage* secondary_page =
|
||||
reinterpret_cast<PartitionPage*>(page_char_ptr);
|
||||
secondary_page->page_offset = i;
|
||||
}
|
||||
}
|
||||
|
||||
ALWAYS_INLINE char* PartitionBucket::AllocAndFillFreelist(PartitionPage* page) {
|
||||
DCHECK(page != PartitionPage::get_sentinel_page());
|
||||
uint16_t num_slots = page->num_unprovisioned_slots;
|
||||
DCHECK(num_slots);
|
||||
// We should only get here when _every_ slot is either used or unprovisioned.
|
||||
// (The third state is "on the freelist". If we have a non-empty freelist, we
|
||||
// should not get here.)
|
||||
DCHECK(num_slots + page->num_allocated_slots == get_slots_per_span());
|
||||
// Similarly, make explicitly sure that the freelist is empty.
|
||||
DCHECK(!page->freelist_head);
|
||||
DCHECK(page->num_allocated_slots >= 0);
|
||||
|
||||
size_t size = slot_size;
|
||||
char* base = reinterpret_cast<char*>(PartitionPage::ToPointer(page));
|
||||
char* return_object = base + (size * page->num_allocated_slots);
|
||||
char* first_freelist_pointer = return_object + size;
|
||||
char* first_freelist_pointer_extent =
|
||||
first_freelist_pointer + sizeof(PartitionFreelistEntry*);
|
||||
// Our goal is to fault as few system pages as possible. We calculate the
|
||||
// page containing the "end" of the returned slot, and then allow freelist
|
||||
// pointers to be written up to the end of that page.
|
||||
char* sub_page_limit = reinterpret_cast<char*>(
|
||||
RoundUpToSystemPage(reinterpret_cast<size_t>(first_freelist_pointer)));
|
||||
char* slots_limit = return_object + (size * num_slots);
|
||||
char* freelist_limit = sub_page_limit;
|
||||
if (UNLIKELY(slots_limit < freelist_limit))
|
||||
freelist_limit = slots_limit;
|
||||
|
||||
uint16_t num_new_freelist_entries = 0;
|
||||
if (LIKELY(first_freelist_pointer_extent <= freelist_limit)) {
|
||||
// Only consider used space in the slot span. If we consider wasted
|
||||
// space, we may get an off-by-one when a freelist pointer fits in the
|
||||
// wasted space, but a slot does not.
|
||||
// We know we can fit at least one freelist pointer.
|
||||
num_new_freelist_entries = 1;
|
||||
// Any further entries require space for the whole slot span.
|
||||
num_new_freelist_entries += static_cast<uint16_t>(
|
||||
(freelist_limit - first_freelist_pointer_extent) / size);
|
||||
}
|
||||
|
||||
// We always return an object slot -- that's the +1 below.
|
||||
// We do not neccessarily create any new freelist entries, because we cross
|
||||
// sub page boundaries frequently for large bucket sizes.
|
||||
DCHECK(num_new_freelist_entries + 1 <= num_slots);
|
||||
num_slots -= (num_new_freelist_entries + 1);
|
||||
page->num_unprovisioned_slots = num_slots;
|
||||
page->num_allocated_slots++;
|
||||
|
||||
if (LIKELY(num_new_freelist_entries)) {
|
||||
char* freelist_pointer = first_freelist_pointer;
|
||||
PartitionFreelistEntry* entry =
|
||||
reinterpret_cast<PartitionFreelistEntry*>(freelist_pointer);
|
||||
page->freelist_head = entry;
|
||||
while (--num_new_freelist_entries) {
|
||||
freelist_pointer += size;
|
||||
PartitionFreelistEntry* next_entry =
|
||||
reinterpret_cast<PartitionFreelistEntry*>(freelist_pointer);
|
||||
entry->next = PartitionFreelistEntry::Encode(next_entry);
|
||||
entry = next_entry;
|
||||
}
|
||||
entry->next = PartitionFreelistEntry::Encode(nullptr);
|
||||
} else {
|
||||
page->freelist_head = nullptr;
|
||||
}
|
||||
return return_object;
|
||||
}
|
||||
|
||||
bool PartitionBucket::SetNewActivePage() {
|
||||
PartitionPage* page = active_pages_head;
|
||||
if (page == PartitionPage::get_sentinel_page())
|
||||
return false;
|
||||
|
||||
PartitionPage* next_page;
|
||||
|
||||
for (; page; page = next_page) {
|
||||
next_page = page->next_page;
|
||||
DCHECK(page->bucket == this);
|
||||
DCHECK(page != empty_pages_head);
|
||||
DCHECK(page != decommitted_pages_head);
|
||||
|
||||
if (LIKELY(page->is_active())) {
|
||||
// This page is usable because it has freelist entries, or has
|
||||
// unprovisioned slots we can create freelist entries from.
|
||||
active_pages_head = page;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Deal with empty and decommitted pages.
|
||||
if (LIKELY(page->is_empty())) {
|
||||
page->next_page = empty_pages_head;
|
||||
empty_pages_head = page;
|
||||
} else if (LIKELY(page->is_decommitted())) {
|
||||
page->next_page = decommitted_pages_head;
|
||||
decommitted_pages_head = page;
|
||||
} else {
|
||||
DCHECK(page->is_full());
|
||||
// If we get here, we found a full page. Skip over it too, and also
|
||||
// tag it as full (via a negative value). We need it tagged so that
|
||||
// free'ing can tell, and move it back into the active page list.
|
||||
page->num_allocated_slots = -page->num_allocated_slots;
|
||||
++num_full_pages;
|
||||
// num_full_pages is a uint16_t for efficient packing so guard against
|
||||
// overflow to be safe.
|
||||
if (UNLIKELY(!num_full_pages))
|
||||
OnFull();
|
||||
// Not necessary but might help stop accidents.
|
||||
page->next_page = nullptr;
|
||||
}
|
||||
}
|
||||
|
||||
active_pages_head = PartitionPage::get_sentinel_page();
|
||||
return false;
|
||||
}
|
||||
|
||||
void* PartitionBucket::SlowPathAlloc(PartitionRootBase* root,
|
||||
int flags,
|
||||
size_t size,
|
||||
bool* is_already_zeroed) {
|
||||
// The slow path is called when the freelist is empty.
|
||||
DCHECK(!active_pages_head->freelist_head);
|
||||
|
||||
PartitionPage* new_page = nullptr;
|
||||
*is_already_zeroed = false;
|
||||
|
||||
// For the PartitionRootGeneric::Alloc() API, we have a bunch of buckets
|
||||
// marked as special cases. We bounce them through to the slow path so that
|
||||
// we can still have a blazing fast hot path due to lack of corner-case
|
||||
// branches.
|
||||
//
|
||||
// Note: The ordering of the conditionals matter! In particular,
|
||||
// SetNewActivePage() has a side-effect even when returning
|
||||
// false where it sweeps the active page list and may move things into
|
||||
// the empty or decommitted lists which affects the subsequent conditional.
|
||||
bool return_null = flags & PartitionAllocReturnNull;
|
||||
if (UNLIKELY(is_direct_mapped())) {
|
||||
DCHECK(size > kGenericMaxBucketed);
|
||||
DCHECK(this == get_sentinel_bucket());
|
||||
DCHECK(active_pages_head == PartitionPage::get_sentinel_page());
|
||||
if (size > kGenericMaxDirectMapped) {
|
||||
if (return_null)
|
||||
return nullptr;
|
||||
PartitionExcessiveAllocationSize(size);
|
||||
}
|
||||
new_page = PartitionDirectMap(root, flags, size);
|
||||
*is_already_zeroed = true;
|
||||
} else if (LIKELY(SetNewActivePage())) {
|
||||
// First, did we find an active page in the active pages list?
|
||||
new_page = active_pages_head;
|
||||
DCHECK(new_page->is_active());
|
||||
} else if (LIKELY(empty_pages_head != nullptr) ||
|
||||
LIKELY(decommitted_pages_head != nullptr)) {
|
||||
// Second, look in our lists of empty and decommitted pages.
|
||||
// Check empty pages first, which are preferred, but beware that an
|
||||
// empty page might have been decommitted.
|
||||
while (LIKELY((new_page = empty_pages_head) != nullptr)) {
|
||||
DCHECK(new_page->bucket == this);
|
||||
DCHECK(new_page->is_empty() || new_page->is_decommitted());
|
||||
empty_pages_head = new_page->next_page;
|
||||
// Accept the empty page unless it got decommitted.
|
||||
if (new_page->freelist_head) {
|
||||
new_page->next_page = nullptr;
|
||||
break;
|
||||
}
|
||||
DCHECK(new_page->is_decommitted());
|
||||
new_page->next_page = decommitted_pages_head;
|
||||
decommitted_pages_head = new_page;
|
||||
}
|
||||
if (UNLIKELY(!new_page) && LIKELY(decommitted_pages_head != nullptr)) {
|
||||
new_page = decommitted_pages_head;
|
||||
DCHECK(new_page->bucket == this);
|
||||
DCHECK(new_page->is_decommitted());
|
||||
decommitted_pages_head = new_page->next_page;
|
||||
void* addr = PartitionPage::ToPointer(new_page);
|
||||
root->RecommitSystemPages(addr, new_page->bucket->get_bytes_per_span());
|
||||
new_page->Reset();
|
||||
// TODO(https://crbug.com/890752): Optimizing here might cause pages to
|
||||
// not be zeroed.
|
||||
// *is_already_zeroed = true;
|
||||
}
|
||||
DCHECK(new_page);
|
||||
} else {
|
||||
// Third. If we get here, we need a brand new page.
|
||||
uint16_t num_partition_pages = get_pages_per_slot_span();
|
||||
void* raw_pages = AllocNewSlotSpan(root, flags, num_partition_pages);
|
||||
if (LIKELY(raw_pages != nullptr)) {
|
||||
new_page = PartitionPage::FromPointerNoAlignmentCheck(raw_pages);
|
||||
InitializeSlotSpan(new_page);
|
||||
// TODO(https://crbug.com/890752): Optimizing here causes pages to not be
|
||||
// zeroed on at least macOS.
|
||||
// *is_already_zeroed = true;
|
||||
}
|
||||
}
|
||||
|
||||
// Bail if we had a memory allocation failure.
|
||||
if (UNLIKELY(!new_page)) {
|
||||
DCHECK(active_pages_head == PartitionPage::get_sentinel_page());
|
||||
if (return_null)
|
||||
return nullptr;
|
||||
root->OutOfMemory(size);
|
||||
}
|
||||
|
||||
// TODO(ajwong): Is there a way to avoid the reading of bucket here?
|
||||
// It seems like in many of the conditional branches above, |this| ==
|
||||
// |new_page->bucket|. Maybe pull this into another function?
|
||||
PartitionBucket* bucket = new_page->bucket;
|
||||
DCHECK(bucket != get_sentinel_bucket());
|
||||
bucket->active_pages_head = new_page;
|
||||
new_page->set_raw_size(size);
|
||||
|
||||
// If we found an active page with free slots, or an empty page, we have a
|
||||
// usable freelist head.
|
||||
if (LIKELY(new_page->freelist_head != nullptr)) {
|
||||
PartitionFreelistEntry* entry = new_page->freelist_head;
|
||||
PartitionFreelistEntry* new_head =
|
||||
EncodedPartitionFreelistEntry::Decode(entry->next);
|
||||
new_page->freelist_head = new_head;
|
||||
new_page->num_allocated_slots++;
|
||||
return entry;
|
||||
}
|
||||
// Otherwise, we need to build the freelist.
|
||||
DCHECK(new_page->num_unprovisioned_slots);
|
||||
return AllocAndFillFreelist(new_page);
|
||||
}
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,129 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_BUCKET_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_BUCKET_H_
|
||||
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
|
||||
#include "base/base_export.h"
|
||||
#include "base/compiler_specific.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
struct PartitionPage;
|
||||
struct PartitionRootBase;
|
||||
|
||||
struct PartitionBucket {
|
||||
// Accessed most in hot path => goes first.
|
||||
PartitionPage* active_pages_head;
|
||||
|
||||
PartitionPage* empty_pages_head;
|
||||
PartitionPage* decommitted_pages_head;
|
||||
uint32_t slot_size;
|
||||
uint32_t num_system_pages_per_slot_span : 8;
|
||||
uint32_t num_full_pages : 24;
|
||||
|
||||
// Public API.
|
||||
void Init(uint32_t new_slot_size);
|
||||
|
||||
// Sets |is_already_zeroed| to true if the allocation was satisfied by
|
||||
// requesting (a) new page(s) from the operating system, or false otherwise.
|
||||
// This enables an optimization for when callers use |PartitionAllocZeroFill|:
|
||||
// there is no need to call memset on fresh pages; the OS has already zeroed
|
||||
// them. (See |PartitionRootBase::AllocFromBucket|.)
|
||||
//
|
||||
// Note the matching Free() functions are in PartitionPage.
|
||||
BASE_EXPORT NOINLINE void* SlowPathAlloc(PartitionRootBase* root,
|
||||
int flags,
|
||||
size_t size,
|
||||
bool* is_already_zeroed);
|
||||
|
||||
ALWAYS_INLINE bool is_direct_mapped() const {
|
||||
return !num_system_pages_per_slot_span;
|
||||
}
|
||||
ALWAYS_INLINE size_t get_bytes_per_span() const {
|
||||
// TODO(ajwong): Change to CheckedMul. https://crbug.com/787153
|
||||
// https://crbug.com/680657
|
||||
return num_system_pages_per_slot_span * kSystemPageSize;
|
||||
}
|
||||
ALWAYS_INLINE uint16_t get_slots_per_span() const {
|
||||
// TODO(ajwong): Change to CheckedMul. https://crbug.com/787153
|
||||
// https://crbug.com/680657
|
||||
return static_cast<uint16_t>(get_bytes_per_span() / slot_size);
|
||||
}
|
||||
|
||||
static ALWAYS_INLINE size_t get_direct_map_size(size_t size) {
|
||||
// Caller must check that the size is not above the kGenericMaxDirectMapped
|
||||
// limit before calling. This also guards against integer overflow in the
|
||||
// calculation here.
|
||||
DCHECK(size <= kGenericMaxDirectMapped);
|
||||
return (size + kSystemPageOffsetMask) & kSystemPageBaseMask;
|
||||
}
|
||||
|
||||
// TODO(ajwong): Can this be made private? https://crbug.com/787153
|
||||
static PartitionBucket* get_sentinel_bucket();
|
||||
|
||||
// This helper function scans a bucket's active page list for a suitable new
|
||||
// active page. When it finds a suitable new active page (one that has
|
||||
// free slots and is not empty), it is set as the new active page. If there
|
||||
// is no suitable new active page, the current active page is set to
|
||||
// PartitionPage::get_sentinel_page(). As potential pages are scanned, they
|
||||
// are tidied up according to their state. Empty pages are swept on to the
|
||||
// empty page list, decommitted pages on to the decommitted page list and full
|
||||
// pages are unlinked from any list.
|
||||
//
|
||||
// This is where the guts of the bucket maintenance is done!
|
||||
bool SetNewActivePage();
|
||||
|
||||
private:
|
||||
static void OutOfMemory(const PartitionRootBase* root);
|
||||
static void OutOfMemoryWithLotsOfUncommitedPages();
|
||||
|
||||
static NOINLINE void OnFull();
|
||||
|
||||
// Returns a natural number of PartitionPages (calculated by
|
||||
// get_system_pages_per_slot_span()) to allocate from the current
|
||||
// SuperPage when the bucket runs out of slots.
|
||||
ALWAYS_INLINE uint16_t get_pages_per_slot_span();
|
||||
|
||||
// Returns the number of system pages in a slot span.
|
||||
//
|
||||
// The calculation attemps to find the best number of System Pages to
|
||||
// allocate for the given slot_size to minimize wasted space. It uses a
|
||||
// heuristic that looks at number of bytes wasted after the last slot and
|
||||
// attempts to account for the PTE usage of each System Page.
|
||||
uint8_t get_system_pages_per_slot_span();
|
||||
|
||||
// Allocates a new slot span with size |num_partition_pages| from the
|
||||
// current extent. Metadata within this slot span will be uninitialized.
|
||||
// Returns nullptr on error.
|
||||
ALWAYS_INLINE void* AllocNewSlotSpan(PartitionRootBase* root,
|
||||
int flags,
|
||||
uint16_t num_partition_pages);
|
||||
|
||||
// Each bucket allocates a slot span when it runs out of slots.
|
||||
// A slot span's size is equal to get_pages_per_slot_span() number of
|
||||
// PartitionPages. This function initializes all PartitionPage within the
|
||||
// span to point to the first PartitionPage which holds all the metadata
|
||||
// for the span and registers this bucket as the owner of the span. It does
|
||||
// NOT put the slots into the bucket's freelist.
|
||||
ALWAYS_INLINE void InitializeSlotSpan(PartitionPage* page);
|
||||
|
||||
// Allocates one slot from the given |page| and then adds the remainder to
|
||||
// the current bucket. If the |page| was freshly allocated, it must have been
|
||||
// passed through InitializeSlotSpan() first.
|
||||
ALWAYS_INLINE char* AllocAndFillFreelist(PartitionPage* page);
|
||||
|
||||
static PartitionBucket sentinel_bucket_;
|
||||
};
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_BUCKET_H_
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_COOKIE_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_COOKIE_H_
|
||||
|
||||
#include "base/compiler_specific.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
#if DCHECK_IS_ON()
|
||||
// Handles alignment up to XMM instructions on Intel.
|
||||
static constexpr size_t kCookieSize = 16;
|
||||
|
||||
static constexpr unsigned char kCookieValue[kCookieSize] = {
|
||||
0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xD0, 0x0D,
|
||||
0x13, 0x37, 0xF0, 0x05, 0xBA, 0x11, 0xAB, 0x1E};
|
||||
#endif
|
||||
|
||||
ALWAYS_INLINE void PartitionCookieCheckValue(void* ptr) {
|
||||
#if DCHECK_IS_ON()
|
||||
unsigned char* cookie_ptr = reinterpret_cast<unsigned char*>(ptr);
|
||||
for (size_t i = 0; i < kCookieSize; ++i, ++cookie_ptr)
|
||||
DCHECK(*cookie_ptr == kCookieValue[i]);
|
||||
#endif
|
||||
}
|
||||
|
||||
ALWAYS_INLINE size_t PartitionCookieSizeAdjustAdd(size_t size) {
|
||||
#if DCHECK_IS_ON()
|
||||
// Add space for cookies, checking for integer overflow. TODO(palmer):
|
||||
// Investigate the performance and code size implications of using
|
||||
// CheckedNumeric throughout PA.
|
||||
DCHECK(size + (2 * kCookieSize) > size);
|
||||
size += 2 * kCookieSize;
|
||||
#endif
|
||||
return size;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void* PartitionCookieFreePointerAdjust(void* ptr) {
|
||||
#if DCHECK_IS_ON()
|
||||
// The value given to the application is actually just after the cookie.
|
||||
ptr = static_cast<char*>(ptr) - kCookieSize;
|
||||
#endif
|
||||
return ptr;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE size_t PartitionCookieSizeAdjustSubtract(size_t size) {
|
||||
#if DCHECK_IS_ON()
|
||||
// Remove space for cookies.
|
||||
DCHECK(size >= 2 * kCookieSize);
|
||||
size -= 2 * kCookieSize;
|
||||
#endif
|
||||
return size;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionCookieWriteValue(void* ptr) {
|
||||
#if DCHECK_IS_ON()
|
||||
unsigned char* cookie_ptr = reinterpret_cast<unsigned char*>(ptr);
|
||||
for (size_t i = 0; i < kCookieSize; ++i, ++cookie_ptr)
|
||||
*cookie_ptr = kCookieValue[i];
|
||||
#endif
|
||||
}
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_COOKIE_H_
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_DIRECT_MAP_EXTENT_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_DIRECT_MAP_EXTENT_H_
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_bucket.h"
|
||||
#include "base/allocator/partition_allocator/partition_page.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
struct PartitionDirectMapExtent {
|
||||
PartitionDirectMapExtent* next_extent;
|
||||
PartitionDirectMapExtent* prev_extent;
|
||||
PartitionBucket* bucket;
|
||||
size_t map_size; // Mapped size, not including guard pages and meta-data.
|
||||
|
||||
ALWAYS_INLINE static PartitionDirectMapExtent* FromPage(PartitionPage* page);
|
||||
};
|
||||
|
||||
ALWAYS_INLINE PartitionDirectMapExtent* PartitionDirectMapExtent::FromPage(
|
||||
PartitionPage* page) {
|
||||
DCHECK(page->bucket->is_direct_mapped());
|
||||
return reinterpret_cast<PartitionDirectMapExtent*>(
|
||||
reinterpret_cast<char*>(page) + 3 * kPageMetadataSize);
|
||||
}
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_DIRECT_MAP_EXTENT_H_
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_FREELIST_ENTRY_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_FREELIST_ENTRY_H_
|
||||
|
||||
#include <stdint.h>
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
|
||||
#include "base/compiler_specific.h"
|
||||
#include "base/sys_byteorder.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
struct EncodedPartitionFreelistEntry;
|
||||
|
||||
struct PartitionFreelistEntry {
|
||||
EncodedPartitionFreelistEntry* next;
|
||||
|
||||
PartitionFreelistEntry() = delete;
|
||||
~PartitionFreelistEntry() = delete;
|
||||
|
||||
ALWAYS_INLINE static EncodedPartitionFreelistEntry* Encode(
|
||||
PartitionFreelistEntry* ptr) {
|
||||
return reinterpret_cast<EncodedPartitionFreelistEntry*>(Transform(ptr));
|
||||
}
|
||||
|
||||
private:
|
||||
friend struct EncodedPartitionFreelistEntry;
|
||||
static ALWAYS_INLINE void* Transform(void* ptr) {
|
||||
// We use bswap on little endian as a fast mask for two reasons:
|
||||
// 1) If an object is freed and its vtable used where the attacker doesn't
|
||||
// get the chance to run allocations between the free and use, the vtable
|
||||
// dereference is likely to fault.
|
||||
// 2) If the attacker has a linear buffer overflow and elects to try and
|
||||
// corrupt a freelist pointer, partial pointer overwrite attacks are
|
||||
// thwarted.
|
||||
// For big endian, similar guarantees are arrived at with a negation.
|
||||
#if defined(ARCH_CPU_BIG_ENDIAN)
|
||||
uintptr_t masked = ~reinterpret_cast<uintptr_t>(ptr);
|
||||
#else
|
||||
uintptr_t masked = ByteSwapUintPtrT(reinterpret_cast<uintptr_t>(ptr));
|
||||
#endif
|
||||
return reinterpret_cast<void*>(masked);
|
||||
}
|
||||
};
|
||||
|
||||
struct EncodedPartitionFreelistEntry {
|
||||
char scrambled[sizeof(PartitionFreelistEntry*)];
|
||||
|
||||
EncodedPartitionFreelistEntry() = delete;
|
||||
~EncodedPartitionFreelistEntry() = delete;
|
||||
|
||||
ALWAYS_INLINE static PartitionFreelistEntry* Decode(
|
||||
EncodedPartitionFreelistEntry* ptr) {
|
||||
return reinterpret_cast<PartitionFreelistEntry*>(
|
||||
PartitionFreelistEntry::Transform(ptr));
|
||||
}
|
||||
};
|
||||
|
||||
static_assert(sizeof(PartitionFreelistEntry) ==
|
||||
sizeof(EncodedPartitionFreelistEntry),
|
||||
"Should not have padding");
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_FREELIST_ENTRY_H_
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_oom.h"
|
||||
|
||||
#include "base/allocator/partition_allocator/oom.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
void NOINLINE PartitionExcessiveAllocationSize(size_t size) {
|
||||
OOM_CRASH(size);
|
||||
}
|
||||
|
||||
#if !defined(ARCH_CPU_64_BITS)
|
||||
NOINLINE void PartitionOutOfMemoryWithLotsOfUncommitedPages(size_t size) {
|
||||
OOM_CRASH(size);
|
||||
}
|
||||
#endif
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
// Holds functions for generating OOM errors from PartitionAlloc. This is
|
||||
// distinct from oom.h in that it is meant only for use in PartitionAlloc.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_OOM_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_OOM_H_
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
#include "base/compiler_specific.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
NOINLINE void PartitionExcessiveAllocationSize(size_t size);
|
||||
|
||||
#if !defined(ARCH_CPU_64_BITS)
|
||||
NOINLINE void PartitionOutOfMemoryWithLotsOfUncommitedPages(size_t size);
|
||||
#endif
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_OOM_H_
|
||||
|
|
@ -0,0 +1,164 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_page.h"
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_direct_map_extent.h"
|
||||
#include "base/allocator/partition_allocator/partition_root_base.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
namespace {
|
||||
|
||||
ALWAYS_INLINE void PartitionDirectUnmap(PartitionPage* page) {
|
||||
PartitionRootBase* root = PartitionRootBase::FromPage(page);
|
||||
const PartitionDirectMapExtent* extent =
|
||||
PartitionDirectMapExtent::FromPage(page);
|
||||
size_t unmap_size = extent->map_size;
|
||||
|
||||
// Maintain the doubly-linked list of all direct mappings.
|
||||
if (extent->prev_extent) {
|
||||
DCHECK(extent->prev_extent->next_extent == extent);
|
||||
extent->prev_extent->next_extent = extent->next_extent;
|
||||
} else {
|
||||
root->direct_map_list = extent->next_extent;
|
||||
}
|
||||
if (extent->next_extent) {
|
||||
DCHECK(extent->next_extent->prev_extent == extent);
|
||||
extent->next_extent->prev_extent = extent->prev_extent;
|
||||
}
|
||||
|
||||
// Add on the size of the trailing guard page and preceeding partition
|
||||
// page.
|
||||
unmap_size += kPartitionPageSize + kSystemPageSize;
|
||||
|
||||
size_t uncommitted_page_size = page->bucket->slot_size + kSystemPageSize;
|
||||
root->DecreaseCommittedPages(uncommitted_page_size);
|
||||
DCHECK(root->total_size_of_direct_mapped_pages >= uncommitted_page_size);
|
||||
root->total_size_of_direct_mapped_pages -= uncommitted_page_size;
|
||||
|
||||
DCHECK(!(unmap_size & kPageAllocationGranularityOffsetMask));
|
||||
|
||||
char* ptr = reinterpret_cast<char*>(PartitionPage::ToPointer(page));
|
||||
// Account for the mapping starting a partition page before the actual
|
||||
// allocation address.
|
||||
ptr -= kPartitionPageSize;
|
||||
|
||||
FreePages(ptr, unmap_size);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionRegisterEmptyPage(PartitionPage* page) {
|
||||
DCHECK(page->is_empty());
|
||||
PartitionRootBase* root = PartitionRootBase::FromPage(page);
|
||||
|
||||
// If the page is already registered as empty, give it another life.
|
||||
if (page->empty_cache_index != -1) {
|
||||
DCHECK(page->empty_cache_index >= 0);
|
||||
DCHECK(static_cast<unsigned>(page->empty_cache_index) < kMaxFreeableSpans);
|
||||
DCHECK(root->global_empty_page_ring[page->empty_cache_index] == page);
|
||||
root->global_empty_page_ring[page->empty_cache_index] = nullptr;
|
||||
}
|
||||
|
||||
int16_t current_index = root->global_empty_page_ring_index;
|
||||
PartitionPage* page_to_decommit = root->global_empty_page_ring[current_index];
|
||||
// The page might well have been re-activated, filled up, etc. before we get
|
||||
// around to looking at it here.
|
||||
if (page_to_decommit)
|
||||
page_to_decommit->DecommitIfPossible(root);
|
||||
|
||||
// We put the empty slot span on our global list of "pages that were once
|
||||
// empty". thus providing it a bit of breathing room to get re-used before
|
||||
// we really free it. This improves performance, particularly on Mac OS X
|
||||
// which has subpar memory management performance.
|
||||
root->global_empty_page_ring[current_index] = page;
|
||||
page->empty_cache_index = current_index;
|
||||
++current_index;
|
||||
if (current_index == kMaxFreeableSpans)
|
||||
current_index = 0;
|
||||
root->global_empty_page_ring_index = current_index;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// static
|
||||
PartitionPage PartitionPage::sentinel_page_;
|
||||
|
||||
PartitionPage* PartitionPage::get_sentinel_page() {
|
||||
return &sentinel_page_;
|
||||
}
|
||||
|
||||
void PartitionPage::FreeSlowPath() {
|
||||
DCHECK(this != get_sentinel_page());
|
||||
if (LIKELY(num_allocated_slots == 0)) {
|
||||
// Page became fully unused.
|
||||
if (UNLIKELY(bucket->is_direct_mapped())) {
|
||||
PartitionDirectUnmap(this);
|
||||
return;
|
||||
}
|
||||
// If it's the current active page, change it. We bounce the page to
|
||||
// the empty list as a force towards defragmentation.
|
||||
if (LIKELY(this == bucket->active_pages_head))
|
||||
bucket->SetNewActivePage();
|
||||
DCHECK(bucket->active_pages_head != this);
|
||||
|
||||
set_raw_size(0);
|
||||
DCHECK(!get_raw_size());
|
||||
|
||||
PartitionRegisterEmptyPage(this);
|
||||
} else {
|
||||
DCHECK(!bucket->is_direct_mapped());
|
||||
// Ensure that the page is full. That's the only valid case if we
|
||||
// arrive here.
|
||||
DCHECK(num_allocated_slots < 0);
|
||||
// A transition of num_allocated_slots from 0 to -1 is not legal, and
|
||||
// likely indicates a double-free.
|
||||
CHECK(num_allocated_slots != -1);
|
||||
num_allocated_slots = -num_allocated_slots - 2;
|
||||
DCHECK(num_allocated_slots == bucket->get_slots_per_span() - 1);
|
||||
// Fully used page became partially used. It must be put back on the
|
||||
// non-full page list. Also make it the current page to increase the
|
||||
// chances of it being filled up again. The old current page will be
|
||||
// the next page.
|
||||
DCHECK(!next_page);
|
||||
if (LIKELY(bucket->active_pages_head != get_sentinel_page()))
|
||||
next_page = bucket->active_pages_head;
|
||||
bucket->active_pages_head = this;
|
||||
--bucket->num_full_pages;
|
||||
// Special case: for a partition page with just a single slot, it may
|
||||
// now be empty and we want to run it through the empty logic.
|
||||
if (UNLIKELY(num_allocated_slots == 0))
|
||||
FreeSlowPath();
|
||||
}
|
||||
}
|
||||
|
||||
void PartitionPage::Decommit(PartitionRootBase* root) {
|
||||
DCHECK(is_empty());
|
||||
DCHECK(!bucket->is_direct_mapped());
|
||||
void* addr = PartitionPage::ToPointer(this);
|
||||
root->DecommitSystemPages(addr, bucket->get_bytes_per_span());
|
||||
|
||||
// We actually leave the decommitted page in the active list. We'll sweep
|
||||
// it on to the decommitted page list when we next walk the active page
|
||||
// list.
|
||||
// Pulling this trick enables us to use a singly-linked page list for all
|
||||
// cases, which is critical in keeping the page metadata structure down to
|
||||
// 32 bytes in size.
|
||||
freelist_head = nullptr;
|
||||
num_unprovisioned_slots = 0;
|
||||
DCHECK(is_decommitted());
|
||||
}
|
||||
|
||||
void PartitionPage::DecommitIfPossible(PartitionRootBase* root) {
|
||||
DCHECK(empty_cache_index >= 0);
|
||||
DCHECK(static_cast<unsigned>(empty_cache_index) < kMaxFreeableSpans);
|
||||
DCHECK(this == root->global_empty_page_ring[empty_cache_index]);
|
||||
empty_cache_index = -1;
|
||||
if (is_empty())
|
||||
Decommit(root);
|
||||
}
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,293 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_PAGE_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_PAGE_H_
|
||||
|
||||
#include <string.h>
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
|
||||
#include "base/allocator/partition_allocator/partition_bucket.h"
|
||||
#include "base/allocator/partition_allocator/partition_cookie.h"
|
||||
#include "base/allocator/partition_allocator/partition_freelist_entry.h"
|
||||
#include "base/allocator/partition_allocator/random.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
struct PartitionRootBase;
|
||||
|
||||
// Some notes on page states. A page can be in one of four major states:
|
||||
// 1) Active.
|
||||
// 2) Full.
|
||||
// 3) Empty.
|
||||
// 4) Decommitted.
|
||||
// An active page has available free slots. A full page has no free slots. An
|
||||
// empty page has no free slots, and a decommitted page is an empty page that
|
||||
// had its backing memory released back to the system.
|
||||
// There are two linked lists tracking the pages. The "active page" list is an
|
||||
// approximation of a list of active pages. It is an approximation because
|
||||
// full, empty and decommitted pages may briefly be present in the list until
|
||||
// we next do a scan over it.
|
||||
// The "empty page" list is an accurate list of pages which are either empty
|
||||
// or decommitted.
|
||||
//
|
||||
// The significant page transitions are:
|
||||
// - free() will detect when a full page has a slot free()'d and immediately
|
||||
// return the page to the head of the active list.
|
||||
// - free() will detect when a page is fully emptied. It _may_ add it to the
|
||||
// empty list or it _may_ leave it on the active list until a future list scan.
|
||||
// - malloc() _may_ scan the active page list in order to fulfil the request.
|
||||
// If it does this, full, empty and decommitted pages encountered will be
|
||||
// booted out of the active list. If there are no suitable active pages found,
|
||||
// an empty or decommitted page (if one exists) will be pulled from the empty
|
||||
// list on to the active list.
|
||||
//
|
||||
// TODO(ajwong): Evaluate if this should be named PartitionSlotSpanMetadata or
|
||||
// similar. If so, all uses of the term "page" in comments, member variables,
|
||||
// local variables, and documentation that refer to this concept should be
|
||||
// updated.
|
||||
struct PartitionPage {
|
||||
PartitionFreelistEntry* freelist_head;
|
||||
PartitionPage* next_page;
|
||||
PartitionBucket* bucket;
|
||||
// Deliberately signed, 0 for empty or decommitted page, -n for full pages:
|
||||
int16_t num_allocated_slots;
|
||||
uint16_t num_unprovisioned_slots;
|
||||
uint16_t page_offset;
|
||||
int16_t empty_cache_index; // -1 if not in the empty cache.
|
||||
|
||||
// Public API
|
||||
|
||||
// Note the matching Alloc() functions are in PartitionPage.
|
||||
BASE_EXPORT NOINLINE void FreeSlowPath();
|
||||
ALWAYS_INLINE void Free(void* ptr);
|
||||
|
||||
void Decommit(PartitionRootBase* root);
|
||||
void DecommitIfPossible(PartitionRootBase* root);
|
||||
|
||||
// Pointer manipulation functions. These must be static as the input |page|
|
||||
// pointer may be the result of an offset calculation and therefore cannot
|
||||
// be trusted. The objective of these functions is to sanitize this input.
|
||||
ALWAYS_INLINE static void* ToPointer(const PartitionPage* page);
|
||||
ALWAYS_INLINE static PartitionPage* FromPointerNoAlignmentCheck(void* ptr);
|
||||
ALWAYS_INLINE static PartitionPage* FromPointer(void* ptr);
|
||||
|
||||
ALWAYS_INLINE const size_t* get_raw_size_ptr() const;
|
||||
ALWAYS_INLINE size_t* get_raw_size_ptr() {
|
||||
return const_cast<size_t*>(
|
||||
const_cast<const PartitionPage*>(this)->get_raw_size_ptr());
|
||||
}
|
||||
|
||||
ALWAYS_INLINE size_t get_raw_size() const;
|
||||
ALWAYS_INLINE void set_raw_size(size_t size);
|
||||
|
||||
ALWAYS_INLINE void Reset();
|
||||
|
||||
// TODO(ajwong): Can this be made private? https://crbug.com/787153
|
||||
BASE_EXPORT static PartitionPage* get_sentinel_page();
|
||||
|
||||
// Page State accessors.
|
||||
// Note that it's only valid to call these functions on pages found on one of
|
||||
// the page lists. Specifically, you can't call these functions on full pages
|
||||
// that were detached from the active list.
|
||||
//
|
||||
// This restriction provides the flexibity for some of the status fields to
|
||||
// be repurposed when a page is taken off a list. See the negation of
|
||||
// |num_allocated_slots| when a full page is removed from the active list
|
||||
// for an example of such repurposing.
|
||||
ALWAYS_INLINE bool is_active() const;
|
||||
ALWAYS_INLINE bool is_full() const;
|
||||
ALWAYS_INLINE bool is_empty() const;
|
||||
ALWAYS_INLINE bool is_decommitted() const;
|
||||
|
||||
private:
|
||||
// g_sentinel_page is used as a sentinel to indicate that there is no page
|
||||
// in the active page list. We can use nullptr, but in that case we need
|
||||
// to add a null-check branch to the hot allocation path. We want to avoid
|
||||
// that.
|
||||
//
|
||||
// Note, this declaration is kept in the header as opposed to an anonymous
|
||||
// namespace so the getter can be fully inlined.
|
||||
static PartitionPage sentinel_page_;
|
||||
};
|
||||
static_assert(sizeof(PartitionPage) <= kPageMetadataSize,
|
||||
"PartitionPage must be able to fit in a metadata slot");
|
||||
|
||||
ALWAYS_INLINE char* PartitionSuperPageToMetadataArea(char* ptr) {
|
||||
uintptr_t pointer_as_uint = reinterpret_cast<uintptr_t>(ptr);
|
||||
DCHECK(!(pointer_as_uint & kSuperPageOffsetMask));
|
||||
// The metadata area is exactly one system page (the guard page) into the
|
||||
// super page.
|
||||
return reinterpret_cast<char*>(pointer_as_uint + kSystemPageSize);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE PartitionPage* PartitionPage::FromPointerNoAlignmentCheck(
|
||||
void* ptr) {
|
||||
uintptr_t pointer_as_uint = reinterpret_cast<uintptr_t>(ptr);
|
||||
char* super_page_ptr =
|
||||
reinterpret_cast<char*>(pointer_as_uint & kSuperPageBaseMask);
|
||||
uintptr_t partition_page_index =
|
||||
(pointer_as_uint & kSuperPageOffsetMask) >> kPartitionPageShift;
|
||||
// Index 0 is invalid because it is the metadata and guard area and
|
||||
// the last index is invalid because it is a guard page.
|
||||
DCHECK(partition_page_index);
|
||||
DCHECK(partition_page_index < kNumPartitionPagesPerSuperPage - 1);
|
||||
PartitionPage* page = reinterpret_cast<PartitionPage*>(
|
||||
PartitionSuperPageToMetadataArea(super_page_ptr) +
|
||||
(partition_page_index << kPageMetadataShift));
|
||||
// Partition pages in the same slot span can share the same page object.
|
||||
// Adjust for that.
|
||||
size_t delta = page->page_offset << kPageMetadataShift;
|
||||
page =
|
||||
reinterpret_cast<PartitionPage*>(reinterpret_cast<char*>(page) - delta);
|
||||
return page;
|
||||
}
|
||||
|
||||
// Resturns start of the slot span for the PartitionPage.
|
||||
ALWAYS_INLINE void* PartitionPage::ToPointer(const PartitionPage* page) {
|
||||
uintptr_t pointer_as_uint = reinterpret_cast<uintptr_t>(page);
|
||||
|
||||
uintptr_t super_page_offset = (pointer_as_uint & kSuperPageOffsetMask);
|
||||
|
||||
// A valid |page| must be past the first guard System page and within
|
||||
// the following metadata region.
|
||||
DCHECK(super_page_offset > kSystemPageSize);
|
||||
// Must be less than total metadata region.
|
||||
DCHECK(super_page_offset < kSystemPageSize + (kNumPartitionPagesPerSuperPage *
|
||||
kPageMetadataSize));
|
||||
uintptr_t partition_page_index =
|
||||
(super_page_offset - kSystemPageSize) >> kPageMetadataShift;
|
||||
// Index 0 is invalid because it is the superpage extent metadata and the
|
||||
// last index is invalid because the whole PartitionPage is set as guard
|
||||
// pages for the metadata region.
|
||||
DCHECK(partition_page_index);
|
||||
DCHECK(partition_page_index < kNumPartitionPagesPerSuperPage - 1);
|
||||
uintptr_t super_page_base = (pointer_as_uint & kSuperPageBaseMask);
|
||||
void* ret = reinterpret_cast<void*>(
|
||||
super_page_base + (partition_page_index << kPartitionPageShift));
|
||||
return ret;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE PartitionPage* PartitionPage::FromPointer(void* ptr) {
|
||||
PartitionPage* page = PartitionPage::FromPointerNoAlignmentCheck(ptr);
|
||||
// Checks that the pointer is a multiple of bucket size.
|
||||
DCHECK(!((reinterpret_cast<uintptr_t>(ptr) -
|
||||
reinterpret_cast<uintptr_t>(PartitionPage::ToPointer(page))) %
|
||||
page->bucket->slot_size));
|
||||
return page;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE const size_t* PartitionPage::get_raw_size_ptr() const {
|
||||
// For single-slot buckets which span more than one partition page, we
|
||||
// have some spare metadata space to store the raw allocation size. We
|
||||
// can use this to report better statistics.
|
||||
if (bucket->slot_size <= kMaxSystemPagesPerSlotSpan * kSystemPageSize)
|
||||
return nullptr;
|
||||
|
||||
DCHECK((bucket->slot_size % kSystemPageSize) == 0);
|
||||
DCHECK(bucket->is_direct_mapped() || bucket->get_slots_per_span() == 1);
|
||||
|
||||
const PartitionPage* the_next_page = this + 1;
|
||||
return reinterpret_cast<const size_t*>(&the_next_page->freelist_head);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE size_t PartitionPage::get_raw_size() const {
|
||||
const size_t* ptr = get_raw_size_ptr();
|
||||
if (UNLIKELY(ptr != nullptr))
|
||||
return *ptr;
|
||||
return 0;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionPage::Free(void* ptr) {
|
||||
#if DCHECK_IS_ON()
|
||||
size_t slot_size = bucket->slot_size;
|
||||
const size_t raw_size = get_raw_size();
|
||||
if (raw_size) {
|
||||
slot_size = raw_size;
|
||||
}
|
||||
|
||||
// If these asserts fire, you probably corrupted memory.
|
||||
PartitionCookieCheckValue(ptr);
|
||||
PartitionCookieCheckValue(reinterpret_cast<char*>(ptr) + slot_size -
|
||||
kCookieSize);
|
||||
|
||||
memset(ptr, kFreedByte, slot_size);
|
||||
#endif
|
||||
|
||||
DCHECK(num_allocated_slots);
|
||||
// Catches an immediate double free.
|
||||
CHECK(ptr != freelist_head);
|
||||
// Look for double free one level deeper in debug.
|
||||
DCHECK(!freelist_head ||
|
||||
ptr != EncodedPartitionFreelistEntry::Decode(freelist_head->next));
|
||||
internal::PartitionFreelistEntry* entry =
|
||||
static_cast<internal::PartitionFreelistEntry*>(ptr);
|
||||
entry->next = internal::PartitionFreelistEntry::Encode(freelist_head);
|
||||
freelist_head = entry;
|
||||
--num_allocated_slots;
|
||||
if (UNLIKELY(num_allocated_slots <= 0)) {
|
||||
FreeSlowPath();
|
||||
} else {
|
||||
// All single-slot allocations must go through the slow path to
|
||||
// correctly update the size metadata.
|
||||
DCHECK(get_raw_size() == 0);
|
||||
}
|
||||
}
|
||||
|
||||
ALWAYS_INLINE bool PartitionPage::is_active() const {
|
||||
DCHECK(this != get_sentinel_page());
|
||||
DCHECK(!page_offset);
|
||||
return (num_allocated_slots > 0 &&
|
||||
(freelist_head || num_unprovisioned_slots));
|
||||
}
|
||||
|
||||
ALWAYS_INLINE bool PartitionPage::is_full() const {
|
||||
DCHECK(this != get_sentinel_page());
|
||||
DCHECK(!page_offset);
|
||||
bool ret = (num_allocated_slots == bucket->get_slots_per_span());
|
||||
if (ret) {
|
||||
DCHECK(!freelist_head);
|
||||
DCHECK(!num_unprovisioned_slots);
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE bool PartitionPage::is_empty() const {
|
||||
DCHECK(this != get_sentinel_page());
|
||||
DCHECK(!page_offset);
|
||||
return (!num_allocated_slots && freelist_head);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE bool PartitionPage::is_decommitted() const {
|
||||
DCHECK(this != get_sentinel_page());
|
||||
DCHECK(!page_offset);
|
||||
bool ret = (!num_allocated_slots && !freelist_head);
|
||||
if (ret) {
|
||||
DCHECK(!num_unprovisioned_slots);
|
||||
DCHECK(empty_cache_index == -1);
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionPage::set_raw_size(size_t size) {
|
||||
size_t* raw_size_ptr = get_raw_size_ptr();
|
||||
if (UNLIKELY(raw_size_ptr != nullptr))
|
||||
*raw_size_ptr = size;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionPage::Reset() {
|
||||
DCHECK(is_decommitted());
|
||||
|
||||
num_unprovisioned_slots = bucket->get_slots_per_span();
|
||||
DCHECK(num_unprovisioned_slots);
|
||||
|
||||
next_page = nullptr;
|
||||
}
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_PAGE_H_
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/partition_root_base.h"
|
||||
|
||||
#include "base/allocator/partition_allocator/oom.h"
|
||||
#include "base/allocator/partition_allocator/partition_oom.h"
|
||||
#include "base/allocator/partition_allocator/partition_page.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
namespace internal {
|
||||
|
||||
NOINLINE void PartitionRootBase::OutOfMemory(size_t size) {
|
||||
#if !defined(ARCH_CPU_64_BITS)
|
||||
// Check whether this OOM is due to a lot of super pages that are allocated
|
||||
// but not committed, probably due to http://crbug.com/421387.
|
||||
if (total_size_of_super_pages + total_size_of_direct_mapped_pages -
|
||||
total_size_of_committed_pages >
|
||||
kReasonableSizeOfUnusedPages) {
|
||||
PartitionOutOfMemoryWithLotsOfUncommitedPages(size);
|
||||
}
|
||||
#endif
|
||||
if (PartitionRootBase::g_oom_handling_function)
|
||||
(*PartitionRootBase::g_oom_handling_function)(size);
|
||||
OOM_CRASH(size);
|
||||
}
|
||||
|
||||
void PartitionRootBase::DecommitEmptyPages() {
|
||||
for (size_t i = 0; i < kMaxFreeableSpans; ++i) {
|
||||
internal::PartitionPage* page = global_empty_page_ring[i];
|
||||
if (page)
|
||||
page->DecommitIfPossible(this);
|
||||
global_empty_page_ring[i] = nullptr;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,200 @@
|
|||
// Copyright (c) 2018 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ROOT_BASE_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ROOT_BASE_H_
|
||||
|
||||
#include "base/allocator/partition_allocator/page_allocator.h"
|
||||
#include "base/allocator/partition_allocator/partition_alloc_constants.h"
|
||||
#include "base/allocator/partition_allocator/partition_bucket.h"
|
||||
#include "base/allocator/partition_allocator/partition_direct_map_extent.h"
|
||||
#include "base/allocator/partition_allocator/partition_page.h"
|
||||
#include "base/logging.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
typedef void (*OomFunction)(size_t);
|
||||
|
||||
namespace internal {
|
||||
|
||||
struct PartitionPage;
|
||||
struct PartitionRootBase;
|
||||
|
||||
// An "extent" is a span of consecutive superpages. We link to the partition's
|
||||
// next extent (if there is one) to the very start of a superpage's metadata
|
||||
// area.
|
||||
struct PartitionSuperPageExtentEntry {
|
||||
PartitionRootBase* root;
|
||||
char* super_page_base;
|
||||
char* super_pages_end;
|
||||
PartitionSuperPageExtentEntry* next;
|
||||
};
|
||||
static_assert(
|
||||
sizeof(PartitionSuperPageExtentEntry) <= kPageMetadataSize,
|
||||
"PartitionSuperPageExtentEntry must be able to fit in a metadata slot");
|
||||
|
||||
struct BASE_EXPORT PartitionRootBase {
|
||||
PartitionRootBase();
|
||||
virtual ~PartitionRootBase();
|
||||
size_t total_size_of_committed_pages = 0;
|
||||
size_t total_size_of_super_pages = 0;
|
||||
size_t total_size_of_direct_mapped_pages = 0;
|
||||
// Invariant: total_size_of_committed_pages <=
|
||||
// total_size_of_super_pages +
|
||||
// total_size_of_direct_mapped_pages.
|
||||
unsigned num_buckets = 0;
|
||||
unsigned max_allocation = 0;
|
||||
bool initialized = false;
|
||||
char* next_super_page = nullptr;
|
||||
char* next_partition_page = nullptr;
|
||||
char* next_partition_page_end = nullptr;
|
||||
PartitionSuperPageExtentEntry* current_extent = nullptr;
|
||||
PartitionSuperPageExtentEntry* first_extent = nullptr;
|
||||
PartitionDirectMapExtent* direct_map_list = nullptr;
|
||||
PartitionPage* global_empty_page_ring[kMaxFreeableSpans] = {};
|
||||
int16_t global_empty_page_ring_index = 0;
|
||||
uintptr_t inverted_self = 0;
|
||||
|
||||
// Public API
|
||||
|
||||
// Allocates out of the given bucket. Properly, this function should probably
|
||||
// be in PartitionBucket, but because the implementation needs to be inlined
|
||||
// for performance, and because it needs to inspect PartitionPage,
|
||||
// it becomes impossible to have it in PartitionBucket as this causes a
|
||||
// cyclical dependency on PartitionPage function implementations.
|
||||
//
|
||||
// Moving it a layer lower couples PartitionRootBase and PartitionBucket, but
|
||||
// preserves the layering of the includes.
|
||||
//
|
||||
// Note the matching Free() functions are in PartitionPage.
|
||||
ALWAYS_INLINE void* AllocFromBucket(PartitionBucket* bucket,
|
||||
int flags,
|
||||
size_t size);
|
||||
|
||||
ALWAYS_INLINE static bool IsValidPage(PartitionPage* page);
|
||||
ALWAYS_INLINE static PartitionRootBase* FromPage(PartitionPage* page);
|
||||
|
||||
// g_oom_handling_function is invoked when PartitionAlloc hits OutOfMemory.
|
||||
static OomFunction g_oom_handling_function;
|
||||
NOINLINE void OutOfMemory(size_t size);
|
||||
|
||||
ALWAYS_INLINE void IncreaseCommittedPages(size_t len);
|
||||
ALWAYS_INLINE void DecreaseCommittedPages(size_t len);
|
||||
ALWAYS_INLINE void DecommitSystemPages(void* address, size_t length);
|
||||
ALWAYS_INLINE void RecommitSystemPages(void* address, size_t length);
|
||||
|
||||
// Frees memory from this partition, if possible, by decommitting pages.
|
||||
// |flags| is an OR of base::PartitionPurgeFlags.
|
||||
virtual void PurgeMemory(int flags) = 0;
|
||||
void DecommitEmptyPages();
|
||||
};
|
||||
|
||||
ALWAYS_INLINE void* PartitionRootBase::AllocFromBucket(PartitionBucket* bucket,
|
||||
int flags,
|
||||
size_t size) {
|
||||
bool zero_fill = flags & PartitionAllocZeroFill;
|
||||
bool is_already_zeroed = false;
|
||||
|
||||
PartitionPage* page = bucket->active_pages_head;
|
||||
// Check that this page is neither full nor freed.
|
||||
DCHECK(page->num_allocated_slots >= 0);
|
||||
void* ret = page->freelist_head;
|
||||
if (LIKELY(ret != 0)) {
|
||||
// If these DCHECKs fire, you probably corrupted memory. TODO(palmer): See
|
||||
// if we can afford to make these CHECKs.
|
||||
DCHECK(PartitionRootBase::IsValidPage(page));
|
||||
|
||||
// All large allocations must go through the slow path to correctly update
|
||||
// the size metadata.
|
||||
DCHECK(page->get_raw_size() == 0);
|
||||
internal::PartitionFreelistEntry* new_head =
|
||||
internal::EncodedPartitionFreelistEntry::Decode(
|
||||
page->freelist_head->next);
|
||||
page->freelist_head = new_head;
|
||||
page->num_allocated_slots++;
|
||||
} else {
|
||||
ret = bucket->SlowPathAlloc(this, flags, size, &is_already_zeroed);
|
||||
// TODO(palmer): See if we can afford to make this a CHECK.
|
||||
DCHECK(!ret ||
|
||||
PartitionRootBase::IsValidPage(PartitionPage::FromPointer(ret)));
|
||||
}
|
||||
|
||||
#if DCHECK_IS_ON()
|
||||
if (!ret) {
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
page = PartitionPage::FromPointer(ret);
|
||||
// TODO(ajwong): Can |page->bucket| ever not be |this|? If not, can this just
|
||||
// be bucket->slot_size?
|
||||
size_t new_slot_size = page->bucket->slot_size;
|
||||
size_t raw_size = page->get_raw_size();
|
||||
if (raw_size) {
|
||||
DCHECK(raw_size == size);
|
||||
new_slot_size = raw_size;
|
||||
}
|
||||
size_t no_cookie_size = PartitionCookieSizeAdjustSubtract(new_slot_size);
|
||||
char* char_ret = static_cast<char*>(ret);
|
||||
// The value given to the application is actually just after the cookie.
|
||||
ret = char_ret + kCookieSize;
|
||||
|
||||
// Fill the region kUninitializedByte or 0, and surround it with 2 cookies.
|
||||
PartitionCookieWriteValue(char_ret);
|
||||
if (!zero_fill) {
|
||||
memset(ret, kUninitializedByte, no_cookie_size);
|
||||
} else if (!is_already_zeroed) {
|
||||
memset(ret, 0, no_cookie_size);
|
||||
}
|
||||
PartitionCookieWriteValue(char_ret + kCookieSize + no_cookie_size);
|
||||
#else
|
||||
if (ret && zero_fill && !is_already_zeroed) {
|
||||
memset(ret, 0, size);
|
||||
}
|
||||
#endif
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE bool PartitionRootBase::IsValidPage(PartitionPage* page) {
|
||||
PartitionRootBase* root = PartitionRootBase::FromPage(page);
|
||||
return root->inverted_self == ~reinterpret_cast<uintptr_t>(root);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE PartitionRootBase* PartitionRootBase::FromPage(
|
||||
PartitionPage* page) {
|
||||
PartitionSuperPageExtentEntry* extent_entry =
|
||||
reinterpret_cast<PartitionSuperPageExtentEntry*>(
|
||||
reinterpret_cast<uintptr_t>(page) & kSystemPageBaseMask);
|
||||
return extent_entry->root;
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionRootBase::IncreaseCommittedPages(size_t len) {
|
||||
total_size_of_committed_pages += len;
|
||||
DCHECK(total_size_of_committed_pages <=
|
||||
total_size_of_super_pages + total_size_of_direct_mapped_pages);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionRootBase::DecreaseCommittedPages(size_t len) {
|
||||
total_size_of_committed_pages -= len;
|
||||
DCHECK(total_size_of_committed_pages <=
|
||||
total_size_of_super_pages + total_size_of_direct_mapped_pages);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionRootBase::DecommitSystemPages(void* address,
|
||||
size_t length) {
|
||||
::base::DecommitSystemPages(address, length);
|
||||
DecreaseCommittedPages(length);
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void PartitionRootBase::RecommitSystemPages(void* address,
|
||||
size_t length) {
|
||||
CHECK(::base::RecommitSystemPages(address, length, PageReadWrite));
|
||||
IncreaseCommittedPages(length);
|
||||
}
|
||||
|
||||
} // namespace internal
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_PARTITION_ROOT_BASE_H_
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
// Copyright 2019 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/random.h"
|
||||
|
||||
#include "base/allocator/partition_allocator/spin_lock.h"
|
||||
#include "base/logging.h"
|
||||
#include "base/no_destructor.h"
|
||||
#include "base/rand_util.h"
|
||||
#include "base/synchronization/lock.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
namespace {
|
||||
|
||||
Lock& GetLock() {
|
||||
static NoDestructor<Lock> lock;
|
||||
return *lock;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// This is the same PRNG as used by tcmalloc for mapping address randomness;
|
||||
// see http://burtleburtle.net/bob/rand/smallprng.html.
|
||||
struct RandomContext {
|
||||
bool initialized;
|
||||
uint32_t a;
|
||||
uint32_t b;
|
||||
uint32_t c;
|
||||
uint32_t d;
|
||||
};
|
||||
|
||||
static RandomContext g_context GUARDED_BY(GetLock());
|
||||
|
||||
namespace {
|
||||
|
||||
RandomContext& GetRandomContext() EXCLUSIVE_LOCKS_REQUIRED(GetLock()) {
|
||||
if (UNLIKELY(!g_context.initialized)) {
|
||||
const uint64_t r1 = RandUint64();
|
||||
const uint64_t r2 = RandUint64();
|
||||
g_context.a = static_cast<uint32_t>(r1);
|
||||
g_context.b = static_cast<uint32_t>(r1 >> 32);
|
||||
g_context.c = static_cast<uint32_t>(r2);
|
||||
g_context.d = static_cast<uint32_t>(r2 >> 32);
|
||||
g_context.initialized = true;
|
||||
}
|
||||
return g_context;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
uint32_t RandomValue() {
|
||||
AutoLock guard(GetLock());
|
||||
RandomContext& x = GetRandomContext();
|
||||
#define rot(x, k) (((x) << (k)) | ((x) >> (32 - (k))))
|
||||
uint32_t e = x.a - rot(x.b, 27);
|
||||
x.a = x.b ^ rot(x.c, 17);
|
||||
x.b = x.c + x.d;
|
||||
x.c = x.d + e;
|
||||
x.d = e + x.a;
|
||||
return x.d;
|
||||
#undef rot
|
||||
}
|
||||
|
||||
void SetMmapSeedForTesting(uint64_t seed) {
|
||||
AutoLock guard(GetLock());
|
||||
RandomContext& x = GetRandomContext();
|
||||
x.a = x.b = static_cast<uint32_t>(seed);
|
||||
x.c = x.d = static_cast<uint32_t>(seed >> 32);
|
||||
x.initialized = true;
|
||||
}
|
||||
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
// Copyright 2019 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_RANDOM_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_RANDOM_H_
|
||||
|
||||
#include <stdint.h>
|
||||
|
||||
#include "base/base_export.h"
|
||||
|
||||
namespace base {
|
||||
|
||||
// Returns a random value. The generator's internal state is initialized with
|
||||
// `base::RandUint64` which is very unpredictable, but which is expensive due to
|
||||
// the need to call into the kernel. Therefore this generator uses a fast,
|
||||
// entirely user-space function after initialization.
|
||||
BASE_EXPORT uint32_t RandomValue();
|
||||
|
||||
// Sets the seed for the random number generator to a known value, to cause the
|
||||
// RNG to generate a predictable sequence of outputs. May be called multiple
|
||||
// times.
|
||||
BASE_EXPORT void SetMmapSeedForTesting(uint64_t seed);
|
||||
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_RANDOM_H_
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
// Copyright 2015 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#include "base/allocator/partition_allocator/spin_lock.h"
|
||||
|
||||
#include "base/threading/platform_thread.h"
|
||||
#include "build/build_config.h"
|
||||
|
||||
#if defined(OS_WIN)
|
||||
#include <windows.h>
|
||||
#elif defined(OS_POSIX) || defined(OS_FUCHSIA)
|
||||
#include <sched.h>
|
||||
#endif
|
||||
|
||||
// The YIELD_PROCESSOR macro wraps an architecture specific-instruction that
|
||||
// informs the processor we're in a busy wait, so it can handle the branch more
|
||||
// intelligently and e.g. reduce power to our core or give more resources to the
|
||||
// other hyper-thread on this core. See the following for context:
|
||||
// https://software.intel.com/en-us/articles/benefitting-power-and-performance-sleep-loops
|
||||
//
|
||||
// The YIELD_THREAD macro tells the OS to relinquish our quantum. This is
|
||||
// basically a worst-case fallback, and if you're hitting it with any frequency
|
||||
// you really should be using a proper lock (such as |base::Lock|)rather than
|
||||
// these spinlocks.
|
||||
#if defined(OS_WIN)
|
||||
|
||||
#define YIELD_PROCESSOR YieldProcessor()
|
||||
#define YIELD_THREAD SwitchToThread()
|
||||
|
||||
#elif defined(OS_POSIX) || defined(OS_FUCHSIA)
|
||||
|
||||
#if defined(ARCH_CPU_X86_64) || defined(ARCH_CPU_X86)
|
||||
#define YIELD_PROCESSOR __asm__ __volatile__("pause")
|
||||
#elif (defined(ARCH_CPU_ARMEL) && __ARM_ARCH >= 6) || defined(ARCH_CPU_ARM64)
|
||||
#define YIELD_PROCESSOR __asm__ __volatile__("yield")
|
||||
#elif defined(ARCH_CPU_MIPSEL)
|
||||
// The MIPS32 docs state that the PAUSE instruction is a no-op on older
|
||||
// architectures (first added in MIPS32r2). To avoid assembler errors when
|
||||
// targeting pre-r2, we must encode the instruction manually.
|
||||
#define YIELD_PROCESSOR __asm__ __volatile__(".word 0x00000140")
|
||||
#elif defined(ARCH_CPU_MIPS64EL) && __mips_isa_rev >= 2
|
||||
// Don't bother doing using .word here since r2 is the lowest supported mips64
|
||||
// that Chromium supports.
|
||||
#define YIELD_PROCESSOR __asm__ __volatile__("pause")
|
||||
#elif defined(ARCH_CPU_PPC64_FAMILY)
|
||||
#define YIELD_PROCESSOR __asm__ __volatile__("or 31,31,31")
|
||||
#elif defined(ARCH_CPU_S390_FAMILY)
|
||||
// just do nothing
|
||||
#define YIELD_PROCESSOR ((void)0)
|
||||
#endif // ARCH
|
||||
|
||||
#ifndef YIELD_PROCESSOR
|
||||
#warning "Processor yield not supported on this architecture."
|
||||
#define YIELD_PROCESSOR ((void)0)
|
||||
#endif
|
||||
|
||||
#define YIELD_THREAD sched_yield()
|
||||
|
||||
#else // Other OS
|
||||
|
||||
#warning "Thread yield not supported on this OS."
|
||||
#define YIELD_THREAD ((void)0)
|
||||
|
||||
#endif // OS_WIN
|
||||
|
||||
namespace base {
|
||||
namespace subtle {
|
||||
|
||||
void SpinLock::LockSlow() {
|
||||
// The value of |kYieldProcessorTries| is cargo culted from TCMalloc, Windows
|
||||
// critical section defaults, and various other recommendations.
|
||||
// TODO(jschuh): Further tuning may be warranted.
|
||||
static const int kYieldProcessorTries = 1000;
|
||||
// The value of |kYieldThreadTries| is completely made up.
|
||||
static const int kYieldThreadTries = 10;
|
||||
int yield_thread_count = 0;
|
||||
do {
|
||||
do {
|
||||
for (int count = 0; count < kYieldProcessorTries; ++count) {
|
||||
// Let the processor know we're spinning.
|
||||
YIELD_PROCESSOR;
|
||||
if (!lock_.load(std::memory_order_relaxed) &&
|
||||
LIKELY(!lock_.exchange(true, std::memory_order_acquire)))
|
||||
return;
|
||||
}
|
||||
|
||||
if (yield_thread_count < kYieldThreadTries) {
|
||||
++yield_thread_count;
|
||||
// Give the OS a chance to schedule something on this core.
|
||||
YIELD_THREAD;
|
||||
} else {
|
||||
// At this point, it's likely that the lock is held by a lower priority
|
||||
// thread that is unavailable to finish its work because of higher
|
||||
// priority threads spinning here. Sleeping should ensure that they make
|
||||
// progress.
|
||||
PlatformThread::Sleep(TimeDelta::FromMilliseconds(1));
|
||||
}
|
||||
} while (lock_.load(std::memory_order_relaxed));
|
||||
} while (UNLIKELY(lock_.exchange(true, std::memory_order_acquire)));
|
||||
}
|
||||
|
||||
} // namespace subtle
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
// Copyright (c) 2013 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_PARTITION_ALLOCATOR_SPIN_LOCK_H_
|
||||
#define BASE_ALLOCATOR_PARTITION_ALLOCATOR_SPIN_LOCK_H_
|
||||
|
||||
#include <atomic>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
|
||||
#include "base/base_export.h"
|
||||
#include "base/compiler_specific.h"
|
||||
|
||||
// Spinlock is a simple spinlock class based on the standard CPU primitive of
|
||||
// atomic increment and decrement of an int at a given memory address. These are
|
||||
// intended only for very short duration locks and assume a system with multiple
|
||||
// cores. For any potentially longer wait you should use a real lock, such as
|
||||
// |base::Lock|.
|
||||
namespace base {
|
||||
namespace subtle {
|
||||
|
||||
class BASE_EXPORT SpinLock {
|
||||
public:
|
||||
constexpr SpinLock() = default;
|
||||
~SpinLock() = default;
|
||||
using Guard = std::lock_guard<SpinLock>;
|
||||
|
||||
ALWAYS_INLINE void lock() {
|
||||
static_assert(sizeof(lock_) == sizeof(int),
|
||||
"int and lock_ are different sizes");
|
||||
if (LIKELY(!lock_.exchange(true, std::memory_order_acquire)))
|
||||
return;
|
||||
LockSlow();
|
||||
}
|
||||
|
||||
ALWAYS_INLINE void unlock() { lock_.store(false, std::memory_order_release); }
|
||||
|
||||
private:
|
||||
// This is called if the initial attempt to acquire the lock fails. It's
|
||||
// slower, but has a much better scheduling and power consumption behavior.
|
||||
void LockSlow();
|
||||
|
||||
std::atomic_int lock_{0};
|
||||
};
|
||||
|
||||
} // namespace subtle
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_PARTITION_ALLOCATOR_SPIN_LOCK_H_
|
||||
|
|
@ -0,0 +1,208 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
// This code should move into the default Windows shim once the win-specific
|
||||
// allocation shim has been removed, and the generic shim has becaome the
|
||||
// default.
|
||||
|
||||
#include "winheap_stubs_win.h"
|
||||
|
||||
#include <limits.h>
|
||||
#include <malloc.h>
|
||||
#include <new.h>
|
||||
#include <windows.h>
|
||||
#include <algorithm>
|
||||
#include <limits>
|
||||
|
||||
#include "base/bits.h"
|
||||
#include "base/logging.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
bool g_is_win_shim_layer_initialized = false;
|
||||
|
||||
namespace {
|
||||
|
||||
const size_t kWindowsPageSize = 4096;
|
||||
const size_t kMaxWindowsAllocation = INT_MAX - kWindowsPageSize;
|
||||
|
||||
inline HANDLE get_heap_handle() {
|
||||
return reinterpret_cast<HANDLE>(_get_heap_handle());
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void* WinHeapMalloc(size_t size) {
|
||||
if (size < kMaxWindowsAllocation)
|
||||
return HeapAlloc(get_heap_handle(), 0, size);
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
void WinHeapFree(void* ptr) {
|
||||
if (!ptr)
|
||||
return;
|
||||
|
||||
HeapFree(get_heap_handle(), 0, ptr);
|
||||
}
|
||||
|
||||
void* WinHeapRealloc(void* ptr, size_t size) {
|
||||
if (!ptr)
|
||||
return WinHeapMalloc(size);
|
||||
if (!size) {
|
||||
WinHeapFree(ptr);
|
||||
return nullptr;
|
||||
}
|
||||
if (size < kMaxWindowsAllocation)
|
||||
return HeapReAlloc(get_heap_handle(), 0, ptr, size);
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
size_t WinHeapGetSizeEstimate(void* ptr) {
|
||||
if (!ptr)
|
||||
return 0;
|
||||
|
||||
return HeapSize(get_heap_handle(), 0, ptr);
|
||||
}
|
||||
|
||||
// Call the new handler, if one has been set.
|
||||
// Returns true on successfully calling the handler, false otherwise.
|
||||
bool WinCallNewHandler(size_t size) {
|
||||
#ifdef _CPPUNWIND
|
||||
#error "Exceptions in allocator shim are not supported!"
|
||||
#endif // _CPPUNWIND
|
||||
// Get the current new handler.
|
||||
_PNH nh = _query_new_handler();
|
||||
if (!nh)
|
||||
return false;
|
||||
// Since exceptions are disabled, we don't really know if new_handler
|
||||
// failed. Assume it will abort if it fails.
|
||||
return nh(size) ? true : false;
|
||||
}
|
||||
|
||||
// The Windows _aligned_* functions are implemented by creating an allocation
|
||||
// with enough space to create an aligned allocation internally. The offset to
|
||||
// the original allocation is prefixed to the aligned allocation so that it can
|
||||
// be correctly freed.
|
||||
|
||||
namespace {
|
||||
|
||||
struct AlignedPrefix {
|
||||
// Offset to the original allocation point.
|
||||
unsigned int original_allocation_offset;
|
||||
// Make sure an unsigned int is enough to store the offset
|
||||
static_assert(
|
||||
kMaxWindowsAllocation < std::numeric_limits<unsigned int>::max(),
|
||||
"original_allocation_offset must be able to fit into an unsigned int");
|
||||
#if DCHECK_IS_ON()
|
||||
// Magic value used to check that _aligned_free() and _aligned_realloc() are
|
||||
// only ever called on an aligned allocated chunk.
|
||||
static constexpr unsigned int kMagic = 0x12003400;
|
||||
unsigned int magic;
|
||||
#endif // DCHECK_IS_ON()
|
||||
};
|
||||
|
||||
// Compute how large an allocation we need to fit an allocation with the given
|
||||
// size and alignment and space for a prefix pointer.
|
||||
size_t AdjustedSize(size_t size, size_t alignment) {
|
||||
// Minimal alignment is the prefix size so the prefix is properly aligned.
|
||||
alignment = std::max(alignment, alignof(AlignedPrefix));
|
||||
return size + sizeof(AlignedPrefix) + alignment - 1;
|
||||
}
|
||||
|
||||
// Align the allocation and write the prefix.
|
||||
void* AlignAllocation(void* ptr, size_t alignment) {
|
||||
// Minimal alignment is the prefix size so the prefix is properly aligned.
|
||||
alignment = std::max(alignment, alignof(AlignedPrefix));
|
||||
|
||||
uintptr_t address = reinterpret_cast<uintptr_t>(ptr);
|
||||
address = base::bits::Align(address + sizeof(AlignedPrefix), alignment);
|
||||
|
||||
// Write the prefix.
|
||||
AlignedPrefix* prefix = reinterpret_cast<AlignedPrefix*>(address) - 1;
|
||||
prefix->original_allocation_offset =
|
||||
address - reinterpret_cast<uintptr_t>(ptr);
|
||||
#if DCHECK_IS_ON()
|
||||
prefix->magic = AlignedPrefix::kMagic;
|
||||
#endif // DCHECK_IS_ON()
|
||||
return reinterpret_cast<void*>(address);
|
||||
}
|
||||
|
||||
// Return the original allocation from an aligned allocation.
|
||||
void* UnalignAllocation(void* ptr) {
|
||||
AlignedPrefix* prefix = reinterpret_cast<AlignedPrefix*>(ptr) - 1;
|
||||
#if DCHECK_IS_ON()
|
||||
DCHECK_EQ(prefix->magic, AlignedPrefix::kMagic);
|
||||
#endif // DCHECK_IS_ON()
|
||||
void* unaligned =
|
||||
static_cast<uint8_t*>(ptr) - prefix->original_allocation_offset;
|
||||
CHECK_LT(unaligned, ptr);
|
||||
CHECK_LE(
|
||||
reinterpret_cast<uintptr_t>(ptr) - reinterpret_cast<uintptr_t>(unaligned),
|
||||
kMaxWindowsAllocation);
|
||||
return unaligned;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void* WinHeapAlignedMalloc(size_t size, size_t alignment) {
|
||||
CHECK(base::bits::IsPowerOfTwo(alignment));
|
||||
|
||||
size_t adjusted = AdjustedSize(size, alignment);
|
||||
if (adjusted >= kMaxWindowsAllocation)
|
||||
return nullptr;
|
||||
|
||||
void* ptr = WinHeapMalloc(adjusted);
|
||||
if (!ptr)
|
||||
return nullptr;
|
||||
|
||||
return AlignAllocation(ptr, alignment);
|
||||
}
|
||||
|
||||
void* WinHeapAlignedRealloc(void* ptr, size_t size, size_t alignment) {
|
||||
CHECK(base::bits::IsPowerOfTwo(alignment));
|
||||
|
||||
if (!ptr)
|
||||
return WinHeapAlignedMalloc(size, alignment);
|
||||
if (!size) {
|
||||
WinHeapAlignedFree(ptr);
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
size_t adjusted = AdjustedSize(size, alignment);
|
||||
if (adjusted >= kMaxWindowsAllocation)
|
||||
return nullptr;
|
||||
|
||||
// Try to resize the allocation in place first.
|
||||
void* unaligned = UnalignAllocation(ptr);
|
||||
if (HeapReAlloc(get_heap_handle(), HEAP_REALLOC_IN_PLACE_ONLY, unaligned,
|
||||
adjusted)) {
|
||||
return ptr;
|
||||
}
|
||||
|
||||
// Otherwise manually perform an _aligned_malloc() and copy since an
|
||||
// unaligned allocation from HeapReAlloc() would force us to copy the
|
||||
// allocation twice.
|
||||
void* new_ptr = WinHeapAlignedMalloc(size, alignment);
|
||||
if (!new_ptr)
|
||||
return nullptr;
|
||||
|
||||
size_t gap =
|
||||
reinterpret_cast<uintptr_t>(ptr) - reinterpret_cast<uintptr_t>(unaligned);
|
||||
size_t old_size = WinHeapGetSizeEstimate(unaligned) - gap;
|
||||
memcpy(new_ptr, ptr, std::min(size, old_size));
|
||||
WinHeapAlignedFree(ptr);
|
||||
return new_ptr;
|
||||
}
|
||||
|
||||
void WinHeapAlignedFree(void* ptr) {
|
||||
if (!ptr)
|
||||
return;
|
||||
|
||||
void* original_allocation = UnalignAllocation(ptr);
|
||||
WinHeapFree(original_allocation);
|
||||
}
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
// Copyright 2016 The Chromium Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
// Thin allocation wrappers for the windows heap. This file should be deleted
|
||||
// once the win-specific allocation shim has been removed, and the generic shim
|
||||
// has becaome the default.
|
||||
|
||||
#ifndef BASE_ALLOCATOR_WINHEAP_STUBS_H_
|
||||
#define BASE_ALLOCATOR_WINHEAP_STUBS_H_
|
||||
|
||||
#include <stdint.h>
|
||||
|
||||
#include "base/base_export.h"
|
||||
|
||||
namespace base {
|
||||
namespace allocator {
|
||||
|
||||
// Set to true if the link-time magic has successfully hooked into the CRT's
|
||||
// heap initialization.
|
||||
extern bool g_is_win_shim_layer_initialized;
|
||||
|
||||
// Thin wrappers to implement the standard C allocation semantics on the
|
||||
// CRT's Windows heap.
|
||||
void* WinHeapMalloc(size_t size);
|
||||
void WinHeapFree(void* ptr);
|
||||
void* WinHeapRealloc(void* ptr, size_t size);
|
||||
|
||||
// Returns a lower-bound estimate for the full amount of memory consumed by the
|
||||
// the allocation |ptr|.
|
||||
size_t WinHeapGetSizeEstimate(void* ptr);
|
||||
|
||||
// Call the new handler, if one has been set.
|
||||
// Returns true on successfully calling the handler, false otherwise.
|
||||
bool WinCallNewHandler(size_t size);
|
||||
|
||||
// Wrappers to implement the interface for the _aligned_* functions on top of
|
||||
// the CRT's Windows heap. Exported for tests.
|
||||
BASE_EXPORT void* WinHeapAlignedMalloc(size_t size, size_t alignment);
|
||||
BASE_EXPORT void* WinHeapAlignedRealloc(void* ptr,
|
||||
size_t size,
|
||||
size_t alignment);
|
||||
BASE_EXPORT void WinHeapAlignedFree(void* ptr);
|
||||
|
||||
} // namespace allocator
|
||||
} // namespace base
|
||||
|
||||
#endif // BASE_ALLOCATOR_WINHEAP_STUBS_H_
|
||||
Loading…
Add table
Add a link
Reference in a new issue