前言
在Java中分配直接内存大概有如下三种主要方式：
- Unsafe.allocateMemory()
- ByteBuffer.allocateDirect()
- native方法
Unsafe类
Java提供了Unsafe类用来进行直接内存的分配与释放
1
2
| public native long allocateMemory(long var1);
public native void freeMemory(long var1);
|
示例
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| public class DirectMemoryMain {
public static void main(String[] args) throws InterruptedException {
Unsafe unsafe = getUnsafe();
while (true) {
for (int i = 0; i < 10000; i++) {
long address = unsafe.allocateMemory(10000);
}
Thread.sleep(1);
}
}
private static Unsafe getUnsafe() {
try {
Class clazz = Unsafe.class;
Field field = clazz.getDeclaredField("theUnsafe");
field.setAccessible(true);
return (Unsafe) field.get(null);
} catch (IllegalAccessException | NoSuchFieldException e) {
throw new RuntimeException(e);
}
}
}
|
下面为这段代码的演示效果，其中JVM最大内存设为64M，而真实内存则可以无限增长。
![image-20181225122431853]()
DirectByteBuffer类
内存分配
虽然Unsafe可以通过反射调用来进行内存分配，但是按照其设计方式，它并不是给开发者来使用的，而且Unsafe里面的方法也十分原始，更像是一个底层设施。而其上层的封装则是DirectByteBuffer，这个才是最终留给开发者使用的。DirectByteBuffer的分配是通过ByteBuffer.allocateDirect(int capacity)方法来实现的。
DirectByteBuffer申请内存的源码如下：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
| DirectByteBuffer(int cap) {
super(-1, 0, cap, cap);
boolean pa = VM.isDirectMemoryPageAligned();
int ps = Bits.pageSize();
long size = Math.max(1L, (long)cap + (pa ? ps : 0));
Bits.reserveMemory(size, cap);
long base = 0;
try {
base = unsafe.allocateMemory(size);
} catch (OutOfMemoryError x) {
Bits.unreserveMemory(size, cap);
throw x;
}
unsafe.setMemory(base, size, (byte) 0);
if (pa && (base % ps != 0)) {
address = base + ps - (base & (ps - 1));
} else {
address = base;
}
cleaner = Cleaner.create(this, new Deallocator(base, size, cap));
att = null;
}
|
整个DirectByteBuffer分配过程中，比较需要关注的Bits.reserveMemory()和Cleaner,Deallocator，其中Bits.reserveMemory()与分配相关，Cleaner、Deallocator则与内存释放相关。
Bits.reserveMemory()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
| static void reserveMemory(long size, int cap) {
if (!memoryLimitSet && VM.isBooted()) {
maxMemory = VM.maxDirectMemory();
memoryLimitSet = true;
}
if (tryReserveMemory(size, cap)) {
return;
}
final JavaLangRefAccess jlra = SharedSecrets.getJavaLangRefAccess();
while (jlra.tryHandlePendingReference()) {
if (tryReserveMemory(size, cap)) {
return;
}
}
System.gc();
boolean interrupted = false;
try {
long sleepTime = 1;
int sleeps = 0;
while (true) {
if (tryReserveMemory(size, cap)) {
return;
}
if (sleeps >= MAX_SLEEPS) {
break;
}
if (!jlra.tryHandlePendingReference()) {
try {
Thread.sleep(sleepTime);
sleepTime <<= 1;
sleeps++;
} catch (InterruptedException e) {
interrupted = true;
}
}
}
throw new OutOfMemoryError("Direct buffer memory");
} finally {
if (interrupted) {
Thread.currentThread().interrupt();
}
}
}
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
private static boolean tryReserveMemory(long size, int cap) {
long totalCap;
while (cap <= maxMemory - (totalCap = totalCapacity.get())) {
if (totalCapacity.compareAndSet(totalCap, totalCap + cap)) {
reservedMemory.addAndGet(size);
count.incrementAndGet();
return true;
}
}
return false;
}
|
内存释放
内存释放是通过Cleaner和Deallocator来实现的。
Deallocator
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| private static class Deallocator implements Runnable {
private static Unsafe unsafe = Unsafe.getUnsafe();
private long address;
private long size;
private int capacity;
private Deallocator(long address, long size, int capacity) {
assert (address != 0);
this.address = address;
this.size = size;
this.capacity = capacity;
}
public void run() {
if (address == 0) {
return;
}
unsafe.freeMemory(address);
address = 0;
Bits.unreserveMemory(size, capacity);
}
}
|
这个类中主要方法为run()，里面的步骤也很简单，包含两步
- 使用unsafe释放内存
- 利用Bits管理内存的释放，就是标记一下该内存已释放
每个DirectByteBuffer都有一个相对应的Deallocator，而Deallocator则是由Cleaner来进行调度。
Cleaner
Cleaner的数据结构为一个双向链表，如下
1
2
3
4
| private static Cleaner first = null;
private Cleaner next = null;
private Cleaner prev = null;
private final Runnable thunk;
|
Cleaner中主要包含如下操作，add, remove,clean
主要操作
1. add
1
2
3
4
5
6
7
8
9
| private static synchronized Cleaner add(Cleaner var0) {
if (first != null) {
var0.next = first;
first.prev = var0;
}
first = var0;
return var0;
}
|
add操作就是不断地将新的Cleaner节点添加在链表头部，之后将头节点指针指向新的Cleaner
2. remove
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| private static synchronized boolean remove(Cleaner var0) {
if (var0.next == var0) {
return false;
} else {
if (first == var0) {
if (var0.next != null) {
first = var0.next;
} else {
first = var0.prev;
}
}
if (var0.next != null) {
var0.next.prev = var0.prev;
}
if (var0.prev != null) {
var0.prev.next = var0.next;
}
var0.next = var0;
var0.prev = var0;
return true;
}
}
|
remove操作就是将Cleaner节点从链表中删除
3. clean
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| public void clean() {
if (remove(this)) {
try {
this.thunk.run();
} catch (final Throwable var2) {
AccessController.doPrivileged(new PrivilegedAction<Void>() {
public Void run() {
if (System.err != null) {
(new Error("Cleaner terminated abnormally", var2)).printStackTrace();
}
System.exit(1);
return null;
}
});
}
}
}
|
clean操作则是移除Cleaner节点并调用Deallocator的run()方法
清理过程
疑问 Cleaner.clean()又是由谁在何时调用的呢？
仔细观察可以发现，Cleaner继承了PhantomReference，其referent为DirectByteBuffer
Reference
在Reference初次加载的过程中会调用一段静态代码
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| static {
ThreadGroup tg = Thread.currentThread().getThreadGroup();
for (ThreadGroup tgn = tg;
tgn != null;
tg = tgn, tgn = tg.getParent());
Thread handler = new ReferenceHandler(tg, "Reference Handler");
handler.setPriority(Thread.MAX_PRIORITY);
handler.setDaemon(true);
handler.start();
SharedSecrets.setJavaLangRefAccess(new JavaLangRefAccess() {
@Override
public boolean tryHandlePendingReference() {
return tryHandlePending(false);
}
});
}
|
这段代码中包含了两种可以调用Cleaner的方式：
- ReferenceHandler，会不停地循环调用tryHandlePending
- SharedSecrets.JavaLangRefAccess，在Bits.reserveMemory()中被调用
事实上直接内存的回收过程也的确是由这两种方式混合组成，这两种方式有一个共同点，他们都会调用Reference.tryHandlePending()方法。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
| static boolean tryHandlePending(boolean waitForNotify) {
Reference<Object> r;
Cleaner c;
try {
synchronized (lock) {
if (pending != null) {
r = pending;
c = r instanceof Cleaner ? (Cleaner) r : null;
pending = r.discovered;
r.discovered = null;
} else {
if (waitForNotify) {
lock.wait();
}
return waitForNotify;
}
}
} catch (OutOfMemoryError x) {
Thread.yield();
return true;
} catch (InterruptedException x) {
return true;
}
if (c != null) {
c.clean();
return true;
}
ReferenceQueue<? super Object> q = r.queue;
if (q != ReferenceQueue.NULL) q.enqueue(r);
return true;
}
|
其中pending和discovered由JVM来操作，两个共同组成一个等待队列链表，对于PhantomReference的情况，当对象不存在其他引用，便会直接加入等待队列。每当等待队列中出现Cleaner，就会执行其clean()方法。
总结
1. 整个DirectByteBuffer的分配与释放流程如下
![image-20181225181100276]()
2. -XX:MaxDirectMemorySize参数只对由DirectByteBuffer分配的内存有效，对Unsafe直接分配的内存无效
native方法
疑问 native方法中分配的内存是否是属于DirectByteBuffer对象呢?
这个疑问来自于一次内存泄漏问题的排查，一直没有机会去研究，正好借这次机会寻找一下该问题的答案。
demo
写了一个简单的demo程序如下
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
public class NativeMain {
public native void allocateMemory();
static {
System.setProperty("java.library.path", ".");
System.loadLibrary("nativemain");
}
public static void main(String[] args) throws Exception {
NativeMain nativeMain = new NativeMain();
while (true) {
for (int i = 0; i < 10000; i++) {
nativeMain.allocateMemory();
}
Thread.sleep(1);
}
}
}
|
1
2
3
4
5
6
7
8
|
#include "jni.h"
#include "NativeMain.h"
#include <stdlib.h>
JNIEXPORT void JNICALL Java_NativeMain_allocateMemory(JNIEnv *, jobject) {
char *ptr = (char*)malloc(1000);
}
|
![image-20181225185252446]()
![image-20181225185325785]()
运行发现native方法分配的内存并不会产生DirectByteBuffer对象，同样的也不受-XX:MaxDirectMemorySize影响。
