1 介绍
Codec2是Android中多媒体相关的软件框架，是MediaCodec的中间件，往上对接MediaCodec Native层，往下提供新的API标准供芯片底层的编解码去实现，也就是说适配了Codec2，就可以通过MediaCodec来调用芯片的硬件编解码的能力，来完成一些多媒体相关的功能。这篇文章先从下到上讲解适配Codec2需要实现的接口，然后再从上到下分析MediaCodec的流程来分析这些接口是如何调用的。主要抓住以下两条主线
- 输入buffer是如何送到编解码组件的
- 编解码完成之后输入buffer和输出buffer是如何上报的
开始之前需要如下前置知识
- Android异步消息机制
- Android HIDL
- 视频编解码基本流程
2 适配
下面以Android中的软件Hevc编码器的实现为例，分析如何适配Codec2接口，首先看codec2的基本架构，分为4层，第一层是sfplugin，负责和上层的stage fright对接，下面是HIDL，是各个组件的硬件抽象层，再往下是Core，封装了一个组件需要实现的接口，最后是具体的Component实现，这里以Hevc软编码器为例，再往下就是具体的编解码库了，软编码器调用的是ihevce相关的接口
2.1 目录结构
Android中的codec2目录在frameworks/av/media/codec2
.
├── Android.mk
├── components # 适配的组件，如h264、hevc软件编解码等，由HIDL调用
├── core # codec2内核，对接component
├── docs
├── faultinjection
├── fuzzer
├── hidl # hal层实现
├── OWNERS
├── sfplugin # 和stagefright的对接层
├── TEST_MAPPING
├── tests
└── vndkcore层组织了components的运行方式，这里先分析core层，其中主要的文件是：core/include/C2Component.h，其中包含了C2Component和C2ComponentInterface两个类
2.2 C2Component
C2Component中定义了一个组件需要实现的接口，定义如下，这里需要关注的两个重要接口
queue_nb：可以看作送帧/送流的接口，在编解码之前，将需要处理的原始数据送入，该接口必现设计为非阻塞的onWorkDone_nb：当一帧数据处理完了之后会回调该接口
class C2Component {
public:
class Listener {
public:
virtual void onWorkDone_nb(std::weak_ptr<C2Component> component,
std::list<std::unique_ptr<C2Work>> workItems) = 0;
virtual void onTripped_nb(std::weak_ptr<C2Component> component,
std::vector<std::shared_ptr<C2SettingResult>> settingResult) = 0;
virtual void onError_nb(std::weak_ptr<C2Component> component,
uint32_t errorCode) = 0;
virtual ~Listener() = default;
};
...
/* Queues up work for the component. */
virtual c2_status_t queue_nb(std::list<std::unique_ptr<C2Work>>* const items) = 0;
/*
* Announces a work to be queued later for the component. This reserves a slot for the queue
* to ensure correct work ordering even if the work is queued later.
*/
virtual c2_status_t announce_nb(const std::vector<C2WorkOutline> &items) = 0;
enum flush_mode_t : uint32_t {
/// flush work from this component only
FLUSH_COMPONENT,
/// flush work from this component and all components connected downstream from it via
/// tunneling
FLUSH_CHAIN = (1 << 16),
};
/*
* Discards and abandons any pending work for the component, and optionally any component
* downstream.
*/
virtual c2_status_t flush_sm(flush_mode_t mode, std::list<std::unique_ptr<C2Work>>* const flushedWork) = 0;
enum drain_mode_t : uint32_t {
DRAIN_COMPONENT_WITH_EOS,
DRAIN_COMPONENT_NO_EOS = (1 << 0),
DRAIN_CHAIN = (1 << 16),
};
/*
* Drains the component, and optionally downstream components. This is a signalling method;
* as such it does not wait for any work completion.
* Marks last work item as "drain-till-here", so component is notified not to wait for further
* work before it processes work already queued. This method can also used to set the
* end-of-stream flag after work has been queued. Client can continue to queue further work
* immediately after this method returns.
*/
virtual c2_status_t drain_nb(drain_mode_t mode) = 0;
// STATE CHANGE METHODS
// =============================================================================================
virtual c2_status_t start() = 0;
virtual c2_status_t stop() = 0;
virtual c2_status_t reset() = 0;
virtual c2_status_t release() = 0;
virtual std::shared_ptr<C2ComponentInterface> intf() = 0;
virtual ~C2Component() = default;
};2.3 C2ComponentInterface
TODO
2.4 SimpleC2Component
SimpleC2Component提供了一种组件的实现，后面不同的实现只需要继承该实现即可，其中重要的接口设计如下。可以看到SimpleC2Component继承了C2Component，并将实现了一个AMessage和AHandle的异步消息机制，下面分别对相应的函数进行分析
class SimpleC2Component
: public C2Component, public std::enable_shared_from_this<SimpleC2Component> {
public:
explicit SimpleC2Component(
const std::shared_ptr<C2ComponentInterface> &intf);
virtual ~SimpleC2Component();
// 设置对应的回调
virtual c2_status_t setListener_vb(
const std::shared_ptr<Listener> &listener, c2_blocking_t mayBlock) override;
// 实现queue_nb接口
virtual c2_status_t queue_nb(std::list<std::unique_ptr<C2Work>>* const items) override;
...
// 实际处理一帧数据的函数
bool processQueue();
protected:
...
// 具体的编解码过程，由子类实现
virtual void process(
const std::unique_ptr<C2Work> &work,
const std::shared_ptr<C2BlockPool> &pool) = 0;
...
private:
const std::shared_ptr<C2ComponentInterface> mIntf;
class WorkHandler : public AHandler {
public:
...
protected:
// 异步消息处理接口
void onMessageReceived(const sp<AMessage> &msg) override;
private:
...
};
...
struct ExecState {
ExecState() : mState(UNINITIALIZED) {}
int mState;
std::shared_ptr<C2Component::Listener> mListener;
};
// 状态机
Mutexed<ExecState> mExecState;
sp<ALooper> mLooper;
sp<WorkHandler> mHandler;
class WorkQueue {
...
}
// 待处理的工作队列
Mutexed<WorkQueue> mWorkQueue;
...2.4.1 setListener_vb
该接口将上面的传递的listener设置到状态机中，后续时机合适时再回调相应接口
代码语言：cpp复制c2_status_t SimpleC2Component::setListener_vb(
const std::shared_ptr<C2Component::Listener> &listener, c2_blocking_t mayBlock) {
mHandler->setComponent(shared_from_this());
Mutexed<ExecState>::Locked state(mExecState);
...
state->mListener = listener;
return C2_OK;
}2.4.2 queue_nb
该函数实现了C2Component.queue_nb，实际只是将work放到队列中，并且发起一个异步消息然后返回，满足非阻塞的要求
c2_status_t SimpleC2Component::queue_nb(std::list<std::unique_ptr<C2Work>> * const items) {
...
bool queueWasEmpty = false;
{
Mutexed<WorkQueue>::Locked queue(mWorkQueue);
queueWasEmpty = queue->empty();
while (!items->empty()) {
queue->push_back(std::move(items->front()));
items->pop_front();
}
}
if (queueWasEmpty) {
// 发起一次数据处理的消息，会将mWorkQueue中帧处理完
(new AMessage(WorkHandler::kWhatProcess, mHandler))->post();
}
return C2_OK;
}2.4.3 onMessageReceived
该函数是异步消息处理接口，当发出对应的消息时最终会调用到该函数中，这里processQueue()函数是处理一帧数据，然后返回当前队列是否还有未处理的数据
void SimpleC2Component::WorkHandler::onMessageReceived(const sp<AMessage> &msg) {
...
switch (msg->what()) {
case kWhatProcess: {
if (mRunning) {
// 如果processQueue()返回true，重新发一个kWhatProcess消息
if (thiz->processQueue()) {
(new AMessage(kWhatProcess, this))->post();
}
} else {
ALOGV("Ignore process message as we're not running");
}
break;
}
...
}
}2.4.4 processQueue
该函数完成编解码的实际操作，代码如下，其中process和onWorkDone_nb都由子类实现，work先从之前的mWorkQueue队列中拿出，再调用process进行处理，process是一个虚函数，由子类实现，处理完成之后再调用listener->onWorkDone_nb通知处理完成事件onWorkDone_nb也是一个虚函数，由子类实现
bool SimpleC2Component::processQueue() {
...
bool hasQueuedWork = false;
{
Mutexed<WorkQueue>::Locked queue(mWorkQueue);
if (queue->empty()) {
return false;
}
...
work = queue->pop_front();
hasQueuedWork = !queue->empty();
}
if (!mOutputBlockPool) {
c2_status_t err = [this] {
...
std::shared_ptr<C2BlockPool> blockPool;
err = GetCodec2BlockPool(poolId, shared_from_this(), &blockPool);
ALOGD("Using output block pool with poolID %llu => got %llu - %d",
(unsigned long long)poolId,
(unsigned long long)(
blockPool ? blockPool->getLocalId() : 111000111),
err);
if (err == C2_OK) {
// mOutputBlockPool作用是什么？
mOutputBlockPool = std::make_shared<BlockingBlockPool>(blockPool);
}
return err;
}();
...
}
...
process(work, mOutputBlockPool);
...
// 查看工作链中已处理完的数量，其实就是看proess是否处理成功？
if (work->workletsProcessed != 0u) {
queue.unlock();
Mutexed<ExecState>::Locked state(mExecState);
ALOGV("returning this work");
std::shared_ptr<C2Component::Listener> listener = state->mListener;
state.unlock();
listener->onWorkDone_nb(shared_from_this(), vec(work));
} else {
...
}
return hasQueuedWork;
}2.5 C2SoftHevcEnc
下面分析process的实现，以Android Hevc软编码器为例，类继承自SimpleC2Component，实现如下，主要流程是从work中取出输入buffer，然后进行一帧编码，然后再把输出设置到work中
struct C2SoftHevcEnc : public SimpleC2Component {
...
void process(const std::unique_ptr<C2Work>& work,
const std::shared_ptr<C2BlockPool>& pool) override;
...
}
void C2SoftHevcEnc::process(const std::unique_ptr<C2Work>& work,
const std::shared_ptr<C2BlockPool>& pool) {
...
std::shared_ptr<const C2GraphicView> view;
std::shared_ptr<C2Buffer> inputBuffer = nullptr;
...
if (!work->input.buffers.empty()) {
// 获取work中的input buffer
inputBuffer = work->input.buffers[0];
// 将buffer与view绑定
view = std::make_shared<const C2GraphicView>(
inputBuffer->data().graphicBlocks().front().map().get());
...
}
...
ihevce_inp_buf_t s_encode_ip{};
ihevce_out_buf_t s_encode_op{};
...
// 将view转换为s_encode_ip
status = setEncodeArgs(&s_encode_ip, view.get(), workIndex);
...
memset(&s_encode_op, 0, sizeof(s_encode_op));
...
if (inputBuffer) {
// 以s_encode_ip为输入，完成一帧hevc编码
err = ihevce_encode(mCodecCtx, &s_encode_ip, &s_encode_op);
...
}
...
if (s_encode_op.i4_bytes_generated) {
// s_encode_op有数据，将其配置到work中
finishWork(s_encode_op.u8_pts, work, pool, &s_encode_op);
}
}下面分析finishWork，可见软编码输出的buffer最终是拷贝到了一个C2Buffer中，最终再放入work->worklets.front()->output.buffers队列
void C2SoftHevcEnc::finishWork(uint64_t index,
const std::unique_ptr<C2Work>& work,
const std::shared_ptr<C2BlockPool>& pool,
ihevce_out_buf_t* ps_encode_op) {
std::shared_ptr<C2LinearBlock> block;
...
// 获取一个LinearBlock
c2_status_t status =
pool->fetchLinearBlock(ps_encode_op->i4_bytes_generated, usage, &block);
...
// 将block映射到view
C2WriteView wView = block->map().get();
...
// 将输出buffer内容拷贝到block中
memcpy(wView.data(), ps_encode_op->pu1_output_buf,
ps_encode_op->i4_bytes_generated);
// 由block创建一个C2Buffer
std::shared_ptr<C2Buffer> buffer =
createLinearBuffer(block, 0, ps_encode_op->i4_bytes_generated);
...
// 将buffer放到work中
auto fillWork = [buffer](const std::unique_ptr<C2Work>& work) {
work->worklets.front()->output.flags = (C2FrameData::flags_t)0;
work->worklets.front()->output.buffers.clear();
work->worklets.front()->output.buffers.push_back(buffer);
work->worklets.front()->output.ordinal = work->input.ordinal;
work->workletsProcessed = 1u;
};
if (work && c2_cntr64_t(index) == work->input.ordinal.frameIndex) {
fillWork(work);
...
} else {
finish(index, fillWork);
}
}3 HIDL
3.1 概念
C2Component的上层是HIDL层，可以理解为Android的HAL层，这一层的头文件所继承的接口由一种叫做HIDL（Hardware Interface Definition Language）的语言动态生成，输出到out目录下，例如其中的IComponent头文件位于：
out/soong/.intermediates/hardware/interfaces/media/c2/1.0/android.hardware.media.c2@1.0_genc _headers/gen/android/hardware/media/c2/1.0/IComponent.h
其定义如下
代码语言：cpp复制struct IComponent : public ::android::hidl::base::V1_0::IBase {
typedef ::android::hardware::details::i_tag _hidl_tag;
static const char* descriptor;
virtual bool isRemote() const override { return false; }
virtual ::android::hardware::Return<::android::hardware::media::c2::V1_0::Status> queue(const ::android::hardware::media::c2::V1_0::WorkBundle& workBundle) = 0;
using flush_cb = std::function<void(::android::hardware::media::c2::V1_0::Status status, const ::android::hardware::media::c2::V1_0::WorkBundle& flushedWorkBundle)>;
virtual ::android::hardware::Return<void> flush(flush_cb _hidl_cb) = 0;
...
}对应的HIDL文件为hardware/interfaces/media/c2/1.0/IComponent.hal
interface IComponent {
queue(WorkBundle workBundle) generates (Status status);
flush(
) generates (
Status status,
WorkBundle flushedWorkBundle
);
...
}3.2 Component
Component继承自IComponent.h，也就会实现其中的接口，通过Component就可以调用上面讲到的C2Component，其定义如下，我们重点看下其中的queue和Listener。
struct Component : public IComponent,
public std::enable_shared_from_this<Component> {
...
virtual Return<Status> queue(const WorkBundle& workBundle) override;
...
protected:
...
struct Listener;
...
}3.2.1 queue
queue的实现如下，这里的mComponent实际上就是C2Component，这里如何实现的暂且不表，后面再分析，以软编码为例，因此这里最终调用的queue_nb实际上调用的是SimpleC2Component.queue_nb
Return<Status> Component::queue(const WorkBundle& workBundle) {
std::list<std::unique_ptr<C2Work>> c2works;
if (!objcpy(&c2works, workBundle)) {
return Status::CORRUPTED;
}
// Register input buffers.
for (const std::unique_ptr<C2Work>& work : c2works) {
if (work) {
InputBufferManager::
registerFrameData(mListener, work->input);
}
}
/* 调用C2Component.queue_nb，如果组件继承自SimpleC2Component，
* 则调用SimpleC2Component.queue_nb
* 注意这里是非阻塞的
*/
return static_cast<Status>(mComponent->queue_nb(&c2works));
}3.2.1 Listener
再看Listener的定义，这里实际继承的是C2Component::Listener，并且对onWorkDone_nb进行了实现，因为onWorkDone_nb是回调函数，因此由调用者实现也是符合预期的。这里其实就是调用了另一个回调listener->onWorkDone
struct Component::Listener : public C2Component::Listener {
Listener(const sp<Component>& component) :
mComponent(component),
mListener(component->mListener) {
}
...
virtual void onWorkDone_nb(
std::weak_ptr<C2Component> /* c2component */,
std::list<std::unique_ptr<C2Work>> c2workItems) override {
...
sp<IComponentListener> listener = mListener.promote();
if (listener) {
WorkBundle workBundle;
// 拷贝到workBundle
sp<Component> strongComponent = mComponent.promote();
beginTransferBufferQueueBlocks(c2workItems, true);
if (!objcpy(&workBundle, c2workItems, strongComponent ?
&strongComponent->mBufferPoolSender : nullptr)) {
...
}
// 回调
Return<void> transStatus = listener->onWorkDone(workBundle);
...
endTransferBufferQueueBlocks(c2workItems, true, true);
}
}以上的Listener只是一个定义，还要看该Listener是在哪里声明的，以及是什么时候注册的。首先第一个问题，Listener声明是在SimpleC2Component::ExecState.mListener，只要继承了SimpleC2Component内部就有该成员，第二个问题，注册是在Component::initListener函数中，定义如下，同样以软编码为例，这里mComponent->setListener_vb调用实际是SimpleC2Component.setListener_vb
void Component::initListener(const sp<Component>& self) {
std::shared_ptr<C2Component::Listener> c2listener =
std::make_shared<Listener>(self);
/* 调用C2Component.setListener_vb，如果组件继承自SimpleC2Component，
* 那么调用SimpleC2Component.setListener_vb
*/
c2_status_t res = mComponent->setListener_vb(c2listener, C2_DONT_BLOCK);
...
}关于HIDL还有很长的一个调用流程，这里暂且分析到这里，后续再从MediaCodec从上往下分析，看如何调用到HIDL的
4 MediaCodec
MediaCodec是Android app层来进行多媒体编解码的模块，分为java层和cpp层，这里只从cpp层切入
4.1 调用流程
首先来看MediaCodec是如何使用的，由于MediaCodec也基于AMessage机制，因此先要创建一个ALooper，然后传递到MediaCodec中，下面以创建Hevc编码器为例，伪代码如下
代码语言：cpp复制sp<android::ALooper> looper = new android::ALooper; // 创建ALooper
looper->setName("TestLooper");
looper->start();
// 创建编码器，自动查找合适的编码器组件
mMediaCodec = MediaCodec::CreateByType(looper, "video/hevc", true);
mMediaCodec->start();
for (uint32_t i = 0; i < 100; i ) {
// 从MediaCodec取一个输入buffer的index
mMediaCodec->dequeueInputBuffer(&inputBufIdx, sTimeOut);
// 获取输入buffer
sp<MediaCodecBuffer> inputBuf;
mMediaCodec->getInputBuffer(inputBufIdx, &inputBuf);
// 写YUV到输入buffer
int readSize = writeYUVToBuffer(inputBuf->data(), inputBuf->size());
uint32_t flags = 0;
if (readSize == 0) {
flags = BUFFER_FLAG_END_OF_STREAM;
}
// 将输入buffer送回MediaCodec
mMediaCodec->queueInputBuffer(inputBufIdx, 0, readSize, getCurTimeUs(), flags);
// 从MediaCodec取一个输出buffer的index
size_t outputBufIdx, outputOffset, outputSize;
mMediaCodec->dequeueOutputBuffer(&outputBufIdx, &outputOffset, &outputSize,
&outputPts, &outputFlags, sTimeOut);
// 获取输出buffer
sp<MediaCodecBuffer> outputBuf;
mMediaCodec->getOutputBuffer(outputBufIdx, &outputBuf);
// 将码流写到文件
writeStreamToFile(outputBuf);
// 向MediaCodec释放输出buffer
mMediaCodec->releaseOutputBuffer(outputBufIdx)
}4.2 发送输入buffer流程
4.2.1 queueInputBuffer
首先分析queueInputBuffer，看YUV是如何送到具体的编码器组件的，可以看到实际这里只是发了一个异步消息，将index送进去，然后调用PostAndAwaitResponse阻塞等待消息响应
status_t MediaCodec::queueInputBuffer(
size_t index,
size_t offset,
size_t size,
int64_t presentationTimeUs,
uint32_t flags,
AString *errorDetailMsg) {
if (errorDetailMsg != NULL) {
errorDetailMsg->clear();
}
sp<AMessage> msg = new AMessage(kWhatQueueInputBuffer, this);
msg->setSize("index", index);
msg->setSize("offset", offset);
msg->setSize("size", size);
msg->setInt64("timeUs", presentationTimeUs);
msg->setInt32("flags", flags);
msg->setPointer("errorDetailMsg", errorDetailMsg);
sp<AMessage> response;
// 注意这里是阻塞等待
return PostAndAwaitResponse(msg, &response);
}4.2.2 QueueInputBuffer消息处理
再查看MediaCodec的消息处理函数，由于queueInputBuffer是阻塞等待的，因此这里要调用PostReplyWithError之后，queueInputBuffer才返回，这里往后是调用了onQueueInputBuffer函数
void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
switch (msg->what()) {
...
case kWhatQueueInputBuffer:
{
sp<AReplyToken> replyID;
CHECK(msg->senderAwaitsResponse(&replyID));
...
status_t err = UNKNOWN_ERROR;
if (!mLeftover.empty()) {
mLeftover.push_back(msg);
size_t index;
msg->findSize("index", &index);
err = handleLeftover(index);
} else {
err = onQueueInputBuffer(msg);
}
// 消息响应
PostReplyWithError(replyID, err);
break;
}
...
}
}下面分析onQueueInputBuffer，可以看到这里实际是通过index获取到MediaCodecBuffer，并且送到了mBufferChannel->queueInputBuffer中，注意执行到此处最外层的queueInputBuffer仍然在等待消息响应，因此到这里为止都是阻塞的
struct MediaCodec : public AHandler {
private:
...
// 两个port，输入是0，输出是1
std::vector<BufferInfo> mPortBuffers[2];
...
std::shared_ptr<BufferChannelBase> mBufferChannel;
}
status_t MediaCodec::onQueueInputBuffer(const sp<AMessage> &msg) {
size_t index;
...
CHECK(msg->findSize("index", &index));
...
BufferInfo *info = &mPortBuffers[kPortIndexInput][index];
sp<MediaCodecBuffer> buffer = info->mData;
...
if (hasCryptoOrDescrambler() && !c2Buffer && !memory) {
// 安全编码流程
...
} else {
// 非安编码流程
mBufferChannel->queueInputBuffer(buffer);
}
...
}4.3 获取输入buffer流程
4.3.1 dequeueInputBuffer
下面看dequeueInputBuffer函数，该函数是获取一个空闲的输入buffer，这里是否空闲仍然需要底层的组件来通知，因此需要分析这里的向上通知的流程。可以看到该函数仍然是发起一个异步消息，然后阻塞等待响应
status_t MediaCodec::dequeueInputBuffer(size_t *index, int64_t timeoutUs) {
sp<AMessage> msg = new AMessage(kWhatDequeueInputBuffer, this);
msg->setInt64("timeoutUs", timeoutUs);
sp<AMessage> response;
status_t err;
if ((err = PostAndAwaitResponse(msg, &response)) != OK) {
return err;
}
CHECK(response->findSize("index", index));
return OK;
}4.3.2 kWhatDequeueInputBuffer消息处理
再回到onMessageReceived函数，最终会调用到dequeuePortBuffer函数，可以看到MediaCodec中有一个mAvailPortBuffers链表，存储着当前可用的buffer的index，当dequeue的时候只需要从这个链表中拿出第一个index就行了
void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
switch (msg->what()) {
...
case kWhatDequeueInputBuffer:
{
sp<AReplyToken> replyID;
CHECK(msg->senderAwaitsResponse(&replyID));
handleDequeueInputBuffer(replyID, true /* new request */);
...
break;
}
...
}
}
bool MediaCodec::handleDequeueInputBuffer(const sp<AReplyToken> &replyID, bool newRequest) {
...
ssize_t index = dequeuePortBuffer(kPortIndexInput);
...
sp<AMessage> response = new AMessage;
response->setSize("index", index);
// 响应消息
response->postReply(replyID);
return true;
}
struct MediaCodec : public AHandler {
private:
...
// 存储可用的index
List<size_t> mAvailPortBuffers[2];
...
}
ssize_t MediaCodec::dequeuePortBuffer(int32_t portIndex) {
...
List<size_t> *availBuffers = &mAvailPortBuffers[portIndex];
size_t index = *availBuffers->begin();
...
availBuffers->erase(availBuffers->begin());
return index;
}4.3.3 updateBuffers
继续分析availBuffers是什么时候更新的，查看updateBuffers函数，发现其中的index是通过异步消息上报的，updateBuffers可以更新输入队列和输出队列，我们只看输入队列，发现是在收到kWhatFillThisBuffer消息时更新的
size_t MediaCodec::updateBuffers(
int32_t portIndex, const sp<AMessage> &msg) {
CHECK(portIndex == kPortIndexInput || portIndex == kPortIndexOutput);
size_t index;
// 从消息中获取index
CHECK(msg->findSize("index", &index));
sp<RefBase> obj;
CHECK(msg->findObject("buffer", &obj));
// 获取buffer的实体
sp<MediaCodecBuffer> buffer = static_cast<MediaCodecBuffer *>(obj.get());
{
Mutex::Autolock al(mBufferLock);
// 如果index大于了队列的大小，则扩充队列
if (mPortBuffers[portIndex].size() <= index) {
mPortBuffers[portIndex].resize(align(index 1, kNumBuffersAlign));
}
// 将buffer放到队列中，索引为index
mPortBuffers[portIndex][index].mData = buffer;
}
// 将index放入可用的链表
mAvailPortBuffers[portIndex].push_back(index);
return index;
}
void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
switch (msg->what()) {
...
case kWhatFillThisBuffer:
{
updateBuffers(kPortIndexInput, msg);
...
break;
}
...
}4.3.4 BufferCallback
至此可以发现，MediaCodec中的输入和输出buffer是否可用时由异步消息通知的，而异步消息又是底层的组件通过回调MediaCodec的接口发送的，这部分代码在BufferCallback中，实际上BufferCallback是继承了CodecBase::BufferCallback类，并实现了其中的接口，可以看到输入和输出buffer都是在这里回调的
class BufferCallback : public CodecBase::BufferCallback {
public:
explicit BufferCallback(const sp<AMessage> ¬ify);
virtual ~BufferCallback() = default;
virtual void onInputBufferAvailable(
size_t index, const sp<MediaCodecBuffer> &buffer) override;
virtual void onOutputBufferAvailable(
size_t index, const sp<MediaCodecBuffer> &buffer) override;
private:
const sp<AMessage> mNotify;
};
BufferCallback::BufferCallback(const sp<AMessage> ¬ify)
: mNotify(notify) {}
void BufferCallback::onInputBufferAvailable(
size_t index, const sp<MediaCodecBuffer> &buffer) {
sp<AMessage> notify(mNotify->dup());
notify->setInt32("what", kWhatFillThisBuffer);
notify->setSize("index", index);
notify->setObject("buffer", buffer);
notify->post();
}CodecBase实际上又是codec2中的接口了，属于sfplugin模块，篇幅原因这里暂时不往下分析了
4.4 获取输出buffer流程
4.4.1 dequeueOutputBuffer
获取输出buffer和获取输入buffer是一个流程，都是发出一个消息然后等待响应，这里输出buffer是底层组件处理好的数据，因此也需要底层组件来通知
代码语言：cpp复制status_t MediaCodec::dequeueOutputBuffer(
size_t *index,
size_t *offset,
size_t *size,
int64_t *presentationTimeUs,
uint32_t *flags,
int64_t timeoutUs) {
sp<AMessage> msg = new AMessage(kWhatDequeueOutputBuffer, this);
msg->setInt64("timeoutUs", timeoutUs);
sp<AMessage> response;
status_t err;
if ((err = PostAndAwaitResponse(msg, &response)) != OK) {
return err;
}
CHECK(response->findSize("index", index));
CHECK(response->findSize("offset", offset));
CHECK(response->findSize("size", size));
CHECK(response->findInt64("timeUs", presentationTimeUs));
CHECK(response->findInt32("flags", (int32_t *)flags));
return OK;
}4.4.2 kWhatDequeueOutputBuffer消息处理
获取输出buffer和获取输入buffer相同，都是调用的dequeuePortBuffer函数，只不过传入参数不同，也就是这里仍然是从mAvailPortBuffers队列中获取以及处理完成的buffer index
void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
switch (msg->what()) {
...
case kWhatDequeueOutputBuffer:
{
sp<AReplyToken> replyID;
CHECK(msg->senderAwaitsResponse(&replyID));
...
handleDequeueOutputBuffer(replyID, true /* new request */))
...
break;
}
...
}
}
bool MediaCodec::handleDequeueOutputBuffer(const sp<AReplyToken> &replyID, bool newRequest) {
...
} else {
sp<AMessage> response = new AMessage;
...
dequeuePortBuffer(kPortIndexOutput);
...
response->postReply(replyID);
}
}4.4.3 mAvailPortBuffers的更新
获取输入buffer仍然是调用updateBuffers进行mAvailPortBuffers的更新，只不过这里是从kWhatDrainThisBuffer更新的，而发出该消息的地方也是另一个onOutputBufferAvailable回调
void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
switch (msg->what()) {
...
case kWhatDrainThisBuffer:
{
...
updateBuffers(kPortIndexInput, msg);
...
break;
}
...
}
void BufferCallback::onOutputBufferAvailable(
size_t index, const sp<MediaCodecBuffer> &buffer) {
sp<AMessage> notify(mNotify->dup());
notify->setInt32("what", kWhatDrainThisBuffer);
notify->setSize("index", index);
notify->setObject("buffer", buffer);
notify->post();
}4.5 释放输出buffer
4.5.1 releaseOutputBuffer
释放输出buffer流程比较简单，也是发一个异步消息然后阻塞等待，在消息处理中将mPortBuffers队列对应的buffer清除，然后最后调用mBufferChannel->discardBuffer通知底层组件
status_t MediaCodec::releaseOutputBuffer(size_t index) {
sp<AMessage> msg = new AMessage(kWhatReleaseOutputBuffer, this);
msg->setSize("index", index);
sp<AMessage> response;
return PostAndAwaitResponse(msg, &response);
}
void MediaCodec::onMessageReceived(const sp<AMessage> &msg) {
switch (msg->what()) {
...
case kWhatDrainThisBuffer:
{
...
onReleaseOutputBuffer(msg);
PostReplyWithError(replyID, err);
break;
}
...
}
status_t MediaCodec::onReleaseOutputBuffer(const sp<AMessage> &msg) {
size_t index;
CHECK(msg->findSize("index", &index));
...
BufferInfo *info = &mPortBuffers[kPortIndexOutput][index];
...
sp<MediaCodecBuffer> buffer;
{
Mutex::Autolock al(mBufferLock);
info->mOwnedByClient = false;
buffer = info->mData;
info->mData.clear();
}
if (render && buffer->size() != 0) {
...
} else {
mBufferChannel->discardBuffer(buffer);
}
}5 总结
总结以上流程，除开HIDL和sfplugin，这里关于使用MediaCodec调用codec2进行编解码的流程进行了大体的分析，首先看整体的运作流程和实现关系如下
