序
本文主要研究一下rocketmq-proxy的popMessage
MessagingProcessor
org/apache/rocketmq/proxy/processor/MessagingProcessor.java
代码语言：javascript复制public interface MessagingProcessor extends StartAndShutdown {
//......
CompletableFuture<PopResult> popMessage(
ProxyContext ctx,
QueueSelector queueSelector,
String consumerGroup,
String topic,
int maxMsgNums,
long invisibleTime,
long pollTime,
int initMode,
SubscriptionData subscriptionData,
boolean fifo,
PopMessageResultFilter popMessageResultFilter,
String attemptId,
long timeoutMillis
);
//......
}
MessagingProcessor接口定义了popMessage方法
DefaultMessagingProcessor
org/apache/rocketmq/proxy/processor/DefaultMessagingProcessor.java
代码语言：javascript复制public class DefaultMessagingProcessor extends AbstractStartAndShutdown implements MessagingProcessor {
//......
public CompletableFuture<PopResult> popMessage(
ProxyContext ctx,
QueueSelector queueSelector,
String consumerGroup,
String topic,
int maxMsgNums,
long invisibleTime,
long pollTime,
int initMode,
SubscriptionData subscriptionData,
boolean fifo,
PopMessageResultFilter popMessageResultFilter,
String attemptId,
long timeoutMillis
) {
return this.consumerProcessor.popMessage(ctx, queueSelector, consumerGroup, topic, maxMsgNums,
invisibleTime, pollTime, initMode, subscriptionData, fifo, popMessageResultFilter, attemptId, timeoutMillis);
}
//......
}
DefaultMessagingProcessor的popMessage方法委托给了consumerProcessor.popMessage
ConsumerProcessor
org/apache/rocketmq/proxy/processor/ConsumerProcessor.java
代码语言：javascript复制 public CompletableFuture<PopResult> popMessage(
ProxyContext ctx,
QueueSelector queueSelector,
String consumerGroup,
String topic,
int maxMsgNums,
long invisibleTime,
long pollTime,
int initMode,
SubscriptionData subscriptionData,
boolean fifo,
PopMessageResultFilter popMessageResultFilter,
String attemptId,
long timeoutMillis
) {
CompletableFuture<PopResult> future = new CompletableFuture<>();
try {
AddressableMessageQueue messageQueue = queueSelector.select(ctx, this.serviceManager.getTopicRouteService().getCurrentMessageQueueView(ctx, topic));
if (messageQueue == null) {
throw new ProxyException(ProxyExceptionCode.FORBIDDEN, "no readable queue");
}
return popMessage(ctx, messageQueue, consumerGroup, topic, maxMsgNums, invisibleTime, pollTime, initMode,
subscriptionData, fifo, popMessageResultFilter, attemptId, timeoutMillis);
} catch (Throwable t) {
future.completeExceptionally(t);
}
return future;
}
public CompletableFuture<PopResult> popMessage(
ProxyContext ctx,
AddressableMessageQueue messageQueue,
String consumerGroup,
String topic,
int maxMsgNums,
long invisibleTime,
long pollTime,
int initMode,
SubscriptionData subscriptionData,
boolean fifo,
PopMessageResultFilter popMessageResultFilter,
String attemptId,
long timeoutMillis
) {
CompletableFuture<PopResult> future = new CompletableFuture<>();
try {
if (maxMsgNums > ProxyUtils.MAX_MSG_NUMS_FOR_POP_REQUEST) {
log.warn("change maxNums from {} to {} for pop request, with info: topic:{}, group:{}",
maxMsgNums, ProxyUtils.MAX_MSG_NUMS_FOR_POP_REQUEST, topic, consumerGroup);
maxMsgNums = ProxyUtils.MAX_MSG_NUMS_FOR_POP_REQUEST;
}
PopMessageRequestHeader requestHeader = new PopMessageRequestHeader();
requestHeader.setConsumerGroup(consumerGroup);
requestHeader.setTopic(topic);
requestHeader.setQueueId(messageQueue.getQueueId());
requestHeader.setMaxMsgNums(maxMsgNums);
requestHeader.setInvisibleTime(invisibleTime);
requestHeader.setPollTime(pollTime);
requestHeader.setInitMode(initMode);
requestHeader.setExpType(subscriptionData.getExpressionType());
requestHeader.setExp(subscriptionData.getSubString());
requestHeader.setOrder(fifo);
requestHeader.setAttemptId(attemptId);
future = this.serviceManager.getMessageService().popMessage(
ctx,
messageQueue,
requestHeader,
timeoutMillis)
.thenApplyAsync(popResult -> {
if (PopStatus.FOUND.equals(popResult.getPopStatus()) &&
popResult.getMsgFoundList() != null &&
!popResult.getMsgFoundList().isEmpty() &&
popMessageResultFilter != null) {
List<MessageExt> messageExtList = new ArrayList<>();
for (MessageExt messageExt : popResult.getMsgFoundList()) {
try {
fillUniqIDIfNeed(messageExt);
String handleString = createHandle(messageExt.getProperty(MessageConst.PROPERTY_POP_CK), messageExt.getCommitLogOffset());
if (handleString == null) {
log.error("[BUG] pop message from broker but handle is empty. requestHeader:{}, msg:{}", requestHeader, messageExt);
messageExtList.add(messageExt);
continue;
}
MessageAccessor.putProperty(messageExt, MessageConst.PROPERTY_POP_CK, handleString);
PopMessageResultFilter.FilterResult filterResult =
popMessageResultFilter.filterMessage(ctx, consumerGroup, subscriptionData, messageExt);
switch (filterResult) {
case NO_MATCH:
this.messagingProcessor.ackMessage(
ctx,
ReceiptHandle.decode(handleString),
messageExt.getMsgId(),
consumerGroup,
topic,
MessagingProcessor.DEFAULT_TIMEOUT_MILLS);
break;
case TO_DLQ:
this.messagingProcessor.forwardMessageToDeadLetterQueue(
ctx,
ReceiptHandle.decode(handleString),
messageExt.getMsgId(),
consumerGroup,
topic,
MessagingProcessor.DEFAULT_TIMEOUT_MILLS);
break;
case MATCH:
default:
messageExtList.add(messageExt);
break;
}
} catch (Throwable t) {
log.error("process filterMessage failed. requestHeader:{}, msg:{}", requestHeader, messageExt, t);
messageExtList.add(messageExt);
}
}
popResult.setMsgFoundList(messageExtList);
}
return popResult;
}, this.executor);
} catch (Throwable t) {
future.completeExceptionally(t);
}
return FutureUtils.addExecutor(future, this.executor);
}
ConsumerProcessor的popMessage方法先通过queueSelector.select选择messageQueue，之后再执行popMessage；这里限制了每次pop消息的数量不能超过MAX_MSG_NUMS_FOR_POP_REQUEST(
32)，构建PopMessageRequestHeader之后再委托给了this.serviceManager.getMessageService().popMessage，之后返回popResult，拉取到的消息放到了msgFoundList属性
MessageService
org/apache/rocketmq/proxy/service/message/MessageService.java
代码语言：javascript复制 CompletableFuture<PopResult> popMessage(
ProxyContext ctx,
AddressableMessageQueue messageQueue,
PopMessageRequestHeader requestHeader,
long timeoutMillis
);
MessageService接口定义了popMessage方法，它有两个实现类，分别是LocalMessageService
LocalMessageService
org/apache/rocketmq/proxy/service/message/LocalMessageService.java
代码语言：javascript复制 public CompletableFuture<PopResult> popMessage(ProxyContext ctx, AddressableMessageQueue messageQueue,
PopMessageRequestHeader requestHeader, long timeoutMillis) {
requestHeader.setBornTime(System.currentTimeMillis());
RemotingCommand request = LocalRemotingCommand.createRequestCommand(RequestCode.POP_MESSAGE, requestHeader, ctx.getLanguage());
CompletableFuture<RemotingCommand> future = new CompletableFuture<>();
SimpleChannel channel = channelManager.createInvocationChannel(ctx);
InvocationContext invocationContext = new InvocationContext(future);
channel.registerInvocationContext(request.getOpaque(), invocationContext);
ChannelHandlerContext simpleChannelHandlerContext = channel.getChannelHandlerContext();
try {
RemotingCommand response = brokerController.getPopMessageProcessor().processRequest(simpleChannelHandlerContext, request);
if (response != null) {
invocationContext.handle(response);
channel.eraseInvocationContext(request.getOpaque());
}
} catch (Exception e) {
future.completeExceptionally(e);
channel.eraseInvocationContext(request.getOpaque());
log.error("Failed to process popMessage command", e);
}
return future.thenApply(r -> {
PopStatus popStatus;
List<MessageExt> messageExtList = new ArrayList<>();
switch (r.getCode()) {
case ResponseCode.SUCCESS:
popStatus = PopStatus.FOUND;
ByteBuffer byteBuffer = ByteBuffer.wrap(r.getBody());
messageExtList = MessageDecoder.decodesBatch(
byteBuffer,
true,
false,
true
);
break;
case ResponseCode.POLLING_FULL:
popStatus = PopStatus.POLLING_FULL;
break;
case ResponseCode.POLLING_TIMEOUT:
case ResponseCode.PULL_NOT_FOUND:
popStatus = PopStatus.POLLING_NOT_FOUND;
break;
default:
throw new ProxyException(ProxyExceptionCode.INTERNAL_SERVER_ERROR, r.getRemark());
}
PopResult popResult = new PopResult(popStatus, messageExtList);
PopMessageResponseHeader responseHeader = (PopMessageResponseHeader) r.readCustomHeader();
if (popStatus == PopStatus.FOUND) {
Map<String, Long> startOffsetInfo;
Map<String, List<Long>> msgOffsetInfo;
Map<String, Integer> orderCountInfo;
popResult.setInvisibleTime(responseHeader.getInvisibleTime());
popResult.setPopTime(responseHeader.getPopTime());
startOffsetInfo = ExtraInfoUtil.parseStartOffsetInfo(responseHeader.getStartOffsetInfo());
msgOffsetInfo = ExtraInfoUtil.parseMsgOffsetInfo(responseHeader.getMsgOffsetInfo());
orderCountInfo = ExtraInfoUtil.parseOrderCountInfo(responseHeader.getOrderCountInfo());
// <topicMark@queueId, msg queueOffset>
Map<String, List<Long>> sortMap = new HashMap<>(16);
for (MessageExt messageExt : messageExtList) {
// Value of POP_CK is used to determine whether it is a pop retry,
// cause topic could be rewritten by broker.
String key = ExtraInfoUtil.getStartOffsetInfoMapKey(messageExt.getTopic(),
messageExt.getProperty(MessageConst.PROPERTY_POP_CK), messageExt.getQueueId());
if (!sortMap.containsKey(key)) {
sortMap.put(key, new ArrayList<>(4));
}
sortMap.get(key).add(messageExt.getQueueOffset());
}
Map<String, String> map = new HashMap<>(5);
for (MessageExt messageExt : messageExtList) {
if (startOffsetInfo == null) {
// we should set the check point info to extraInfo field , if the command is popMsg
// find pop ck offset
String key = messageExt.getTopic() messageExt.getQueueId();
if (!map.containsKey(messageExt.getTopic() messageExt.getQueueId())) {
map.put(key, ExtraInfoUtil.buildExtraInfo(messageExt.getQueueOffset(), responseHeader.getPopTime(), responseHeader.getInvisibleTime(), responseHeader.getReviveQid(),
messageExt.getTopic(), messageQueue.getBrokerName(), messageExt.getQueueId()));
}
messageExt.getProperties().put(MessageConst.PROPERTY_POP_CK, map.get(key) MessageConst.KEY_SEPARATOR messageExt.getQueueOffset());
} else {
if (messageExt.getProperty(MessageConst.PROPERTY_POP_CK) == null) {
String key = ExtraInfoUtil.getStartOffsetInfoMapKey(messageExt.getTopic(), messageExt.getQueueId());
int index = sortMap.get(key).indexOf(messageExt.getQueueOffset());
Long msgQueueOffset = msgOffsetInfo.get(key).get(index);
if (msgQueueOffset != messageExt.getQueueOffset()) {
log.warn("Queue offset [{}] of msg is strange, not equal to the stored in msg, {}", msgQueueOffset, messageExt);
}
messageExt.getProperties().put(MessageConst.PROPERTY_POP_CK,
ExtraInfoUtil.buildExtraInfo(startOffsetInfo.get(key), responseHeader.getPopTime(), responseHeader.getInvisibleTime(),
responseHeader.getReviveQid(), messageExt.getTopic(), messageQueue.getBrokerName(), messageExt.getQueueId(), msgQueueOffset)
);
if (requestHeader.isOrder() && orderCountInfo != null) {
Integer count = orderCountInfo.get(key);
if (count != null && count > 0) {
messageExt.setReconsumeTimes(count);
}
}
}
}
messageExt.getProperties().computeIfAbsent(MessageConst.PROPERTY_FIRST_POP_TIME, k -> String.valueOf(responseHeader.getPopTime()));
messageExt.setBrokerName(messageExt.getBrokerName());
messageExt.setTopic(messageQueue.getTopic());
}
}
return popResult;
});
}
LocalMessageService的popMessage方法构建RequestCode.POP_MESSAGE命令，然后通过brokerController.getPopMessageProcessor().processRequest来拉取消息
ClusterMessageService
org/apache/rocketmq/proxy/service/message/ClusterMessageService.java
代码语言：javascript复制 public CompletableFuture<PopResult> popMessage(ProxyContext ctx, AddressableMessageQueue messageQueue,
PopMessageRequestHeader requestHeader, long timeoutMillis) {
return this.mqClientAPIFactory.getClient().popMessageAsync(
messageQueue.getBrokerAddr(),
messageQueue.getBrokerName(),
requestHeader,
timeoutMillis
);
}
ClusterMessageService的popMessage方法则是委托给了mqClientAPIFactory.getClient().popMessageAsync
popMessageAsync
org/apache/rocketmq/client/impl/mqclient/MQClientAPIExt.java
代码语言：javascript复制 public CompletableFuture<PopResult> popMessageAsync(
String brokerAddr,
String brokerName,
PopMessageRequestHeader requestHeader,
long timeoutMillis
) {
CompletableFuture<PopResult> future = new CompletableFuture<>();
try {
this.popMessageAsync(brokerName, brokerAddr, requestHeader, timeoutMillis, new PopCallback() {
@Override
public void onSuccess(PopResult popResult) {
future.complete(popResult);
}
@Override
public void onException(Throwable t) {
future.completeExceptionally(t);
}
});
} catch (Throwable t) {
future.completeExceptionally(t);
}
return future;
}
public void popMessageAsync(
final String brokerName, final String addr, final PopMessageRequestHeader requestHeader,
final long timeoutMillis, final PopCallback popCallback
) throws RemotingException, InterruptedException {
final RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.POP_MESSAGE, requestHeader);
this.remotingClient.invokeAsync(addr, request, timeoutMillis, new InvokeCallback() {
@Override
public void operationComplete(ResponseFuture responseFuture) {
}
@Override
public void operationSucceed(RemotingCommand response) {
try {
PopResult popResult = MQClientAPIImpl.this.processPopResponse(brokerName, response, requestHeader.getTopic(), requestHeader);
popCallback.onSuccess(popResult);
} catch (Exception e) {
popCallback.onException(e);
}
}
@Override
public void operationFail(Throwable throwable) {
popCallback.onException(throwable);
}
});
}
popMessageAsync也是构建RequestCode.POP_MESSAGE命令，然后通过remotingClient去发送请求，之后再通过processPopResponse来处理返回结果
PopMessageProcessor
org/apache/rocketmq/broker/processor/PopMessageProcessor.java
代码语言：javascript复制 public RemotingCommand processRequest(final ChannelHandlerContext ctx, RemotingCommand request)
throws RemotingCommandException {
final long beginTimeMills = this.brokerController.getMessageStore().now();
request.addExtFieldIfNotExist(BORN_TIME, String.valueOf(System.currentTimeMillis()));
if (Objects.equals(request.getExtFields().get(BORN_TIME), "0")) {
request.addExtField(BORN_TIME, String.valueOf(System.currentTimeMillis()));
}
Channel channel = ctx.channel();
RemotingCommand response = RemotingCommand.createResponseCommand(PopMessageResponseHeader.class);
final PopMessageResponseHeader responseHeader = (PopMessageResponseHeader) response.readCustomHeader();
final PopMessageRequestHeader requestHeader =
(PopMessageRequestHeader) request.decodeCommandCustomHeader(PopMessageRequestHeader.class, true);
StringBuilder startOffsetInfo = new StringBuilder(64);
StringBuilder msgOffsetInfo = new StringBuilder(64);
StringBuilder orderCountInfo = null;
if (requestHeader.isOrder()) {
orderCountInfo = new StringBuilder(64);
}
brokerController.getConsumerManager().compensateBasicConsumerInfo(requestHeader.getConsumerGroup(),
ConsumeType.CONSUME_POP, MessageModel.CLUSTERING);
response.setOpaque(request.getOpaque());
if (brokerController.getBrokerConfig().isEnablePopLog()) {
POP_LOGGER.info("receive PopMessage request command, {}", request);
}
if (requestHeader.isTimeoutTooMuch()) {
response.setCode(ResponseCode.POLLING_TIMEOUT);
response.setRemark(String.format("the broker[%s] pop message is timeout too much",
this.brokerController.getBrokerConfig().getBrokerIP1()));
return response;
}
if (!PermName.isReadable(this.brokerController.getBrokerConfig().getBrokerPermission())) {
response.setCode(ResponseCode.NO_PERMISSION);
response.setRemark(String.format("the broker[%s] pop message is forbidden",
this.brokerController.getBrokerConfig().getBrokerIP1()));
return response;
}
if (requestHeader.getMaxMsgNums() > 32) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark(String.format("the broker[%s] pop message's num is greater than 32",
this.brokerController.getBrokerConfig().getBrokerIP1()));
return response;
}
if (!brokerController.getMessageStore().getMessageStoreConfig().isTimerWheelEnable()) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark(String.format("the broker[%s] pop message is forbidden because timerWheelEnable is false",
this.brokerController.getBrokerConfig().getBrokerIP1()));
return response;
}
TopicConfig topicConfig =
this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic());
if (null == topicConfig) {
POP_LOGGER.error("The topic {} not exist, consumer: {} ", requestHeader.getTopic(),
RemotingHelper.parseChannelRemoteAddr(channel));
response.setCode(ResponseCode.TOPIC_NOT_EXIST);
response.setRemark(String.format("topic[%s] not exist, apply first please! %s", requestHeader.getTopic(),
FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL)));
return response;
}
if (!PermName.isReadable(topicConfig.getPerm())) {
response.setCode(ResponseCode.NO_PERMISSION);
response.setRemark("the topic[" requestHeader.getTopic() "] peeking message is forbidden");
return response;
}
if (requestHeader.getQueueId() >= topicConfig.getReadQueueNums()) {
String errorInfo = String.format("queueId[%d] is illegal, topic:[%s] topicConfig.readQueueNums:[%d] "
"consumer:[%s]",
requestHeader.getQueueId(), requestHeader.getTopic(), topicConfig.getReadQueueNums(),
channel.remoteAddress());
POP_LOGGER.warn(errorInfo);
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark(errorInfo);
return response;
}
SubscriptionGroupConfig subscriptionGroupConfig =
this.brokerController.getSubscriptionGroupManager().findSubscriptionGroupConfig(requestHeader.getConsumerGroup());
if (null == subscriptionGroupConfig) {
response.setCode(ResponseCode.SUBSCRIPTION_GROUP_NOT_EXIST);
response.setRemark(String.format("subscription group [%s] does not exist, %s",
requestHeader.getConsumerGroup(), FAQUrl.suggestTodo(FAQUrl.SUBSCRIPTION_GROUP_NOT_EXIST)));
return response;
}
if (!subscriptionGroupConfig.isConsumeEnable()) {
response.setCode(ResponseCode.NO_PERMISSION);
response.setRemark("subscription group no permission, " requestHeader.getConsumerGroup());
return response;
}
BrokerConfig brokerConfig = brokerController.getBrokerConfig();
SubscriptionData subscriptionData = null;
ExpressionMessageFilter messageFilter = null;
if (requestHeader.getExp() != null && !requestHeader.getExp().isEmpty()) {
try {
subscriptionData = FilterAPI.build(requestHeader.getTopic(), requestHeader.getExp(), requestHeader.getExpType());
brokerController.getConsumerManager().compensateSubscribeData(requestHeader.getConsumerGroup(),
requestHeader.getTopic(), subscriptionData);
String retryTopic = KeyBuilder.buildPopRetryTopic(requestHeader.getTopic(), requestHeader.getConsumerGroup(), brokerConfig.isEnableRetryTopicV2());
SubscriptionData retrySubscriptionData = FilterAPI.build(retryTopic, SubscriptionData.SUB_ALL, requestHeader.getExpType());
brokerController.getConsumerManager().compensateSubscribeData(requestHeader.getConsumerGroup(),
retryTopic, retrySubscriptionData);
ConsumerFilterData consumerFilterData = null;
if (!ExpressionType.isTagType(subscriptionData.getExpressionType())) {
consumerFilterData = ConsumerFilterManager.build(
requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getExp(),
requestHeader.getExpType(), System.currentTimeMillis()
);
if (consumerFilterData == null) {
POP_LOGGER.warn("Parse the consumer's subscription[{}] failed, group: {}",
requestHeader.getExp(), requestHeader.getConsumerGroup());
response.setCode(ResponseCode.SUBSCRIPTION_PARSE_FAILED);
response.setRemark("parse the consumer's subscription failed");
return response;
}
}
messageFilter = new ExpressionMessageFilter(subscriptionData, consumerFilterData,
brokerController.getConsumerFilterManager());
} catch (Exception e) {
POP_LOGGER.warn("Parse the consumer's subscription[{}] error, group: {}", requestHeader.getExp(),
requestHeader.getConsumerGroup());
response.setCode(ResponseCode.SUBSCRIPTION_PARSE_FAILED);
response.setRemark("parse the consumer's subscription failed");
return response;
}
} else {
try {
subscriptionData = FilterAPI.build(requestHeader.getTopic(), "*", ExpressionType.TAG);
brokerController.getConsumerManager().compensateSubscribeData(requestHeader.getConsumerGroup(),
requestHeader.getTopic(), subscriptionData);
String retryTopic = KeyBuilder.buildPopRetryTopic(requestHeader.getTopic(), requestHeader.getConsumerGroup(), brokerConfig.isEnableRetryTopicV2());
SubscriptionData retrySubscriptionData = FilterAPI.build(retryTopic, "*", ExpressionType.TAG);
brokerController.getConsumerManager().compensateSubscribeData(requestHeader.getConsumerGroup(),
retryTopic, retrySubscriptionData);
} catch (Exception e) {
POP_LOGGER.warn("Build default subscription error, group: {}", requestHeader.getConsumerGroup());
}
}
int randomQ = random.nextInt(100);
int reviveQid;
if (requestHeader.isOrder()) {
reviveQid = KeyBuilder.POP_ORDER_REVIVE_QUEUE;
} else {
reviveQid = (int) Math.abs(ckMessageNumber.getAndIncrement() % this.brokerController.getBrokerConfig().getReviveQueueNum());
}
GetMessageResult getMessageResult = new GetMessageResult(requestHeader.getMaxMsgNums());
ExpressionMessageFilter finalMessageFilter = messageFilter;
StringBuilder finalOrderCountInfo = orderCountInfo;
// Due to the design of the fields startOffsetInfo, msgOffsetInfo, and orderCountInfo,
// a single POP request could only invoke the popMsgFromQueue method once
// for either a normal topic or a retry topic's queue. Retry topics v1 and v2 are
// considered the same type because they share the same retry flag in previous fields.
// Therefore, needRetryV1 is designed as a subset of needRetry, and within a single request,
// only one type of retry topic is able to call popMsgFromQueue.
boolean needRetry = randomQ % 5 == 0;
boolean needRetryV1 = false;
if (brokerConfig.isEnableRetryTopicV2() && brokerConfig.isRetrieveMessageFromPopRetryTopicV1()) {
needRetryV1 = randomQ % 2 == 0;
}
long popTime = System.currentTimeMillis();
CompletableFuture<Long> getMessageFuture = CompletableFuture.completedFuture(0L);
if (needRetry && !requestHeader.isOrder()) {
if (needRetryV1) {
String retryTopic = KeyBuilder.buildPopRetryTopicV1(requestHeader.getTopic(), requestHeader.getConsumerGroup());
getMessageFuture = popMsgFromTopic(retryTopic, true, getMessageResult, requestHeader, reviveQid, channel,
popTime, finalMessageFilter, startOffsetInfo, msgOffsetInfo, orderCountInfo, randomQ, getMessageFuture);
} else {
String retryTopic = KeyBuilder.buildPopRetryTopic(requestHeader.getTopic(), requestHeader.getConsumerGroup(), brokerConfig.isEnableRetryTopicV2());
getMessageFuture = popMsgFromTopic(retryTopic, true, getMessageResult, requestHeader, reviveQid, channel,
popTime, finalMessageFilter, startOffsetInfo, msgOffsetInfo, orderCountInfo, randomQ, getMessageFuture);
}
}
if (requestHeader.getQueueId() < 0) {
// read all queue
getMessageFuture = popMsgFromTopic(topicConfig, false, getMessageResult, requestHeader, reviveQid, channel,
popTime, finalMessageFilter, startOffsetInfo, msgOffsetInfo, orderCountInfo, randomQ, getMessageFuture);
} else {
int queueId = requestHeader.getQueueId();
getMessageFuture = getMessageFuture.thenCompose(restNum ->
popMsgFromQueue(topicConfig.getTopicName(), requestHeader.getAttemptId(), false,
getMessageResult, requestHeader, queueId, restNum, reviveQid, channel, popTime, finalMessageFilter,
startOffsetInfo, msgOffsetInfo, finalOrderCountInfo));
}
// if not full , fetch retry again
if (!needRetry && getMessageResult.getMessageMapedList().size() < requestHeader.getMaxMsgNums() && !requestHeader.isOrder()) {
if (needRetryV1) {
String retryTopicV1 = KeyBuilder.buildPopRetryTopicV1(requestHeader.getTopic(), requestHeader.getConsumerGroup());
getMessageFuture = popMsgFromTopic(retryTopicV1, true, getMessageResult, requestHeader, reviveQid, channel,
popTime, finalMessageFilter, startOffsetInfo, msgOffsetInfo, orderCountInfo, randomQ, getMessageFuture);
} else {
String retryTopic = KeyBuilder.buildPopRetryTopic(requestHeader.getTopic(), requestHeader.getConsumerGroup(), brokerConfig.isEnableRetryTopicV2());
getMessageFuture = popMsgFromTopic(retryTopic, true, getMessageResult, requestHeader, reviveQid, channel,
popTime, finalMessageFilter, startOffsetInfo, msgOffsetInfo, orderCountInfo, randomQ, getMessageFuture);
}
}
final RemotingCommand finalResponse = response;
SubscriptionData finalSubscriptionData = subscriptionData;
getMessageFuture.thenApply(restNum -> {
if (!getMessageResult.getMessageBufferList().isEmpty()) {
finalResponse.setCode(ResponseCode.SUCCESS);
getMessageResult.setStatus(GetMessageStatus.FOUND);
if (restNum > 0) {
// all queue pop can not notify specified queue pop, and vice versa
popLongPollingService.notifyMessageArriving(
requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getConsumerGroup(),
null, 0L, null, null);
}
} else {
PollingResult pollingResult = popLongPollingService.polling(
ctx, request, new PollingHeader(requestHeader), finalSubscriptionData, finalMessageFilter);
if (PollingResult.POLLING_SUC == pollingResult) {
return null;
} else if (PollingResult.POLLING_FULL == pollingResult) {
finalResponse.setCode(ResponseCode.POLLING_FULL);
} else {
finalResponse.setCode(ResponseCode.POLLING_TIMEOUT);
}
getMessageResult.setStatus(GetMessageStatus.NO_MESSAGE_IN_QUEUE);
}
responseHeader.setInvisibleTime(requestHeader.getInvisibleTime());
responseHeader.setPopTime(popTime);
responseHeader.setReviveQid(reviveQid);
responseHeader.setRestNum(restNum);
responseHeader.setStartOffsetInfo(startOffsetInfo.toString());
responseHeader.setMsgOffsetInfo(msgOffsetInfo.toString());
if (requestHeader.isOrder() && finalOrderCountInfo != null) {
responseHeader.setOrderCountInfo(finalOrderCountInfo.toString());
}
finalResponse.setRemark(getMessageResult.getStatus().name());
switch (finalResponse.getCode()) {
case ResponseCode.SUCCESS:
if (this.brokerController.getBrokerConfig().isTransferMsgByHeap()) {
final byte[] r = this.readGetMessageResult(getMessageResult, requestHeader.getConsumerGroup(),
requestHeader.getTopic(), requestHeader.getQueueId());
this.brokerController.getBrokerStatsManager().incGroupGetLatency(requestHeader.getConsumerGroup(),
requestHeader.getTopic(), requestHeader.getQueueId(),
(int) (this.brokerController.getMessageStore().now() - beginTimeMills));
finalResponse.setBody(r);
} else {
final GetMessageResult tmpGetMessageResult = getMessageResult;
try {
FileRegion fileRegion =
new ManyMessageTransfer(finalResponse.encodeHeader(getMessageResult.getBufferTotalSize()),
getMessageResult);
channel.writeAndFlush(fileRegion)
.addListener((ChannelFutureListener) future -> {
tmpGetMessageResult.release();
Attributes attributes = RemotingMetricsManager.newAttributesBuilder()
.put(LABEL_REQUEST_CODE, RemotingHelper.getRequestCodeDesc(request.getCode()))
.put(LABEL_RESPONSE_CODE, RemotingHelper.getResponseCodeDesc(finalResponse.getCode()))
.put(LABEL_RESULT, RemotingMetricsManager.getWriteAndFlushResult(future))
.build();
RemotingMetricsManager.rpcLatency.record(request.getProcessTimer().elapsed(TimeUnit.MILLISECONDS), attributes);
if (!future.isSuccess()) {
POP_LOGGER.error("Fail to transfer messages from page cache to {}",
channel.remoteAddress(), future.cause());
}
});
} catch (Throwable e) {
POP_LOGGER.error("Error occurred when transferring messages from page cache", e);
getMessageResult.release();
}
return null;
}
break;
default:
return finalResponse;
}
return finalResponse;
}).thenAccept(result -> NettyRemotingAbstract.writeResponse(channel, request, result));
return null;
}
PopMessageProcessor的processRequest方法用于处理RequestCode.POP_MESSAGE请求，它构建了GetMessageResult，之后通过popMsgFromTopic或者popMsgFromQueue去拉取消息
popMsgFromTopic
代码语言：javascript复制 private CompletableFuture<Long> popMsgFromTopic(TopicConfig topicConfig, boolean isRetry, GetMessageResult getMessageResult,
PopMessageRequestHeader requestHeader, int reviveQid, Channel channel, long popTime,
ExpressionMessageFilter messageFilter, StringBuilder startOffsetInfo,
StringBuilder msgOffsetInfo, StringBuilder orderCountInfo, int randomQ, CompletableFuture<Long> getMessageFuture) {
if (topicConfig != null) {
for (int i = 0; i < topicConfig.getReadQueueNums(); i ) {
int queueId = (randomQ i) % topicConfig.getReadQueueNums();
getMessageFuture = getMessageFuture.thenCompose(restNum ->
popMsgFromQueue(topicConfig.getTopicName(), requestHeader.getAttemptId(), isRetry,
getMessageResult, requestHeader, queueId, restNum, reviveQid, channel, popTime, messageFilter,
startOffsetInfo, msgOffsetInfo, orderCountInfo));
}
}
return getMessageFuture;
}
popMsgFromTopic根据(randomQ i) % topicConfig.getReadQueueNums()确定queueId，然后执行popMsgFromQueue
popMsgFromQueue
代码语言：javascript复制 private CompletableFuture<Long> popMsgFromQueue(String topic, String attemptId, boolean isRetry, GetMessageResult getMessageResult,
PopMessageRequestHeader requestHeader, int queueId, long restNum, int reviveQid,
Channel channel, long popTime, ExpressionMessageFilter messageFilter, StringBuilder startOffsetInfo,
StringBuilder msgOffsetInfo, StringBuilder orderCountInfo) {
String lockKey =
topic PopAckConstants.SPLIT requestHeader.getConsumerGroup() PopAckConstants.SPLIT queueId;
boolean isOrder = requestHeader.isOrder();
long offset = getPopOffset(topic, requestHeader.getConsumerGroup(), queueId, requestHeader.getInitMode(),
false, lockKey, false);
CompletableFuture<Long> future = new CompletableFuture<>();
if (!queueLockManager.tryLock(lockKey)) {
restNum = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId) - offset restNum;
future.complete(restNum);
return future;
}
if (isPopShouldStop(topic, requestHeader.getConsumerGroup(), queueId)) {
POP_LOGGER.warn("Too much msgs unacked, then stop poping. topic={}, group={}, queueId={}", topic, requestHeader.getConsumerGroup(), queueId);
restNum = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId) - offset restNum;
future.complete(restNum);
return future;
}
try {
future.whenComplete((result, throwable) -> queueLockManager.unLock(lockKey));
offset = getPopOffset(topic, requestHeader.getConsumerGroup(), queueId, requestHeader.getInitMode(),
true, lockKey, true);
if (isOrder && brokerController.getConsumerOrderInfoManager().checkBlock(attemptId, topic,
requestHeader.getConsumerGroup(), queueId, requestHeader.getInvisibleTime())) {
future.complete(this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId) - offset restNum);
return future;
}
if (isOrder) {
this.brokerController.getPopInflightMessageCounter().clearInFlightMessageNum(
topic,
requestHeader.getConsumerGroup(),
queueId
);
}
if (getMessageResult.getMessageMapedList().size() >= requestHeader.getMaxMsgNums()) {
restNum = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId) - offset restNum;
future.complete(restNum);
return future;
}
} catch (Exception e) {
POP_LOGGER.error("Exception in popMsgFromQueue", e);
future.complete(restNum);
return future;
}
AtomicLong atomicRestNum = new AtomicLong(restNum);
AtomicLong atomicOffset = new AtomicLong(offset);
long finalOffset = offset;
return this.brokerController.getMessageStore()
.getMessageAsync(requestHeader.getConsumerGroup(), topic, queueId, offset,
requestHeader.getMaxMsgNums() - getMessageResult.getMessageMapedList().size(), messageFilter)
.thenCompose(result -> {
if (result == null) {
return CompletableFuture.completedFuture(null);
}
// maybe store offset is not correct.
if (GetMessageStatus.OFFSET_TOO_SMALL.equals(result.getStatus())
|| GetMessageStatus.OFFSET_OVERFLOW_BADLY.equals(result.getStatus())
|| GetMessageStatus.OFFSET_FOUND_NULL.equals(result.getStatus())) {
// commit offset, because the offset is not correct
// If offset in store is greater than cq offset, it will cause duplicate messages,
// because offset in PopBuffer is not committed.
POP_LOGGER.warn("Pop initial offset, because store is no correct, {}, {}->{}",
lockKey, atomicOffset.get(), result.getNextBeginOffset());
this.brokerController.getConsumerOffsetManager().commitOffset(channel.remoteAddress().toString(), requestHeader.getConsumerGroup(), topic,
queueId, result.getNextBeginOffset());
atomicOffset.set(result.getNextBeginOffset());
return this.brokerController.getMessageStore().getMessageAsync(requestHeader.getConsumerGroup(), topic, queueId, atomicOffset.get(),
requestHeader.getMaxMsgNums() - getMessageResult.getMessageMapedList().size(), messageFilter);
}
return CompletableFuture.completedFuture(result);
}).thenApply(result -> {
if (result == null) {
atomicRestNum.set(brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId) - atomicOffset.get() atomicRestNum.get());
return atomicRestNum.get();
}
if (!result.getMessageMapedList().isEmpty()) {
this.brokerController.getBrokerStatsManager().incBrokerGetNums(requestHeader.getTopic(), result.getMessageCount());
this.brokerController.getBrokerStatsManager().incGroupGetNums(requestHeader.getConsumerGroup(), topic,
result.getMessageCount());
this.brokerController.getBrokerStatsManager().incGroupGetSize(requestHeader.getConsumerGroup(), topic,
result.getBufferTotalSize());
Attributes attributes = BrokerMetricsManager.newAttributesBuilder()
.put(LABEL_TOPIC, requestHeader.getTopic())
.put(LABEL_CONSUMER_GROUP, requestHeader.getConsumerGroup())
.put(LABEL_IS_SYSTEM, TopicValidator.isSystemTopic(requestHeader.getTopic()) || MixAll.isSysConsumerGroup(requestHeader.getConsumerGroup()))
.put(LABEL_IS_RETRY, isRetry)
.build();
BrokerMetricsManager.messagesOutTotal.add(result.getMessageCount(), attributes);
BrokerMetricsManager.throughputOutTotal.add(result.getBufferTotalSize(), attributes);
if (isOrder) {
this.brokerController.getConsumerOrderInfoManager().update(requestHeader.getAttemptId(), isRetry, topic,
requestHeader.getConsumerGroup(),
queueId, popTime, requestHeader.getInvisibleTime(), result.getMessageQueueOffset(),
orderCountInfo);
this.brokerController.getConsumerOffsetManager().commitOffset(channel.remoteAddress().toString(),
requestHeader.getConsumerGroup(), topic, queueId, finalOffset);
} else {
if (!appendCheckPoint(requestHeader, topic, reviveQid, queueId, finalOffset, result, popTime, this.brokerController.getBrokerConfig().getBrokerName())) {
return atomicRestNum.get() result.getMessageCount();
}
}
ExtraInfoUtil.buildStartOffsetInfo(startOffsetInfo, topic, queueId, finalOffset);
ExtraInfoUtil.buildMsgOffsetInfo(msgOffsetInfo, topic, queueId,
result.getMessageQueueOffset());
} else if ((GetMessageStatus.NO_MATCHED_MESSAGE.equals(result.getStatus())
|| GetMessageStatus.OFFSET_FOUND_NULL.equals(result.getStatus())
|| GetMessageStatus.MESSAGE_WAS_REMOVING.equals(result.getStatus())
|| GetMessageStatus.NO_MATCHED_LOGIC_QUEUE.equals(result.getStatus()))
&& result.getNextBeginOffset() > -1) {
if (isOrder) {
this.brokerController.getConsumerOffsetManager().commitOffset(channel.remoteAddress().toString(), requestHeader.getConsumerGroup(), topic,
queueId, result.getNextBeginOffset());
} else {
popBufferMergeService.addCkMock(requestHeader.getConsumerGroup(), topic, queueId, finalOffset,
requestHeader.getInvisibleTime(), popTime, reviveQid, result.getNextBeginOffset(), brokerController.getBrokerConfig().getBrokerName());
}
}
atomicRestNum.set(result.getMaxOffset() - result.getNextBeginOffset() atomicRestNum.get());
String brokerName = brokerController.getBrokerConfig().getBrokerName();
for (SelectMappedBufferResult mapedBuffer : result.getMessageMapedList()) {
// We should not recode buffer when popResponseReturnActualRetryTopic is true or topic is not retry topic
if (brokerController.getBrokerConfig().isPopResponseReturnActualRetryTopic() || !isRetry) {
getMessageResult.addMessage(mapedBuffer);
} else {
List<MessageExt> messageExtList = MessageDecoder.decodesBatch(mapedBuffer.getByteBuffer(),
true, false, true);
mapedBuffer.release();
for (MessageExt messageExt : messageExtList) {
try {
String ckInfo = ExtraInfoUtil.buildExtraInfo(finalOffset, popTime, requestHeader.getInvisibleTime(),
reviveQid, messageExt.getTopic(), brokerName, messageExt.getQueueId(), messageExt.getQueueOffset());
messageExt.getProperties().putIfAbsent(MessageConst.PROPERTY_POP_CK, ckInfo);
// Set retry message topic to origin topic and clear message store size to recode
messageExt.setTopic(requestHeader.getTopic());
messageExt.setStoreSize(0);
byte[] encode = MessageDecoder.encode(messageExt, false);
ByteBuffer buffer = ByteBuffer.wrap(encode);
SelectMappedBufferResult tmpResult =
new SelectMappedBufferResult(mapedBuffer.getStartOffset(), buffer, encode.length, null);
getMessageResult.addMessage(tmpResult);
} catch (Exception e) {
POP_LOGGER.error("Exception in recode retry message buffer, topic={}", topic, e);
}
}
}
}
this.brokerController.getPopInflightMessageCounter().incrementInFlightMessageNum(
topic,
requestHeader.getConsumerGroup(),
queueId,
result.getMessageCount()
);
return atomicRestNum.get();
}).whenComplete((result, throwable) -> {
if (throwable != null) {
POP_LOGGER.error("Pop message error, {}", lockKey, throwable);
}
queueLockManager.unLock(lockKey);
});
}
popMsgFromQueue针对顺序消息会执行brokerController.getConsumerOrderInfoManager().checkBlock(attemptId, topic, requestHeader.getConsumerGroup(), queueId, requestHeader.getInvisibleTime())校验，需要block的话会直接返回
checkBlock
org/apache/rocketmq/broker/offset/ConsumerOrderInfoManager.java
代码语言：javascript复制 public boolean checkBlock(String attemptId, String topic, String group, int queueId, long invisibleTime) {
String key = buildKey(topic, group);
ConcurrentHashMap<Integer/*queueId*/, OrderInfo> qs = table.get(key);
if (qs == null) {
qs = new ConcurrentHashMap<>(16);
ConcurrentHashMap<Integer/*queueId*/, OrderInfo> old = table.putIfAbsent(key, qs);
if (old != null) {
qs = old;
}
}
OrderInfo orderInfo = qs.get(queueId);
if (orderInfo == null) {
return false;
}
return orderInfo.needBlock(attemptId, invisibleTime);
}
@JSONField(serialize = false, deserialize = false)
public boolean needBlock(String attemptId, long currentInvisibleTime) {
if (offsetList == null || offsetList.isEmpty()) {
return false;
}
if (this.attemptId != null && this.attemptId.equals(attemptId)) {
return false;
}
int num = offsetList.size();
int i = 0;
if (this.invisibleTime == null || this.invisibleTime <= 0) {
this.invisibleTime = currentInvisibleTime;
}
long currentTime = System.currentTimeMillis();
for (; i < num; i ) {
if (isNotAck(i)) {
long nextVisibleTime = popTime invisibleTime;
if (offsetNextVisibleTime != null) {
Long time = offsetNextVisibleTime.get(this.getQueueOffset(i));
if (time != null) {
nextVisibleTime = time;
}
}
if (currentTime < nextVisibleTime) {
return true;
}
}
}
return false;
}
checkBlock先找到指定的orderInfo，然后执行orderInfo.needBlock(attemptId, invisibleTime)；它先判断该offsetList是否都已经ack了，都ack了则返回false，否则计算nextVisibleTime，若有当前时间小于nextVisibleTime则返回true
小结
ConsumerProcessor的popMessage方法先通过queueSelector.select选择messageQueue，之后再执行popMessage；这里限制了每次pop消息的数量不能超过MAX_MSG_NUMS_FOR_POP_REQUEST(32)，构建PopMessageRequestHeader之后再委托给了this.serviceManager.getMessageService().popMessage，之后返回popResult，拉取到的消息放到了msgFoundList属性。对于顺序消息会执行checkBlock判断，若该offsetList有未ack的且时间还小于nextVisibleTime则返回true。
doc
- RocketMQ 原理和架构
