SpringBoot入门建站全系列(二十四)使用Sharding-JDBC进行分库分表
一、概述
一个系统最初的线上业务量并不会很大,比如说单库的数据量在百万级别以下(事实上千万级别以下都还能支撑),那么MySQL的单库即可完成任何增/删/改/查的业务操作。随着业务的发展,单个DB中保存的数据量(用户、订单、计费明细和权限规则等数据)呈现指数级增长,那么各种业务处理操作都会面临单DB的IO读写瓶颈带来的性能问题。
Sharding-JDBC分库分表就是其中一个解决方法,目前用的还挺广泛的,虽然还是有蛮多问题,但是对于公司的普通应用已经足够了。
其实,如果仅仅是分表,Mybatis等中间件就可以帮我们实现简单分表功能,不需要使用Sharding-JDBC,但是Sharding-JDBC可以支持分库,而且支持分库的本地事务(弱事务):
Sharding-JDBC本地事务官方说明:
- 完全支持非跨库事务,例如:仅分表,或分库但是路由的结果在单库中。
- 完全支持因逻辑异常导致的跨库事务。例如:同一事务中,跨两个库更新。更新完毕后,抛出空指针,则两个库的内容都能回滚。
- 不支持因网络、硬件异常导致的跨库事务。例如:同一事务中,跨两个库更新,更新完毕后、未提交之前,第一个库宕机,则只有第二个库数据提交。
在《Spring整合Sharding-JDBC分库分表详情》一篇中,已经使用Spring mvc 整合Spring-data-jpa、Sharding-JDBC 进行分库分表操作;这一篇将使用SpringBoot与Mybatis整合Sharding-JDBC 进行分库分表操作。
二、配置
本文假设你已经引入spring-boot-starter-web。已经是个SpringBoot项目了,如果不会搭建,可以打开这篇文章看一看《SpringBoot入门建站全系列(一)项目建立》。
本篇使用Mybatis做数据库访问,并整合Sharding-JDBC。
2.1 Maven依赖
需要引入mybatis-spring-boot-starter,这里要访问数据库进行操作,所以要依赖数据库相关jar包。还要引入sharding-jdbc-spring-boot-starter。
代码语言:txt复制<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-dbcp2</artifactId>
</dependency>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>sharding-jdbc-spring-boot-starter</artifactId>
<version>${sharding-sphere.version}</version>
</dependency>
2.2 配置文件
在application.properties 中需要添加下面的配置:
代码语言:txt复制worker.id=1
spring.shardingsphere.datasource.names=master,slave
spring.shardingsphere.datasource.master.type=org.apache.commons.dbcp2.BasicDataSource
spring.shardingsphere.datasource.master.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.master.url=jdbc:mysql://127.0.0.1:3306/boot?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC
spring.shardingsphere.datasource.master.username=cff
spring.shardingsphere.datasource.master.password=123456
spring.shardingsphere.datasource.master.max-wait-millis=60000
spring.shardingsphere.datasource.master.min-idle=20
spring.shardingsphere.datasource.master.initial-size=2
spring.shardingsphere.datasource.master.validation-query=SELECT 1
spring.shardingsphere.datasource.master.connection-properties=characterEncoding=utf8
spring.shardingsphere.datasource.master.validation-query=SELECT 1
spring.shardingsphere.datasource.master.test-while-idle=true
spring.shardingsphere.datasource.master.test-on-borrow=true
spring.shardingsphere.datasource.master.test-on-return=false
spring.shardingsphere.datasource.slave.type=org.apache.commons.dbcp2.BasicDataSource
spring.shardingsphere.datasource.slave.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.slave.url=jdbc:mysql://127.0.0.1:3306/cff?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC
spring.shardingsphere.datasource.slave.username=cff
spring.shardingsphere.datasource.slave.password=123456
spring.shardingsphere.datasource.slave.max-wait-millis=60000
spring.shardingsphere.datasource.slave.min-idle=20
spring.shardingsphere.datasource.slave.initial-size=2
spring.shardingsphere.datasource.slave.validation-query=SELECT 1
spring.shardingsphere.datasource.slave.connection-properties=characterEncoding=utf8
spring.shardingsphere.datasource.slave.validation-query=SELECT 1
spring.shardingsphere.datasource.slave.test-while-idle=true
spring.shardingsphere.datasource.slave.test-on-borrow=true
spring.shardingsphere.datasource.slave.test-on-return=false
spring.shardingsphere.sharding.tables.t_chat_info.actual-data-nodes=ds0.t_chat_info_$->{0..9}
spring.shardingsphere.sharding.tables.t_chat_info.table-strategy.inline.sharding-column=live_id
spring.shardingsphere.sharding.tables.t_chat_info.table-strategy.inline.algorithm-expression=t_chat_info_$->{live_id % 10}
spring.shardingsphere.sharding.tables.t_chat_info.key-generator.column=chat_no
spring.shardingsphere.sharding.tables.t_chat_info.key-generator.type=SNOWFLAKE
spring.shardingsphere.sharding.binding-tables=t_chat_info
spring.shardingsphere.sharding.master-slave-rules.ds0.master-data-source-name=master
spring.shardingsphere.sharding.master-slave-rules.ds0.slave-data-source-names=slave
mybatis.configuration.log-impl=org.apache.ibatis.logging.stdout.StdOutImpl
这里的配置主要就是数据库及数据源、sharding-jdbc的配置。
- worker.id 是雪花算法要求的机器id。自定义的属性。
- spring.shardingsphere.datasource.names 指明了有哪些数据源要被sharding-jdbc管理。
- spring.shardingsphere.datasource.master.* 是master数据源的配置,包含连接池的配置。
- spring.shardingsphere.datasource.slave.* 是slave数据源的配置,包含连接池的配置。
- spring.shardingsphere.sharding.master-slave-rules.ds0.* 指明了sharding-jdbc数据源ds0的主从数据源配置。 spring.shardingsphere.sharding.binding-tables 指明了分库分表要处理的表。
- spring.shardingsphere.sharding.tables.t_chat_info.actual-data-nodes 指明了t_chat_info表分库分表的具体表所在位置是ds0的t_chat_info_0到t_chat_info_9;
- spring.shardingsphere.sharding.tables.t_chat_info.table-strategy.inline.sharding-column指明了t_chat_info表的分表字段。
- spring.shardingsphere.sharding.tables.t_chat_info.table-strategy.inline.algorithm-expression指明了分表策略。
- spring.shardingsphere.sharding.tables.t_chat_info.key-generator.column 指明了主键。
- spring.shardingsphere.sharding.tables.t_chat_info.key-generator.type 指明了主键生成方式。(雪花算法它是不会帮你自动生成主键的,自增主键不确定。)
- mybatis.configuration.log-impl,mybatis配置日志输出。
三、测试Sharding-jdbc功能
上面都配置完成之后,代码的实现上和普通的数据库操作没区别了。
需要注意的是,增删改都要带上分表字段,查询可以不带分表字段;分表字段就是上面定义的sharding-column="live_id"
中的live_id
.
3.1 Mapper数据访问层
下面写好了增删改查,因为要带分表字段,参数都用实体来接收。
ChatInfoMapper :
代码语言:txt复制package com.cff.springbootwork.sharding.jdbc.dao;
import java.util.List;
import org.apache.ibatis.annotations.Delete;
import org.apache.ibatis.annotations.Insert;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select;
import org.apache.ibatis.annotations.Update;
import com.cff.springbootwork.sharding.jdbc.domain.ChatInfo;
@Mapper
public interface ChatInfoMapper {
@Select({
"<script>",
"SELECT ",
"chat_no as chatNo,user_id as userId,live_id as liveId,nick_name as nickName,create_time as createTime,delete_flag as deleteFlag,read_flag as readFlag",
"FROM t_chat_info",
"</script>"})
List<ChatInfo> findAll();
@Select({
"<script>",
"SELECT ",
"chat_no as chatNo,user_id as userId,live_id as liveId,nick_name as nickName,create_time as createTime,delete_flag as deleteFlag,read_flag as readFlag",
"FROM t_chat_info",
"<trim prefix=" where " prefixOverrides="AND">",
"<if test='paramIn.chatNo != null '> and chat_no = #{paramIn.chatNo} </if>",
"<if test='paramIn.userId != null '> and user_id = #{paramIn.userId} </if>",
"<if test='paramIn.liveId != null '> and live_id = #{paramIn.liveId} </if>",
"<if test='paramIn.nickName != null '> and nick_name = #{paramIn.nickName} </if>",
"<if test='paramIn.createTime != null '> and create_time = #{paramIn.createTime} </if>",
"<if test='paramIn.deleteFlag != null '> and delete_flag = #{paramIn.deleteFlag} </if>",
"<if test='paramIn.readFlag != null '> and read_flag = #{paramIn.readFlag} </if>",
"</trim>",
"</script>"})
public ChatInfo findByCondition(@Param("paramIn") ChatInfo paramIn);
@Insert({
"<script> ",
"INSERT INTO t_chat_info",
"( chat_no,user_id,live_id,nick_name,create_time,delete_flag,read_flag ) ",
" values ",
"( #{chatNo},#{userId},#{liveId},#{nickName},#{createTime},#{deleteFlag},#{readFlag} ) ",
"</script>"})
int save(ChatInfo item);
@Update({
"<script>",
"update t_chat_info",
"<trim prefix="set" suffixoverride=","> ",
"<if test='paramIn.userId != null '> user_id = #{paramIn.userId}, </if>",
"<if test='paramIn.liveId != null '> live_id = #{paramIn.liveId}, </if>",
"<if test='paramIn.nickName != null '> nick_name = #{paramIn.nickName}, </if>",
"<if test='paramIn.createTime != null '> create_time = #{paramIn.createTime}, </if>",
"<if test='paramIn.deleteFlag != null '> delete_flag = #{paramIn.deleteFlag}, </if>",
"<if test='paramIn.readFlag != null '> read_flag = #{paramIn.readFlag}, </if>",
"</trim>",
"where chat_no = #{chatNo}",
"</script>"
})
int update(@Param("updateIn") ChatInfo updateIn, @Param("chatNo") Long chatNo);
@Delete({
"<script>",
" delete from t_chat_info",
"<trim prefix=" where " prefixOverrides="AND">",
"<if test='paramIn.chatNo != null '> and chat_no = #{paramIn.chatNo} </if>",
"<if test='paramIn.userId != null '> and user_id = #{paramIn.userId} </if>",
"<if test='paramIn.liveId != null '> and live_id = #{paramIn.liveId} </if>",
"<if test='paramIn.nickName != null '> and nick_name = #{paramIn.nickName} </if>",
"<if test='paramIn.createTime != null '> and create_time = #{paramIn.createTime} </if>",
"<if test='paramIn.deleteFlag != null '> and delete_flag = #{paramIn.deleteFlag} </if>",
"<if test='paramIn.readFlag != null '> and read_flag = #{paramIn.readFlag} </if>",
"</trim>",
"</script>"
})
int delete(@Param("paramIn") ChatInfo paramIn);
}
3.2 Service逻辑层
没啥逻辑,就是个调用。注意主键要自己生成填入。
ChatInfoService:
代码语言:txt复制package com.cff.springbootwork.sharding.jdbc.service;
import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.cff.springbootwork.sharding.jdbc.dao.ChatInfoMapper;
import com.cff.springbootwork.sharding.jdbc.domain.ChatInfo;
@Service
public class ChatInfoService {
@Autowired
ChatInfoMapper chatInfoMapper;
@Autowired
SnowflakeIdGenerator snowflakeIdGenerator;
public void save(ChatInfo tChatInfo) {
tChatInfo.setChatNo(snowflakeIdGenerator.nextId());
chatInfoMapper.save(tChatInfo);
}
public void delete(ChatInfo tChatInfo) {
chatInfoMapper.delete(tChatInfo);
}
public void update(ChatInfo tChatInfo) {
chatInfoMapper.update(tChatInfo, tChatInfo.getChatNo());
}
public List<ChatInfo> findAll() {
return chatInfoMapper.findAll();
}
public ChatInfo findByChatNoAndLiveId(Integer liveId, Long id) {
ChatInfo condition = new ChatInfo();
condition.setChatNo(id);
condition.setLiveId(liveId);
return chatInfoMapper.findByCondition(condition);
}
public ChatInfo findById(Long id) {
ChatInfo condition = new ChatInfo();
condition.setChatNo(id);
return chatInfoMapper.findByCondition(condition);
}
}
3.3 Web请求层
下面的web请求层,涵盖了sharding-jdbc大多数情况下的使用。
ShardingRest :
代码语言:txt复制package com.cff.springbootwork.sharding.jdbc.web;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import com.cff.springbootwork.sharding.jdbc.domain.ChatInfo;
import com.cff.springbootwork.sharding.jdbc.dto.ResultModel;
import com.cff.springbootwork.sharding.jdbc.service.ChatInfoService;
@RestController
@RequestMapping("/sharing")
public class ShardingRest {
@Autowired
ChatInfoService chatInfoService;
@RequestMapping(value = "/add", method = { RequestMethod.POST })
public ResultModel add(@RequestBody ChatInfo chatInfo) {
chatInfoService.save(chatInfo);
return ResultModel.ok();
}
/**
* 查询也要携带分表字段,方便查找数据
* @param chatInfo
* @return
*/
@RequestMapping(value = "/info", method = { RequestMethod.GET })
public ResultModel info(@RequestParam("liveId") Integer liveId, @RequestParam("chatNo") Long chatNo) {
return ResultModel.ok(chatInfoService.findByChatNoAndLiveId(liveId,chatNo));
}
/**
* 不带分表字段查询
* @param chatInfo
* @return
*/
@RequestMapping(value = "/detail", method = { RequestMethod.GET })
public ResultModel detail(@RequestParam("chatNo") Long chatNo) {
return ResultModel.ok(chatInfoService.findById(chatNo));
}
@RequestMapping(value = "/delete", method = { RequestMethod.POST })
public ResultModel delete(@RequestBody ChatInfo chatInfo) {
chatInfoService.delete(chatInfo);
return ResultModel.ok();
}
@RequestMapping(value = "/update", method = { RequestMethod.POST })
public ResultModel update(@RequestBody ChatInfo chatInfo) {
chatInfoService.update(chatInfo);
return ResultModel.ok();
}
}
四、雪花算法
SnowflakeIdGenerator :
代码语言:txt复制package com.cff.springbootwork.sharding.jdbc.service;
import javax.annotation.PostConstruct;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
/**
* SnowFlake 算法修改
* 0 - 0000000000 00 - 0000000000 0000000000 0000000000 000000000 - 0000 - 00000000
* 符号位 -12位年月位(表示yyMM,最大4096,即可用至2040年)-39位时间戳 (可用17年,即可用至2035年)-4位机器ID(最大16,即可部署16个节点)-8位序列号(z最大256)
* @author yujinlong
*
*/
@Component
public class SnowflakeIdGenerator {
// ==============================Fields===========================================
/** 开始时间截 (2018-01-01) */
private final long twepoch = 1514736000000L;
/** 时间戳占的位数 */
public static final long timestampBits = 39L;
/** 机器id所占的位数 */
public static final long workerIdBits = 4L;
/** 支持的最大机器id,结果是15 (这个移位算法可以很快的计算出几位二进制数所能表示的最大十进制数) */
private final long maxWorkerId = -1L ^ (-1L << workerIdBits);
/** 序列在id中占的位数 */
public static final long sequenceBits = 8L;
/** 机器ID向左移6位 */
private final long workerIdShift = sequenceBits;
/** 时间截向左移12位(4 8) */
private final long timestampLeftShift = sequenceBits workerIdBits;
/** 年月标识左移51位(39 4 8)*/
private final long yearMonthLeftShift = sequenceBits workerIdBits timestampBits;
/** 生成序列的掩码,这里为255 */
private final long sequenceMask = -1L ^ (-1L << sequenceBits);
/** 工作机器ID(0~16) */
@Value("${worker.id}")
private long workerId;
/** 毫秒内序列(0~256) */
private long sequence = 0L;
/** 上次生成ID的时间截 */
private long lastTimestamp = -1L;
// ==============================Methods==========================================
@PostConstruct
public void init(){
System.out.println(workerId);
if(this.workerId < 0 || this.workerId > maxWorkerId){
throw new RuntimeException("workerId(" this.workerId ") is out of range [0, 15]");
}
}
/**
* 获得下一个ID (该方法是线程安全的)
* @return SnowflakeId
*/
public synchronized long nextId(long yyMM) {
long timestamp = timeGen();
//如果当前时间小于上一次ID生成的时间戳,说明系统时钟回退过这个时候应当抛出异常
if (timestamp < lastTimestamp) {
throw new RuntimeException(
String.format("Clock moved backwards. Refusing to generate id for %d milliseconds", lastTimestamp - timestamp));
}
//如果是同一时间生成的,则进行毫秒内序列
if (lastTimestamp == timestamp) {
sequence = (sequence 1) & sequenceMask;
//毫秒内序列溢出
if (sequence == 0) {
//阻塞到下一个毫秒,获得新的时间戳
timestamp = tilNextMillis(lastTimestamp);
}
} else {
//时间戳改变,毫秒内序列重置
sequence = 0L;
}
//上次生成ID的时间截
lastTimestamp = timestamp;
//移位并通过或运算拼到一起组成64位的ID
long preId = (yyMM << yearMonthLeftShift) | ((timestamp - twepoch) << timestampLeftShift) | (workerId << workerIdShift) | sequence;
return preId;
}
/**
* 获得不带年月位的id
* @return
*/
public synchronized long nextId() {
long timestamp = timeGen();
//如果当前时间小于上一次ID生成的时间戳,说明系统时钟回退过这个时候应当抛出异常
if (timestamp < lastTimestamp) {
throw new RuntimeException(
String.format("Clock moved backwards. Refusing to generate id for %d milliseconds", lastTimestamp - timestamp));
}
//如果是同一时间生成的,则进行毫秒内序列
if (lastTimestamp == timestamp) {
sequence = (sequence 1) & sequenceMask;
//毫秒内序列溢出
if (sequence == 0) {
//阻塞到下一个毫秒,获得新的时间戳
timestamp = tilNextMillis(lastTimestamp);
}
} else {
//时间戳改变,毫秒内序列重置
sequence = 0L;
}
//上次生成ID的时间截
lastTimestamp = timestamp;
//移位并通过或运算拼到一起组成64位的ID
long preId = ((timestamp - twepoch) << timestampLeftShift) | (workerId << workerIdShift) | sequence;
return preId;
}
/**
* 阻塞到下一个毫秒,直到获得新的时间戳
* @param lastTimestamp 上次生成ID的时间截
* @return 当前时间戳
*/
protected long tilNextMillis(long lastTimestamp) {
long timestamp = timeGen();
while (timestamp <= lastTimestamp) {
timestamp = timeGen();
}
return timestamp;
}
/**
* 返回以毫秒为单位的当前时间
* @return 当前时间(毫秒)
*/
protected long timeGen() {
return System.currentTimeMillis();
}
public void setWorkerId(long workerId) {
this.workerId = workerId;
}
}
五、过程中用到的完整实体
ChatInfo:
代码语言:txt复制package com.cff.springbootwork.sharding.jdbc.domain;
import java.util.Date;
public class ChatInfo {
private Long chatNo;
private Integer userId;
private Integer liveId;
private String nickName;
private Date createTime;
private Integer deleteFlag;
private Integer readFlag;
public void setChatNo(Long chatNo) {
this.chatNo = chatNo;
}
public Long getChatNo() {
return chatNo;
}
public Integer getUserId() {
return userId;
}
public void setUserId(Integer userId) {
this.userId = userId;
}
public Integer getLiveId() {
return liveId;
}
public void setLiveId(Integer liveId) {
this.liveId = liveId;
}
public String getNickName() {
return nickName;
}
public void setNickName(String nickName) {
this.nickName = nickName;
}
public Date getCreateTime() {
return createTime;
}
public void setCreateTime(Date createTime) {
this.createTime = createTime;
}
public Integer getDeleteFlag() {
return deleteFlag;
}
public void setDeleteFlag(Integer deleteFlag) {
this.deleteFlag = deleteFlag;
}
public Integer getReadFlag() {
return readFlag;
}
public void setReadFlag(Integer readFlag) {
this.readFlag = readFlag;
}
}
ResultModel:
代码语言:txt复制package com.cff.springbootwork.sharding.jdbc.dto;
public class ResultModel {
private String errorCode;
private String message;
private Object data;
public ResultModel() {
}
public ResultModel(String errorCode) {
this.errorCode = errorCode;
}
public ResultModel(String errorCode, String message) {
this.errorCode = errorCode;
this.message = message;
}
public ResultModel(String errorCode, String message, Object data) {
this.errorCode = errorCode;
this.message = message;
this.data = data;
}
public String getErrorCode() {
return errorCode;
}
public void setErrorCode(String errorCode) {
this.errorCode = errorCode;
}
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
public Object getData() {
return data;
}
public void setData(Object data) {
this.data = data;
}
public static ResultModel ok() {
ResultModel resultModel = new ResultModel("0000", "成功");
return resultModel;
}
public static ResultModel ok(Object data) {
ResultModel resultModel = new ResultModel("0000", "成功");
resultModel.setData(data);
return resultModel;
}
public static ResultModel error() {
ResultModel resultModel = new ResultModel("1111", "失败");
return resultModel;
}
public static ResultModel resultModel(String message) {
ResultModel resultModel = new ResultModel("1111", message);
return resultModel;
}
}
六、过程中得到的结论
得出的结论如下:
- sharding-jdbc不支持mysql-connector-java6.0以上版本,不得不降级到5.0版本。
- sharding-jdbc不能实现主从同步,主从同步还是会依赖于数据库自身机制
- RandomMasterSlaveLoadBalanceAlgorithm算法默认到从库列表中随机选择一个查询。
- 更新时,查询到从库查询,更新到主库; 如果从库有,主库无,会报乐观锁更新失败这种逗逼错误,大概是Jpa以为其他线程修改了主库。 如果从库无,主库有,更新会提示主键重复,因为它是根据从库来判断是否存在这条记录的。 两边一致后,可以正常更新,当然这个更新还只是更新主库。
- 不带分表字段也能实现查询,但肯定是所有表扫描的,sharding-jdbc没打印日志,但jpa打印日志不同,增加了好几步。
- 删除也是删除主库的,删除从库有主库无的记录会提示找不到记录的错误。删除必须带分表字段。提示错误:
Parameter
null
should extends Comparable for sharding value.