前由
1. 我们在企业内部实际使用中，通常需要使用vlan划分几个网段。
PVE默认使用Linux自带的网桥提供网络交换服务，在划分vlan的时候还需要修改IP路由表文件，配置稍显繁琐。
这里推荐使用专门为虚拟化设计的Open vSwitch。
Open vSwitch是一个高质量的、多层虚拟交换机，使用开源Apache2.0许可协议，由Nicira Networks开发，主要实现代码为可移植的C代码。它的目的是让大规模网络自动化可以通过编程扩展,同时仍然支持标准的管理接口和协议（例如NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag）。此外,它被设计位支持跨越多个物理服务器的分布式环境，类似于VMware的vNetwork分布式vswitch或Cisco Nexus 1000 V。Open vSwitch支持多种linux 虚拟化技术，包括Xen/XenServer， KVM和irtualBox。 https://www.openvswitch.org/
2. 服务器通常有不止一块网卡。
很多人喜欢把几块网卡配置为主备模式，这样虽然有冗余，可是也浪费了一半的带宽。我个人更喜欢使用LACP(链路聚合控制协议)，让多条链路汇聚，既增加了带宽提高了流量，同时也保证了链路的安全冗余。
安装
代码语言：javascript复制# apt install openvswitch-switch -y配置
1. PVE配置
我一直不习惯使用PVE的网络配置界面，反正不管怎么，最后修改都是这一个文件：/etc/network/interfaces
下面的例子是把两块物理网卡(enp9s0 enp10s0)绑定为一块逻辑网卡(bond0，绑定模式为LACP)，然后在上面架设网关(vmbr0)，供vlan10/vlan11/vlan12通行
代码语言：javascript复制# cat /etc/network/interfaces
auto lo
iface lo inet loopback
allow-vmbr1 bond0
iface bond0 inet manual
ovs_bonds enp9s0 enp10s0
ovs_type OVSBond
ovs_bridge vmbr1
ovs_options bond_mode=balance-slb lacp=active
pre-up ( ip link set mtu 9000 dev enp9s0 && ip link set mtu 9000 dev enp10s0 )
mtu 9000
auto vmbr1
iface vmbr1 inet manual
ovs_type OVSBridge
ovs_ports bond0 vlan10 vlan11 vlan12
allow-vmbr1 vlan10
iface vlan10 inet static
address 192.168.10.11
netmask 255.255.255.0
gateway 192.168.10.254
ovs_type OVSIntPort
ovs_bridge vmbr1
ovs_options tag=10
ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
allow-vmbr1 vlan11
iface vlan11 inet static
address 192.168.11.11
netmask 255.255.255.0
gateway 192.168.10.254
ovs_type OVSIntPort
ovs_bridge vmbr1
ovs_options tag=11
ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
allow-vmbr1 vlan12
iface vlan12 inet static
address 192.168.12.11
netmask 255.255.255.0
gateway 192.168.10.254
ovs_type OVSIntPort
ovs_bridge vmbr1
ovs_options tag=12
ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif重启网络服务：
代码语言：javascript复制# /etc/init.d/networking restart如果没有生效就重启一遍PVE服务器：
代码语言：javascript复制# reboot2. 交换机配置
瑞捷S5750-24GT/8SFP-S
绑定示例：G0/1和G0/2，G0/3和G0/4，G0/5和G0/6
代码语言：javascript复制ruijie02#show run
......
interface GigabitEthernet 0/1
port-group 1 mode active
!
interface GigabitEthernet 0/2
port-group 1 mode active
!
interface GigabitEthernet 0/3
port-group 2 mode active
!
interface GigabitEthernet 0/4
port-group 2 mode active
!
interface GigabitEthernet 0/5
port-group 3 mode active
!
interface GigabitEthernet 0/6
port-group 3 mode active
......
interface AggregatePort 1
switchport mode trunk
switchport trunk allowed vlan remove 1-9,13-4094
!
interface AggregatePort 2
switchport mode trunk
switchport trunk allowed vlan remove 1-9,13-4094
!
interface AggregatePort 3
switchport mode trunk
switchport trunk allowed vlan remove 1-9,13-4094
......测试
在PVE服务器上查看网络接口
使用linux自带的ip命令
代码语言：javascript复制# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
#下面可以看到两块网卡都属于ovs-system
2: enp9s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
link/ether xx:xx:xx:75:d2:d7 brd ff:ff:ff:ff:ff:ff
inet6 fe80::325a:3aff:fe75:d2d7/64 scope link
valid_lft forever preferred_lft forever
3: enp10s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
link/ether xx:xx:xx:75:d2:d8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::325a:3aff:fe75:d2d8/64 scope link
valid_lft forever preferred_lft forever
......
#下面是虚拟出来的网络交换设备，包括bond0,网关vmbr0和vlan
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether fa:2f:ff:73:d0:f9 brd ff:ff:ff:ff:ff:ff
7: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether xx:xx:xx:75:d2:d8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::325a:3aff:fe75:d2d8/64 scope link
valid_lft forever preferred_lft forever
8: bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether f6:af:9e:5e:5b:49 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f4af:9eff:fe5e:5b49/64 scope link
valid_lft forever preferred_lft forever
......
10: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 36:1c:86:ed:a6:d3 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.11/24 brd 192.168.10.255 scope global vlan10
valid_lft forever preferred_lft forever
inet6 fe80::341c:86ff:feed:a6d3/64 scope link
valid_lft forever preferred_lft forever
11: vlan11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether da:26:e8:18:85:35 brd ff:ff:ff:ff:ff:ff
inet 192.168.11.11/24 brd 192.168.11.255 scope global vlan11
valid_lft forever preferred_lft forever
inet6 fe80::d826:e8ff:fe18:8535/64 scope link
valid_lft forever preferred_lft forever
valid_lft forever preferred_lft forever
......
＃下面是分配给虚拟机的网卡
14: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master ovs-system state UNKNOWN group default qlen 1000
link/ether fa:4e:69:07:e9:73 brd ff:ff:ff:ff:ff:ff
......直接使用Open vSwitch工具查看：
代码语言：javascript复制# ovs-appctl bond/show
---- bond0 ----
bond_mode: balance-slb
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 2454 ms
lacp_status: negotiated
lacp_fallback_ab: false
active slave mac: xx:xx:xx:75:d2:d8(enp10s0)
slave enp10s0: enabled
active slave
may_enable: true
hash 8: 3 kB load
hash 38: 31 kB load
hash 66: 1 kB load
hash 145: 2 kB load
hash 146: 7 kB load
slave enp9s0: enabled
may_enable: true
hash 33: 171 kB load
# ovs-appctl lacp/show
---- bond0 ----
status: active negotiated
sys_id: xx:xx:xx:75:d2:d8
sys_priority: 65534
aggregation key: 1
lacp_time: slow
slave: enp10s0: current attached
port_id: 2
port_priority: 65535
may_enable: true
actor sys_id: xx:xx:xx:75:d2:d8
actor sys_priority: 65534
actor port_id: 2
actor port_priority: 65535
actor key: 1
actor state: activity aggregation synchronized collecting distributing
partner sys_id: xx:xx:xx:34:69:52
partner sys_priority: 32768
partner port_id: 2
partner port_priority: 32768
partner key: 1
partner state: activity aggregation synchronized collecting distributing
slave: enp9s0: current attached
port_id: 1
port_priority: 65535
may_enable: true
actor sys_id: xx:xx:xx:75:d2:d8
actor sys_priority: 65534
actor port_id: 1
actor port_priority: 65535
actor key: 1
actor state: activity aggregation synchronized collecting distributing
partner sys_id: xx:xx:xx:34:69:52
partner sys_priority: 32768
partner port_id: 1
partner port_priority: 32768
partner key: 1
partner state: activity aggregation synchronized collecting distributing我们最关心的应该还是流量是否分担到了两条链路上:
代码语言：javascript复制# apt install nload -y
# nload -m enp9s0 enp10s0
可以看到两块物理网卡都有流量，也可以到交换机上查看：
后附
1. Open vSwitch自带的命令很多，创建／删除／查看都有，可以自己多试试；
2. 华为和华三交换机(还是S5700系列，为什么各家的网络交换机取名都差不多？)的配置示例：
代码语言：javascript复制......
#
interface Eth-Trunk1
port link-type trunk
port trunk allow-pass vlan 2 to 3999
mode lacp
#
interface Eth-Trunk2
port link-type trunk
port trunk allow-pass vlan 2 to 3999
mode lacp
#
interface Eth-Trunk3
port link-type trunk
port trunk allow-pass vlan 2 to 3999
mode lacp
......
interface GigabitEthernet0/0/1
eth-trunk 1
#
interface GigabitEthernet0/0/2
eth-trunk 1
#
interface GigabitEthernet0/0/3
eth-trunk 2
#
interface GigabitEthernet0/0/4
eth-trunk 2
#
interface GigabitEthernet0/0/5
eth-trunk 3
#
interface GigabitEthernet0/0/6
eth-trunk 3
......3. 主备模式
如果你的交换机不支持LACP，用主备也可以的，只需要
修改绑定模式从lacp为主备
代码语言：javascript复制#ovs_options bond_mode=balance-slb lacp=active
ovs_options bond_mode=active-backup记得交换机那边相应的取消LACP。
4. 不做绑定
即使不做任何绑定，单个网卡也可以使用Open vSwitch替换PVE自带的Linux网桥，这里就不专门讲解了。
