OpenStack networking - with Open vSwitch VLAN, GRE
Paul Sim Cloud Consultant
[email protected]
Index ● Prior Knowledge ● OpenStack Networking - VLAN ● OpenStack Networking - GRE ● Security Group, Floating-IP, NameSpace ● Neutron ML2
Prior Knowledge - Network NameSpace without Network NameSpace Process
with Network NameSpace
Process
Process
Process
Process
Process Process Process
Share
Network Resources Routing table
BMW NameSpace
Ford NameSpace
Benz NameSpace
Network Resources
Network Resources
Network Resources
Address Netfilter rules eth0
eth1
Network Resources eth2
eth0
eth1
eth2
Network NameSpace provides isolation of the system resources associated with networking. Thus, each network namespace has its own network devices, IP addresses, IP routing tables, /proc/net directory, port numbers, and so on. - http://lwn.net/Articles/531114/
Prior Knowledge - VLAN, GRE VLAN - Virtual LAN
802.1Q Header TPIC : 16bit - 0x8100 TCI : 16bit PCP : 3bit DEI : 1bit VID : 12bit (0 ~ 4095)
GRE - Generic Routing Encapsulation
16 Bytes Header + IP header Key field : 32bit - identify an individual traffic flow within a tunnel
OpenStack Installation - Grizzly External network 192.168.122.0/24
eth0
eth0
Controller node
Network node Quantum L3-agent
Nova
Keystone
Glance
Horizon
Quantum openvswitch-agent Quantum metadata-agent
Quantum - Server
Quantum dhcpagent
eth1
eth1
eth2
eth2
eth0
eth0
Compute node - 1
Compute node - 2
Quantum openvswitch-agent
Quantum openvswitch-agent
Nova compute
Nova compute
eth1
eth2
Management 192.168.20.0/24 Data 192.168.10.0/24
eth1
eth2
Network Topology
● ● ● ●
ext_net : external network - 192.168.122.0/24 net_proj_one : “user_one” tenant - 50.50.1.0/24 net_proj_two : “user_one” tenant - 50.50.2.0/24 net_proj_new : “user_new” tenant - 60.60.1.0/24
Big picture - VLAN OpenStack Grizzly OpenvSwitch plug-in VLAN mode
Network node net_proj_one
net_proj_two
tap~
qr~
tap~
qr~
qg~
br-ex
net_proj_new
qg~
VM
VM
VM
tap~ tag: 1
tap~ tag:2
tap~ tag:2
tap~
qr~
br-int qg~
Compute node - 1
int-br-eth1
phy-br-eth1
br-eth1
int-br-eth1 Data 192.168.10.0 /24
eth1
br-int
phy-br-eth1
eth1
br-eth1
eth0 OVS port OVS Bridge
● ●
qg~~~ : external gateway interface qr~~~ : virtual router interface
VLAN - Compute node OpenStack Grizzly OpenvSwitch plug-in VLAN mode
br-eth1
eth1
Compute node - 1
VM
VM
VM
VM
tap~ tag: 1
tap~ tag:2
tap~ tag:2
tap~ tag:3
veth pair
phy-br-eth1
int-br-eth1
br-int
Packet conversion mod_vlan_vid mod_vlan_vid
Security Group[1]
VLAN - Compute node Packet conversion janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-eth1 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=90455.716s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=2 actions=drop cookie=0x0, duration=89606.096s, table=0, n_packets=9484, n_bytes=2312018, priority=4,in_port=2,dl_vlan=1 actions=mod_vlan_vid:1024,NORMAL cookie=0x0, duration=90456.248s, table=0, n_packets=6813, n_bytes=1325511, priority=1 actions=NORMAL janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=90458.482s, table=0, n_packets=64, n_bytes=4644, priority=2,in_port=1 actions=drop cookie=0x0, duration=89608.755s, table=0, n_packets=6499, n_bytes=1283680, priority=3,in_port=1,dl_vlan=1024 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=90459.075s, table=0, n_packets=9820, n_bytes=2323195, priority=1 actions=NORMAL
openvswitch-agent.log Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0, idle_timeout=0,priority=3,in_port=1,dl_vlan=1024,actions=mod_vl an_vid:1,normal'] Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-eth1', 'hard_timeout=0, idle_timeout=0,priority=4,in_port=2,dl_vlan=1,actions=mod_vlan _vid:1024,normal']
VLAN - Network node OpenStack Grizzly OpenvSwitch plug-in VLAN mode Network node tap~
Namespcae
tap~
Namespcae
qr~ qg~
qr~ qg~ veth pair
br-int
int-br-eth1
phy-br-eth1
br-ex eth0
Packet conversion
net_proj_one
mod_vlan_id net_proj_two
Floating-IP(NAT) net_proj_new
mod_vlan_id
eth1
qg~
Namespcae
br-eth1
qr~
tap~
VLAN - Network node Packet conversion janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=7370.307s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=6 actions=drop cookie=0x0, duration=7368.424s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=6,dl_vlan=2048 actions=mod_vlan_vid:2,NORMAL cookie=0x0, duration=7367.991s, table=0, n_packets=764, n_bytes=191460, priority=3,in_port=6,dl_vlan=1024 actions=mod_vlan_vid:3, NORMAL cookie=0x0, duration=7369.073s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=6,dl_vlan=500 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=7370.924s, table=0, n_packets=549, n_bytes=104066, priority=1 actions=NORMAL janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-eth1 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=7373.826s, table=0, n_packets=14, n_bytes=1104, priority=2,in_port=2 actions=drop cookie=0x0, duration=7372.725s, table=0, n_packets=13, n_bytes=922, priority=4,in_port=2,dl_vlan=1 actions=mod_vlan_vid:500,NORMAL cookie=0x0, duration=7371.663s, table=0, n_packets=519, n_bytes=103966, priority=4,in_port=2,dl_vlan=3 actions=mod_vlan_vid:1024, NORMAL cookie=0x0, duration=7372.09s, table=0, n_packets=9, n_bytes=634, priority=4,in_port=2,dl_vlan=2 actions=mod_vlan_vid:2048,NORMAL cookie=0x0, duration=7374.384s, table=0, n_packets=764, n_bytes=191460, priority=1 actions=NORMAL
Big picture - GRE OpenStack Grizzly OpenvSwitch plug-in GRE tunneling
Network node
qr~
qr~
Tunnel gre~
qg~
patch
patch
br-int
qg~
Data 192.168.10.0 /24
tap~
br-tun
qr~
tap~
patch
tap~
net_proj_new
br-tun
net_proj_two
gre~
net_proj_one
Compute node - 1
qg~
VM
VM
tap~ tag: 1
tap~ tag:2
patch
br-int
br-ex eth0 OVS port OVS Bridge
● ●
qg~~~ : external gateway interface qr~~~ : virtual router interface
GRE - Compute node OpenStack Grizzly OpenvSwitch plug-in GRE tunneling
br-tun
Tunnel
gre~
Compute node - 1
patch
VM
VM
VM
VM
tap~ tag: 1
tap~ tag:2
tap~ tag:2
tap~ tag:3
patch
br-int
Packet conversion mod_vlan_vid set_tunnel id
Security Group[1]
GRE - Compute node
Packet conversion janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=87770.027s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00: 00 actions=mod_vlan_vid:1,output:1 cookie=0x0, duration=87770.09s, table=0, n_packets=8786, n_bytes=1893724, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL cookie=0x0, duration=87769.693s, table=0, n_packets=3031, n_bytes=617650, priority=3,tun_id=0x1,dl_dst=fa:16:3e:db:08:63 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=87769.966s, table=0, n_packets=6320, n_bytes=4432680, priority=3,tun_id=0x1,dl_dst=fa:16:3e:e0:73:95 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=87771.753s, table=0, n_packets=2921, n_bytes=951454, priority=1 actions=drop
GRE - Network node OpenStack Grizzly OpenvSwitch plug-in GRE tunneling Network node tap~
Namespcae
tap~
Namespcae
qr~
Namespcae
qr~ qg~
patch
patch
br-int br-ex eth0
Packet conversion
net_proj_one
set_tunnel id
net_proj_two
Floating-IP(NAT) net_proj_new
mod_vlan_id
Tunnel gre~
qg~
qr~
br-tun
qg~
tap~
GRE - Network node Packet conversion janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-tun NXST_FLOW reply (xid=0x4): cookie=0x0, duration=474674.446s, table=0, n_packets=7899, n_bytes=2572502, priority=3,tun_id=0x3,dl_dst=01:00:00:00:00:00/01:00:00:00:00: 00 actions=mod_vlan_vid:2,output:1 cookie=0x0, duration=473163.123s, table=0, n_packets=7876, n_bytes=2565284, priority=3,tun_id=0x4,dl_dst=01:00:00:00:00:00/01:00:00:00:00: 00 actions=mod_vlan_vid:3,output:1 cookie=0x0, duration=633937.826s, table=0, n_packets=10543, n_bytes=3426814, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00: 00 actions=mod_vlan_vid:1,output:1 cookie=0x0, duration=473163.329s, table=0, n_packets=16484, n_bytes=3348666, priority=4,in_port=1,dl_vlan=3 actions=set_tunnel:0x4, NORMAL cookie=0x0, duration=474674.541s, table=0, n_packets=16864, n_bytes=3389132, priority=4,in_port=1,dl_vlan=2 actions=set_tunnel:0x3, NORMAL cookie=0x0, duration=633937.905s, table=0, n_packets=62044, n_bytes=37320316, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1, NORMAL cookie=0x0, duration=472911.069s, table=0, n_packets=16335, n_bytes=3551350, priority=3,tun_id=0x4,dl_dst=fa:16:3e:89:fd:ce actions=mod_vlan_vid:3,NORMAL cookie=0x0, duration=474336.184s, table=0, n_packets=16360, n_bytes=3560332, priority=3,tun_id=0x3,dl_dst=fa:16:3e:d8:d5:29 actions=mod_vlan_vid:2,NORMAL cookie=0x0, duration=474674.351s, table=0, n_packets=525, n_bytes=52427, priority=3,tun_id=0x3,dl_dst=fa:16:3e:69:ca:97 actions=mod_vlan_vid:2,NORMAL cookie=0x0, duration=473162.912s, table=0, n_packets=197, n_bytes=19365, priority=3,tun_id=0x4,dl_dst=fa:16:3e:d6:b8:07 actions=mod_vlan_vid:3,NORMAL cookie=0x0, duration=633937.746s, table=0, n_packets=6207, n_bytes=630043, priority=3,tun_id=0x1,dl_dst=fa:16:3e:c7:ec:bd actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=474794.912s, table=0, n_packets=36912, n_bytes=7440964, priority=3,tun_id=0x1,dl_dst=fa:16:3e:8b:a6:d7 actions=mod_vlan_vid:1,NORMAL cookie=0x0, duration=636252.069s, table=0, n_packets=163, n_bytes=36046, priority=1 actions=drop
Security Group - VLAN, GRE FORWARD
quantum-filter-top quantum-openvswi-local Security group is applied here quantum-openvswi-FORWARD quantum-openvswi-sg-chain quantum-openvswi-iTAP_NUMBER quantum-openvswi-sg-fallback quantum-openvswi-oTAP_NUMBER quantum-openvswi-sg-fallback
Security Group - VLAN, GRE Chain quantum-openvswi-sg-chain (4 references) target prot opt source destination quantum-openvswi-i21767f1f-4 all -- 0.0.0.0/0 0.0.0.0/0 quantum-openvswi-o21767f1f-4 all -- 0.0.0.0/0 0.0.0.0/0 quantum-openvswi-i7903fd30-7 all -- 0.0.0.0/0 0.0.0.0/0 quantum-openvswi-o7903fd30-7 all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
PHYSDEV match --physdev-out tap21767f1f-45 --physdev-is-bridged PHYSDEV match --physdev-in tap21767f1f-45 --physdev-is-bridged PHYSDEV match --physdev-out tap7903fd30-74 --physdev-is-bridged PHYSDEV match --physdev-in tap7903fd30-74 --physdev-is-bridged
Chain quantum-openvswi-i7903fd30-7 (1 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 RETURN udp -- 50.50.1.3 0.0.0.0/0 udp spt:67 dpt:68 quantum-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain quantum-openvswi-o7903fd30-7 (2 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 MAC ! FA:16:3E:DB:08:63 RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 DROP all -- !50.50.1.2 0.0.0.0/0 DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN all -- 0.0.0.0/0 0.0.0.0/0 quantum-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
[1] Note, OpenStack uses iptables rules on the TAP devices such as “tap~~” to implement security groups,. However, Open vSwitch is not compatible with iptables rules that are applied directly on TAP devices that are connected to an Open vSwitch port.
Network NameSpace
janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 ifconfig lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 qg-fa243f49-d6 Link encap:Ethernet HWaddr fa:16:3e:9f:4b:63 inet addr:192.168.122.50 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe9f:4b63/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 qr-bc654dc2-f1 Link encap:Ethernet HWaddr fa:16:3e:c7:ec:bd inet addr:50.50.1.1 Bcast:50.50.1.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fec7:ecbd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 192.168.122.1 0.0.0.0 UG 0 0 0 qg-fa243f49-d6 50.50.1.0 * 255.255.255.0 U 0 0 0 qr-bc654dc2-f1 192.168.122.0 * 255.255.255.0 U 0 0 0 qg-fa243f49-d6
Floating-IP(NAT) - VLAN, GRE NameSpace janghoon@Network-node:~$ sudo ip netns show qdhcp-4c2f2346-ffaa-41a0-ab76-34cadf0163f5 qrouter-e1b88ce4-51e9-4744-be80-d70d04c6a59b qdhcp-c19e22a0-1700-4b3b-91e5-2c961ef0a353 qrouter-244fff3f-f935-4bdd-949d-739f1ce81dd0 qdhcp-f37b681a-4be8-47b8-8063-3d17d24ee1ae qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0
Floating-IP(NAT) janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 iptables -L -n -t nat Chain quantum-l3-agent-PREROUTING (1 references) target prot opt source destination REDIRECT tcp -- 0.0.0.0/0 169.254.169.254 tcp dpt:80 redir ports 9697 DNAT all -- 0.0.0.0/0 192.168.122.51 to:50.50.1.2 Chain quantum-l3-agent-float-snat (1 references) target prot opt source destination SNAT all -- 50.50.1.2 0.0.0.0/0 to:192.168.122.51 Chain quantum-l3-agent-snat (1 references) target prot opt source destination quantum-l3-agent-float-snat all -- 0.0.0.0/0 SNAT all -- 50.50.1.0/24 0.0.0.0/0
0.0.0.0/0 to:192.168.122.50
Neutron ML2 The Modular Layer 2 (ML2) plugin is a framework allowing OpenStack Networking to simultaneously utilize the variety of layer 2 networking technologies found in complex real-world data centers. It currently works with the existing openvswitch, linuxbridge, and hyperv L2 agents, and is intended to replace and deprecate the monolithic plugins associated with those L2 agents.
Neutron ML2 Plugin TypeDriver
Cisco Nexus
Arista
Flat
OpenDaylight
VxLAN
Hyper-V
GRE
OpenvSwitch
VLAN
MechanismDriver
pSwitch
TypeDriver : TypeDrivers maintain any needed type-specific network state, and perform provider network validation and tenant network allocation. MechanismDriver : The MechanismDriver is responsible for taking the information established by the TypeDriver and ensuring that it is properly applied given the specific networking mechanisms that have been enabled. https://wiki.openstack.org/wiki/Neutron/ML2
Neutron ML2
eth0
eth0
eth0
Network node
Compute node - 1
Compute node - 2
Neutron ML2-agent
Neutron ML2-agent
Nova compute
Nova compute
Neutron L3-agent Neutron ML2 plugin Neutron metadataagent Neutron dhcpagent eth1
eth2
eth1
eth2
eth1
eth2
