Projet

Général

Profil

DPDK » Historique » Version 30

Laurent GUERBY, 23/12/2015 19:29

1 1 Laurent GUERBY
{{>toc}}
2 1 Laurent GUERBY
3 1 Laurent GUERBY
h1. DPDK
4 1 Laurent GUERBY
5 1 Laurent GUERBY
Traitement haute performance en logiciel, 10 Gbit/s en petit paquets line rate
6 1 Laurent GUERBY
7 1 Laurent GUERBY
h2. Liens
8 1 Laurent GUERBY
9 1 Laurent GUERBY
* http://dpdk.org
10 1 Laurent GUERBY
* [[AtelierPPS2012]]
11 16 Laurent GUERBY
* GIT http://chiliproject.tetaneutral.net/projects/git-tetaneutral-net/repository/dpdk
12 16 Laurent GUERBY
<pre>
13 16 Laurent GUERBY
$ git clone ssh://git@git.tetaneutral.net/dpdk.git
14 16 Laurent GUERBY
</pre>
15 16 Laurent GUERBY
16 17 Laurent GUERBY
* Nerim Antoine https://twitter.com/nerim_net/status/517592380841930752
17 17 Laurent GUERBY
** Du code maison fondé sur #DPDK en prod dans le réseau : première étape vers l'élimination des systèmes propriétaires ?
18 2 Laurent GUERBY
19 2 Laurent GUERBY
h2. Misc
20 2 Laurent GUERBY
21 26 Laurent GUERBY
* https://www.kernel.org/doc/Documentation/networking/switchdev.txt
22 25 Laurent GUERBY
* https://www.usenix.org/conference/osdi14/technical-sessions/presentation/belay
23 25 Laurent GUERBY
** IX: A Protected Dataplane Operating System for High Throughput and Low Latency
24 25 Laurent GUERBY
** http://www.linuxfoundation.org/images/1/1c/Network_data_flow_through_kernel.png
25 25 Laurent GUERBY
** http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow
26 23 Laurent GUERBY
* https://blog.cloudflare.com/how-to-receive-a-million-packets/
27 23 Laurent GUERBY
** How to receive a million packets per second
28 24 Laurent GUERBY
* https://blog.cloudflare.com/how-to-achieve-low-latency/
29 21 Laurent GUERBY
* http://www.cs.cmu.edu/~dongz/papers/cuckooswitch.pdf
30 22 Laurent GUERBY
** We show that CUCKOOSWITCH can process 92.22 million minimum sized packets per second on a commodity server equipped with eight 10 Gbps Ethernet interfaces while maintaining a forwarding table of one billion forwarding entries.
31 18 Laurent GUERBY
* Improving Linux networking performance https://lwn.net/Articles/629155/
32 18 Laurent GUERBY
** https://www.youtube.com/watch?v=3XG9-X777Jo
33 19 Laurent GUERBY
** http://people.netfilter.org/hawk/presentations/LCA2015/net_stack_challenges_100G_LCA2015.pdf
34 20 Laurent GUERBY
* http://www.bigswitch.com/sites/default/files/presentations/onug-baremetal-2014-final.pdf
35 2 Laurent GUERBY
* Intel PCI id http://www.intel.com/support/network/sb/cs-012904.htm
36 3 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-products.html?iid=embed_portal+hdprod_ethernet#s1=Gigabit%20Ethernet&s2=all&s3=all
37 3 Laurent GUERBY
* Intel Pro/1000 PT vs CT http://hardforum.com/showthread.php?t=1386042
38 4 Laurent GUERBY
* http://images.ncix.com/forumimages/C9864B87-583B-10FB-B0DE80742781CE07.jpg
39 4 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82572EI&s3=all
40 4 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82574L&s3=all
41 5 Laurent GUERBY
* https://github.com/Pktgen/Pktgen-DPDK
42 6 Laurent GUERBY
* Evaluating the Suitability of Server Network Cards for Software Routers http://kfall.net/ucbpage/papers/snc.pdf
43 13 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/guides/intel-dpdk-sample-applications-user-guide.pdf
44 13 Laurent GUERBY
*  rte_eth_rx_burst
45 13 Laurent GUERBY
** http://dpdk.org/doc/api/rte__ethdev_8h.html#a5ac93faa0f3f69545ca121dcfd9433f5
46 13 Laurent GUERBY
** http://dpdk.info/ml/archives/dev/2014-January/001223.html
47 14 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/white-papers/aes-ipsec-performance-linux-paper.pdf
48 15 Laurent GUERBY
* http://networkbuilders.intel.com/docs/Network_Builders_RA_vBRAS_Final.pdf
49 15 Laurent GUERBY
50 15 Laurent GUERBY
* https://github.com/SnabbCo/snabbswitch
51 15 Laurent GUERBY
** https://github.com/SnabbCo/snabbswitch/blob/master/src/apps/intel/intel10g.lua
52 15 Laurent GUERBY
** http://blog.ipspace.net/2014/06/snabb-switch-and-nfv-on-openstack-in.html
53 15 Laurent GUERBY
** https://github.com/SnabbCo/snabbswitch/wiki/Snabb-Lab
54 15 Laurent GUERBY
** https://twitter.com/lukego/status/327714050219847680
55 7 Mehdi Abaakouk
56 7 Mehdi Abaakouk
h2. Note sileht dpdk:
57 7 Mehdi Abaakouk
58 7 Mehdi Abaakouk
Extract from: http://www.intel.com/content/dam/www/public/us/en/documents/guides/intel-dpdk-getting-started-guide.pdf
59 7 Mehdi Abaakouk
60 7 Mehdi Abaakouk
h3. configuration hugepages: 
61 7 Mehdi Abaakouk
62 7 Mehdi Abaakouk
* 2M (1024*2k): hugepages=1024 (at runtime: echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
63 7 Mehdi Abaakouk
* 4G (4x1G): default_hugepagesz=1G hugepagesz=1G hugepages=4 (only works at boot time via grub)
64 7 Mehdi Abaakouk
65 7 Mehdi Abaakouk
<pre>
66 7 Mehdi Abaakouk
mkdir /mnt/huge
67 7 Mehdi Abaakouk
mount -t hugetlbfs nodev /mnt/huge
68 7 Mehdi Abaakouk
</pre>
69 7 Mehdi Abaakouk
70 7 Mehdi Abaakouk
h3. Compile and load modules: 
71 7 Mehdi Abaakouk
72 7 Mehdi Abaakouk
_Note: source tools/setup.sh is a helper tools for this but works only with IGB driver not e1000e_
73 7 Mehdi Abaakouk
74 1 Laurent GUERBY
<pre>
75 8 Mehdi Abaakouk
# apt-get install make gcc linux-headers-amd64
76 8 Mehdi Abaakouk
# make config T=x86_64-default-linuxapp-gcc
77 7 Mehdi Abaakouk
# make T=x86_64-default-linuxapp-gcc
78 7 Mehdi Abaakouk
..
79 7 Mehdi Abaakouk
Build complete
80 7 Mehdi Abaakouk
81 7 Mehdi Abaakouk
# modprobe uio  (I think this is not useful)
82 7 Mehdi Abaakouk
# insmod build/kmod/rte_kni.ko (I think this is not useful)
83 7 Mehdi Abaakouk
# insmod build/kmod/igb_uio.ko (I think this is not useful)
84 7 Mehdi Abaakouk
# ./tools/pci_unbind.py --status
85 7 Mehdi Abaakouk
86 7 Mehdi Abaakouk
Network devices using IGB_UIO driver
87 7 Mehdi Abaakouk
====================================
88 7 Mehdi Abaakouk
<none>
89 7 Mehdi Abaakouk
90 7 Mehdi Abaakouk
Network devices using kernel driver
91 7 Mehdi Abaakouk
===================================
92 9 Mehdi Abaakouk
0000:00:19.0 '82579LM Gigabit Network Connection' if=eth1 drv=e1000e unused=<none>
93 9 Mehdi Abaakouk
0000:01:00.0 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth6 drv=ixgbe unused=igb_uio *Active*
94 9 Mehdi Abaakouk
0000:01:00.1 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth7 drv=ixgbe unused=igb_uio
95 9 Mehdi Abaakouk
0000:04:00.0 '82571EB Gigabit Ethernet Controller' if=eth2 drv=e1000e unused=igb_uio *Active*
96 9 Mehdi Abaakouk
0000:04:00.1 '82571EB Gigabit Ethernet Controller' if=eth3 drv=e1000e unused=igb_uio
97 9 Mehdi Abaakouk
0000:05:00.0 '82571EB Gigabit Ethernet Controller' if=eth4 drv=e1000e unused=igb_uio
98 9 Mehdi Abaakouk
0000:05:00.1 '82571EB Gigabit Ethernet Controller' if=eth5 drv=e1000e unused=igb_uio
99 9 Mehdi Abaakouk
0000:06:00.0 'RTL8111/8168B PCI Express Gigabit Ethernet controller' if=eth0 drv=r8169 unused=<none> *Active*
100 1 Laurent GUERBY
101 1 Laurent GUERBY
Other network devices
102 1 Laurent GUERBY
=====================
103 1 Laurent GUERBY
<none>
104 9 Mehdi Abaakouk
</pre>
105 7 Mehdi Abaakouk
106 9 Mehdi Abaakouk
test sur eth7:
107 9 Mehdi Abaakouk
108 9 Mehdi Abaakouk
<pre>
109 9 Mehdi Abaakouk
# ip link set eth7 down
110 9 Mehdi Abaakouk
# ./tools/pci_unbind.py --bind=igb_uio 0000:01:00.1
111 7 Mehdi Abaakouk
</pre>
112 7 Mehdi Abaakouk
113 7 Mehdi Abaakouk
h3. Prepare examples programs:
114 7 Mehdi Abaakouk
115 7 Mehdi Abaakouk
<pre>
116 7 Mehdi Abaakouk
# export RTE_SDK=/root/sileht/dpdk-1.6.0r1
117 7 Mehdi Abaakouk
# export RTE_TARGET=build
118 7 Mehdi Abaakouk
# cd /root/sileht/
119 7 Mehdi Abaakouk
# cp -r $RTE_SDK/examples/helloworld my_rte_app
120 7 Mehdi Abaakouk
# cd my_rte_app
121 7 Mehdi Abaakouk
# make
122 7 Mehdi Abaakouk
</pre>
123 7 Mehdi Abaakouk
124 7 Mehdi Abaakouk
125 7 Mehdi Abaakouk
h3. Tests
126 7 Mehdi Abaakouk
127 7 Mehdi Abaakouk
<pre>
128 1 Laurent GUERBY
</pre>
129 10 Mehdi Abaakouk
130 11 Mehdi Abaakouk
h2. Limitation SPF carte intel:
131 10 Mehdi Abaakouk
132 11 Mehdi Abaakouk
Une fois le module igb_uio changé sur les:
133 10 Mehdi Abaakouk
134 10 Mehdi Abaakouk
<pre>
135 10 Mehdi Abaakouk
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
136 10 Mehdi Abaakouk
</pre>
137 10 Mehdi Abaakouk
138 11 Mehdi Abaakouk
Un flags de l'eeprom change et le module ixgbe ne charge plus les ports de la carte qui n'ont pas des SFP+ intel avec le message d'erreur suivant:
139 10 Mehdi Abaakouk
140 10 Mehdi Abaakouk
<pre>
141 10 Mehdi Abaakouk
[  563.069683] ixgbe 0000:01:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
142 10 Mehdi Abaakouk
[  563.069786] ixgbe 0000:01:00.0: Reload the driver after installing a supported module.
143 10 Mehdi Abaakouk
</pre>
144 10 Mehdi Abaakouk
145 10 Mehdi Abaakouk
Pour contourner le problème:
146 10 Mehdi Abaakouk
147 10 Mehdi Abaakouk
<pre>
148 10 Mehdi Abaakouk
modprobe ixgbe allow_unsupported_sfp=1 # allow_any_sfp=1 suivant le kernel
149 10 Mehdi Abaakouk
</pre>
150 10 Mehdi Abaakouk
151 10 Mehdi Abaakouk
ou 
152 10 Mehdi Abaakouk
153 1 Laurent GUERBY
<pre>
154 1 Laurent GUERBY
echo "options ixgbe allow_unsupported_sfp=1" > /etc/modprobe.d/ixgbe_any_sfp.conf
155 1 Laurent GUERBY
update-initramfs -u -k all
156 11 Mehdi Abaakouk
</pre>
157 11 Mehdi Abaakouk
158 11 Mehdi Abaakouk
La carte fonctionnera à nouveau avec le message suivant:
159 11 Mehdi Abaakouk
160 11 Mehdi Abaakouk
<pre>
161 11 Mehdi Abaakouk
[  908.361767] ixgbe 0000:01:00.0 (unregistered net_device): WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics.  Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter.  Intel Corporation is not responsible for any harm caused by using untested modules.
162 10 Mehdi Abaakouk
</pre>
163 12 Laurent GUERBY
164 27 Laurent GUERBY
h2. Packet Journey
165 1 Laurent GUERBY
166 27 Laurent GUERBY
https://www.gandi.net/news/en/2015-12-16/6308-packet-journey_a_free_software_router_for_linux_based_on_dpdk/
167 27 Laurent GUERBY
https://github.com/gandi/packet-journey
168 27 Laurent GUERBY
169 27 Laurent GUERBY
h2. Atelier 2016
170 27 Laurent GUERBY
171 1 Laurent GUERBY
<pre>
172 28 Laurent GUERBY
core i7 2600 @ 3.40GHz
173 28 Laurent GUERBY
16G RAM
174 28 Laurent GUERBY
SSD 250G
175 28 Laurent GUERBY
</pre>
176 28 Laurent GUERBY
177 28 Laurent GUERBY
<pre>
178 28 Laurent GUERBY
host eth DHCP          10G-a                                           10G-b
179 28 Laurent GUERBY
t1   54:04:a6:ef:a8:c7 00:1b:21:d9:28:55 fe80::21b:21ff:fed9:2855(t2b) 00:1b:21:d9:28:54 fe80::21b:21ff:fed9:2854(t3b)
180 28 Laurent GUERBY
t2   e0:69:95:2e:e6:a2 00:1b:21:d9:1a:e1 fe80::21b:21ff:fed9:1ae1(t3a) 00:1b:21:d9:1a:e0 fe80::21b:21ff:fed9:1ae0(t1a)
181 28 Laurent GUERBY
t3   bc:ae:c5:e1:b2:6d 00:1b:21:d9:df:3d fe80::21b:21ff:fed9:df3d(t2a) 00:1b:21:d9:df:3c fe80::21b:21ff:fed9:df3c(t1b)
182 27 Laurent GUERBY
</pre>
183 29 Laurent GUERBY
184 30 Laurent GUERBY
http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html
185 30 Laurent GUERBY
186 29 Laurent GUERBY
<pre>
187 30 Laurent GUERBY
modprobe uio_pci_generic
188 29 Laurent GUERBY
wget http://dpdk.org/browse/dpdk/snapshot/dpdk-2.2.0.tar.gz
189 29 Laurent GUERBY
git clone https://github.com/Gandi/packet-journey.git
190 29 Laurent GUERBY
git clone http://dpdk.org/git/apps/pktgen-dpdk
191 29 Laurent GUERBY
git clone http://dpdk.org/git/dpdk-stable
192 30 Laurent GUERBY
tar xf dpdk-2.2.0.tar.gz
193 30 Laurent GUERBY
cd dpdk-2.2.0
194 30 Laurent GUERBY
make config T=x86_64-native-linuxapp-gcc
195 30 Laurent GUERBY
make install T=x86_64-native-linuxapp-gcc
196 29 Laurent GUERBY
</pre>