Projet

Général

Profil

DPDK » Historique » Version 63

Laurent GUERBY, 29/12/2015 17:56

1 1 Laurent GUERBY
{{>toc}}
2 1 Laurent GUERBY
3 1 Laurent GUERBY
h1. DPDK
4 1 Laurent GUERBY
5 1 Laurent GUERBY
Traitement haute performance en logiciel, 10 Gbit/s en petit paquets line rate
6 1 Laurent GUERBY
7 1 Laurent GUERBY
h2. Liens
8 1 Laurent GUERBY
9 1 Laurent GUERBY
* http://dpdk.org
10 1 Laurent GUERBY
* [[AtelierPPS2012]]
11 16 Laurent GUERBY
* GIT http://chiliproject.tetaneutral.net/projects/git-tetaneutral-net/repository/dpdk
12 16 Laurent GUERBY
<pre>
13 16 Laurent GUERBY
$ git clone ssh://git@git.tetaneutral.net/dpdk.git
14 16 Laurent GUERBY
</pre>
15 16 Laurent GUERBY
16 17 Laurent GUERBY
* Nerim Antoine https://twitter.com/nerim_net/status/517592380841930752
17 17 Laurent GUERBY
** Du code maison fondé sur #DPDK en prod dans le réseau : première étape vers l'élimination des systèmes propriétaires ?
18 2 Laurent GUERBY
19 2 Laurent GUERBY
h2. Misc
20 2 Laurent GUERBY
21 62 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf
22 45 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf
23 45 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/datasheets/82598-10-gbe-controller-datasheet.pdf
24 44 Laurent GUERBY
* MoonGen: A Scriptable High-Speed Packet Generator
25 44 Laurent GUERBY
** http://arxiv.org/pdf/1410.3322.pdf
26 44 Laurent GUERBY
** http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/MoonGen_IMC2015.pdf
27 44 Laurent GUERBY
** https://github.com/emmericp/MoonGen
28 26 Laurent GUERBY
* https://www.kernel.org/doc/Documentation/networking/switchdev.txt
29 25 Laurent GUERBY
* https://www.usenix.org/conference/osdi14/technical-sessions/presentation/belay
30 25 Laurent GUERBY
** IX: A Protected Dataplane Operating System for High Throughput and Low Latency
31 25 Laurent GUERBY
** http://www.linuxfoundation.org/images/1/1c/Network_data_flow_through_kernel.png
32 25 Laurent GUERBY
** http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow
33 23 Laurent GUERBY
* https://blog.cloudflare.com/how-to-receive-a-million-packets/
34 23 Laurent GUERBY
** How to receive a million packets per second
35 24 Laurent GUERBY
* https://blog.cloudflare.com/how-to-achieve-low-latency/
36 21 Laurent GUERBY
* http://www.cs.cmu.edu/~dongz/papers/cuckooswitch.pdf
37 22 Laurent GUERBY
** We show that CUCKOOSWITCH can process 92.22 million minimum sized packets per second on a commodity server equipped with eight 10 Gbps Ethernet interfaces while maintaining a forwarding table of one billion forwarding entries.
38 18 Laurent GUERBY
* Improving Linux networking performance https://lwn.net/Articles/629155/
39 18 Laurent GUERBY
** https://www.youtube.com/watch?v=3XG9-X777Jo
40 19 Laurent GUERBY
** http://people.netfilter.org/hawk/presentations/LCA2015/net_stack_challenges_100G_LCA2015.pdf
41 20 Laurent GUERBY
* http://www.bigswitch.com/sites/default/files/presentations/onug-baremetal-2014-final.pdf
42 2 Laurent GUERBY
* Intel PCI id http://www.intel.com/support/network/sb/cs-012904.htm
43 3 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-products.html?iid=embed_portal+hdprod_ethernet#s1=Gigabit%20Ethernet&s2=all&s3=all
44 3 Laurent GUERBY
* Intel Pro/1000 PT vs CT http://hardforum.com/showthread.php?t=1386042
45 4 Laurent GUERBY
* http://images.ncix.com/forumimages/C9864B87-583B-10FB-B0DE80742781CE07.jpg
46 4 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82572EI&s3=all
47 4 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82574L&s3=all
48 5 Laurent GUERBY
* https://github.com/Pktgen/Pktgen-DPDK
49 6 Laurent GUERBY
* Evaluating the Suitability of Server Network Cards for Software Routers http://kfall.net/ucbpage/papers/snc.pdf
50 13 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/guides/intel-dpdk-sample-applications-user-guide.pdf
51 13 Laurent GUERBY
*  rte_eth_rx_burst
52 13 Laurent GUERBY
** http://dpdk.org/doc/api/rte__ethdev_8h.html#a5ac93faa0f3f69545ca121dcfd9433f5
53 13 Laurent GUERBY
** http://dpdk.info/ml/archives/dev/2014-January/001223.html
54 14 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/white-papers/aes-ipsec-performance-linux-paper.pdf
55 15 Laurent GUERBY
* http://networkbuilders.intel.com/docs/Network_Builders_RA_vBRAS_Final.pdf
56 15 Laurent GUERBY
57 15 Laurent GUERBY
* https://github.com/SnabbCo/snabbswitch
58 15 Laurent GUERBY
** https://github.com/SnabbCo/snabbswitch/blob/master/src/apps/intel/intel10g.lua
59 15 Laurent GUERBY
** http://blog.ipspace.net/2014/06/snabb-switch-and-nfv-on-openstack-in.html
60 15 Laurent GUERBY
** https://github.com/SnabbCo/snabbswitch/wiki/Snabb-Lab
61 15 Laurent GUERBY
** https://twitter.com/lukego/status/327714050219847680
62 7 Mehdi Abaakouk
63 7 Mehdi Abaakouk
h2. Note sileht dpdk:
64 7 Mehdi Abaakouk
65 7 Mehdi Abaakouk
Extract from: http://www.intel.com/content/dam/www/public/us/en/documents/guides/intel-dpdk-getting-started-guide.pdf
66 7 Mehdi Abaakouk
67 7 Mehdi Abaakouk
h3. configuration hugepages: 
68 7 Mehdi Abaakouk
69 7 Mehdi Abaakouk
* 2M (1024*2k): hugepages=1024 (at runtime: echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
70 7 Mehdi Abaakouk
* 4G (4x1G): default_hugepagesz=1G hugepagesz=1G hugepages=4 (only works at boot time via grub)
71 7 Mehdi Abaakouk
72 7 Mehdi Abaakouk
<pre>
73 7 Mehdi Abaakouk
mkdir /mnt/huge
74 7 Mehdi Abaakouk
mount -t hugetlbfs nodev /mnt/huge
75 7 Mehdi Abaakouk
</pre>
76 7 Mehdi Abaakouk
77 7 Mehdi Abaakouk
h3. Compile and load modules: 
78 7 Mehdi Abaakouk
79 7 Mehdi Abaakouk
_Note: source tools/setup.sh is a helper tools for this but works only with IGB driver not e1000e_
80 7 Mehdi Abaakouk
81 1 Laurent GUERBY
<pre>
82 8 Mehdi Abaakouk
# apt-get install make gcc linux-headers-amd64
83 8 Mehdi Abaakouk
# make config T=x86_64-default-linuxapp-gcc
84 7 Mehdi Abaakouk
# make T=x86_64-default-linuxapp-gcc
85 7 Mehdi Abaakouk
..
86 7 Mehdi Abaakouk
Build complete
87 7 Mehdi Abaakouk
88 7 Mehdi Abaakouk
# modprobe uio  (I think this is not useful)
89 7 Mehdi Abaakouk
# insmod build/kmod/rte_kni.ko (I think this is not useful)
90 7 Mehdi Abaakouk
# insmod build/kmod/igb_uio.ko (I think this is not useful)
91 7 Mehdi Abaakouk
# ./tools/pci_unbind.py --status
92 7 Mehdi Abaakouk
93 7 Mehdi Abaakouk
Network devices using IGB_UIO driver
94 7 Mehdi Abaakouk
====================================
95 7 Mehdi Abaakouk
<none>
96 7 Mehdi Abaakouk
97 7 Mehdi Abaakouk
Network devices using kernel driver
98 7 Mehdi Abaakouk
===================================
99 9 Mehdi Abaakouk
0000:00:19.0 '82579LM Gigabit Network Connection' if=eth1 drv=e1000e unused=<none>
100 9 Mehdi Abaakouk
0000:01:00.0 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth6 drv=ixgbe unused=igb_uio *Active*
101 9 Mehdi Abaakouk
0000:01:00.1 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth7 drv=ixgbe unused=igb_uio
102 9 Mehdi Abaakouk
0000:04:00.0 '82571EB Gigabit Ethernet Controller' if=eth2 drv=e1000e unused=igb_uio *Active*
103 9 Mehdi Abaakouk
0000:04:00.1 '82571EB Gigabit Ethernet Controller' if=eth3 drv=e1000e unused=igb_uio
104 9 Mehdi Abaakouk
0000:05:00.0 '82571EB Gigabit Ethernet Controller' if=eth4 drv=e1000e unused=igb_uio
105 9 Mehdi Abaakouk
0000:05:00.1 '82571EB Gigabit Ethernet Controller' if=eth5 drv=e1000e unused=igb_uio
106 9 Mehdi Abaakouk
0000:06:00.0 'RTL8111/8168B PCI Express Gigabit Ethernet controller' if=eth0 drv=r8169 unused=<none> *Active*
107 1 Laurent GUERBY
108 1 Laurent GUERBY
Other network devices
109 1 Laurent GUERBY
=====================
110 1 Laurent GUERBY
<none>
111 9 Mehdi Abaakouk
</pre>
112 7 Mehdi Abaakouk
113 9 Mehdi Abaakouk
test sur eth7:
114 9 Mehdi Abaakouk
115 9 Mehdi Abaakouk
<pre>
116 9 Mehdi Abaakouk
# ip link set eth7 down
117 9 Mehdi Abaakouk
# ./tools/pci_unbind.py --bind=igb_uio 0000:01:00.1
118 7 Mehdi Abaakouk
</pre>
119 7 Mehdi Abaakouk
120 7 Mehdi Abaakouk
h3. Prepare examples programs:
121 7 Mehdi Abaakouk
122 7 Mehdi Abaakouk
<pre>
123 7 Mehdi Abaakouk
# export RTE_SDK=/root/sileht/dpdk-1.6.0r1
124 7 Mehdi Abaakouk
# export RTE_TARGET=build
125 7 Mehdi Abaakouk
# cd /root/sileht/
126 7 Mehdi Abaakouk
# cp -r $RTE_SDK/examples/helloworld my_rte_app
127 7 Mehdi Abaakouk
# cd my_rte_app
128 7 Mehdi Abaakouk
# make
129 7 Mehdi Abaakouk
</pre>
130 7 Mehdi Abaakouk
131 7 Mehdi Abaakouk
132 7 Mehdi Abaakouk
h3. Tests
133 7 Mehdi Abaakouk
134 7 Mehdi Abaakouk
<pre>
135 1 Laurent GUERBY
</pre>
136 10 Mehdi Abaakouk
137 11 Mehdi Abaakouk
h2. Limitation SPF carte intel:
138 10 Mehdi Abaakouk
139 11 Mehdi Abaakouk
Une fois le module igb_uio changé sur les:
140 10 Mehdi Abaakouk
141 10 Mehdi Abaakouk
<pre>
142 10 Mehdi Abaakouk
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
143 10 Mehdi Abaakouk
</pre>
144 10 Mehdi Abaakouk
145 11 Mehdi Abaakouk
Un flags de l'eeprom change et le module ixgbe ne charge plus les ports de la carte qui n'ont pas des SFP+ intel avec le message d'erreur suivant:
146 10 Mehdi Abaakouk
147 10 Mehdi Abaakouk
<pre>
148 10 Mehdi Abaakouk
[  563.069683] ixgbe 0000:01:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
149 10 Mehdi Abaakouk
[  563.069786] ixgbe 0000:01:00.0: Reload the driver after installing a supported module.
150 10 Mehdi Abaakouk
</pre>
151 10 Mehdi Abaakouk
152 10 Mehdi Abaakouk
Pour contourner le problème:
153 10 Mehdi Abaakouk
154 10 Mehdi Abaakouk
<pre>
155 10 Mehdi Abaakouk
modprobe ixgbe allow_unsupported_sfp=1 # allow_any_sfp=1 suivant le kernel
156 10 Mehdi Abaakouk
</pre>
157 10 Mehdi Abaakouk
158 10 Mehdi Abaakouk
ou 
159 10 Mehdi Abaakouk
160 1 Laurent GUERBY
<pre>
161 1 Laurent GUERBY
echo "options ixgbe allow_unsupported_sfp=1" > /etc/modprobe.d/ixgbe_any_sfp.conf
162 1 Laurent GUERBY
update-initramfs -u -k all
163 11 Mehdi Abaakouk
</pre>
164 11 Mehdi Abaakouk
165 11 Mehdi Abaakouk
La carte fonctionnera à nouveau avec le message suivant:
166 11 Mehdi Abaakouk
167 11 Mehdi Abaakouk
<pre>
168 11 Mehdi Abaakouk
[  908.361767] ixgbe 0000:01:00.0 (unregistered net_device): WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics.  Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter.  Intel Corporation is not responsible for any harm caused by using untested modules.
169 10 Mehdi Abaakouk
</pre>
170 12 Laurent GUERBY
171 27 Laurent GUERBY
h2. Packet Journey
172 1 Laurent GUERBY
173 27 Laurent GUERBY
https://www.gandi.net/news/en/2015-12-16/6308-packet-journey_a_free_software_router_for_linux_based_on_dpdk/
174 27 Laurent GUERBY
https://github.com/gandi/packet-journey
175 27 Laurent GUERBY
176 27 Laurent GUERBY
h2. Atelier 2016
177 27 Laurent GUERBY
178 55 Laurent GUERBY
h3. Atelier Hardware
179 54 Laurent GUERBY
180 1 Laurent GUERBY
<pre>
181 28 Laurent GUERBY
core i7 2600 @ 3.40GHz
182 31 Laurent GUERBY
 fpu vme de pse tsc msr pae mce cx8 
183 31 Laurent GUERBY
 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 
184 31 Laurent GUERBY
 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl 
185 31 Laurent GUERBY
 xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 
186 31 Laurent GUERBY
 cx16 xtpr pdcm pcid sse4_1 sse4_2 
187 31 Laurent GUERBY
 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
188 28 Laurent GUERBY
16G RAM
189 28 Laurent GUERBY
SSD 250G
190 28 Laurent GUERBY
</pre>
191 28 Laurent GUERBY
192 28 Laurent GUERBY
<pre>
193 28 Laurent GUERBY
host eth DHCP          10G-a                                           10G-b
194 28 Laurent GUERBY
t1   54:04:a6:ef:a8:c7 00:1b:21:d9:28:55 fe80::21b:21ff:fed9:2855(t2b) 00:1b:21:d9:28:54 fe80::21b:21ff:fed9:2854(t3b)
195 28 Laurent GUERBY
t2   e0:69:95:2e:e6:a2 00:1b:21:d9:1a:e1 fe80::21b:21ff:fed9:1ae1(t3a) 00:1b:21:d9:1a:e0 fe80::21b:21ff:fed9:1ae0(t1a)
196 28 Laurent GUERBY
t3   bc:ae:c5:e1:b2:6d 00:1b:21:d9:df:3d fe80::21b:21ff:fed9:df3d(t2a) 00:1b:21:d9:df:3c fe80::21b:21ff:fed9:df3c(t1b)
197 56 Laurent GUERBY
198 56 Laurent GUERBY
1G e1000e
199 56 Laurent GUERBY
t1 00:15:17:a6:cb:19
200 56 Laurent GUERBY
t3 00:15:17:16:bd:de
201 27 Laurent GUERBY
</pre>
202 29 Laurent GUERBY
203 60 Laurent GUERBY
<pre>
204 63 Laurent GUERBY
h7   82599ES 8086:10fb 8086:7a11 
205 63 Laurent GUERBY
g1   82599ES (VM maposmatic)
206 60 Laurent GUERBY
g2   X520    8086:154d 8086:7b11
207 60 Laurent GUERBY
g3   82599ES
208 60 Laurent GUERBY
g4   X520 
209 60 Laurent GUERBY
g5   X520
210 60 Laurent GUERBY
g6   X520
211 63 Laurent GUERBY
n7   X520 (VM openstack)
212 60 Laurent GUERBY
g8   X520
213 60 Laurent GUERBY
g9   82598EB 8086:10f1 8086:a21f
214 60 Laurent GUERBY
g10  X520
215 60 Laurent GUERBY
g11  82598EB
216 60 Laurent GUERBY
g12  82598EB
217 60 Laurent GUERBY
stri X520
218 60 Laurent GUERBY
219 60 Laurent GUERBY
t1   82598EB 
220 60 Laurent GUERBY
t2   82598EB 
221 60 Laurent GUERBY
t3   82598EB 
222 60 Laurent GUERBY
223 60 Laurent GUERBY
stock 82598EB
224 60 Laurent GUERBY
stock 82598EB
225 60 Laurent GUERBY
stock X520
226 60 Laurent GUERBY
</pre>
227 60 Laurent GUERBY
228 55 Laurent GUERBY
h3. Atelier Liens
229 54 Laurent GUERBY
230 33 Laurent GUERBY
https://github.com/Gandi/packet-journey
231 30 Laurent GUERBY
http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html
232 40 Laurent GUERBY
http://dpdk.org/doc/quick-start
233 50 Laurent GUERBY
http://dpdk.org/browse/apps/pktgen-dpdk/tree/README.md
234 1 Laurent GUERBY
http://dpdk.org/doc/guides/testpmd_app_ug/index.html
235 47 Laurent GUERBY
http://dpdk.readthedocs.org/en/v2.2.0/
236 54 Laurent GUERBY
https://github.com/emmericp/MoonGen
237 29 Laurent GUERBY
http://www.slideshare.net/movilforum/7-hands-on
238 33 Laurent GUERBY
http://p4.org/wp-content/uploads/2015/11/Princeton-Demo-Poster.pdf
239 59 Laurent GUERBY
http://dpdk.org/ml/archives/users/
240 59 Laurent GUERBY
http://dpdk.org/ml/archives/dev/
241 59 Laurent GUERBY
http://openvswitch.org/support/ovscon2015/
242 59 Laurent GUERBY
243 33 Laurent GUERBY
244 54 Laurent GUERBY
h3. Atelier Issues
245 1 Laurent GUERBY
246 1 Laurent GUERBY
* Issues opened
247 33 Laurent GUERBY
** https://github.com/Gandi/packet-journey/issues/20 
248 1 Laurent GUERBY
** https://github.com/pktgen/Pktgen-DPDK/issues/52
249 1 Laurent GUERBY
** https://github.com/emmericp/MoonGen/issues/106
250 58 Laurent GUERBY
** http://dpdk.org/ml/archives/dev/2015-December/030798.html
251 1 Laurent GUERBY
252 54 Laurent GUERBY
h3. Atelier Build
253 54 Laurent GUERBY
254 1 Laurent GUERBY
<pre>
255 54 Laurent GUERBY
apt-get build-dep linux-image-3.16.0-30-generic 
256 57 Laurent GUERBY
apt-get install linux-tools-3.16.0-57-generic
257 54 Laurent GUERBY
apt-get install emacs24-nox libpcap-dev tcpdump iperf iperf3 screen rsync socat bzip2 xz-utils cmake python-sphinx
258 54 Laurent GUERBY
</pre>
259 54 Laurent GUERBY
260 54 Laurent GUERBY
<pre>
261 54 Laurent GUERBY
# Dans /etc/rc.local par exemple
262 1 Laurent GUERBY
modprobe uio_pci_generic
263 1 Laurent GUERBY
mkdir -p /mnt/huge
264 29 Laurent GUERBY
mount -t hugetlbfs nodev /mnt/huge
265 1 Laurent GUERBY
# si pas GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=4" :
266 1 Laurent GUERBY
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
267 49 Laurent GUERBY
268 54 Laurent GUERBY
# Sources
269 54 Laurent GUERBY
270 1 Laurent GUERBY
mkdir ftp
271 1 Laurent GUERBY
cd ftp
272 1 Laurent GUERBY
wget http://dpdk.org/browse/dpdk/snapshot/dpdk-2.2.0.tar.gz
273 1 Laurent GUERBY
cd ..
274 1 Laurent GUERBY
tar xf ftp/dpdk-2.2.0.tar.gz
275 61 Laurent GUERBY
git clone https://github.com/Gandi/packet-journey.git
276 61 Laurent GUERBY
git clone http://dpdk.org/git/apps/pktgen-dpdk
277 61 Laurent GUERBY
git clone http://dpdk.org/git/dpdk-stable
278 61 Laurent GUERBY
git clone https://github.com/emmericp/MoonGen.git
279 61 Laurent GUERBY
280 52 Laurent GUERBY
281 54 Laurent GUERBY
# Build
282 54 Laurent GUERBY
283 30 Laurent GUERBY
cd dpdk-2.2.0
284 32 Laurent GUERBY
make config T=x86_64-native-linuxapp-gcc
285 1 Laurent GUERBY
make install T=x86_64-native-linuxapp-gcc
286 32 Laurent GUERBY
export RTE_SDK=/root/dpdk-2.2.0
287 38 Laurent GUERBY
export RTE_TARGET=x86_64-native-linuxapp-gcc
288 38 Laurent GUERBY
./tools/dpdk_nic_bind.py -s
289 38 Laurent GUERBY
./tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.0
290 42 Laurent GUERBY
./tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.1
291 37 Laurent GUERBY
./x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -c 0x6 -n 2 -- -i
292 1 Laurent GUERBY
293 33 Laurent GUERBY
cd ../packet-journey
294 37 Laurent GUERBY
make
295 1 Laurent GUERBY
296 33 Laurent GUERBY
cd ../pktgen-dpdk
297 41 Laurent GUERBY
make
298 41 Laurent GUERBY
cd docs
299 41 Laurent GUERBY
make html
300 29 Laurent GUERBY
</pre>