Projet

Général

Profil

DPDK » Historique » Version 38

Laurent GUERBY, 23/12/2015 21:45

1 1 Laurent GUERBY
{{>toc}}
2 1 Laurent GUERBY
3 1 Laurent GUERBY
h1. DPDK
4 1 Laurent GUERBY
5 1 Laurent GUERBY
Traitement haute performance en logiciel, 10 Gbit/s en petit paquets line rate
6 1 Laurent GUERBY
7 1 Laurent GUERBY
h2. Liens
8 1 Laurent GUERBY
9 1 Laurent GUERBY
* http://dpdk.org
10 1 Laurent GUERBY
* [[AtelierPPS2012]]
11 16 Laurent GUERBY
* GIT http://chiliproject.tetaneutral.net/projects/git-tetaneutral-net/repository/dpdk
12 16 Laurent GUERBY
<pre>
13 16 Laurent GUERBY
$ git clone ssh://git@git.tetaneutral.net/dpdk.git
14 16 Laurent GUERBY
</pre>
15 16 Laurent GUERBY
16 17 Laurent GUERBY
* Nerim Antoine https://twitter.com/nerim_net/status/517592380841930752
17 17 Laurent GUERBY
** Du code maison fondé sur #DPDK en prod dans le réseau : première étape vers l'élimination des systèmes propriétaires ?
18 2 Laurent GUERBY
19 2 Laurent GUERBY
h2. Misc
20 2 Laurent GUERBY
21 26 Laurent GUERBY
* https://www.kernel.org/doc/Documentation/networking/switchdev.txt
22 25 Laurent GUERBY
* https://www.usenix.org/conference/osdi14/technical-sessions/presentation/belay
23 25 Laurent GUERBY
** IX: A Protected Dataplane Operating System for High Throughput and Low Latency
24 25 Laurent GUERBY
** http://www.linuxfoundation.org/images/1/1c/Network_data_flow_through_kernel.png
25 25 Laurent GUERBY
** http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow
26 23 Laurent GUERBY
* https://blog.cloudflare.com/how-to-receive-a-million-packets/
27 23 Laurent GUERBY
** How to receive a million packets per second
28 24 Laurent GUERBY
* https://blog.cloudflare.com/how-to-achieve-low-latency/
29 21 Laurent GUERBY
* http://www.cs.cmu.edu/~dongz/papers/cuckooswitch.pdf
30 22 Laurent GUERBY
** We show that CUCKOOSWITCH can process 92.22 million minimum sized packets per second on a commodity server equipped with eight 10 Gbps Ethernet interfaces while maintaining a forwarding table of one billion forwarding entries.
31 18 Laurent GUERBY
* Improving Linux networking performance https://lwn.net/Articles/629155/
32 18 Laurent GUERBY
** https://www.youtube.com/watch?v=3XG9-X777Jo
33 19 Laurent GUERBY
** http://people.netfilter.org/hawk/presentations/LCA2015/net_stack_challenges_100G_LCA2015.pdf
34 20 Laurent GUERBY
* http://www.bigswitch.com/sites/default/files/presentations/onug-baremetal-2014-final.pdf
35 2 Laurent GUERBY
* Intel PCI id http://www.intel.com/support/network/sb/cs-012904.htm
36 3 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-products.html?iid=embed_portal+hdprod_ethernet#s1=Gigabit%20Ethernet&s2=all&s3=all
37 3 Laurent GUERBY
* Intel Pro/1000 PT vs CT http://hardforum.com/showthread.php?t=1386042
38 4 Laurent GUERBY
* http://images.ncix.com/forumimages/C9864B87-583B-10FB-B0DE80742781CE07.jpg
39 4 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82572EI&s3=all
40 4 Laurent GUERBY
* http://www.intel.com/content/www/us/en/ethernet-controllers/ethernet-controllers.html#s1=all&s2=82574L&s3=all
41 5 Laurent GUERBY
* https://github.com/Pktgen/Pktgen-DPDK
42 6 Laurent GUERBY
* Evaluating the Suitability of Server Network Cards for Software Routers http://kfall.net/ucbpage/papers/snc.pdf
43 13 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/guides/intel-dpdk-sample-applications-user-guide.pdf
44 13 Laurent GUERBY
*  rte_eth_rx_burst
45 13 Laurent GUERBY
** http://dpdk.org/doc/api/rte__ethdev_8h.html#a5ac93faa0f3f69545ca121dcfd9433f5
46 13 Laurent GUERBY
** http://dpdk.info/ml/archives/dev/2014-January/001223.html
47 14 Laurent GUERBY
* http://www.intel.fr/content/dam/www/public/us/en/documents/white-papers/aes-ipsec-performance-linux-paper.pdf
48 15 Laurent GUERBY
* http://networkbuilders.intel.com/docs/Network_Builders_RA_vBRAS_Final.pdf
49 15 Laurent GUERBY
50 15 Laurent GUERBY
* https://github.com/SnabbCo/snabbswitch
51 15 Laurent GUERBY
** https://github.com/SnabbCo/snabbswitch/blob/master/src/apps/intel/intel10g.lua
52 15 Laurent GUERBY
** http://blog.ipspace.net/2014/06/snabb-switch-and-nfv-on-openstack-in.html
53 15 Laurent GUERBY
** https://github.com/SnabbCo/snabbswitch/wiki/Snabb-Lab
54 15 Laurent GUERBY
** https://twitter.com/lukego/status/327714050219847680
55 7 Mehdi Abaakouk
56 7 Mehdi Abaakouk
h2. Note sileht dpdk:
57 7 Mehdi Abaakouk
58 7 Mehdi Abaakouk
Extract from: http://www.intel.com/content/dam/www/public/us/en/documents/guides/intel-dpdk-getting-started-guide.pdf
59 7 Mehdi Abaakouk
60 7 Mehdi Abaakouk
h3. configuration hugepages: 
61 7 Mehdi Abaakouk
62 7 Mehdi Abaakouk
* 2M (1024*2k): hugepages=1024 (at runtime: echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
63 7 Mehdi Abaakouk
* 4G (4x1G): default_hugepagesz=1G hugepagesz=1G hugepages=4 (only works at boot time via grub)
64 7 Mehdi Abaakouk
65 7 Mehdi Abaakouk
<pre>
66 7 Mehdi Abaakouk
mkdir /mnt/huge
67 7 Mehdi Abaakouk
mount -t hugetlbfs nodev /mnt/huge
68 7 Mehdi Abaakouk
</pre>
69 7 Mehdi Abaakouk
70 7 Mehdi Abaakouk
h3. Compile and load modules: 
71 7 Mehdi Abaakouk
72 7 Mehdi Abaakouk
_Note: source tools/setup.sh is a helper tools for this but works only with IGB driver not e1000e_
73 7 Mehdi Abaakouk
74 1 Laurent GUERBY
<pre>
75 8 Mehdi Abaakouk
# apt-get install make gcc linux-headers-amd64
76 8 Mehdi Abaakouk
# make config T=x86_64-default-linuxapp-gcc
77 7 Mehdi Abaakouk
# make T=x86_64-default-linuxapp-gcc
78 7 Mehdi Abaakouk
..
79 7 Mehdi Abaakouk
Build complete
80 7 Mehdi Abaakouk
81 7 Mehdi Abaakouk
# modprobe uio  (I think this is not useful)
82 7 Mehdi Abaakouk
# insmod build/kmod/rte_kni.ko (I think this is not useful)
83 7 Mehdi Abaakouk
# insmod build/kmod/igb_uio.ko (I think this is not useful)
84 7 Mehdi Abaakouk
# ./tools/pci_unbind.py --status
85 7 Mehdi Abaakouk
86 7 Mehdi Abaakouk
Network devices using IGB_UIO driver
87 7 Mehdi Abaakouk
====================================
88 7 Mehdi Abaakouk
<none>
89 7 Mehdi Abaakouk
90 7 Mehdi Abaakouk
Network devices using kernel driver
91 7 Mehdi Abaakouk
===================================
92 9 Mehdi Abaakouk
0000:00:19.0 '82579LM Gigabit Network Connection' if=eth1 drv=e1000e unused=<none>
93 9 Mehdi Abaakouk
0000:01:00.0 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth6 drv=ixgbe unused=igb_uio *Active*
94 9 Mehdi Abaakouk
0000:01:00.1 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth7 drv=ixgbe unused=igb_uio
95 9 Mehdi Abaakouk
0000:04:00.0 '82571EB Gigabit Ethernet Controller' if=eth2 drv=e1000e unused=igb_uio *Active*
96 9 Mehdi Abaakouk
0000:04:00.1 '82571EB Gigabit Ethernet Controller' if=eth3 drv=e1000e unused=igb_uio
97 9 Mehdi Abaakouk
0000:05:00.0 '82571EB Gigabit Ethernet Controller' if=eth4 drv=e1000e unused=igb_uio
98 9 Mehdi Abaakouk
0000:05:00.1 '82571EB Gigabit Ethernet Controller' if=eth5 drv=e1000e unused=igb_uio
99 9 Mehdi Abaakouk
0000:06:00.0 'RTL8111/8168B PCI Express Gigabit Ethernet controller' if=eth0 drv=r8169 unused=<none> *Active*
100 1 Laurent GUERBY
101 1 Laurent GUERBY
Other network devices
102 1 Laurent GUERBY
=====================
103 1 Laurent GUERBY
<none>
104 9 Mehdi Abaakouk
</pre>
105 7 Mehdi Abaakouk
106 9 Mehdi Abaakouk
test sur eth7:
107 9 Mehdi Abaakouk
108 9 Mehdi Abaakouk
<pre>
109 9 Mehdi Abaakouk
# ip link set eth7 down
110 9 Mehdi Abaakouk
# ./tools/pci_unbind.py --bind=igb_uio 0000:01:00.1
111 7 Mehdi Abaakouk
</pre>
112 7 Mehdi Abaakouk
113 7 Mehdi Abaakouk
h3. Prepare examples programs:
114 7 Mehdi Abaakouk
115 7 Mehdi Abaakouk
<pre>
116 7 Mehdi Abaakouk
# export RTE_SDK=/root/sileht/dpdk-1.6.0r1
117 7 Mehdi Abaakouk
# export RTE_TARGET=build
118 7 Mehdi Abaakouk
# cd /root/sileht/
119 7 Mehdi Abaakouk
# cp -r $RTE_SDK/examples/helloworld my_rte_app
120 7 Mehdi Abaakouk
# cd my_rte_app
121 7 Mehdi Abaakouk
# make
122 7 Mehdi Abaakouk
</pre>
123 7 Mehdi Abaakouk
124 7 Mehdi Abaakouk
125 7 Mehdi Abaakouk
h3. Tests
126 7 Mehdi Abaakouk
127 7 Mehdi Abaakouk
<pre>
128 1 Laurent GUERBY
</pre>
129 10 Mehdi Abaakouk
130 11 Mehdi Abaakouk
h2. Limitation SPF carte intel:
131 10 Mehdi Abaakouk
132 11 Mehdi Abaakouk
Une fois le module igb_uio changé sur les:
133 10 Mehdi Abaakouk
134 10 Mehdi Abaakouk
<pre>
135 10 Mehdi Abaakouk
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
136 10 Mehdi Abaakouk
</pre>
137 10 Mehdi Abaakouk
138 11 Mehdi Abaakouk
Un flags de l'eeprom change et le module ixgbe ne charge plus les ports de la carte qui n'ont pas des SFP+ intel avec le message d'erreur suivant:
139 10 Mehdi Abaakouk
140 10 Mehdi Abaakouk
<pre>
141 10 Mehdi Abaakouk
[  563.069683] ixgbe 0000:01:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
142 10 Mehdi Abaakouk
[  563.069786] ixgbe 0000:01:00.0: Reload the driver after installing a supported module.
143 10 Mehdi Abaakouk
</pre>
144 10 Mehdi Abaakouk
145 10 Mehdi Abaakouk
Pour contourner le problème:
146 10 Mehdi Abaakouk
147 10 Mehdi Abaakouk
<pre>
148 10 Mehdi Abaakouk
modprobe ixgbe allow_unsupported_sfp=1 # allow_any_sfp=1 suivant le kernel
149 10 Mehdi Abaakouk
</pre>
150 10 Mehdi Abaakouk
151 10 Mehdi Abaakouk
ou 
152 10 Mehdi Abaakouk
153 1 Laurent GUERBY
<pre>
154 1 Laurent GUERBY
echo "options ixgbe allow_unsupported_sfp=1" > /etc/modprobe.d/ixgbe_any_sfp.conf
155 1 Laurent GUERBY
update-initramfs -u -k all
156 11 Mehdi Abaakouk
</pre>
157 11 Mehdi Abaakouk
158 11 Mehdi Abaakouk
La carte fonctionnera à nouveau avec le message suivant:
159 11 Mehdi Abaakouk
160 11 Mehdi Abaakouk
<pre>
161 11 Mehdi Abaakouk
[  908.361767] ixgbe 0000:01:00.0 (unregistered net_device): WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics.  Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter.  Intel Corporation is not responsible for any harm caused by using untested modules.
162 10 Mehdi Abaakouk
</pre>
163 12 Laurent GUERBY
164 27 Laurent GUERBY
h2. Packet Journey
165 1 Laurent GUERBY
166 27 Laurent GUERBY
https://www.gandi.net/news/en/2015-12-16/6308-packet-journey_a_free_software_router_for_linux_based_on_dpdk/
167 27 Laurent GUERBY
https://github.com/gandi/packet-journey
168 27 Laurent GUERBY
169 27 Laurent GUERBY
h2. Atelier 2016
170 27 Laurent GUERBY
171 1 Laurent GUERBY
<pre>
172 28 Laurent GUERBY
core i7 2600 @ 3.40GHz
173 31 Laurent GUERBY
 fpu vme de pse tsc msr pae mce cx8 
174 31 Laurent GUERBY
 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 
175 31 Laurent GUERBY
 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl 
176 31 Laurent GUERBY
 xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 
177 31 Laurent GUERBY
 cx16 xtpr pdcm pcid sse4_1 sse4_2 
178 31 Laurent GUERBY
 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
179 28 Laurent GUERBY
16G RAM
180 28 Laurent GUERBY
SSD 250G
181 28 Laurent GUERBY
</pre>
182 28 Laurent GUERBY
183 28 Laurent GUERBY
<pre>
184 28 Laurent GUERBY
host eth DHCP          10G-a                                           10G-b
185 28 Laurent GUERBY
t1   54:04:a6:ef:a8:c7 00:1b:21:d9:28:55 fe80::21b:21ff:fed9:2855(t2b) 00:1b:21:d9:28:54 fe80::21b:21ff:fed9:2854(t3b)
186 28 Laurent GUERBY
t2   e0:69:95:2e:e6:a2 00:1b:21:d9:1a:e1 fe80::21b:21ff:fed9:1ae1(t3a) 00:1b:21:d9:1a:e0 fe80::21b:21ff:fed9:1ae0(t1a)
187 28 Laurent GUERBY
t3   bc:ae:c5:e1:b2:6d 00:1b:21:d9:df:3d fe80::21b:21ff:fed9:df3d(t2a) 00:1b:21:d9:df:3c fe80::21b:21ff:fed9:df3c(t1b)
188 27 Laurent GUERBY
</pre>
189 29 Laurent GUERBY
190 33 Laurent GUERBY
https://github.com/Gandi/packet-journey
191 30 Laurent GUERBY
http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html
192 34 Laurent GUERBY
http://dpdk.org/browse/apps/pktgen-dpdk/tree/README.md
193 36 Laurent GUERBY
http://dpdk.org/doc/quick-start
194 30 Laurent GUERBY
195 29 Laurent GUERBY
<pre>
196 33 Laurent GUERBY
apt-get build-dep linux-image-3.16.0-30-generic
197 33 Laurent GUERBY
apt-get install emacs24-nox libpcap-dev tcpdump iperf iperf3 screen rsync socat bzip2 xz-utils
198 33 Laurent GUERBY
</pre>
199 33 Laurent GUERBY
200 33 Laurent GUERBY
Patch :
201 33 Laurent GUERBY
https://github.com/Gandi/packet-journey/issues/20
202 33 Laurent GUERBY
203 33 Laurent GUERBY
<pre>
204 29 Laurent GUERBY
modprobe uio_pci_generic
205 37 Laurent GUERBY
mkdir -p /mnt/huge
206 37 Laurent GUERBY
mount -t hugetlbfs nodev /mnt/huge
207 37 Laurent GUERBY
echo 256 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
208 37 Laurent GUERBY
209 29 Laurent GUERBY
git clone https://github.com/Gandi/packet-journey.git
210 1 Laurent GUERBY
git clone http://dpdk.org/git/apps/pktgen-dpdk
211 29 Laurent GUERBY
git clone http://dpdk.org/git/dpdk-stable
212 37 Laurent GUERBY
wget http://dpdk.org/browse/dpdk/snapshot/dpdk-2.2.0.tar.gz
213 30 Laurent GUERBY
tar xf dpdk-2.2.0.tar.gz
214 37 Laurent GUERBY
215 30 Laurent GUERBY
cd dpdk-2.2.0
216 30 Laurent GUERBY
make config T=x86_64-native-linuxapp-gcc
217 32 Laurent GUERBY
make install T=x86_64-native-linuxapp-gcc
218 1 Laurent GUERBY
export RTE_SDK=/root/dpdk-2.2.0
219 32 Laurent GUERBY
export RTE_TARGET=x86_64-native-linuxapp-gcc
220 38 Laurent GUERBY
./tools/dpdk_nic_bind.py -s
221 38 Laurent GUERBY
./tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.0
222 38 Laurent GUERBY
./tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.1
223 38 Laurent GUERBY
./x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -c 0xff -n 2
224 37 Laurent GUERBY
225 1 Laurent GUERBY
cd ../packet-journey
226 33 Laurent GUERBY
make
227 37 Laurent GUERBY
228 1 Laurent GUERBY
cd ../pktgen-dpdk
229 33 Laurent GUERBY
make
230 37 Laurent GUERBY
231 29 Laurent GUERBY
</pre>