Linux Network Performance
RAW ethernet vs. UDP
Abstract
This document describes how to do RAW ethernet programming on Linux and gives a comparison of Linux' networking performance on UDP and ethernet layer.
A visitor of my website has pointed out to me that there is a mistake in the performance measuring code. This means the figures presented are very likely wrong. Currently I do not have the time to rerun the tests. The value of this article is therefore reduced to that it is a tutorial for RAW Ethernet programming on Linux but nothing more.
List of Examples
Chapter 1. RAW ethernet programming
The first chapter gives a short overview of how to establish a RAW ethernet communication between two Linux hosts.
IP networking is using four layers:
Normaly applications are placed above the TCP/UDP layer. Thus, communication between two programs is going two times through all the layers. On the sender side from TCP/ UDP down to the physical layer, on the receiver side from the physical layer up to the TCP/ UDP layer. The actual data transmission is carried out by the physical layer.
However, it is also possible for two programs to communicate on the ethernet layer. In this case the IP and the TCP/UDP layer are not used and as a result an increased data transmission rate is to be expected. It is one purpose of this document to show exact performance figures.
First of all let us take a look at UDP. It is necessary to create a socket. This socket can later be used to send or receive a UDP message.
Example 1.1. Create a UDP socket
#include<string.h>
#include<sys/socket.h>
#include<netinet/in.h>
int create_udp_socket(int port) {
/*socketdescriptor*/
int s;
/*struct used for binding the socket to a local address*/
struct sockaddr_in host_address;
/*create the socket*/
s=socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP);
if (s < 0) { /*errorhandling ....*/}
/*init the host_address the socket is beeing bound to*/
memset((void*)&host_address, 0, sizeof(host_address));
/*set address family*/
host_address.sin_family=PF_INET;
/*accept any incoming messages:*/
host_address.sin_addr.s_addr=INADDR_ANY;
/*the port the socket i to be bound to:*/
host_address.sin_port=htons(port);
/*bind it...*/
if (
bind(s, (struct sockaddr*)&host_address, sizeof(host_address)) < 0
) {
/*errorhandling...*/
}
return s;
}
Example 1.2. Use a UDP socket to send a message
#define BUF_SIZE 1000
#define LOCAL_PORT 5000
#define REMOTE_PORT 5000
...
int s; /*the socket descriptor*/
char buffer[BUF_SIZE]; /*the message to send*/
struct sockaddr_in target_host_address; /*the receiver's address*/
unsigned char* target_address_holder; /*a pointer to the ip address*/
...
/*create the socket*/
s = create_udp_socket(LOCAL_PORT);
if (s == -1) {errorhandling.....}
/*init target address structure*/
target_host_address.sin_family=PF_INET;
target_host_address.sin_port=htons(REMOTE_PORT);
target_address_holder=(unsigned char*)&target_host_address.sin_addr.s_addr;
target_address_holder[0]=10;
target_address_holder[1]=0;
target_address_holder[2]=0;
target_address_holder[3]=1;
/*fill message with random data....*/
for (j = 0; j < BUF_SIZE; j++) {
buffer[j] = (unsigned char)((int) (255.0*rand()/(RAND_MAX+1.0)));
}
/*send it*/
sendto(s, buffer, BUF_SIZE, 0,
(struct sockaddr*)&target_host_address, sizeof(struct sockaddr));
Example 1.3. Use a UDP socket to receive a message
#define BUF_SIZE 1000
#define LOCAL_PORT 5000
...
char buffer[BUF_SIZE];
/*address of the sender will be stored here*/
struct sockaddr_in host_address;
int hst_addr_size = sizeof(host_address);
/*length of the incoming packet*/
int length = 0;
/*socketdescriptor*/
int s;
...
/*create the socket*/
s = create_udp_socket(LOCAL_PORT);
if (s == -1) {errorhandling.....}
/*wait for incoming message*/
length = recvfrom(s, buffer, BUF_SIZE, 0,
(struct sockaddr*)&host_address, &hst_addr_size);
Now let us look at RAW ethernet communication. From the programmers' point of view it is quite similar to UDP. The differences are:
-
the parameters for the function used to create a socket
-
instead of IP addresses MAC addresses are used
-
an ethernet frame needs to be created manually
ethernet Frame Spec (IEEE 802.3)
Example 1.4. Create a RAW ethernet socket
#include <sys/socket.h>
#include <linux/if_packet.h>
#include <linux/if_ether.h>
#include <linux/if_arp.h>
...
int s; /*socketdescriptor*/
s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL));
if (s == -1) { errorhandling ... }
Example 1.5. Send a RAW ethernet frame
#define ETH_FRAME_LEN 1518
...
/*target address*/
struct sockaddr_ll socket_address;
/*buffer for ethernet frame*/
void* buffer = (void*)malloc(ETH_FRAME_LEN);
/*pointer to ethenet header*/
unsigned char* etherhead = buffer;
/*userdata in ethernet frame*/
unsigned char* data = buffer + 14;
/*another pointer to ethernet header*/
struct ethhdr *eh = (struct ethhdr *)etherhead;
int send_result = 0;
/*our MAC address*/
unsigned char src_mac[6] = {0x00, 0x01, 0x02, 0xFA, 0x70, 0xAA};
/*other host MAC address*/
unsigned char dest_mac[6] = {0x00, 0x04, 0x75, 0xC8, 0x28, 0xE5};
/*prepare sockaddr_ll*/
/*RAW communication*/
socket_address.sll_family = PF_PACKET;
/*we don't use a protocoll above ethernet layer
->just use anything here*/
socket_address.sll_protocol = htons(ETH_P_IP);
/*index of the network device
see full code later how to retrieve it*/
socket_address.sll_ifindex = 2;
/*ARP hardware identifier is ethernet*/
socket_address.sll_hatype = ARPHRD_ETHER;
/*target is another host*/
socket_address.sll_pkttype = PACKET_OTHERHOST;
/*address length*/
socket_address.sll_halen = ETH_ALEN;
/*MAC - begin*/
socket_address.sll_addr[0] = 0x00;
socket_address.sll_addr[1] = 0x04;
socket_address.sll_addr[2] = 0x75;
socket_address.sll_addr[3] = 0xC8;
socket_address.sll_addr[4] = 0x28;
socket_address.sll_addr[5] = 0xE5;
/*MAC - end*/
socket_address.sll_addr[6] = 0x00;/*not used*/
socket_address.sll_addr[7] = 0x00;/*not used*/
/*set the frame header*/
memcpy((void*)buffer, (void*)dest_mac, ETH_ALEN);
memcpy((void*)(buffer+ETH_ALEN), (void*)src_mac, ETH_ALEN);
eh->h_proto = 0x00;
/*fill the frame with some data*/
for (j = 0; j < 1500; j++) {
data[j] = (unsigned char)((int) (255.0*rand()/(RAND_MAX+1.0)));
}
/*send the packet*/
send_result = sendto(s, buffer, ETH_FRAME_LEN, 0,
(struct sockaddr*)&socket_address, sizeof(socket_address));
if (send_result == -1) { errorhandling... }
Example 1.6. Receive a RAW ethernet frame
void* buffer = (void*)malloc(ETH_FRAME_LEN); /*Buffer for ethernet frame*/
int length = 0; /*length of the received frame*/
...
length = recvfrom(s, buffer, ETH_FRAME_LEN, 0, NULL, NULL);
if (length == -1) { errorhandling .... }
Chapter 2. Testing Environment
In this chapter the testing environment is described.
The following hardware equipment was used:
-
two hosts were communicating with each other
-
amount of user data in the packets was varying from 50 octets up to 1500 octets. (step: 50 octets)
-
each measurement was carried out for UDP and RAW ethernet
-
the value taken as result is the average of 100.000 measurements for one roundtrip (back and forth both hosts, as shown below)
Chapter 3. Testing Results
This chapter shows the testing results.
Table 3.1. Results
* |
Host3 / Host 4 (SMC/ 3Com) |
Host1 / Host 4 (Intel/ 3Com) |
Host2 / Host 4 (3Com/ 3Com) |
#octets |
RAW (us) |
UDP (us) |
RAW (us) |
UDP (us) |
RAW (us) |
UDP (us) |
|
|
|
|
|
|
|
50 |
29 |
227 |
116 |
100 |
14 |
208 |
100 |
19 |
290 |
104 |
119 |
13 |
227 |
150 |
13 |
296 |
124 |
137 |
14 |
232 |
200 |
21 |
360 |
142 |
154 |
26 |
285 |
250 |
22 |
361 |
159 |
173 |
27 |
300 |
300 |
22 |
418 |
177 |
192 |
27 |
306 |
350 |
20 |
442 |
196 |
209 |
28 |
355 |
400 |
18 |
478 |
214 |
226 |
30 |
366 |
450 |
23 |
511 |
232 |
244 |
30 |
371 |
500 |
17 |
523 |
251 |
263 |
31 |
384 |
550 |
19 |
590 |
116 |
263 |
32 |
426 |
600 |
18 |
601 |
124 |
330 |
33 |
447 |
650 |
18 |
651 |
50 |
317 |
34 |
453 |
700 |
17 |
682 |
10 |
335 |
35 |
467 |
750 |
17 |
708 |
10 |
394 |
35 |
505 |
800 |
17 |
751 |
10 |
372 |
36 |
520 |
850 |
17 |
765 |
10 |
388 |
36 |
524 |
900 |
17 |
816 |
10 |
406 |
37 |
557 |
950 |
17 |
823 |
10 |
424 |
38 |
593 |
1000 |
17 |
874 |
10 |
443 |
38 |
595 |
1050 |
17 |
906 |
10 |
461 |
39 |
611 |
1100 |
18 |
901 |
10 |
478 |
40 |
641 |
1150 |
21 |
915 |
11 |
496 |
41 |
664 |
1200 |
21 |
952 |
11 |
558 |
41 |
671 |
1250 |
21 |
965 |
11 |
532 |
42 |
697 |
1300 |
19 |
973 |
11 |
550 |
43 |
742 |
1350 |
17 |
976 |
11 |
568 |
44 |
739 |
1400 |
17 |
1000 |
11 |
586 |
44 |
745 |
1450 |
22 |
1142 |
12 |
605 |
45 |
781 |
1500 |
19 |
1155 |
12 |
633 |
46 |
767 |
.
Comparison of UDP results
The following graph is showing the results of the UDP measurements for each of the different hardware configurations.
These results are not surprising. The more user data a packet contains the more time is taken for one roundtrip (linear growing). It is also possible to see that the performance is different for different hardware configurations.
Comparison of RAW results
The following graph is showing the results of the RAW ethernet measurements for each of the different hardware configurations.
Looking at SMC/ 3com and 3com/ 3com performance the results are as expected. The time required for one roundtrip is increasing almost linearly, when the amount of user data is increasing. However, the results which were produced by the Intel NIC were unexpected. Although the Intel NIC was very slow when it came to sending packets containing 50 up tp 600 octets of user data, it was outperforming the other two, when the amount of data in a packet is increasing. A possible explanation might be that the driver or the hardware is somehow optimized for larger packages. However this behaviour is not reflected in the UDP comparison graph above.
Comparison of UDP and RAW ethernet results
Finally both graphs are merged into one diagram, in order to compare the performance of UDP and RAW ethernet.
The results show that RAW ethernet in general is much faster than UDP. Looking at the features that are offered by UDP and RAW ethernet it becomes visible that those two are quite similiar. Both are not connection oriented, both are packet based and both do not give any guarantee for success of transmission. Thus, in the view of real world systems which are currently using UDP running over ethernet or systems which require very high network performance, RAW ethernet could offer a very good solution.