nagai/kernel
3
0
Fork 0
forked from mirrors/linux
Code Pull requests Activity

udp: add batching to udp_rmem_release()

Browse source 
If udp_recvmsg() constantly releases sk_rmem_alloc
for every read packet, it gives opportunity for
producers to immediately grab spinlocks and desperatly
try adding another packet, causing false sharing.

We can add a simple heuristic to give the signal
by batches of ~25 % of the queue capacity.

This patch considerably increases performance under
flood by about 50 %, since the thread draining the queue
is no longer slowed by false sharing.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
This commit is contained in:
Eric Dumazet 2016-12-08 11:41:56 -08:00 committed by David S. Miller
parent c84d949057
commit 6b229cf77d
2 changed files with 15 additions and 0 deletions
repo.diff.show_diff_stats Download patch file Download diff file Expand all files Collapse all files

3
include/linux/udp.h
View file

@ -79,6 +79,9 @@ struct udp_sock {
int (*gro_complete)(struct sock *sk, int (*gro_complete)(struct sock *sk,
struct sk_buff *skb, struct sk_buff *skb,
int nhoff); int nhoff);
/* This field is dirtied by udp_recvmsg() */
int forward_deficit;
}; };
static inline struct udp_sock *udp_sk(const struct sock *sk) static inline struct udp_sock *udp_sk(const struct sock *sk)

12
net/ipv4/udp.c
View file

@ -1177,8 +1177,20 @@ int udp_sendpage(struct sock *sk, struct page *page, int offset,
/* fully reclaim rmem/fwd memory allocated for skb */ /* fully reclaim rmem/fwd memory allocated for skb */
static void udp_rmem_release(struct sock *sk, int size, int partial) static void udp_rmem_release(struct sock *sk, int size, int partial)
{ {
struct udp_sock *up = udp_sk(sk);
int amt; int amt;
if (likely(partial)) {
up->forward_deficit += size;
size = up->forward_deficit;
if (size < (sk->sk_rcvbuf >> 2) &&
!skb_queue_empty(&sk->sk_receive_queue))
return;
} else {
size += up->forward_deficit;
}
up->forward_deficit = 0;
atomic_sub(size, &sk->sk_rmem_alloc); atomic_sub(size, &sk->sk_rmem_alloc);
sk->sk_forward_alloc += size; sk->sk_forward_alloc += size;
amt = (sk->sk_forward_alloc - partial) & ~(SK_MEM_QUANTUM - 1); amt = (sk->sk_forward_alloc - partial) & ~(SK_MEM_QUANTUM - 1);