Internet Draft
Internet Engineering Task Force Indra Widjaja
Fujitsu Network Communications
INTERNET DRAFT Anwar Elwalid
Expired in six months Bell Labs, Lucent Technologies
October 1999
MATE: MPLS Adaptive Traffic Engineering
<draft-widjaja-mpls-mate-01.txt>
Status of this Memo
This document is an Internet-Draft. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its areas,
and its working groups. Note that other groups may also distribute
working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as ``work in progress.''
To learn the current status of any Internet-Draft, please check the
``1id-abstracts.txt'' listing contained in the Internet-Drafts Shadow
Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe),
munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or
ftp.isi.edu (US West Coast).
Abstract
This document describes an MPLS Adaptive Traffic Engineering scheme,
called MATE. The main goal of MATE is to avoid network congestion by
balancing the loads among the LSPs. MATE makes minimal assumptions
in that the intermediate LSRs are not required to perform traffic
engineering activities or measurements beside forwarding packets.
Moreover, MATE does not impose any particular scheduling, buffer
management, architecture, or a priori traffic characterization, on
the LSRs.
1.0 Introduction
One of the main advantages of MPLS is its efficient support of expli-
cit routing through the use of Label Switched Paths (LSPs) [1][2].
With destination-based forwarding as in the conventional datagram
networks, explicit routing is usually provided by attaching to each
Widjaja & Elwalid Expired in six months [Page 1]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
packet the network-layer address of each node along the explicit
path. This approach makes the overhead in the packet prohibitively
expensive. Switched forwarding using label swapping in MPLS enables
a label to identify an LSP regardless of whether the LSP is esta-
blished through hop-by-hop or explicit routing. In other words, MPLS
imposes the same amount of packet header overhead irrespective of the
use of explicit routing.
Explicit routing has useful applications such as Virtual Private Net-
works (VPNs) and Traffic Engineering. This memo focuses on engineer-
ing the traffic across multiple explicit routes. The purpose of
traffic engineering is to manage network resources as efficiently as
possible so that congestion is minimized. This may be done by avoid-
ing links that are already heavily stressed. Traffic engineering typ-
ically becomes more effective as the network provides more alternate
paths. Traffic engineering also becomes more critical in large Auto-
nomous Systems where maximal operational efficiency should be
emphasized [3].
It is envisioned that traffic engineering is performed only for
traffic that does not require resource reservation, but may need pro-
visioning on an aggregated basis. Examples include best-effort and
differentiated services. This memo proposes that traffic engineering
be done by establishing multiple LSPs between a given ingress LSR and
a given egress LSR in an MPLS domain. The main objective is to have
the ingress LSR distribute traffic across the multiple LSPs effec-
tively so that network resources among the LSPs are equitably util-
ized. This memo describes a scheme called MPLS Adaptive Traffic
Engineering (MATE). It is to be noted that MATE is intended for
quasi-stationary situations where traffic statistics changes rela-
tively slowly (much longer than the round-trip delay between the
ingress and egress LSRs). Recent measurements in the Internet indi-
cate traffic stationarity in at least 5-min intervals [4].
Some of the features of MATE include:
o Traffic engineering on a per-class basis.
o Distributed load-balancing algorithm.
o End-to-end protocol between ingress and egress LSRs.
o No new hardware or protocol requirement on intermediate LSRs,
nor a priori traffic distributions.
o No assumption on the scheduling or buffer management
schemes at the LSR.
o Optimization decision based on LSP congestion measure.
o Minimal packet reordering due to traffic engineering.
o No clock synchronization between two LSRs.
Widjaja & Elwalid Expired in six months [Page 2]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
2.0 Traffic Engineering Using MATE
This section provides the description of MATE. While the specifics
of the scheme are tailored for best-effort service, the basic princi-
ples are intended to be extensible to other services (e.g., differen-
tiated service) as well.
2.1 Overview
It is assumed that M explicit LSPs have been established between an
ingress LSR and an egress LSR when traffic engineering is to be
facilitated between these two LSRs. The value of M is typically
between 2 and 5 (the case with M=1 is not interesting). The M LSPs
may be chosen to be the "best" M paths calculated by a link-state
protocol or configured manually. Explicit LSPs may be established by
RSVP, LDP, or other means. The mechanism to select and establish the
LSPs is beyond the scope of this document.
Once the LSPs are setup, the ingress LSR tries to distribute the
traffic across the LSPs so that the traffic loads are balanced and
congestion is thus minimized. The traffic to be engineered at the
ingress LSR is the aggregated flow (called traffic trunk in [5]) that
shares the same Forwarding Equivalence Class. This document assumes
that the traffic to be engineered consists of the best-effort ser-
vice. Future work will consider differentiated service.
In order to perform effective traffic distribution, the characteris-
tics of the LSPs and the QoS requirements of the traffic must be
known. In general, the LSP characteristics may include the average
packet delay, packet delay variance, loading factor/utilization,
packet loss rate, bottleneck bandwidth, available bandwidth, etc.
For best-effort traffic, there is no explicit QoS requirement, except
that it is desirable to have minimum packet loss rate. Since MATE is
intended to be as flexible as possible, the pertinent LSP charac-
teristics are not assumed to be given quantities, but must be gath-
ered through some measurement. In MATE, the ingress LSR transmits
probe packets periodically to the egress LSR which will return the
probe packets back to the ingress LSR. Based on the information in
the returning probe packets, the ingress LSR is able to compute the
LSP characteristics. Intermediate LSRs are not required to modify
the contents of the probe packets, but such optional capabilities may
be used to refine the measurement process.
MATE employs a four-phase algorithm for load balancing. The first
phase initializes the congestion measure for each LSP. The conges-
tion measure may be a function of delay derivative and packet loss.
Widjaja & Elwalid Expired in six months [Page 3]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
In the second phase, the algorithm tries to equalize the congestion
measure for each LSP. Once the measures are equalized, the algorithm
moves to the third phase. The algorithm monitors each LSP in the
third phase. If an appreciable change in the network state is
detected, the algorithm moves to the fourth phase where the conges-
tion measures are appropriate adjusted. Then, the algorithm goes to
the second phase and the whole process repeats.
2.2 Traffic Filtering and Distribution
MATE performs a two-stage traffic distribution. First, MATE distri-
butes the engineered traffic for a given ingress-egress pair equally
among N bins at the ingress LSR. If the total incoming traffic to be
engineered is of rate R bps, each bin receives an amount of r = R/N
bps. Then, the traffic from the N bins is mapped into M LSPs accord-
ing to the rule defined below.
The engineered traffic can be filtered and distributed into the N
bins in a number of ways. A simple method is to distribute the
traffic on a per-packet basis without filtering. For example, one
may distribute incoming packets at the ingress LSR to the bins in a
round-robin fashion. Although it does not have to maintain any per-
flow state, the method suffers from potentially having to reorder an
excessive amount of packets for a given flow which is undesirable for
TCP applications.
On the other extreme, one may filter the traffic on a per-flow basis
(e.g., based on tuple), and distribute the
flows to the bins such that the loads are similar. Although per-flow
traffic filtering and distribution preserves packet sequencing, it
has to maintain a large number of states to keep track of each active
flow.
Another method, which is described in [6], is to filter the incoming
packets by using a hash function on the IP field(s). The fields can
be based on the source and destination address pair, or other combi-
nations. A typical hash function is based on CRC. The purpose of
the hash function is to randomize the address space to prevent hot
spots. Traffic can be distributed into the N bins by applying a
modulo-N operation on the hash space. Note that packet sequence for
each flow is maintained with this method.
After the engineered traffic is distributed into the N bins, a second
function maps each bin to the corresponding LSP. The rule for the
second function is very simple. If LSP(i) is to receive twice as
much traffic as LSP(j), then LSP(i) should receive traffic from twice
Widjaja & Elwalid Expired in six months [Page 4]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
as many bins as LSP(j). The value N should be chosen so that the
smallest traffic that can be shifted, which is equal to 1/N of the
total incoming traffic, has reasonable granularity.
2.3 Traffic Measurement
The efficacy of any traffic engineering scheme depends crucially on
the traffic measurement process. MATE does not require each LSR to
perform traffic measurement. Only the ingress and egress LSRs are
required to participate in the measurement process.
For the purpose of balancing the loads on each LSP, the available
bandwidth appears to be a desirable metric to measure. The methods
for measuring the available bandwidth of a given path have been
described in the past (e.g., see [7] and [8]). Based on our experi-
ence, this metric turns out to be difficult to measure accurately
using minimal requirements assumed in MATE.
To this end, we found that packet delays across the LSPs are the most
reliable known metric. The delay of a packet on an LSP can be
obtained by transmitting a probe packet from the ingress LSR to the
egress LSR. The probe packet is time-stamped at the ingress LSR at
time T1 and recorded at the egress LSR at time T2. If the ingress's
clock is faster than the egress's clock by Td, then the total packet
delay (i.e, queueing time, propagation time, and processing time) is
T2-T1+Td. A bunch of probe packets can easily yield an estimate on
the mean packet delay Tm+Td, where Tm is the long-term average of
T2-T1. The reliability of the estimator can be evaluated by
bootstrapping or jacknife to give the confidence interval for the
mean delay. One important point to note is that the value of Td is
not required when only the marginal delay is needed. MATE exploits
this property by relying only on marginal delays rather than absolute
delays. Therefore, clock synchronization is not necessary.
Packet loss probability is another metric that can be estimated by a
bunch of probe packets. In general, only reasonably high packet loss
rates can be reliably observed. Packet loss probability can be
estimated by encoding a sequence number in the probe packet to notify
the egress LSR how many probe packets have been transmitted by the
ingress LSR, and another field in the probe packet to indicate how
many probe packets have been received by the egress LSR. When a
probe packet returns, the ingress LSR is able to estimate the one-way
packet loss probability based on the number of probe packets that has
been transmitted and the number that has been received. The advan-
tage of this approach is that it is resilient to losses in the
reverse direction.
Widjaja & Elwalid Expired in six months [Page 5]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
2.4 Objective Function
For best-effort traffic, our objective is to avoid congested LSPs.
The congestion measure can be characterized by the sensitivity of the
LSP. An LSP is qualitatively said to be sensitive if a perturbation
in the load changes the mean delay significantly. MATE computes the
derivative of the mean packet delay with respect to the offered load
to quantify the sensitivity of an LSP. For a given LSP, the deriva-
tive increases as the load increases. This is evident since the mean
delay is an increasing and convex function with respect to the load.
The derivative can be derived by observing the mean delays at two
different loads. One advantage for using the sensitivity measure is
that the fixed propagation delay has no effect on the derivative.
An LSP is said to be lossless if each LSR along the LSP never dis-
cards packets. Otherwise, the LSP is said to be lossy. Although the
lossless case may not be realistic, it is much simpler to understand.
Here, MATE performs load balancing by simply equalizing the deriva-
tives across the M LSPs. The implementation is much simplified since
only the relative derivatives should be known. MATE essentially
applies a Min-Max operation; that is minimizing the load on an LSP
that has maximum congestion. The objective is also equivalent to
minimizing the sum of all LSP delays. MATE can be easily made to
respond to changes in network state adaptively as long as the traffic
is quasi-stationary. Network changes can be tracked by periodically
measuring packet delays. If the changes in delays exceed a certain
threshold, the algorithm minimized the most congested LSP by re-
equalizing the derivatives.
For the lossy case, the packet loss information must be incorporated
into the load balancing process. The lossy case will be treated in
the later section.
2.5 State Table
MATE keeps a state table for each traffic class that will be
engineered between an ingress LSR and an egress LSR. The state table
needs to maintain the mean delay and derivative of each LSP. One con-
venient realization is depicted below:
Widjaja & Elwalid Expired in six months [Page 6]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
+-----------------+--------------+-------------+---------+
| LSP | D_old | D_new | Inc |
+-----------------+--------------+-------------+---------+
| 1 | | | |
| 2 | | | |
| 3 | | | |
| ... | | | |
| M | | | |
+-----------------+--------------+-------------+---------+
D_old (D_new) denotes the mean packet delay before (after) a traffic
shift is made to the LSP. Inc (for Increments) denotes the number of
bins for which traffic was recently shifted to/from the LSP.
The derivative for LSP(i) can be simply computed by |D_new(i) -
D_old(i)|/Inc(i).
2.6 Algorithm: Lossless Case
We now present the algorithm in an ideal case with quasi-stationary
traffic, zero packet loss, and perfect measurements.
Initially, the traffic to be engineered at the ingress LSR is sent
through the shortest path, say through LSP(1). The algorithm can be
broken into four phases:
Initially the state table is empty, and phase 1 is run.
Phase 1: Initialize delays and derivatives
Inc(1) <- N
for i from 2 to M
Send probe packets on LSP(1) and LSP(i)
Compute mean packet delays D_old(1) and D_old(i)
Shift [N/M] bins of traffic from LSP(1) to LSP(i)
Inc(1) <- N - [N/M]
Inc(i) <- [N/M]
Send probe packets on LSP(1) and LSP(i)
Compute mean packet delays D_new(1) and D_new(i)
Go to Phase 2
Here, [N/M] indicates the largest integer less than or equal to N/M.
At the end of Phase 1, all columns of the state table are filled up.
The traffic at the ingress LSR is distributed almost equally across all M LSPs.
Phase 2: Equalize derivatives
Widjaja & Elwalid Expired in six months [Page 7]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
do
Let LSP(i) be the LSP that has the highest derivative
Let LSP(j) be the LSP that has the lowest derivative
D_old(i) <- D_new(i)
D_old(j) <- D_new(j)
Shift n bins of traffic from LSP(i) to LSP(j)
Send probe packets on LSP(i) and LSP(j)
Compute mean packet delays D_new(i) and D_new(j)
Inc(i) <- n
Inc(j) <- n
while (|D_old(i) + D_old(j)| > |D_new(i) + D_new(j)|)
Go to Phase 3
The main stopping rule for the optimization process is
when the total LSP delay can no longer be decreased.
This is achieved when all derivatives are approximately equal.
Note that phase 2 actually stops when the total LSP delay overshoots.
A compensation could be made by shifting back
the traffic from LSP(j) to LSP(i) after the overshoot is detected.
The parameter n (n = 1, 2, ..., N)
determines the amount of traffic to be shifted.
A small value of n may cause the "noise" to mask the traffic
that is shifted. On the other hand, a large value of n may
overshoot and overload the LSP. Thus, this value should
be properly engineered, and can be made adaptive.
It is to be understood that MATE cannot shift more bins
than it currently has for a given LSP.
The algorithm terminates if no bin is available from
the LSP with the lowest derivative.
By the mean value theorem, the computed derivative is
equal to the actual derivative between the points before and after
the traffic is shifted.
Phase 3: Track network dynamics
Send probe packets intermittently on each LSP(i)
Compute running mean packet delay D(i)
If (|D(i) - D_new(i)| > alpha * |D_new(i) - D_old(i)|)
Go to Phase 4
The condition in Phase 3 basically detects a network change by moni-
toring an appreciable change in an LSP load.
The amount of probe packets sent at phase 3 should be negligible com-
pared to the engineered traffic that is sent to the same LSP (say,
less than 1 percent). The parameter alpha determines the threshold on
Widjaja & Elwalid Expired in six months [Page 8]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
the change in load.
The mean packet delay on LSP(i) is updated by a low-pass filter:
D(i) <- beta * D(i) + (1 - beta) * Measured delay
where beta is weighting constant between 0 and 1, and D(i) is ini-
tialized to D_new(i).
In order to minimize synchronization due to multiple ingress LSRs
detecting the network change simultaneously, it may be useful to ran-
domize the detection time with a random backoff time. There are other
schemes to eliminate synchronization.
Phase 4: Refresh state table
LSP(i) detects a significant change in its marginal delay
set Flag(i)
if (D(i) > D_new(i))
D_old(i) <- D(i) - 2 * (D_new(i) - D_old(i))
D_new(i) <- D(i)
Inc(i) <- 1
else
D_old(i) <- D(i) - (D_new(i) - D_old(i)) / 2
D_new(i) <- D(i)
Inc(i) <- 1
For each LSP k such that (k != i)
D_old(k) <- D(k) - (D_new(k) - D_old(k))
D_new(k) <- D(k)
Go to Phase 2
When LSP(i) detects a significant change in its marginal delay, it
changes doubles or halfs its derivative depending on whether the
delay has increased or decreased, respectively. The associated flag
is set to ensure that LSP(i) will be considered in Phase 2. D_new's
and D_old's of the rest of the LSPs are updated with the latest
information. All computations should check for illegal values.
2.7 Time Scales
Since MATE relies on measurements to estimate LSP characteristics, it
is important to understand the time scale each phase typically
operates. After the initialization in Phase 1, MATE alternates in
Phase 2, Phase 3 and Phase 4. MATE is designed to execute in Phase 2
on the order of 1 minute. The effectiveness in this phase depends
critically on the stationarity of the traffic. Recent measurements
indicate that traffic in the wide area network is stationary in at
least a 5-minute interval [4]. Phase 3 is intended to detect a
Widjaja & Elwalid Expired in six months [Page 9]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
persistent and appreciable change in the network after the algorithm
has stabilized in Phase 2. In general, Phase 3 operates at a much
longer time scale (on the order of 30 minutes or longer). Phase 4 is
executed to update the State Table immediately after Phase 3. Phase 4
typically operates on the order of a second.
2.8 Monotonous Property of Delay versus Load
To cope with "noise" in the cross/background traffic and measurement
errors, we describe some improvements to the above algorithm by using
the property of delay-load curve.
It is possible for the computed mean delay to violate the monotonous
property of delay versus load. A violation may occur in two cases.
In the first case, the mean delay decreases when traffic is shifted
into an LSP. In another case, the mean delay increases when traffic
is shifted out of an LSP. The violation can be noted by associating
an attribute flag to each delay variable in the state table.
In order to correct an anomaly due to violation, MATE always repeats
the measurement on the pair that is involved in violation until there
is no more violation detected. In the repetition, traffic is shifted
back and forth between the pair until no more violation is detected
in each LSP.
2.9 Inflection
Another observation is that the mean delay is typically a convex
function of the offered load, except beyond a certain point (called
the inflection point) where the LSP becomes heavily congested.
When MATE encounters an inflection point during the iteration pro-
cess, it should adjust D_old so that the higher derivative between
the previous and the current iterations is used. This decision makes
the derivative estimate more conservative when the inflection point
is due to heavy congestion.
2.10 Dynamic Shift
The number of bins to be shifted (Inc) can be dynamically adjusted as
follows. If one of the LSPs to be shifted consists of an LSP whose
new mean delay was invalidated previously due to a violation, then
the number of bins to be shifted will be doubled from the previous
value, subject to the maximum number of bins that can be shifted.
Otherwise, the number of bins to be shifted will be halfed from the
Widjaja & Elwalid Expired in six months [Page 10]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
previous value, subject to the minimum number of bins that is
allowed.
2.11 Lossy Case
The base algorithm described in the previous section assumes an ideal
case where there is no packet loss. This section incorporates the
effect of packet loss on the algorithm by adjusting the derivative
with a loss factor.
MPLS traffic is likely to be dominated by TCP applications, in which
case the packet loss inside an MPLS domain should be tightly con-
trolled since TCP reacts to packet losses. It is well-known that TCP
achieves bandwidth that is inversely proportional to the square root
of the packet loss probability (p), under idealized condition. Note
that this relationship is typically reasonable only in a certain
regime of p (roughly, p is not too close to zero or one).
We use a typical delay-load function which is convex and increasing
in load. The delay-load function blows up when the arrival rate
approaches the service rate, which can be approximated by the simple
formula:
D = A / (mu - lambda), (1)
where mu is the offered traffic, lambda is the average service rate,
and A is some constant. Most queueing systems would have the same
factor in the denominator as the dominant one. The derivative of the
delay with respect to the offered traffic can then be computed by
D' = A / (mu - lambda)^2. (2)
Since the available bandwidth for the lossless case is
BW_lossless = mu - lambda, (3)
we can relate the lossless available bandwidth to the derivative by
BW_lossless^2 = A / D'. (4)
Because the bandwidth achieved by TCP is reduced by a factor of
square root of p when there are packet losses, the available
bandwidth for the lossy case can be correspondingly reduced by the
same factor. In other words, we can write
BW_lossy^2 = BW_lossless^2 * C / p, (5)
Widjaja & Elwalid Expired in six months [Page 11]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
where C is set to the lowest observable loss probability through the
probe packets. If MATE cannot determine the packet loss probability
because of its rare occurrences, it sets p to C. Comparing (4) and
(5), the lossy available bandwidth can be related to the derivative
by
BW_lossy^2 = C / (p * D'). (6)
Therefore, equalizing (p * D') is equivalent to equalizing the avail-
able bandwidth for the lossy case.
It is interesting to see that one may define another metric called
Power which is defined as the ratio of throughput to delay. If the
objective is to equalize power, one will find that this is equivalent
to equalizing (p * D').
For the lossy case, Phase 2 of the algorithm stops when the weighted
delay increases. In Phase 3, it is not possible to measure the
packet loss rate since the probe traffic is low. Instead, a lost
probe packet is accounted for by assigning D(i) to the maximum
measurable delay.
3.0 Protocol Specification
To be added.
4.0 Security Considerations
Security considerations are not addressed in the present document.
5.0 Acknowledgement
The authors thank Cheng Jin for implementing MATE and suggesting
improvements, and Dimitri Stiliadis and Eric Gray for providing
insightful comments.
6.0 References
[1] R. Callon, P. Doolan, N. Feldman, A. Fredette,
G. Swallow and A. Viswanathan,
Widjaja & Elwalid Expired in six months [Page 12]
Internet Draft MPLS Adaptive Traffic Engineering October 1999
"A Framework for Multiprotocol Label Switching,"
Internet Draft <draft-ietf-mpls-framework-02.txt>,
Nov 1997.
[2] E.C. Rosen, A. Viswanathan, and R. Callon,
"Multiprotocol Label Switching Architecture,"
Internet Draft <draft-ietf-mpls-arch-01.txt>, Mar 1998.
[3] D.O. Awduche et. al., "Requirements for Traffic
Engineering over MPLS,"
Internet Draft <draft-awduche-mpls-traffic-eng-00.txt>,
April 1998.
[4] K. Thompson, G.J. Miller, and R. Wilder, "Wide-Area
Internet Traffic Patterns and Characteristics, "
IEEE Networks, vol. 6, no. 6, Dec. 1997.
[5] T. Li and Y. Rekhter, "Provider Architecture for
Differentiated Services and Traffic Engineering (PASTE),"
Internet Draft <draft-li-paste-00.txt>, Jan 1998.
[6] C. Villamizar, "OSPF Optimized Multipath (OSPF-OMP),"
Internet Draft , Mar 1998.
[7] S. Keshav, "A Control-Theoretic Approach to Flow
Control,"
Proc. SIGCOMM'91, pp.3-15, Sep. 1991.
[8] R.L. Carter and M.E. Crovella, "Measuring Bottleneck
Link Speed in Packet-Switched Networks," Technical Report,
BU-CS-96-006, Boston University, Mar. 1996.
Author Information:
Indra Widjaja
Fujitsu Network Communications
Two Blue Hill Plaza
Pearl River, NY 10965, USA
Phone: 914 731 2244
Email: Indra.Widjaja@tddny.fujitsu.com
Anwar Elwalid
Bell Labs Lucent Technologies
Room 2C-324
Murray Hill, NJ 07974, USA
Phone: 908 582-7589
Email: anwar@lucent.com
Widjaja & Elwalid Expired in six months [Page 13]