Internet Draft



Internet Engineering Task Force                        K.K. Ramakrishnan
INTERNET DRAFT                                         Gisli Hjalmtysson
                                                     Kobus van der Merwe
                                                  (AT&T Labs.  Research)
                                                           Flavio Bonomi
                                                           Sateesh Kumar
                                                            Michael Wong
                                                 (CSI Zeitnet/Cabletron)
                                                             August 1998


            UNITE: An Architecture for Lightweight Signaling
                 <draft-ramakrishnan-mpls-unite-00.txt>



Status of This Memo

   This document is an Internet-Draft.  Internet-Drafts are working
   documents of the Internet Engineering Task Force (IETF), its areas,
   and its working groups.  Note that other groups may also distribute
   working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at
   any time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   To view the entire list of current Internet-Drafts, please check
   the "1id-abstracts.txt" listing contained in the Internet-Drafts
   Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern
   Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific
   Rim), ftp.ietf.org US East Coast), or ftp.isi.edu (US West Coast).



   (Note that this ID is also available in Postscript and PDF formats)



Ramakrishnan et al.            Expires February 1999            [Page i]

Internet Draft                     UNITE                     August 1998



Abstract

   Communication networks need to support a wide range of applications
   with diverse service quality requirements.  The current widespread
   use of best-effort communication also suggests that the overhead
   for establishing communication both in processing and latency needs
   to be kept at a minimum.  With ATM signaling, every flow, including
   a best-effort flow, suffers the overhead of end-to-end connection
   establishment.  ATM signaling complexity is further exacerbated by
   having variable length messages with a large number of information
   elements using a very flexible encoding, sent on a single control
   channel.  The inclusion of QoS processing and connectivity in
   the initial setup of a connection requires sequential hop-by-hop
   processing.  Variable length messages involves both a single point of
   resequencing as well as relatively slow, software based processing.
   In recognition of these shortcomings, the MPLS working group has
   opted to use topology driven label distribution as its default
   label distribution mechanism, while at the same time acknowledging
   the possible need for on-demand label distribution.  We see these
   different approaches as points on a range of solutions and we do
   not wish to open a debate concerning the relative merits of each
   approach.  However, we believe that if there is a need for on-demand
   label distribution, then there is a need to do this very efficiently.
   In this light we have decided to bring to the MPLS working group our
   architecture for lightweight signaling.  While in its current form it
   is applicable to an ATM environment, we believe that it represent a
   step forward in the evolution of signaling for high speed networks.
   It holds the promise of processing signaling in hardware, thereby
   enabling substantial speed up of connection setup, so as to meet the
   needs of contemporary applications.

   Our proposed lightweight architecture for ATM signaling is called
   UNITE. The fundamental philosophy of UNITE is the separation of
   connectivity from QoS control.  This has the potential to eliminate
   the round-trip connection setup delay, before initiating data
   transmission.  Using a single cell with proper encoding, we avoid the
   overhead of reassembly and segmentation on the signaling channel.
   With fixed formats, we believe that a hardware implementation is
   feasible.  Performing QoS negotiation in-band allows switches in the
   path to process QoS-requests in parallel, facilitates connection
   specific control policies, supports both sender and receiver
   initiated QoS, and allows for uniform treatment of unicast and
   multicast connections.



Ramakrishnan et al.            Expires February 1999           [Page ii]

Internet Draft                     UNITE                     August 1998



Note on Applicability

   This Internet Draft is based on an ATM Forum contribution and as
   such is written within an ATM context.  However, we believe that
   the UNITE approach to signaling might also be of value within the
   context of MPLS and have therefore decided to present it to the MPLS
   working group to solicit feedback.  We hope to extend and modify this
   Internet Draft to be applicable for on-demand label distribution in
   MPLS based on the feedback received.



1. Introduction

   The goal of lightweight signaling is to reduce the penalty of
   connection setup, while supporting service guarantees.  A lightweight
   signaling protocol should ideally support and enhance both
   connectionless and connection-oriented services.  Because of a desire
   to foresee the signaling needs of any and all applications that are
   likely to use the network, current ATM signaling is complex and
   slow, multiple messages are required to set up a connection, and
   considerable processing is required to parse the complex signaling
   messages.

   In this internet draft, we describe UNITE, a lightweight signaling
   protocol for ATM networks.  We are motivated by the need to more
   efficiently support data applications that typify current Internet
   traffic while providing facilities to support applications that
   require stringent quality-of-service such as telephony.  Furthermore,
   this work is aimed at reducing the complexity of ATM signaling,
   improving the performance of ATM call processing, and improving ATM
   as a general purpose transport infrastructure.

   The principal idea behind UNITE is a complete separation of
   connectivity from quality-of- service, or more generally, service
   attributes.  The connectivity setup message is reduced to a single
   ATM cell, with fixed field sizes and positions, avoiding the overhead
   of reassembly and segmentation on the signaling channel, allowing
   it to be fully processed in hardware.  Exploiting per-VC queueing,
   data can be forwarded immediately after a one-hop exchange, rather
   than suffering a full-round-trip latency.  However, we recognize
   that not all switches are likely to have per-VC queues, and switches
   may initially want to support connection establishment in software.
   For this reason UNITE accommodates both software processing and
   FIFO switches using a marker/marker-acknowledgment protocol between



Ramakrishnan et al.            Expires February 1999            [Page 1]

Internet Draft                     UNITE                     August 1998



   switches.  UNITE reduces connection setup cost sufficiently, so
   that establishing connectivity becomes comparable to forwarding
   and populating a cache in a router.  A UNITE switch can therefore
   reasonably be expected to setup new connections at a rate competitive
   with routing in a connectionless networks.  Conversely, IP-type
   best effort data flows suffer sufficiently small delay penalty for
   establishing a connection over the ATM infrastructure that it becomes
   viable to set up a connection even for the shortest of flows.  Thus,
   UNITE is ideally matched to carry Internet traffic (IP) over ATM
   networks.

   UNITE uses in-band messages for QoS establishment.  It builds on
   the extensive work done for QoS in ATM networks, including the
   specification of classes of service, admission control and related
   issues such as conformance and policing.  Because the QoS messages
   are sent on the established VC, we can exploit parallelism to improve
   the throughput and latency for QoS establishment.

   In part due to its simplicity, UNITE supports both source and
   destination initiated QoS, supports multipoint-to-multipoint
   connections and recognizes the possible need for variable QoS to
   different participants [i, ii] (variegated multicast trees).

   UNITE has been implemented in a software prototype.  Early
   performance measurements confirm our expectations for a higher
   signaling throughput and lower call setup latency.  In the next
   section we describe UNITEs connection setup for best effort
   connections.  Subsequently, in Section 3, we describe UNITEs support
   for multicast.  In Section 4, we provide details of UNITEs QoS
   Management, and then deal with interoperability issues, both with UNI
   as well as with existing switches.  Section 7 summarizes the benefits
   of UNITE and then we conclude.  Finally, in Section 9, we briefly
   consider the applicability of UNITE in an MPLS environment.



2. UNITE Connection Setup

   UNITE uses a separate, initial mechanism for setup of connectivity to
   enable a fast connection setup.  This is shown in Figure 1.

   Figure 1:  The UNITE Connection Setup



Ramakrishnan et al.            Expires February 1999            [Page 2]

Internet Draft                     UNITE                     August 1998



   The calling station issues a micro-setup, which is a single cell, on
   the signaling channel that includes all the information necessary
   to establish a best-effort connection with a remote called station.
   The switch that receives the micro-setup determines the route (based
   on the destination address and a broad QoS class identification)
   to forward the micro-setup on the correct output port.  After
   allocating VC resources on the upstream link for the connection
   and forwarding the micro-setup, the switch returns a single cell
   micro-acknowledgment to the upstream node on the signaling channel.
   If appropriate, the switch may allocate per- VC buffers for the
   switch at that time also.  On receiving the micro-ack, the upstream
   node transmits a single cell marker on the established VC (in-band).
   This marker serves as the 3rd step of a three-way handshake.
   Subsequent to transmitting the marker, the upstream node may
   transmit data on the VC, on a best-effort basis.  The above sequence
   of steps is repeated at each hop.  Virtual Circuits established
   are bi-directional, with VC-ids allocated in the conventional
   manner by switches.  While we believe we can accommodate multiple
   address formats, we are currently using existing NSAP addresses
   and address allocation methodologies.  We assume the existence of
   link-layer management, such as ILMI. The micro- setup is routed to
   the destination on a hop-by-hop basis, using routing tables that
   are setup based on existing PNNI information dissemination and
   route-computations.  We also use existing cell-formats and currently
   defined AAL5 framing.

   The commitment provided by the connection is that data is transmitted
   on a best-effort basis.  Since the QoS class information is provided
   in the micro-setup, the path selected even for the best-effort
   connection may be on a more informed basis than pure best-effort with
   no a priori knowledge.  Data may begin flowing from an upstream node
   to the downstream node immediately upon completion of the micro-setup
   on that hop.  The latency suffered by a best- effort flow to use the
   connection-oriented nature of ATM is thus only a single hop round-
   trip propagation delay, plus the time needed to setup state on that
   hop.  Data is buffered on a switch (with per-VC buffers) when it
   arrives on a VCid for which a forwarding table entry has yet to be
   setup.  In a subsequent section in this draft, we describe methods
   to accommodate FIFO switches, and also when the processing of the
   signaling messages is performed in software.  This is enabled by the
   use of an optional marker-acknowledge, that allows for a downstream
   switch (or node) to require the upstream switch (or node) to delay
   transmitting of data until it is ready to receive data.  To ensure
   that no persistent loops form, UNITE uses a combination of a unique
   Flow-ID for the connection and an end-end acknowledgement.  When
   the destination receives the micro-setup, it sends an in-band (on



Ramakrishnan et al.            Expires February 1999            [Page 3]

Internet Draft                     UNITE                     August 1998



   the established VC) end-end ack to the source.  This indicates to
   the source that a loop-free path has been established.  Only upon
   receiving the end-end ack does the source issue a RELEASE at any time
   in the future when it needs it.  Issuing a RELEASE prior to receiving
   the end-end ack may erase the Flow-ID maintained at a switch.  This
   is undesirable because it will be unable to recognize the micro-setup
   that may come back as a result of a loop.  The combination of the
   unique Flow-ID and holding back the RELEASE until the end-end ack is
   received enables us to avoid loops.



2.1. ESTABLISHING CONNECTIVITY, PHASE 1:  THE MICRO-SETUP

   The micro-setup and the associated micro-acknowledgment are sent on
   a well-known signaling VC. The processing of the micro-setup at the
   switch includes the following functions, in order:

   1.  A route lookup for the micro-setup, identifying the port on which
   to forward the micro- setup.

   2.  Allocation of a VC from the VC address space on the upstream
   link.  We assume that all connections are created bi-directional (to
   minimize the overhead of both ends establishing connections).

   3.  Allocation of a reasonable amount of buffering at the switch for
   that VC, if appropriate.

   4.  Initiating an ACK-timer for the micro-setup.  This timer is for
   error recovery when the micro- setup is lost or when the downstream
   switch does not successfully progress the micro- setup.

   5.  Forwarding the micro-setup downstream on the link that the
   route-lookup function determined as the best path towards the
   destination end-system.

   6.  Mark the incoming VC state as DISCARD, so that the switch
   discards all incoming cells on this VC. This enables us to clear
   previously buffered cells for the upstream link on the newly assigned
   VC, if there are any.  The VC state transitions to FORWARD state
   subsequently, when a MARKER acknowledging the ACK is received.



Ramakrishnan et al.            Expires February 1999            [Page 4]

Internet Draft                     UNITE                     August 1998



   7.  Finally, a VC id is returned to the upstream switch in the
   micro-ACK. The upstream node may begin transmitting data on receipt
   of the micro-ACK. The forwarding of the data to the downstream next
   hop has to await the completion of the processing at the next hop
   switch and the return of a corresponding VC id for the flow.

   We have chosen to provide reliable delivery within the UNITE
   signaling framework itself, rather than layering it on top of
   another reliable transport mechanism.  Current ATM UNI signaling
   uses a reliable transport protocol, SSCOP for transporting signaling
   messages thus re-incorporating some of the overhead for processing a
   signaling message, and makes it difficult to implement in hardware.
   The 3-way handshake obviates the need for a reliable transport for
   carrying signaling messages.

   A simple, efficient encoding of the setup is vital:  we use a single
   cell for the micro-setup, with only essential components in it, thus
   allowing for hardware implementation.  In addition, it allows for
   distributed call setup to be implemented in a switch (especially
   important when there are a large number of ports).  The micro-setup
   uses a unique end-to-end Flow-id.  All control exchanges use this
   Flow-id.  Included in the micro-setup is whether the call is unicast
   or multicast capable.  Multicast and unicast connections have nearly
   identical mechanisms for both connection setup and QoS setup.

   UNITE adopts hop-by-hop routing of the micro-setup, in contrast
   to the traditional source- routing used in ATMs PNNI routing
   protocols.  However, source-routing has been used to avoid loops
   in connection-oriented networks.  Since UNITEs Flow-id is a unique
   end-to- end call-reference identifier, this may be used to detect
   loops.  When a duplicate micro-setup is received with the same
   Flow-id, without it being a retransmission (or on a different port
   than the port the earlier copy was received on) it indicates a
   routing loop.  UNITE suppresses multiple micro-setups (a mechanism
   we also use for multicast connections for normal operation).  A
   controller might also send a release in the backward direction for
   the Flow-id (or allow timers to subsequently close the connection).
   This mechanism along with the rules for issuing a RELEASE after
   an end-end acknowledge is received by the source ensures that a
   successful connection does not contain a loop.  Routing loops are
   mostly transient inconsistencies in routing tables, which we expect
   to be corrected by subsequent updates as part of the normal operation
   of the routing protocols.



Ramakrishnan et al.            Expires February 1999            [Page 5]

Internet Draft                     UNITE                     August 1998



   The micro-setup being a single cell allows the switch to avoid
   re-assembly and segmentation.  In addition, all of the requirements
   to keep the cells in sequence may be ignored:  a micro-setup cell
   may be served in any order, relative to the others.  Thus, we could
   choose to process the micro- setup for some classes in hardware,
   and others in software, if so desired.  Furthermore, it allows for
   a truly distributed implementation of the micro-setup because there
   is no need for a single point of re-sequencing the cell streams for
   signaling messages arriving on different ports.  A fixed format
   micro-setup cell also assists hardware implementations.

   The fields of the micro-setup cell are as follows, with reference to
   Figure 2:

   1.  Flow-id (8 bytes) - A unique (end-to-end) Flow-id identifying the
   micro-setup from source.  This comprises two sub-fields:

   a) A unique source identifier.  For example, this could be the host
   Ethernet address, that is unique through the use of an address ROM (6
   bytes).

   b) A source unique sequence number (2 bytes).

   2.  Type (1 byte) - type of signaling cell.  Includes a Retransmit
   bit.

   3.  QoS Class (1 byte) - for minimal QoS sensitive routing.
   (Potentially broken up into a nibble for class definition and a
   nibble for specification of the size of the dominant parameter for
   that class.

   4.  Reserved (1 byte) - for future use.  Anticipating the potential
   use of a Virtual Private Network Identifier, we could include 3 bytes
   for a VPN ID by removing the User-User Information byte from the AAL5
   trailer.  The use of such a VPN ID is for further discussion.

   5.  Protocol ID (5 bytes) - allows the caller to specify the network
   layer entity addressed at the called station and eliminates a need
   for a second exchange to establish this connectivity.  SNAP encoding
   is assumed by default.  The 5 bytes includes the OUI and PID fields.

   6.  Destination Address (20 bytes) - destination NSAP address.



Ramakrishnan et al.            Expires February 1999            [Page 6]

Internet Draft                     UNITE                     August 1998



   7.  A VPI/VCI that is assigned by the upstream node for the
   connection when it is appropriate.  This is determined by which
   end of a link is supposed to allocate the VPI/VCI value for a new
   connection, just like the current convention.

   8.  AAL5 Trailer (8 bytes) - the standard ATM AAL5 trailer including
   the CRC and length.  In addition, of course, is the 5 byte ATM cell
   header.  The VC id on which the micro-setup is transmitted is a
   common, well-known signaling VC.

   A switch maintains a timer associated with the micro-setup that
   has been transmitted to the downstream hop.  This timer is cleared
   upon receiving the ACK from the downstream switch.  A switch that
   has timed out after transmission of the micro-setup retransmits the
   micro-setup request.  The re-transmitted micro-setup is identical to
   the previous except for a retransmit bit in the type field.  As a
   result it can be retransmitted by hardware.



2.2. Establishing connectivity, Phase 2:  The ACK for the Micro-setup

   The micro-Acknowledgment of the connection setup upon successful
   processing of the micro-setup is returned upstream to the previous
   switch or host.  The information provided has to be adequate for
   appropriate processing at the upstream switch or the original
   host requesting the connection.  The downstream switch maintains
   a timer associated with the micro-ACK transmitted upstream, for
   re-transmitting micro-ACKs.  (This timer is cleared when the MARKER
   is received in the third phase of the micro-setup and therefore also
   protects against loss of the MARKER.) The micro-ACK has the following
   fields:

   1.  Flow-id (8 bytes):  the Flow-id received in the micro-setup, to
   enable to upstream node to match this ACK to the request.

   2.  VPI/VCI returned for the request (3 bytes)

   3.  The NSAP address that was the same as the one received in the
   micro-setup.

   4.  There is also a bit to indicate to the upstream whether a
   Marker-Acknowledge is to be expected or not, before transmitting
   data.  A second bit is used to inform the upstream switch on whether



Ramakrishnan et al.            Expires February 1999            [Page 7]

Internet Draft                     UNITE                     August 1998



   it should delay sending its marker-acknowledge until it has received
   one from downstream (Refer to Section 5.2).



2.3. Establishing connectivity, Phase 3:  The Use of a Marker

   The final part of the 3-way handshake on the hop-by-hop micro-setup
   is an in-band MARKER. The MARKER serves not only to acknowledge the
   micro-ACK message, but is also essential to mark the beginning of the
   new data flow.  The use of the 3-way handshake ensures that data at
   both the upstream node and the link related to the newly allocated
   VC id are flushed of old data at the time the downstream switch
   receives the MARKER. The 3-way handshake also allows for recovery
   from loss of the micro-ACK. The MARKER is the first cell sent in-
   band by the upstream node.  Everything that follows this marker is a
   valid data cell for the new flow.  The MARKER includes the Flow ID,
   the NSAP address of the connection initiator (source), and a bit to
   indicate if the version of the MARKER is a retransmission or not.
   The switch controller may, for example, use the source NSAP address
   for functions, such as VC re-routing or generating an out-of-band
   RELEASE.

   The upstream node, after sending the MARKER, sends data on this VC id
   if the downstream node has not requested that a Marker-Acknowledge
   is to be expected.  If a Marker-Acknowledge is to be expected,
   then the upstream node transmits data only after receiving the
   Marker-Acknowledge.



3. Call Setup for Multicast

   UNITE incorporates the functionality of having multipoint-to-multipo*
 *int
   communication [iv] as an integral part of the signaling architecture.
   The simpler cases of point-to- multipoint multicast calls are
   simple sub-cases of this overall multicast architecture.  The simple
   difference between a unicast call and a multicast call is that
   the micro-setup issued indicates that the call is potentially a
   multicast call.  For the purposes of this discussion we assume that
   the underlying network forwarding mechanism can manage issues such as
   cell interleaving [iv].  Therefore, we describe procedures that are
   applicable for core-initiated joins (for core based trees [v,vi]),
   which are similar for source-initiated join for a source- based tree.
   We then describe leaf-initiated joins for other participants that
   join subsequent to the call being setup [vii,viii].



Ramakrishnan et al.            Expires February 1999            [Page 8]

Internet Draft                     UNITE                     August 1998



3.1. Core/Source Initiated Joins

   Core initiated joins (or source initiated joins) are relevant when
   the set of participants is known initially.  The core issues a
   micro-setup knowing the NSAP address of each individual participant.
   Since there is no way to package more than one NSAP address in the
   micro- setup, an individual micro-setup is issued for each of the
   participants.  We think this is not as important, because, (a) the
   micro-setup is relatively cheap and (b) the number of participants
   that subsequently join using the leaf-initiated joins may dominate.

   The first micro-setup issued to a participant includes a label (in
   the Type field) to indicate that it is a multicast-capable call
   setup.  The rest of the micro-setup is similar to that described
   for a unicast call.  The Flow-id is determined by the originator
   (i.e.  the core or sender).  The Flow-id acts as a call-reference
   identifier for the multicast call.  The micro-setup issued for
   joining subsequent participants uses the same Flow-id, again labeled
   as a multicast.  The micro-ACK that comes back from the downstream
   hop returns a VC id as with unicast calls.  The MARKER transmitted by
   the core (or source) is sent in-band, on the VC id returned in the
   ACK.

   The Flow-id used in the micro-setup is retained at the switch, as a
   means of identifying the multicast call.  During joins, the switch
   sending the micro-setup maintains state, which includes the Flow-id
   and the destination NSAP address to which the setup was issued (the
   new leaf).  This way, ACKs that return for the individual setups
   issued may be matched up by the switch, for purposes of managing
   their retransmission.

   Figure 3:  Core initiated join.  Observe, that the marker on the last
   hop to B is generated by the controller at the branch point.

   The initiator of the micro-setup (core or source) sends the MARKER
   when it receives the first micro-ACK. Upon receiving subsequent
   micro-ACKs, the source/core knows that the VC is already open
   (operational) and therefore, doesnt generate a further MARKER. At a
   new branch point on the multicast tree, however, a MARKER is required
   to the new destination:  this is because that branch of the tree
   needs to be flushed of any old data that is currently in existence
   for that VC identifier.  The controller is responsible for generating
   and sending this in-band MARKER. Subsequently, data may be forwarded



Ramakrishnan et al.            Expires February 1999            [Page 9]

Internet Draft                     UNITE                     August 1998



   on that VC id, as a result of a proper 3-way handshake.  Figure 3
   illustrates this scenario.



3.2. Leaf Initiated Joins

   Figure 4 :  Leaf Initiated Join:  As LIJ is progressed to switch
   four.  Bs LIJ is suppressed at switch two.

   The mechanisms for Leaf Initiated Joins (LIJ) are similar to those
   suggested in the conventional ATM Forum UNI 4.0.  However, instead of
   having a separate LIJ and Add- Party mechanism, UNITE uses the same
   mechanisms of the micro-setup for performing a LIJ. Consider Figure
   4, where two participants A and B wish to join the multicast tree,
   that currently ends at Switch 4.  The LIJ is a micro-setup (the Type
   indicator indicates that this is a LIJ for a multicast call) from
   one of the participants, that is directed towards the core/source,
   using the NSAP address corresponding to the core/source.  The Flow ID
   used in the micro setup is the multicast call reference identifier,
   and is stored at the switches as the micro setup is forwarded
   upstream towards the core.  We assume that the underlying call
   routing mechanisms direct the micro-setup towards the source/core
   in accordance with the appropriate criterion (e.g., shortest-path
   or least cost).  When a LIJ arrives at a switch from another
   participant, such as B, the Flow ID is recognized as already existing
   at the switch, and the forwarding of Bs micro-setup is suppressed.
   This may be done only if the core does not wish to be notified of
   the address of an individual leaf joining.  Note that this happens
   even though the LIJ of the first participant added on this branch,
   has not yet reached the tree at Switch 4.  When the micro-setup
   from A is issued, the 3-way handshake results in the marker being
   forwarded by the switches upstream.  This effectively opens up the
   VC from the node A up to the branching point, at Switch 4.  Along
   with the suppression of the micro-setups, subsequent markers are also
   suppressed at the switches.



4. DETAILS OF QoS MANAGEMENT

   The second part of UNITE is a separate means of full QoS
   specification and negotiation.  This allows both a very flexible
   QoS management process, as well as the ability to incorporate QoS
   renegotiation with ease.  As discussed in the previous sections, the
   micro-setup includes a QoS byte that can be used in the original



Ramakrishnan et al.            Expires February 1999           [Page 10]

Internet Draft                     UNITE                     August 1998



   connection setup to support coarse or aggregate level QoS (e.g.,
   by allowing some differentiated decision for the forwarding of the
   micro-setup).  UNITE supports detailed QoS signaling (or full QoS
   signaling) that is performed in-band on the already established
   best-effort VC. We anticipate that a large subset of flows will
   not use the additional phase of a QoS setup for establishing a
   distinctive quality of service.  The QoS class specification that is
   provided in the initial micro-setup may be adequate for a reasonably
   large subset of best-effort flows (e.g., a large class of TCP/IP and
   UDP/IP flows carrying non-real-time data clearly dont need to have a
   subsequent QoS setup phase).  Similarly, well-understood real-time
   flows such as uncompressed telephony traffic (mu-law, a-law) may be
   adequately specified as being delay- sensitive. The assured QoS for
   the flow begins after the QoS negotiation completes, end-to-end,
   in a similar fashion to the conventional UNI QoS setup.  Also, we
   believe that most of the more sophisticated QoS management will be
   handled in software as is the case in the current UNI framework.
   However, UNITEs framework provides a more flexible and efficient QoS
   management in the following dimensions:

   1.  UNITE QoS requests may be initiated by the source or the
   destination of the original best effort connection setup.  In the
   more general case of multicast connections, QoS requests may be
   source/core initiated or leaf initiated.

   2.  UNITE QoS in-band signaling allows QoS renegotiation originating
   from any of the connection end points.

   3.  UNITE QoS in-band signaling enables potentially different QoS
   negotiation modalities and implementations taking advantage of
   parallelism in the processing of the QoS setup across multiple
   switches in the end-to-end path.

   Figure 5 :  Protocol for Establishing QoS in UNITE.

   For those flows that require a detailed QoS negotiation, we use
   the process of QoS setup described in Figure 5.  The QoS request
   may immediately follow the marker, as shown in Figure 5, or may
   be submitted after the call is established.  The receiver, after
   processing the request sends a QoS Commit, that commits the
   reservation.  To adjust over-committed reservations, and to confirm
   the QoS reservation to the receiver, the originator sends a QoS
   Ack.  The delay until a QoS flow begins on the forward path is an
   end-to-end round-trip plus the processing at the destination.  On the



Ramakrishnan et al.            Expires February 1999           [Page 11]

Internet Draft                     UNITE                     August 1998



   reverse path, a confirmed QoS flow begins one round-trip after the
   QoS Commit is issued from the destination.  For compatibility with
   existing ATM, we anticipate that the QoS request, Commit and Ack,
   would be encoded as in the UNI connection setup and connect messages,
   as far as the QoS information is concerned.  For our purposes in
   this section, we treat the end-system that initiates the QoS setup
   request as the QoS source. The end-system that responds to the QoS
   setup request at the other end is the QoS destination. During the QoS
   negotiation, data may still flow on the connection on a best-effort
   basis.  Cells that belong to the QoS negotiation message are marked
   with a Payload-Type Indicator (PTI), possibly as RM cells, so that
   it may flow to the controller on the switch.  Thus, in fact, QoS
   signaling and data cells (or messages) may be interleaved because of
   the PTI value being distinct from one another.

   Figure 6:  Three Way QoS Setup.

   Various alternatives for detailed QoS negotiation can be considered
   here, including the conventional three way setup described in Figure
   6, and one which is consistent with the RSVP-like signaling proposed
   for IP networks.  With reference to Figure 6, the QoS request is
   multicast to all switch controllers in the path and to the next link
   at each switch, facilitating parallel processing in the controllers
   (1).  The Commit message traverses the reverse path, slaloming to
   every controller, collecting the commitments (2).  The QoS Ack.
   multicasts the commitment to all controllers (3).

   In UNITE a QoS request may be initiated by any participant of a
   multicast, the core (if present), source or a leaf.  Moreover,
   unless otherwise dictated by higher level policies, core/source and
   leaf initiated QoS may all be used at different times for the same
   multicast.  As an illustration of the potential of UNITE, we describe
   the case of Leaf Initiated QoS request by referring to Figure 7.

   The leaf initiated QoS requests carry the demand from the receivers
   upstream.  When the QoS request arrives at a switch, the demand is
   noted at the switch.  The switch conveys upstream, the maximum of all
   the demands observed from the different branches (a leaf node or a
   switch may be at the end of the branch).  Note that different leaves
   may issue their QoS requests at different times.  The switch examines
   each QoS request and transmits a request upstream only if the QoS
   request is higher than the current maximum.  When the demands arrive
   at the core/sender, the permit returned is the minimum of the offered
   capacity, the demands received from the leaves and the available link
   capacity.  Note that each switch needs to maintain state, which is



Ramakrishnan et al.            Expires February 1999           [Page 12]

Internet Draft                     UNITE                     August 1998



   the demand and the permit returned for each branch for the multicast
   call.  The leaf may be a source or a receiver, requesting a QoS on a
   shared distribution tree (e.g., CBT).

   Figure 7:  Multicast QoS. Leaf initiated QoS, a) demand phase, b)
   permit phase.



5. Interoperability Issues

   In this section, we describe how to use UNITE with existing switches
   including software based implementations and FIFO switches.



5.1. Interoperability with existing Switches

   The proposed UNITE protocol discussed in Section 2 assumes that
   switches will be able to do per-VC queueing and furthermore will be
   able to handle the processing of the Marker in- band.  Processing of
   the Marker involves changing the state of the per-VC queue so that
   arriving cells are buffered rather than dropped.  (This ensures that
   valid data cells, that might follow the marker back-to-back, will
   be queued, while any invalid cells, e.g.  cells in flight from an
   erroneous connection, will be dropped.)  Current ATM switches do not
   necessarily provide these capabilities, however, and it is crucial
   that UNITE can still function on such legacy switches.  An extra (but
   optional) Marker-Acknowledge message is introduced to deal with these
   issues.

   If a switch is processing the Marker in software, it cannot guarantee
   that queue state will change from discard to queueing in time to
   cater for valid data cells following the Marker.  In fact because of
   different switch architectures and implementations, the amount of
   time it takes to process the Marker in software will vary greatly.
   An upstream node therefore has no way of knowing how long to delay
   before it can start forwarding data cells.  By having a downstream
   node sending the Marker-Acknowledge message only when it is ready
   to receive data from the upstream node, this problem is solved.  An
   illustration of the optional use of the Marker-Acknowledge is given
   in Figure 9.  A downstream node indicates in the micro-Acknowledge
   message whether it requires the upstream node to wait for a
   Marker- Acknowledge or not.  The Marker-Acknowledge mechanism can
   therefore be used on a hop- by-hop basis as dictated by local switch
   capabilities.  When a downstream node has requested the use of the



Ramakrishnan et al.            Expires February 1999           [Page 13]

Internet Draft                     UNITE                     August 1998



   Marker-Acknowledge message, the upstream node starts a timer when
   it sends the Marker downstream.  This timer is cleared when the
   Marker-Acknowledge message is received from downstream, or, if the
   timer expires the Marker is retransmitted.  Note that the penalty
   for using the Marker-Acknowledge is two round-hop worth of delay as
   opposed to one round hop in the ideal case.  The hop-by-hop nature of
   the original protocol is however maintained

   The Marker-Acknowledge message is also used to cater for FIFO
   switches, as illustrated in Figure 10.  A FIFO switch will not
   be able to buffer data cells until it receives an acknowledgment
   from downstream.  (Indeed some FIFO switches might not even be
   able to accept cells into the switch without having received the
   outgoing VCI from the downstream switch.)  A FIFO switch will then
   simply delay sending the Marker-Acknowledge until it is capable of
   forwarding data cells.  This in itself is however not enough.  If
   the upstream switch is itself a FIFO switch, then the second FIFO
   switch has to also indicate to the upstream switch that it should not
   send a Marker-Acknowledge message upstream until it has received a
   Marker-Acknowledge message from downstream.  (In the non-FIFO case
   described above, a switch can send a Marker-Acknowledge message
   upstream, as soon as it is capable of receiving data.  If both the
   upstream and the downstream switches are FIFO switches, however, the
   upstream switch should wait until the downstream switch is capable of
   receiving data.)  The Acknowledgment message therefore also needs to
   indicate to the upstream node whether it should wait for a downstream
   Marker-Acknowledge, before it can send its Marker-Acknowledge
   upstream.  (If the upstream node is not a FIFO switch and is capable
   of buffering data, it can simply ignore this indication in the
   Acknowledgment message.)

   Figure 9:  Use of the Optional Marker Acknowledge to enable
   downstream switches to control upstream switch transmission of data
   until it is ready

   This approach has the effect of changing the hop-by-hop delay of the
   UNITE protocol into a partial end-to-end delay across consecutive
   FIFO switches.  (A setup request will proceed, with data following,
   on a hop-by-hop basis until a FIFO switch or switches are reached.
   Forwarding of data will then be delayed until the last FIFO switch in
   the sequence is ready to receive the data.)



Ramakrishnan et al.            Expires February 1999           [Page 14]

Internet Draft                     UNITE                     August 1998



6. CONSIDERATIONS ON UNITE IMPLEMENTATION.

   The fundamental features of UNITE, namely, the separation of
   connectivity from full QoS processing, the specification of single
   cell signaling messages and the simplified reliability support via
   timers and retransmission of basic messages, enable a broad range of
   implementation scenarios for UNITE.

   At one extreme, UNITE may be implemented completely at the software
   level.  The only functionality required at the hardware level is the
   ability to recognize in-band control cells used for UNITE signaling
   arriving at the switch ports, and to route such cells to the switch
   controller.  In the most basic software implementation per VC queuing
   would not be required, and early data transmission (before end-to-end
   acknowledgment) may not be supported.  We believe that, while the
   full latency improvement potential of UNITE is not achieved with
   such an implementation, significant improvement in call processing
   capacity as well as fairness improvements may indeed be achieved.

   Figure 10:  Use of the Marker Ack with a sequence of FIFO switches.

   At the opposite extreme in the range of implementations of UNITE
   is the scenario where as much call processing functionality is
   implemented in the hardware, most likely located in the switch line
   cards and host adapters, and the switch supports advanced queuing and
   scheduling schemes.  In this scenario the full potential of UNITE
   may be manifested, with close to a single hop round trip latency
   before the inception of data transmission, and vast call set up
   capacity increases for best effort or basic QoS calls.  Such capacity
   increases are naturally scaleable with the switch port capacity
   and the number of ports on the switch, thanks to the distributed
   nature of the implementation enabled by UNITE. We believe that a call
   processing capacity of several thousand calls per second per OC-3
   port is feasible within a UNITE framework.  Note that even in an
   advanced implementation the full QoS call processing would be handled
   at the software level.

   It is reasonable to conceive a UNITE implementation in which the port
   processing modules on the port cards support the following functions
   in hardware:

   1.  Capture/injection of UNITE signaling cells.



Ramakrishnan et al.            Expires February 1999           [Page 15]

Internet Draft                     UNITE                     August 1998



   2.  Management of timers, retransmissions and state changes in the
   call processing state machine.

   3.  Forwarding of micro-setup to the correct outgoing link, based on
   fast address lookup.

   4.  Allocation of incoming labels (i.e., incoming/outgoing VPI/VCI
   and Tags used for routing through the fabric) out of local label
   pools managed (on longer time scales) by the central switch
   controller.

   5.  Basic QoS support.  This may imply forwarding and
   queueing/scheduling based on a the QoS byte in the micro-setup.

   6.  Control of queue scheduling based on UNITE control messages
   received (e.g., blocking until a message is received).

   A subset of the functionality listed above may also be implemented
   within the Adapter SAR ASIC, namely, signaling control cell
   capture/injection and management of timers, retransmissions and state
   changes in the call processing state machine.  The switch control
   processor would, in this scenario, be responsible for:

   1.  Monitoring and management of label pools allocated in real time
   by the Port Processing Modules.

   2.  Call accounting and monitoring.

   3.  Switch level resource management.

   4.  Full QoS call processing, including Call Admission Control and
   support of sophisticated bandwidth reservation schemes and management
   of appropriate scheduling schemes.

   5.  Initialization, monitoring of error conditions and switch level
   management.  A range of UNITE implementations falling in between the
   extremes described above can naturally be conceived, including the
   case of the current generation of switches supporting per VC queuing
   in the switch fabric, but still handling UNITE control cells in
   software.  Large signaling performance advantages could be gained in
   this case by exploiting the early data transmission feature of UNITE.



Ramakrishnan et al.            Expires February 1999           [Page 16]

Internet Draft                     UNITE                     August 1998



   In order to explore the implications of a basic UNITE software
   implementation we developed a UNITE prototype completely in software
   over a network including two ATM switches and two adapter cards.
   A picture of the prototype setup is shown in Figure 11.  In the
   prototype in-band UNITE signaling cells are supported by OAM cells.
   To evaluate the performance improvement with UNITE, we compared the
   performance of UNI 3.0 versus the prototype UNITE implementation.
   The tests used a mature UNI 3.0 implementation, a Radcom test box
   acting as a source and destination, and a Cabletron ATM Switch.  The
   UNITE tests used two Sun workstations with Cabletron/Zeitnet ATM
   adapters.

   One important issue for connection-oriented communication is the
   amount of memory used to keep state for each individual connection.
   At least comparatively, UNITE is efficient in using memory for VC
   state, using only 128 bytes per best-effort VC in our prototype.  In
   contrast, UNI uses almost 1500 bytes for a best-effort VC. Thus,
   there is the potential for UNITE to support a much larger number of
   VCs on switch ports.

   We measured the UNITE connection setup latency and throughput.
   Our preliminary results, using 100 microsecond clock granularity
   in our measurements, were as follows:  The best effort connection
   setup latency through an individual switch was 1.7 ms with UNITE.
   In comparison, a best-effort UNI connection took 10.9 milliseconds.
   The various components of this service time are shown in Table 1.
   In terms of throughput, UNITE got approximately 700-800 calls/sec,
   while with UNI we got approximately 130 calls/sec.  We believe that
   with some simple optimizations, UNITE could easily get over 1000
   calls/sec.  We expect that even more substantial improvement could be
   achieved with UNITE with a streamlined/hardware implementation.



7. BENEFITS OF UNITE

   In this section we summarize and reorganize, as a quick reference,
   the benefits of UNITE discussed in this internet draft.

   1) Separating connectivity from QoS enables UNITE to:

   a) Achieve high throughput for establishing connections.



Ramakrishnan et al.            Expires February 1999           [Page 17]

Internet Draft                     UNITE                     August 1998



   b) Have a very low latency to begin data transmission because we dont
   have to wait for an end-end message exchange.

   c) Have throughput and latency for connection establishment be
   independent of the complexity of the QoS class and other service
   characteristics.  Complex QoS specifications are allowed for those
   connections that need it.

   d) Support QoS establishment and renegotiation in a similar fashion.
   This enables simple ways to change parameters for flows.

   e) Allow for communication on a best-effort basis even upon failure
   of a QoS request.

   2) UNITE is ideally matched to carry Internet traffic over ATM
   networks.

   3) UNITE is optimized for distributed hardware implementation of
   signaling within a switch on a per-port basis.

   a) A single cell, fixed length, fixed format micro-setup allows for
   high-speed processing of the setup message.

   b) No single point of re-sequencing or SAR is needed, and no software
   stack such as SSCOP is required for supporting basic connection
   establishment.

   c) Reliability is achieved via simple timers and retransmissions that
   are easily implemented in hardware.

   d) State and context information for connectivity requires only a
   small amount of memory and can be kept in a distributed fashion, even
   on a line card.

   4) Separation of connectivity and QoS and sending QoS related
   messages in-band allows the network to have QoS setup initiated by
   sources or destinations.

   5) UNITE provides isolation of QoS negotiation to connections that
   require it



Ramakrishnan et al.            Expires February 1999           [Page 18]

Internet Draft                     UNITE                     August 1998



   a) Multiple passes, complex QoS negotiation or other service
   characteristics may be allowed.

   6) Supports a full range of multicast architectures, including
   multipoint-to-multipoint.

   a) QoS can adapt to the capabilities of the branches of the tree.

   7) Builds on the QoS work done for ATM and IP.

   a) Accommodates a wide range of QoS models:  UNI, RSVP and future
   evolution.

   9) Builds on substantial amount of the work done for PNNI for
   QoS-sensitive routing.

   10) Allows communication on a path selected based on a coarse class
   specification.  Hence even simple connectivity can be better than
   true best-effort.

   11) Inter-works with existing UNI switches.

   12) Allows for legacy switches and various levels of hardware
   implementation complexity.



8. SUMMARY

   We have described a protocol for lightweight signaling.  The key idea
   behind the protocol, is the separation of connection establishment
   and QoS processing.  This makes connection setup independent of QoS
   processing complexity, benefiting most flows and best effort flows
   in particular, as the channel becomes immediately available on best
   effort basis.  The separation allows all flow specific signaling,
   i.e., QoS messages to be carried in-band, thus protecting the shared
   signaling resources.  UNITE signaling unifies initial QoS setup and
   renegotiation, and supports source/core initiated QoS as well as
   receiver initiated QoS requests.

   The connectivity setup message is a single ATM cell (called
   micro-setup).  The micro-setup and micro-acknowledgment are exchanged



Ramakrishnan et al.            Expires February 1999           [Page 19]

Internet Draft                     UNITE                     August 1998



   on a hop-by-hop basis on a signaling channel. By incorporating a
   minimal QoS class identification in the micro-setup request, UNITE
   has the ability to provide QoS sensitive routing.  Data flow on
   the best-effort VC may begin without waiting for the setup to be
   completed over the entire end-end path.  The use of per-VC queuing
   permits the source to start sending data on a best-effort basis as
   soon as the connection has been setup on the next hop.

   Subsequently, QoS setup requests and acknowledgments flow in-band on
   the best-effort VC that is initially setup.  The QoS for the flow
   is assured upon completion of the end-end exchange of the QoS setup
   and ack.  The complexity of QoS messages and their processing is
   isolated to those VCs requiring it, without impacting other VCs.  In
   addition, it allows for the QoS requester to be either the source or
   destination of the connection.  The architecture recognizes the need
   for multipoint-to-multipoint connections, and the possible need for
   variable QoS to different participants in the multicast group.



9. Applicability OF UNITE to MPLS

   As we stated earlier, this Internet Draft is based on an ATM Forum
   contribution and as such is written within an ATM context.  However,
   we believe that UNITE might also be of value within the context of
   MPLS and have therefore decided to present it to the MPLS working
   group to solicit feedback.

   Clearly, UNITE currently uses ATM addresses, to be applicable to
   ATM. However, we believe that the protocol could be used with IP
   addresses, with hop-by-hop forwarding of the micro-setup at the
   MPLS switches using conventional link-state routing, such as OSPF.
   Because of the inclusion of the QoS class in the micro-setup, we
   can take advantage of potential enhancements to IP to accommodate
   QoS-sensitive routing.

   We believe that UNITE might be applicable to the following objectives
   of the MPLS working group.  We reiterate below these specific
   objectives:

   1.  Specify standard protocol(s) for maintenance and distribution
   of label binding information to support unicast destination-based
   routing with forwarding based on label-swapping.



Ramakrishnan et al.            Expires February 1999           [Page 20]

Internet Draft                     UNITE                     August 1998



   2.  Specify standard protocol(s) for maintenance and distribution
   of label binding information to support multicast routing with
   forwarding based on label-swapping.

   4.  Specify standard protocol(s) for maintenance and distribution of
   label binding information to support explicit paths different from
   the one constructed by destination-based forwarding with forwarding
   based on label-swapping.

   6.  Specify a standard way to use the ATM user plane

   a) Allow operation/co-existence with standard (ATM Forum, ITU, etc.)
   ATM control plane and/or standard ATM hardware

   b) Specify a label swapping control plane

   c) Take advantage of possible mods/improvements in ATM hardware, for
   example the ability to merge VCs

   7.  Discuss support for QOS (e.g.  RSVP)

   UNITE is a framework that is efficient in providing connectivity,
   with very low latency for a source to begin transmitting data
   when using on-demand label distribution.  An integral part of the
   framework is providing very flexible support for QoS, accommodating
   multiple QoS models including sender and receiver initiated QoS.
   Multicast support fits naturally in UNITE, with common procedures
   applicable for unicast and multicast (both source and receiver
   joins).  UNITE allows the network to scale to large numbers of nodes
   because of the ability to support on-demand label distribution
   efficiently.  Further, UNITE achieves scalability in the following
   dimensions:

   1.  UNITE can achieve high throughput for label distribution.

   2.  UNITE enables initiation of packet forwarding with low latency.

   3.  UNITE minimizes the amount of state needed in the network.

   UNITE uses a QoS hint to route the setup.  The explicit path
   established in this manner may therefore be different from



Ramakrishnan et al.            Expires February 1999           [Page 21]

Internet Draft                     UNITE                     August 1998



   "default" destination based forwarding because it uses QoS sensitive
   destination based routing.

   As it is currently defined, UNITE is an ATM control protocol and is
   therefore directly applicable to objective (6).

   UNITE also directly addresses QoS issues without making any
   assumptions about the specific QoS model that is used.  For example,
   RSVP can be combined with UNITE to perform the actual resource
   reservations.

   If particular MPLS switches do not support native IP forwarding,
   then the need for UNITE appears even more compelling, especially in
   an environment where services other than best- effort are provided.
   (e.g.  in a Diffserv type of environment, it would be wasteful to
   distribute labels for all service classes across the whole network).



10. References

   [i] L. Zhang, S. Deering, D. Estrin, S. Shenker, RSVP: A New Resource
   ReSerVation Protocol, IEEE Network Magazine, Sept.  1993.

   [ii] Danny J. Mitzel, Deborah Estrin, Scott Shenker, and Lixia
   Zhang, An Architectural Comparison of ST-II and RSVP, Proceedings of
   Infocom94, 1994.

   [iv] M. Grossglauser & K.K. Ramakrishnan (1997) SEAM: Scalable and
   Efficient ATM Multicast, Proceedings of IEEE Infocom'97, April 1997,
   Kobe, Japan.

   [v] T. Ballardie, P. Francis, and J. Crowcroft, Core Based Trees
   (CBT), in Proc.  ACM SIGCOMM'93, San Francisco, California, September
   1993.

   [vi] S. Deering, D. Estrin, D. Farinacci, V. Jacobson, C. Liu, and L.
   Wei, An Architecture for Wide-Area Multicast Routing,' in Proc.  ACM
   SIGCOMM'94, London, August 1994.

   [vii] ATM Forum, ATM User-Network-Interface (UNI) Signaling
   specification version 4.0, July 1996.



Ramakrishnan et al.            Expires February 1999           [Page 22]

Internet Draft                     UNITE                     August 1998



   [viii] S. Deering, Multicast Routing in Internetworks and Extended
   LANs, in Proc.  ACM SIGCOMM'88, Stanford, California, August 1988.



Authors' Addresses

   K. K. Ramakrishnan, Gili Hjalmtysson, Kobus Van der Merwe
   AT&T Labs.  Research
   180 Park Avenue, Florham Park, N.J. 07932
   kkrama@research.att.com,gisli@research.att.com,kobus@research.att.com
   Phone:+1 973 360 8766
   Fax:  +1 973 360 8871



   Flavio Bonomi, Sateesh Kumar, Michael Wong
   CSI ZeitNet/Cabletron
   5150 Great America Parkway, Santa Clara, CA, 95054
   fbonomi@ctron.com, skumar@ctron.com, mwong@ctron.com
   Phone:  +1 408 565 9360
   Fax:  +1 408 565 6501



Ramakrishnan et al.            Expires February 1999           [Page 23]