Internet Draft Internet Draft A. Fredette Expires May 1998 Bay Networks, Inc. C. White Bay Networks, Inc. Loa Andersson Ericsson Telecom Paul Doolan Ennovate Networks November 1997 Stream Aggregation <draft-fredette-mpls-aggregation-00.txt> Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as ``work in progress.'' To learn the current status of any Internet-Draft, please check the ``1id-abstracts.txt'' listing contained in the Internet-Drafts Shadow Directories on ds.internic.net (US East Coast), nic.nordu.net (Europe), ftp.isi.edu (US West Coast), or munnari.oz.au (Pacific Rim). Abstract This document describes issues with aggregating streams of data in an MPLS system and suggests alternatives for increasing the level of aggregation possible. 1. Introduction Fredette, White, Andersson, Doolan [Page 1] INTERNET-DRAFT Stream Aggregation Expires May 1998 One of the important areas for the application of MPLS is in ATM switching hardware. It has been recognized that non-VC-merging ATM switches (the most common in existence today) require a significantly larger number of labels than merge-capable switches. This number can be even larger than many have realized if stream aggregation is not applied intelligently. In Section 3, we examine the different levels of stream aggregation that are possible. Then, in Section 4, we discuss possible solutions for achieving the highest possible level of aggregation. 2. Terminology Terms used in this document are aligned as much as possible with definitions provided in the MPLS Framework document [MPLS-F]. Flow A single instance of an application to application flow of data (as in the RSVP and IFMP use of the term "flow") Stream An aggregate of one or more flows, treated as one aggregate for the purpose of forwarding in L2 and/or L3 nodes (e.g., may be described using a single label). In many cases a stream may be the aggregate of a very large number of flows. Synonymous with "aggregate stream". merged stream An aggregate of one or more streams such that the decision to aggregate is local to the merging LSR. aggregated stream An aggregate of one or more streams such that the decision to aggregate is made by some downstream LSR. MPLS domain a contiguous set of nodes which operate MPLS routing and forwarding and which are also in one Routing or Administrative Domain merge capable An LSR capable of merging frames received with different labels onto the same outgoing interface with the same label. The ATM Fredette, White, Andersson, Doolan [Page 2] INTERNET-DRAFT Stream Aggregation Expires May 1998 specific case, VC- merge capable, is the capability of an ATM switch to merge cell streams without interleaving cells from different AAL5 frames. Port Logical port that represents an ingress to an LSR. A single physical interface may support multiple logical ports. Note, in this document, the term stream may refer either to a single flow or an aggregate of multiple flows or streams. A flow, however, can never refer to a stream that is an aggregate. The difference between a merged stream and an aggregated stream is subtle. In particular, the term "aggregated stream" is somewhat overloading the word aggregate. For the purpose of this document, the terms "aggregate" and "aggregation" are used only in reference to "aggregated streams", and are not used in reference to merged streams. 3. Overview It is useful for an MPLS network to aggregate streams that share the same ingress-egress pair through a routing domain. For example, consider the network in Figure 1 S1--A-------+ | +-D1 | | C-------D-+ | | | +-D2 S2--B-------+ Figure 1: Sample MPLS Network MPLS is used on internal links. Given sources S1 and S2 and destinations D1 and D2, consider four best effort traffic streams: - S1-A-C-D-D1 - S1-A-C-D-D2 - S2-B-C-D-D1 - S2-B-C-D-D2 Fredette, White, Andersson, Doolan [Page 3] INTERNET-DRAFT Stream Aggregation Expires May 1998 All four streams use LSR D as the egress LSR of the MPLS domain. Given that all four streams share the same stream descriptor (e.g., they are all best effort), from the perspective of MPLS, it is irrelevant that D1 and D2 are separate destinations. In this figure, relative to these four streams, LSRs A and B are Ingress LSRs and LSR D is an Egress LSR. We assume that LSRs A and B are merge capable. Stream aggregation allows streams that share a common Ingress LSR (or set of Ingress LSRs), Egress LSR and FEC to be aggregated into an aggregate stream, thus reducing label consumption within the MPLS network. Sections 3.1 - 3.4 describe each of four different possibilities for assigning labels. Note that downstream label allocation on demand is used for the examples, although this is not necessarily a requirement. 3.1 No Merge, No Aggregation In the worst case, none of the internal switches are capable of merging and no aggregation is performed. At each LSR, a different label must be allocated for each label request received from an upstream neighbor. There is no means of determining which label requests can be aggregated. For example, in Figure 1, LSR A makes independent label requests for D1 and D2 to LSR C, its next hop. LSR B makes two similar requests. Since LSR C is unable to perform merge, it requests two different labels for D1 and two different labels for D2 from LSR D. LSR D receives four label requests. From LSR Ds perspective, D1 and D2 are in the same forwarding equivalence class (FEC), but because LSR D is unaware of the history of the requests, it must respond conservatively and assign a different label for each request. In this example, the "history" of a request refers to information that indicates the source of the request as well as the path taken. Without this information, it is not possible for LSR D to distinguish requests originated by LSR A from those originated by LSR B. In networks that do not support merge and do not perform aggregation, the number of LSPs setup can be expressed as: LSPs = O(N^2 * D) N = Number of ingress LSRs = Number of egress LSRs (every ingress is also an egress) Fredette, White, Andersson, Doolan [Page 4] INTERNET-DRAFT Stream Aggregation Expires May 1998 D = Number of destination addresses per egress LSR This follows from the fact that for every route that appears in an ingress LSRs routing table, the LSR must request a separate label, generating a separate LSP. Each ingress LSR will generate D LSPs to each of N-1 egress LSRs. There are N ingress LSRs yielding O(N^2 * D) LSPs for the entire network. A label space of this size can be particularly problematical since D can easily be greater than O(N^2) for a given routing domain. 3.2 No Merge With Aggregation If it is possible to aggregate all the streams that share an LSP, labels can be conserved. This is accomplished by including information with each label request that uniquely identifies the path that the request followed. This allows the egress LSR to assign a single label for all requests that share the same path. In the network shown in Figure 1, LSR A still makes two independent label requests for D1 and D2, but attaches information that "A" is the source of each request. LSR C in turn makes two requests for D1 and D2, but retains "A" as the source of the request. Similarly, LSR B generates two requests which are propagated by LSR C indicating "B" as the source. When LSR D receives the four independent requests, it can assign a single label for the two requests from source "A". Likewise it can assign a single label for the two requests from source "B". In networks that do not support merge but do perform aggregation, the number of LSPs setup can be expressed as: LSPs = O(N^2) N = Number of ingress LSRs = Number of egress LSRs D = Number of destination addresses per egress LSR This is an improvement of O(D) over the previous example without aggregation. 3.3 Merge, No Aggregation When LSRs are capable of merging VCs, the number of separate VCs used in the network is significantly reduced. Let's now assume that LSR C in Figure 1 is capable of merging. Here, no aggregation is Fredette, White, Andersson, Doolan [Page 5] INTERNET-DRAFT Stream Aggregation Expires May 1998 performed. LSR A makes independent label request for D1 and D2 to LSR C. LSR B makes two similar requests. Since LSR C is now able to merge, it requests from LSR D only a single label for D1 and a single label for D2. The egress LSR D then receives only two label requests. If LSR D is unaware of the capabilities of LSR C, LSR D must respond conservatively and assign a different label for each request. In networks that support Merge but do not perform aggregation, the number of LSPs setup can be expressed as: LSPs = O(N*D) N = Number of ingress LSRs = Number of egress LSRs D = Number of destination addresses per egress LSR It should be noted, however, that in a system of fully merge capable LSRs, a downstream LSR may aggregate as it desires as is described in the next section. 3.4 Merge With Aggregation The optimal case is when the network is capable of merging and aggregation is performed. In this case, a single multipoint to point LSP is used for all traffic to the egress LSR, regardless of the final destination beyond the LSR. Let's again assume that LSR C in Figure 1 is capable of merging. LSR A makes two independent label requests for D1 and D2, but attaches information that "A" is the source of each request. LSR B makes two similar requests. Since LSR C is now able to perform merge, it requests from LSR D only a single label for D1 and another for D2, however, since LSR C is capable of merge, it changes the attached information to indicate that "C" is the source of the request (acting like an ingress LSR). When LSR D receives the two requests, it can assign a single label for both requests since they share the same "source" LSR. In networks that support both merge and aggregation, the number of LSPs setup can be expressed as: LSPs = O(N) N = Number of ingress LSRs = Number of egress LSRs Fredette, White, Andersson, Doolan [Page 6] INTERNET-DRAFT Stream Aggregation Expires May 1998 D = Number of destination addresses per egress LSR This is an improvement of O(D) over the previous example without aggregation. In general, aggregation will yield an improvement on the order of the number of destination routes per LSR. Using egress-based aggregation significantly reduces label consumption within the MPLS network. This leads to a best case of one label per egress per hop and a worst case of one label per ingress/egress pair per hop in the MPLS domain. The best case can be achieved by a network fully supporting merge. The worst case is seen when using all non-merge capable LSRs, as this leads to a full-mesh of LSPs between all LSRs. 4. Stream Aggregation An aggregated stream can be described in terms of the Merge LSR, Intermediate LSRs, and the Egress LSR. In Figure 2, the Merge LSR (LSR A) is the first LSR of the aggregated stream and is responsible for merging frames from multiple streams onto the same LSP with the same label. Intermediate LSRs (LSR B) simply forward the data along the LSP and require no special forwarding capabilities. The Egress LSR (LSR C) is the last LSR in the aggregated path and is responsible for de- aggregating the streams from the merged LSP. +-D1 | S1--A-------B-------C-+ | +-D2 Figure 2 An Aggregated Stream: Merge LSR, Intermediate LSRs and Egress LSR A common scenario could be where the Merge LSR is an L3 router and is the ingress LSR to the MPLS domain. Similarly, the Egress LSR could be an L3 router and be the egress LSR from the MPLS domain. The following sub-sections describe several possibilities for achieving stream aggregation. 4.1 Merge LSR "Hides" Aggregation Merge LSR A analyzes the routed path for both D1 and D2 and decides that the LSP to Egress LSR C is the same for both destinations. LSR A requests a label only for D1. The LSP for D1 is set up normally. Fredette, White, Andersson, Doolan [Page 7] INTERNET-DRAFT Stream Aggregation Expires May 1998 LSR A forwards data destined for D2 using the label assigned for D1. Note that since the routing protocol boundary need not coincide with the MPLS boundary, additional information is required in the routing protocol to indicate which hops are MPLS LSRs such that when MPLS analyzes the routed path, it can determine the extent of the MPLS path. With OSPF, this could be accomplished by adding an opaque LSA that is advertised by all LSRs. This method hides the fact that D2 data is being aggregated onto the same LSP used for D1. LSR A is the only node that is aware of any aggregation taking place. This approach defeats the capability of Egress LSR D to assign different labels for D1 and D2. When downstream allocation is used, LSR D assigns a label for D1 and may reasonably assume that only D1 traffic will arrive with this label. Traffic for D2 will be unexpected and may inadvertently be dropped as an error. 4.2 Merge LSR Aggregates to Egress LSR In an approach similar to the one described in Section 4.1, each LSR in a given OSPF area can request an LSP to each other LSR in the area. Then, it can use its link-state database to determine which routes should be forwarded to which egress LSR. This approach is preferable the one described in Section 4.1 because the LSP ends at the router and is not overloaded. For example, in Figure 2, LSR A requests labels for LSRs B and C. Then, it uses its link-state database to determine that LSR C is the egress for D1 and D2. After making this determination, LSR A can use the label for C on packets destined for D1 and D2. One may note that this approach is similar to the one described in [ARA] and [HEIN]. 4.3 Merge LSR Groups Requests Again, Merge LSR A analyzes the routed path for both D1 and D2 and decides that the LSP to Egress LSR C is the same for both destinations. LSR A groups label requests for D1 and D2. Intermediate LSR B maintains this grouping until Egress LSR C receives the grouped request. LSR C assigns a single label for all requests in the same group. This approach allows the egress to choose whether to reply with the same label for the two destinations. The major problem with this Fredette, White, Andersson, Doolan [Page 8] INTERNET-DRAFT Stream Aggregation Expires May 1998 approach is the overhead involved in dealing with topology changes. Each topology change that affects the members of a group requires that a new request be generated containing the all members of the group. For example, if D2 were to become available at a different Egress LSR, the original grouping with D1 would have to be re- established with only D1. This is in addition to establishing a group for D2 to the new Egress LSR. 4.4 Merge LSR ID in Requests The key issue in aggregating multiple requests is the ability to identify groups of requests that follow the same path, and can thus be assigned to the same LSP. The previous sections describe methods to solve this with brute force by having the Merge LSR determine and maintain the groups. Use of the Merge LSR ID, and the next section use of a Merge ID rely on the Egress LSR to determine which requests can be grouped. The burden of maintaining the grouping throughout topology changes is distributed among all LSRs in the path. Information could be added to each label request that would uniquely identify the merge LSR. This would allow the Egress LSR an indication about which requests can be aggregated. For example, in the network in Figure 3, LSR A generates a label request for D1 and another for D2, attaching "A" as the Merge LSR ID. LSR C (not merge capable) propagates these requests on to LSR D, leaving the Merge LSR ID as "A" for both requests. LSR D can easily aggregate the two requests since they share the same Merge LSR ID. S1--A-------+ | +-D1 | | C-------D-+ | | | +-D2 S2--B-------+ Figure 3 Using Merge LSR ID for Flow Aggregation Since the Merge LSR ID is attached to each label request independently, topology changes are easily handled. If a new destination D3 appeared off of D, LSR A would generate a single request for D3 with "A" as the Merge LSR ID. LSR C propagates the request to LSR D who can determine that the same label used for the previous requests from Merge LSR A can be used. This approach suffers in that it only identifies unique Merge LSRs and not unique paths from a given Merge LSR. As shown in more detail in the next section, if packets from a given Merge LSR to a given Fredette, White, Andersson, Doolan [Page 9] INTERNET-DRAFT Stream Aggregation Expires May 1998 Egress LSR can take different paths, merging can be inappropriately applied and cell interleaving can occur. This approach, therefore, is not adequate. 4.5 Merge Identifier (MID) in Label Binding Requests A slight modification to the previous algorithm solves the problem of identifying unique paths from a Merge LSR to an Egress LSR. A Merge Identifier (MID) is used to identify unique paths that traverse a given link. Similar to labels, the MID only has local significance between two LSRs and is mapped to a different MID at each hop on a path. MID values are assigned by the upstream neighbor and are included with each label request. As a label request progresses from Merge LSR to Egress LSR, the MID is mapped at each intermediate hop. At any given link, all requests with the same MID are guaranteed to share the same path from the same Merge LSR. Thus, at the Egress LSR, all requests sharing the same MID can be safely aggregated onto the same LSP. Figure 4 shows a possible MID mapping from two Merge LSRs, A and B, to a common Egress LSR D. Note that in this figure, LSR C is not capable of merge, and thus is an Intermediate LSR. MID=0 S1--A-------+ | +-D1 | MID=3 | C-------D-+ | MID=2 | | +-D2 S2--B-------+ MID=0 Figure 4 MID mappings to indicate unique Merge LSRs Each label request that LSR A originates includes an MID=0 (Null). When LSR C receives a label request from A that must be propagated to LSR D, LSR C allocates a unique MID=3 and mapsto . It propagates the request to D with MID=3. LSR B uses MID=0, LSR C maps to . Label requests received at D will have either an MID=3 (corresponding to LSP A-C-D) or MID=2 (corresponding to LSP B-C-D). Regardless of the number of label requests generated by LSR A, LSR D can use the same label as long as the traffic descriptor for the requests is the same. Fredette, White, Andersson, Doolan [Page 10] INTERNET-DRAFT Stream Aggregation Expires May 1998 The need for MID mappings is clear when considering multiple paths from the same Merge LSR to the same Egress LSR. This may result, for example, from multiple equal cost paths generated by the routing protocol yielding a different next hop for different prefixes. Figure 5 illustrates two separate paths from Merge LSR A to Egress LSR E: A-B-D-E and A-C-D-E. If the Merge LSR ID approach from Section 4.4 were used, LSR E would incorrectly assume that it could use the same label for both requests. Then, when the streams are merged at LSR D, cell interleaving can occur. It is important to note here that a similar problem can occur when using VP merge. In a commonly discussed approach, each ingress LSR has a unique VCI that is uses in the transmission of frames on all VPIs. If the multipath condition described above occurs, cell interleaving can occur at LSR D. MID=0 MID=5 +-------B-------+ | | +-D1 | | MID=3 | S1--A D-------E-+ | | MID=2 | | | +-D2 +-------C-------+ MID=0 MID=6 Figure 5 MID mappings to indicate unique paths from same Merge LSR In order for the egress to properly aggregate streams from LSR A, Egress LSR E must be able to discern which path a request used. Using only the Merge LSR ID in the request would result in LSR E assigning the same label for all requests from LSR A. This results in LSR D using the same outgoing label for streams from two different incoming interfaces. As LSR D is not merge capable, it would incorrectly merge/interleave cells from frames received from LSRs B and C. 4.5.1 MID Tree The mapping of MID values from Merge LSR to all possible Egress LSRs forms a point-to-multipoint tree rooted at the Merge LSR. The leaves of the tree are the Egress LSRs. All other LSRs are Intermediate LSRs and are not merge capable. The path from the Merge LSR to any Egress LSR on the MID tree is the same as a valid routed data path. For any link on the MID tree, the MID value used corresponds directly to a unique path from the Merge Fredette, White, Andersson, Doolan [Page 11] INTERNET-DRAFT Stream Aggregation Expires May 1998 LSR. On any given link the number of MID values in use represents the number of unique paths from all Merge LSRs that cross this link. 4.5.2 Merge Capable LSRs In the event that an LSR in the middle of the MPLS domain is merge- capable, the LSR simply acts as an Egress LSR to upstream neighbors and as a Merge LSR to downstream neighbors. For example, in the previous example (shown in Figure 5), if LSR D were merge-capable, it would perform aggregation of requests from upstream neighbors B and C. It would use a MID=0 for requests to LSR E allowing E to use a single label. Figure 6 demonstrates the resulting LSPs and MID mappings. MID=0 MID=5 +-------B-------+ | | +-D1 | | MID=0 | S1--A D-------E-+ | | MID=0 | | | +-D2 +-------C-------+ MID=0 MID=6 Figure 6 Merge LSRs in the middle of an MPLS domain Note that each Merge LSR only uses single MID per interface (assuming it is Merge LSR for all LSPs traversing this LSR). Since the interfaces are independent, the same MID can be used for each interface. In this case, the NULL MID of 0 can always be used. In a network of only Merge- capable LSRs, the MID can always be left as NULL. Similar to label mappings, mapping of MIDs must include port information as well. Thus the tuple maps to . 4.5.3 Operation of Merge LSRs The Merge LSR is the first LSR in the aggregate stream. It makes no decisions regarding which outgoing streams can be aggregated. There is no snooping of the routing table and no changes to the routing protocol. Fredette, White, Andersson, Doolan [Page 12] INTERNET-DRAFT Stream Aggregation Expires May 1998 Operating conservatively, the Merge LSR makes independent label requests just as it would without considering stream aggregation. However, for each request generated, it attaches a MID=0, indicating to the downstream LSR that it is acceptable to aggregate the stream associated with this request with any other previous binding. (Note that for an LSR operating as a Merge LSR for some LSPs and as an Intermediate LSR for others, the MID would not always be NULL.) When label bindings are received from downstream, the Merge LSR simply incorporates the bindings into the interfaces label information base (LIB). If the binding is in response to a previous request (downstream allocation on demand), the stream to which the binding applies and the MID of the binding must match that of the corresponding request. Any stream aggregation is performed downstream and is only noticeable at the Merge LSR by multiple binding using the same label. 4.5.4 Operation of Intermediate LSRs Intermediate LSRs are, in general, not merge capable. With regard to the aggregated stream, the Intermediate LSR will only bind the same label for multiple upstream streams (on the same incoming interface) when the streams are assigned the same label by the next downstream LSR. As with the Merge LSR, all label requests are independent. For each label request received on an incoming interface, a label request will be generated and sent to the next hop according to the local LSR's routing table. A unique MID is attached to each request sent downstream for each unique tuple. Label bindings received from downstream LSRs will contain a stream (e.g., a route prefix), the assigned label and a MID. Flow aggregation performed downstream is noticeable at the Intermediate LSR by multiple streams using the same label and the same MID. It is an error if the same label is used for two streams that do not have the same MID. When allocating a label for a new stream to an upstream LSR, the Intermediate LSR may use a label in use for an existing stream (or streams) on the same incoming interface if the following conditions are met: - The incoming MID of the new stream matches the incoming MID of the existing stream(s) Fredette, White, Andersson, Doolan [Page 13] INTERNET-DRAFT Stream Aggregation Expires May 1998 - The new and existing streams use the same next hop combined with condition 1, this implies that the outgoing MID of the new and existing streams is the same - The same label has been assigned by the downstream LSR for the new and existing streams If any of these conditions are not met, the Intermediate LSR must allocate a new label for the upstream neighbor. If at a later time the conditions are satisfied, the Intermediate LSR may release the previous binding and bind the stream to the label of the exiting stream(s). 4.5.5 Operation of Egress LSRs The Egress LSR is responsible for determining whether multiple streams may be aggregated. All streams that share the same MID are eligible for aggregation. Again, all label requests are independent. Each label request received from upstream will contain an MID. When allocating a label for a new stream to an upstream LSR, the Egress LSR may use a label in use for an existing stream (or streams) on the same incoming interface if the following condition is met: - The incoming MID of the new stream matches the incoming MID of the existing stream(s) If this condition is not met, the Egress LSR must allocate a new label for the upstream neighbor. If at a later time, the condition is satisfied, the Egress LSR may release the previous binding and bind the stream to the label of the exiting stream(s). 4.5.6 Reserved MID Values It may be desirable to designate certain MID values or ranges of values to be used for special circumstances. 4.5.6.1 NoAggregateValue When setting up a point-to-point stream, it may be desirable for a separate LSP to be setup that is not aggregated with any other streams. A special MID=NoAggregateValue indicates to Egress LSRs that a separate label should be allocated for this stream and not used for Fredette, White, Andersson, Doolan [Page 14] INTERNET-DRAFT Stream Aggregation Expires May 1998 any other stream. Possible uses include: - RSVP streams - Dedicated LSPs for control traffic 4.5.6.2 UnassignedMID LSRs that allocate and distribute labels for streams before receiving requests may use this MID value to indicate to the upstream LSR that the associated label can be used for any MID value. The downstream LSR assigning the label awaits an acknowledgment from upstream. A positive acknowledgment must include a MID value associated with this binding. Using the UnassignedMID value allows upstream LSRs to make use of a label before it is associated with a MID. It may shorten the setup time as it does not require a full request to the downstream LSR and then a response. 5. Conclusions Four levels of aggregation and merging were described in this document. It was shown that for non-merge capable LSRs, if left unchecked, lack of aggregation can cause an explosion of the label space. Two viable alternatives were proposed: 1) Use a link-state routing protocol to decide at the ingress what streams to aggregate (Section 4.2). 2) Use a merge ID (MID) in label binding requests (Section 4.5). Furthermore, if option 2) is used, it only places an additional burden on non-VC-merge capable switches. 6. References [ARA] R. Coltun, J Heinanen, "The OSPF Address Resolution Advertisement Option", Internet Draft, November 1997, . [HEIN] Heinanen, J., "Intra-area IP unicast among routers over Fredette, White, Andersson, Doolan [Page 15] INTERNET-DRAFT Stream Aggregation Expires May 1998 legacy ATM", Internet Draft, July 1997, [MPLS-A] E. Rosen, A. Viswanathan, R. Callon, A Proposed Architecture for MPLS, Internet Draft , August 1997. [MPLS-F] R. Callon, P. Doolan, N. Feldman, A. Fredette, G. Swallow, A. Viswanathan, A Framework for Multiprotocol Label Switching, Internet Draft <draft-ietf-mpls-framework-01.txt>, August 1997. Author's Address Andre N. Fredette Bay Networks, Inc. 3 Federal St. Billerica, MA 01821 Phone: (978) 916-8524 EMail: fredette@baynetworks.com Christopher J. White Bay Networks, Inc. 3 Federal St. Billerica, MA 01821 Phone: (978) 916-4712 EMail: cwhite@baynetworks.com Loa Andersson Ericsson Phone: + 46 8 719 52 67 email: loa.andersson@etx.ericsson.se Paul Doolan Ennovate Networks 330 Codman Hill Rd Marlborough MA 01719 Phone: 978 263-2002 email: pdoolan@cisco.com Fredette, White, Andersson, Doolan [Page 16]