% Prototype Method Send State Machine % % This state machine is targeted towards packet based systems such as VIA. % Some simplifications may be possible for stream based model like TCP. % % Tasks for this state machine: % % - send message at the head of the queue % % - message data source exists in two forms: % % - provided by user - potentially needing to be packed but guaranteed to be % available when the CAR is enqueued % % - provided by associated recv CAR - may not be (completely) available % until later (i.e., send could stall) % % - track buffers provided by associated recv CAR % % - provide buffers to associated recv CAR % % - enforce flow control % % % This version of the state machine only handles sending from a user provided % buffer. It needs to be expanded to include the forwarding cases. \begin{verbatim} State [Awaiting CAR],[label="Awaiting\nCAR"] Event [CAR Available] Invoke_Action([Decide Msg Type], [label="got CAR"]) Action [Decide Msg Type] // Note: An otherwise eager message may get converted to a // rndv-RTS message if too many outstanding eager messages have already // been sent to the receiving process. if the type is eager // TODO: we may want to convert it to rndv (rts and eventually data) Invoke_Action([Check Buffers for Ctrl]) Action [Check Buffers for Ctrl], [label="Check Buffers\nfor Ctrl Pkt"] // get buffers if we don't already have some // always need a buffer for control information if buffers are not available Change_State([Awaiting Buffers for Ctrl], [label="no bufs"]) else Invoke_Action([Check Flow for Ctrl], [label="bufs ok"]) State [Awaiting Buffers for Ctrl], [label="Awaiting Buffers\nfor Ctrl Pkt"] Event [Buffers Available] Invoke_Action([Check Flow for Ctrl], [label="bufs ok"]) Action [Check Flow for Ctrl], [label="Check Flow Control\nfor Ctrl Pkt"] if flow restricted // all tensed up Change_State([Awaiting Flow for Ctrl], [label="flow\nrestricted"]) else Invoke_Action([Post Send Ctrl], [label="flow ok"]) State [Awaiting Flow for Ctrl], [label="Awaiting Flow\nfor Ctrl Pkt"] Event [Flow Enabled] Invoke_Action([Post Send Ctrl], [label="flow ok"]) Action [Post Send Ctrl], [label="Post Send\nfor Ctrl Pkt"] type = OD.type // pack meta-data into the packet buffer // meta-data may include the envelope information (context, src rank, // and tag) as well as the ready send bit put type in packet buffer if type is short or eager put the envelope info and rsend bit in the packet buffer else if type is rndv-rts message put the envelope and LIBA(OD) into the packet buffer OD.type = rndv-data else if type is rndv-cts put the two LIBAs in the packet buffer else if type is rndv-data put the LIBA in the packet buffer else if type is flow control // TODO // pack data from DD into packet buffer (as much as will fit) if we have // any data to send if type is short, eager, or rndv-data and a DD exists if direct access flag is set point network send descriptor at the user data (which is pinned) else pack data into the packet buffer (from the data descriptor) post the send to the network device if there are no data packets to send (i.e., short) Invoke_Action([Signal CAR Completion], [label="msg complete"]) else if direct access flag is set Invoke_Action([Check Flow for Data], [label="more data,\ndirect send"]) else Invoke_Action([Check Buffers for Data], [label="more data,\nbuffered send"]) Action [Check Buffers for Data], [label="Check Buffers\nfor Data Pkts"] // try to get buffers if we need them if buffers aren't available Change_State([Awaiting Buffers for Data], [label="no bufs"]) else Invoke_Action([Check Flow for Data], [label="bufs ok"]) State [Awaiting Buffers for Data], [label="Awaiting Buffers\nfor Data Pkts"] Event [Buffers Available] Invoke_Action([Check Flow for Data], [label="bufs ok"]) Action [Check Flow for Data], [label="Check Flow Control\nfor Data Pkts"] if flow restricted Change_State([Awaiting Flow for Data], [label="flow\nrestricted"]) else Invoke_Action([Post Send Data], [label="flow ok"]) State [Awaiting Flow for Data], [label="Awaiting Flow Control\nfor Data Pkts"] Event [Flow Enabled] Invoke_Action([Post Send Data], [label="flow ok"]) // TODO: Add buffer packing into flow Action [Post Send Data], [label="Post Send\nfor Data Pkts"] if direct access flag is set point network send descriptor at the user data (which is pinned) else pack data into the packet buffer (from the data descriptor) post the send to the network device // Note: In the implementation, we may want to post multiple packets sends // at one time, amortizing the cost of checking for buffers and checking // flow control over a set of outgoing packets. We are uncertain how // packing multiple buffers then posting multiple sends will affect // peformance, but clearly amortizing costs when sending straight out of // the user buffer will be a win. if there are more data packets to send if we are sending directly from the user buffer Invoke_Action([Check Flow for Data], [label="more data,\ndirect send"]) else Invoke_Action([Check Buffers for Data], [label="more data,\nbuffered send"]) else Invoke_Action([Signal CAR Completion], [label="msg complete"]) Action [Signal CAR Completion], [label="Check/Signal\nCAR Completion"] if we did not send all data directly from the user buffer reset (local) completion flag if we send some of the data directly from the user buffer lock CAR.mutex decrement CAR outstanding activities counter if outstanding activity counter reaches zero set completion flag unlock CAR.mutex else // all data sent using packet buffers set completion flag if completion flag is set decrement completion counter associated with CAR Invoke_Action([Check Ready CAR]) Action [Check Ready CAR],[label="Check for Ready\nSend CAR"] if more CARs are on the send queue Invoke_Action([Decide Msg Type], [label="ready CAR"]) else Change_State([Awaiting CAR], [label="no more\nCARs"]) \end{verbatim}