Go Saga Pattern | 220809


220809

Saga Pattern has been replicated

_2017 It took longer than I wanted to and stole my will a few times, but we’re here now.

Saga Pattern Operational in Go

Okay so there is still some work to be done on it. I still testing it, but I’ll push what I have now shortly.

Saga Pattern for microservices implementation

I am going to try and explain how this how thing works now (Bear with me)

SagaOrchestrator

This is main collector for the Saga it self. Its got a name, and a slice of Transactions

type SagaOrchestrator struct {  
   Name         string  
  Transactions []Transaction  
}

There’s a SagaOrchestrator Constructor that checks the order of the Transactions to insure they are ordered correctly which is important for proper saga execution. I am still testing this now. The count system was picked cause it seemed the most fast to write and easiest to read. I may go back make it so that you have to use the Constructor in the future, but for now I am fine with it cause I pass it around a little

func NewSagaOrchestrator(name string, transactions []Transaction) (*SagaOrchestrator, error) {  
  var sagaCreationErrorPrefix = "[ERROR] [SAGA_PATTERN] "  
  
  so := new(SagaOrchestrator)  
  so.Name = name  
  
  var countOfCompensatableTransactions = 0  
  var countOfPivotTransactions = 0  
  var countOfRetryableTransactions = 0  
  lastIndex := len(transactions) - 1  
  // Loops through the list of transactions to check for acceptable order has been passed  
  for index, transaction := range transactions {  
  
 //Checks for Compensatable Transactions  
 //Ruling:
 // There cannot be a Pivot Transaction prior to any Compensatable Transaction
 // There cannot be Retryable Transactions prior to any Pivot Transactions  
 if transaction.GetTransactionType() == GetCompensatableTransactionType() {  
         countOfCompensatableTransactions = countOfCompensatableTransactions + 1  
         
	  //Checking if it's before any Pivot Transactions  
	  if countOfPivotTransactions > 0 {  
            return nil, errors.New(sagaCreationErrorPrefix + "[SAGA_CREATION] [Compensatable]| Cannot put Compensatable Transactions after a Pivot Transaction")  
         }  
         //Checking if it's before any Retryable Transactions  
	  if countOfRetryableTransactions > 0 {  
            return nil, errors.New(sagaCreationErrorPrefix + "[SAGA_CREATION] [Compensatable]| Cannot put Compensatable Transactions before any Retryable Transaction(s)")  
         }  
     }  
  
 //Checks for Pivot Transaction  
 //Ruling:
 // There cannot be more than one pivot transaction 
 // There are no Retryable prior to the Pivot  
 if transaction.GetTransactionType() == GetPivotTransactionType() {  
         countOfPivotTransactions = countOfPivotTransactions + 1  
	  //Checks that there is not already a Pivot Transaction in the List of Transactions  
	  if countOfPivotTransactions > 1 {  
            return nil, errors.New(sagaCreationErrorPrefix + "[SAGA_CREATION] [Pivot]| There is more than one Pivot Transaction")  
         }  
	  //Checks that there is not any Retryable Transactions prior in the list of Transactions  
	  if countOfRetryableTransactions > 0 {  
            return nil, errors.New(sagaCreationErrorPrefix + "[SAGA_CREATION] [Pivot]| Cannot put Pivot Transaction After Retryable Transactions")  
         }    
      }  
//Checking Retryable  
//Note: currently 220801 that I do not think I need to check for anything for Retryable, cause the other two cover it. 
//but I will leave this code here just in case
if transaction.GetTransactionType() == GetPivotTransactionType() {  
         countOfRetryableTransactions = countOfRetryableTransactions + 1  
}  
//Checks at last Transaction in the list  
if index == lastIndex {  
	//Checks that there is a pivot if there is any Compensatable Transactions  
	if countOfPivotTransactions < 1 && countOfCompensatableTransactions > 0 {  
            return nil, errors.New(sagaCreationErrorPrefix + "[SAGA_CREATION] | Compensatable Transactions have not Pivot Transactions")  
         }  
      }  
   }  
  // So it passes the checks above its good to go.  
  so.Transactions = transactions  
   return so, nil  
}

Transaction

Transaction is an interface cause there are multiple types( Compensatable, Pivot, & Retryable). There are currently three functions that they have to populate to be consider an Transaction. Sense I am using an interface and won’t access to the data if any directly. Cause golang doesn’t implement classes like other languages(there is no extend). I do have methods in which I can implement to return the data I need for the each of the separate sub types(this took much longer for me really understand and internalize. Which was most of time doing this part).

type Transaction interface {  
  GetTransactionType() sagaTransactionType  
  GetTransactionCommands() []TransactionCommand  
  GetTransactionName() string  
}
GetTransactionType()

This gets a type I created in which I can compare during execution to know what type of Transaction it is. As each of them have there own logic. The custom type was made just to make coding it little easier and to insure an extra space/inconstant spelling/etc.. cause I know myself I will mess up and string compare somehow and lose many hours of my life.

  type Transaction interface {  
  GetTransactionType() sagaTransactionType  
  ...
GetTransactionCommands()

This returns a slice of ‘TrainsactionCommands’ Its a list cause the Compensatable Transaction should have two by definition, even if the second one is empty, but it should have two. The others ones only need one, but this is why its a slice and not just the Command. ‘TransactionCommand’ Cause that is workhorse of this implementation of the Saga Pattern

  type Transaction interface {  
  ...
  GetTransactionCommands() []TransactionCommand  
  ...
GetTransactionName()

just returns the name of the Transaction its nice to have when implementing and debugging

TransactionCommand

// TransactionCommand this action part of the Transaction in which Execute would do it and give the response if any.
// this was made to allow both REST and Kafka thus functioning taking an anonymous interface. Doing this allows for both  
// of the possible solutions Actions to be in the same collection  
type TransactionCommand interface {  
  Execute() (success bool, err error)  
  GetTransactionCommandType() string  
  GetTransactionCommandName() string  
}

GetTransactionCommandType() & GetTransactionCommandName()

both just used for debugging. The GetTransactionCommandType() is leftover from earlier prototyping, but I think it could be useful later… maybe.

type TransactionCommand interface {  
  ...
  GetTransactionCommandType() string  
  GetTransactionCommandName() string  

Execute()

This interface method allows for many implementations of it. In which it only needs to return two things a proceed boolean and an error. When going threw this I wanted to allow for just about any type of ‘Command’ as the book puts it and boiled that down in an Execute function that could a HTTP call to another service, Kafka Message, g/RPC, even Service side DB calls, essentially whatever. The Book did not go over this, but its what I took from it. Essentially a list of Actions with a pivot points to undo or complete.

NOTE: I am not positive on the bool, error parms cause I think should be able to just send an error for either and let implementation of Execute deal with which was the main goal of this whole ‘TransactionCommand’. I just don’t want to leave out the option of needing to send an error for some reason, but still wanting to proceed.

type TransactionCommand interface {  
  Execute() (success bool, err error)
  ....  

Saga Execution

A Runner for the SagaOrchestrator its just a simple loop with some logic in it to ad hear to the rules of the saga pattern. The order of Transaction should be confirmed by using the SagaOrchestrator constructor. There are sub functions to deal with undo the necessary Transactions and retrying the ones that allow it. Still testing this logic but it seems to work enough for the write up and any changes will be in the git. I left a bunch of notes/comments in the code to help me write it and remind me when I would come back to it after the weekend.

There are a lot ghost prints in it, but that a future me problem I am still tweaking it in testing.

const RetryableTransactionRetryLimit = 3  
  
type sagaRunner struct {  
   logger             *log.Logger  
  generalErrorPrefix error  
}  
  
//ExecuteSaga  
/*  
  
 Order of Transaction is import in passed ListOfTransactions - Compensatable Transactions: -- NOTE: Compensatable Transactions should have two Transaction Commands. The first one for the intended Transaction and the second one to Roll back that Transaction encase of failure/issues in any of the following Transaction  
 -- Can only be in place prior in order to the Pivot Transaction and never after -- If there is no Pivot Transaction Action then there cannot be any Compensatable Transactions - Pivot Transaction:  
 -- There can only be one per list of Transactions or - Retryable Transactions: -- If the Pivot Transaction exist in the List then Retryable Transactions can only be in after that Pivot Transaction  
 -- Retryable Transactions should never be in the list of Transactions prior to the Pivot Transaction if it exists  
 - NOTE Other Use Cases: -- You can have a Pivot Transaction without Compensatable Transaction(s)  
 -- You can just have list of Transactions without Pivot nor Compensatable Transactions(So just Retryable Transactions (Granted idk why you would tho))*/  
func ExecuteSaga(orchestrator SagaOrchestrator) error {  
   prefixSagaExecutionError := errors.New(prefixSagaExecutionError + "[" + orchestrator.Name + "]")  
   sagaRunnerLogger := sagaRunner{  
      logger:             log.New(os.Stdout, orchestrator.Name+" | ", log.LstdFlags),  
  generalErrorPrefix: prefixSagaExecutionError,  
  }  
   //This is the list of the undo transaction in the event of a failure or  
  var compensatableTransactionCommands []TransactionCommand  
  var hasCompensatableTransaction = false  
  
  sagaRunnerLogger.logger.Println("------- Saga Execution Start --------")  
   for _, transaction := range orchestrator.Transactions {  
      //Pull the List of Transactions Commands  
 // [1] is the intended transaction Command // [2] would be a compensatableTransaction Command  transactionCommands := transaction.GetTransactionCommands()  
  
      sagaRunnerLogger.logger.Println("Transaction Name: " + transaction.GetTransactionName())  
      sagaRunnerLogger.logger.Printf("Transaction Type: %+v\n", transaction.GetTransactionType())  
  
      //Execute the Transaction  
 // To handle result a lil later  canProceed, err := transactionCommands[0].Execute()  
  
      sagaRunnerLogger.logger.Println("--- Transaction Executed Results ---")  
      sagaRunnerLogger.logger.Printf("Can Proceed to Next Transaction: %t\n", canProceed)  
      sagaRunnerLogger.logger.Printf("Was There an Error In that Transaction: %v\n", err)  
      //COMPENSATABLE TRANSACTION  
 //If it's a Compensatable Transaction Type add the second transaction to the list undo Transaction Commands  if transaction.GetTransactionType() == GetCompensatableTransactionType() {  
         hasCompensatableTransaction = true  
  compensatableTransactionCommands = append(compensatableTransactionCommands, transactionCommands[2])  
      }  
        
      //FAILURE IN COMPENSATABLE OR PIVOT TRANSACTION  
 // If get the do not proceed flag or an error (the canProceed is for Transaction Commands that get responses)  if !canProceed || err != nil {  
         // This is just in case you get a Pivot Transaction without any CompensatableTransaction(s) in an effort to try and save some compute  
  if !hasCompensatableTransaction {  
            if transaction.GetTransactionType() == GetPivotTransactionType() {  
               if err == nil {  
                  err = errors.New("got a Do Not Proceed, but No Error(So now you get an Error)")  
               }  
               log.Println(err)  
               return err  
            }  
         }  
         //Check if the Transaction Type is Pivot/Compensatable then undo the Transactions providing the list of undo Transaction Commands  
  if transaction.GetTransactionType() == GetCompensatableTransactionType() || transaction.GetTransactionType() == GetPivotTransactionType() {  
            undoErr := undoCompensatableTransactions(compensatableTransactionCommands, &sagaRunnerLogger)  
            if undoErr != nil {  
               // This process fails I really don't want to tell the end user. So I guess I'll just log it.  
  sagaRunnerLogger.logger.Println(undoErr)  
            }  
            if err == nil {  
               err = errors.New("got a Do Not Proceed, but No Error(So now you get an Error)")  
            }  
            return err  
  
         }  
      }  
      //Check if the Transaction Type is Retryable then try to reattempt them to preset retry limit  
  if transaction.GetTransactionType() == GetRetriableTransactionType() {  
         if !canProceed || err != nil {  
            didItFinallyWork, err := retryRetractableTransactionToPresetLimit(transactionCommands[0], &sagaRunnerLogger)  
            if err != nil {  
               return err  
  }  
            if !didItFinallyWork {  
               retriesFailed := fmt.Errorf("%v [UNDO] | After the Preset Number (#%d) of Retries this RetryTabiable Transaction(%v) has still Failed. Look at contruction of this Transaction and its Transaction Command(%v)", sagaRunnerLogger.generalErrorPrefix, RetryableTransactionRetryLimit, transaction.GetTransactionName(), transaction.GetTransactionCommands()[0].GetTransactionCommandName())  
               sagaRunnerLogger.logger.Println(retriesFailed)  
               // I am not positive If I am supposed to return this error I will for now  
  return fmt.Errorf("%v | %v ", retriesFailed, err)  
            }  
         }  
  
      }  
      sagaRunnerLogger.logger.Println("------- Next Transaction --------")  
   } //End of loop  
  sagaRunnerLogger.logger.Println("------- End of Saga Execution --------")  
   return nil  
}  
func retryRetractableTransactionToPresetLimit(command TransactionCommand, runner *sagaRunner) (bool, error) {  
   var didItFinallyWork = false  
 var err error  
 for i := 1; i <= RetryableTransactionRetryLimit; i++ {  
      runner.logger.Println(fmt.Sprintf("%v [RETRYABLE] Retrying Retryable Transaction %d of %d", runner.generalErrorPrefix, i, RetryableTransactionRetryLimit))  
      var err1 error  
  didItFinallyWork, err1 = command.Execute()  
      //TODO Learn proper error handling  
  err = fmt.Errorf("retry #: %d | error: %v | %v", i, err1, err)  
  
   }  
   return didItFinallyWork, err  
}  
  
func undoCompensatableTransactions(listOfCompensatableTransactions []TransactionCommand, runner *sagaRunner) error {  
   //TODO Need to learn to Wrap errors  
  undo := errors.New("[UNDO] |")  
   errCollector := errors.New("")  
   for _, transactionCommand := range listOfCompensatableTransactions {  
      workAsExpected, err := transactionCommand.Execute()  
      if err != nil {  
         runner.logger.Printf(" %v %v | TransactionCommandName: %v, Error: %v", runner.generalErrorPrefix, undo, transactionCommand.GetTransactionCommandName(), err)  
         errCollector = fmt.Errorf("%v | %v | %v", runner.generalErrorPrefix, undo, err)  
         return errCollector  
      }  
      if !workAsExpected {  
         runner.logger.Printf("%v %v | | TransactionCommandName: %v . Got an do not proceed on the undo that's not good so here is your sign", runner.generalErrorPrefix, undo, transactionCommand.GetTransactionCommandName())  
      }  
   }  
   return nil  
}

Other items

For future me if you see this error in the stack it most likely means the http request timed out and just add some time to the timeout.
“context deadline exceeded”

enter image description here