AutoSleep - Sleep Your Dyno When You Sleep

26/6/2021
4-minute read

In my current organisation, we have 20+ web applications and each application has its own staging, UAT, pre-prod, and production environments on Heroku.

These different sets of environments consume a huge number of dynos, dyno usage are even high if some application has multiple processes (sidekiq workers). But except for production, other application environments are not in use 24X7. They are mostly used in developers/QA’s working hours. Even possible that not all the environments used at Once.

This is a waste of resources they should be scaled down when not in use but doing it manually for this many applications is not possible. So we build a tool to do this job that we call autosleep.

How we do it

First, we need to find a way to detect if an application is in use or not. For this, I would say if there is no request come on the server for a finite time period then it’s not in use. To get this we will use Heroku Router Logs. we will configure log drains on the target application to forward logs to our tool. And we will keep track of the last router log to identify if it’s in use or not.
After identifying if the application in use or not we need to Scale down dynos to zero if it’s not in use and scale back to the original config when the first requests come
We will be using Heroku Platform APIs for this purpose

Let’s Start working

Planning

We would need a CRUD to manage application metadata like Heroku app name, Original Configuration, a timestamp for the last request, heroku_api_key to call Heroku platform APIs, etc.
We would need an API that will consume log drains and keep the last request timestamp save
And the last we would need a background job that will keep checking if the application is not running for the defined time duration and scale down the dyno

Tech stack

I will use GIN which is a web framework in golang. Even I’m a rails developer I had to choose Golang here because of its performance benefits and we don’t want to bear more cost to save cost.
Along with gin i had used heroku-go to intrect with platform APIs and work for background job.

Start working

Putting complete code in the blog post will make this post too long. I had already explained the logic to build the tool. In case you are looking for a complete code you can clone it from autosleep

still, we can look into important logic here,

First one the processing log drains, the log drains send by Heroku log drain is in the form of string body. In this API

we need to check if the log contains any router log if yes then we will assume the application is in use
If it’s in use check the current status of the application - if the current status is OFF then scale up the dynos and Enqueue job to check its state later - else update the current timestamp for the application
If it’s not in use (no router logs found) then returned. code

func ProcessDrain(c *gin.Context){

  buf := make([]byte, 1024)
  num, _ := c.Request.Body.Read(buf)
  reqBody := string(buf[0:num])
  is_running := strings.Contains(reqBody, "router") && !strings.Contains(reqBody, "well-known")

  if is_running == false {
    c.JSON(http.StatusOK, gin.H{"status": is_running})
    return
  }
  var app models.Application
  if err := models.DB.Where("heroku_app_name = ?", c.Param("app_id")).First(&app).Error; err != nil {
    c.JSON(http.StatusBadRequest, gin.H{"error": "Record not found!"})
    return
  }
  fmt.Println("app_id ", c.Param("app_id"), " CurrentStatus: ", app.CurrentStatus," is_running: ", is_running)
  if app.CurrentStatus == false &&  is_running == true && app.ManualMode == false{
    ScaleUpDynos(app)
    models.DB.Model(&app).Update("CurrentStatus", true)
    var enqueuer = work.NewEnqueuer("auto_ideal", models.REDIS)
    _, err := enqueuer.EnqueueUniqueIn("sleep_chacker",app.CheckInterval,work.Q{"app_id": app.HerokuAppName})
    if err != nil {
      fmt.Println(err)
    }
  }
  if is_running {
    models.DB.Model(&app).Update("RecentActivityAt", time.Now())
  }
  c.JSON(http.StatusOK, gin.H{"status": is_running})
}

Second one is our background job to check if need to scale down an app

Here we will find the time difference between the current time and the last request on the application.
If the difference is greater than the threshold we will Scale down the dynos
else we will Re-Enqueue to check this later. code:

func (c *Context) SleepChecker(job *work.Job) error {
	app_id := job.ArgString("app_id")
	var app models.Application

	if err := models.DB.Where("heroku_app_name = ?", app_id).First(&app).Error; err != nil {
		return nil
	}
  loc, _ := time.LoadLocation("UTC")
	current := time.Now().In(loc)

	fmt.Println("SleepChecker: Current time is ",current, "dyno was active at ", app.RecentActivityAt)
  diff := current.Sub(app.RecentActivityAt)
	if diff.Seconds() > app.IdealTime && app.CurrentStatus == true {
  	formation_list := ScaleDownDynos(app)
  	models.DB.Model(&app).Updates(map[string]interface{}{"CurrentStatus": false, "CurrentConfig": formation_list})    
  }
  
  if diff.Seconds() <= app.IdealTime && app.CurrentStatus == true {
  	var enqueuer = work.NewEnqueuer("auto_ideal", models.REDIS)
  	_, err := enqueuer.EnqueueUniqueIn("sleep_chacker", app.CheckInterval, work.Q{"app_id": app.HerokuAppName})
  	if err != nil {
  		fmt.Println(err)
  	}
  }
	return nil
}

that’s it in this blog, Refer to my GitHub repo autosleep for the complete code.

Enjoy Saving Cost!!!

Heroku Golang