When building data products, often the end result is either a recurring report or an API which allows for an interface with other services. In these cases, the product might be run by a cron job, or even manually, depending on the situation. When the product is part of (more frequently, at the end of) a larger data processing pipeline, this can become inefficient or problematic since steps earlier in the pipeline can cause the product to supply incorrect information or fail altogether.

To get around this problem it is often desirable to have previous steps in the data processing pipeline directly trigger the data product to run. This ensures that earlier steps of the pipeline have completed successfully and also eliminates the need for buffer time before the cron job starts. A way this coupling is often accomplished is via an HTTP endpoint which can be hit by some calling entity, thus triggering the processing job to start.

When the cron job is already running, building such an HTTP endpoint can be very straightforward since you need only run a local command on the system in question (responding with HTTP code 200, OK) and then while the command is running any subsequent requests can provide an HTTP 503 (Service Unavailable) response.

With that in mind, here is a small example in Go which accomplishes the task. It will listen on some port for requests to a /start endpoint. When that URL is accessed the command will be run in a separate goroutine, and any requests which come in while the command is running will receive a 503 response. After the command completes and errors are handled the system is available and can be restarted again.

This is just a very rough outline and has some nasty things (e.g., global variables, security problems) but it provides the necessary endpoint and does what is needed with minimal effort.

package main

import (
	"net/http"
	"os/exec"
)

const (
	command = "commandTorun"
	args    = "argumentsForTheCommand"
	port    = ":8080"
)

var jobRunning = false

func runJob() {
	jobRunning = true
	err := exec.Command(command, args).Run()
	if err != nil {
		// Error was returned from system command.  Do something.
	}
	jobRunning = false
}

func jobStartEndpoint(w http.ResponseWriter, r *http.Request) {
	if jobRunning {
		w.WriteHeader(503)
	} else {
		go runJob()
		w.WriteHeader(200)

	}
}

func main() {
	http.HandleFunc("/start", jobStartEndpoint)
	http.ListenAndServe(port, nil)
}