Spark Singularity

Spark Singularity

README.md Use Spark on Heroku in a single dyno. Experiment inexpensively with Spark in the Common Runtime. Production-quality Spark clusters may be deployed into Private Spaces using spark-in-space. This buildpack provides the following three processes, children of the main web process: Nginx proxy for basic password authentication set via environment variable SPACE_PROXY_BASIC_AUTH format username:{PLAIN}password Spark master web UI https://your-spark-app.herokuapp.com/ REST API https://your-spark-app.herokuapp.com/rest one Spark worker https://your-spark-app.herokuapp.com/worker ? This app should not be scaled beyond a single dyno. (There is no coordination mechanism between multiple instances; implicitly use 127.0.0.1:7077 as Spark Master.) Submitting & controlling jobs Because Spark Singularity is contained in a single dyno with only port 80 exposed, there are two options for submitting jobs: Spark’s REST API, proxied at https://your-spark-app.herokuapp.com/rest Declare the Spark jobs to submit on start-up…


Link to Full Article: Spark Singularity