Skip to content
July 21, 2014 / edeustace

Another Mongo Seed library

A while ago I wrote a mongo seed utility for SBT.

Well I’ve written another one, this time as a node library as we have a few node projects that can benefit from a bit of db seeding from time to time. It’s incomplete, but follows a similar format as the sbt plugin, but adds the ability to write .ejs files that will first get preprocessed. Good if you need to add lots of text to a json property.

The new lib is here https://github.com/corespring/mongo-seeder.

June 7, 2014 / edeustace

Roll your own Heroku slugs (part 2) – without a buildpack

My last post was about self building heroku slugs on your own machine.

This has been our approach until the last month or so, but now we have changed.

We have created a new command line tool called cs-builder that builds slugs but doesn’t use a buildpack. The reason we do this: speed. The buildpack we use (the scala buildpack) was really inefficient, as it would rebuild the project from scratch even though the project had already been built by the previous continuous integration steps like test and integration test. This tool makes use of any assets from previous build steps for slug generation.

It has a number of commands, but the most interesting ones for this post are:

    cs-builder build-from-git --branch=BRANCH --git=GIT --build-assets my-binaries

Which builds your project from your git repo and archives the build-assets with a commit hash.

    cs-builder git-slug --branch=BRANCH --git=GIT --template=the_slug_template

Which merges your archived build assets with whichever slug template you’re using. It comes with 2 preinstalled templates for node and the jdk 1.7 and it’s easy to add your own.

    cs-builder heroku-deploy-slug --branch=BRANCH --git=GIT --heroku-app=HEROKU_APP

Which deploys the slug using the heroku release api.

Using this tool has greatly reduced our build/slug times. We were averaging 6 minutes using the heroku buildpack – now it takes 1 minute.

The full details are in the project: https://github.com/corespring/cs-builder. Also this command line tool is inspired by the following article: https://devcenter.heroku.com/articles/platform-api-deploying-slugs.

 

October 13, 2013 / edeustace

Roll your own Heroku slugs

heroku.com is a great tool for simplifying deployments of your webapps. You just add the heroku git repository to your app and push your source and it’ll build your app for you.

On my current project we’ve been using this mechanism up until recently, but we have since changed our set up. Instead of pushing our source to the heroku git repository, we build the heroku slug ourselves then ‘release’ the slug to the target server.

What’s a slug?

A slug is an archive of resources that make up your web app. Here’s the heroku definition.

Why? 

Build time

As our application grew in size, the build time grew also (we are compiling a Play! scala app – so some compilation work needs to be done). The build time on the target servers was starting to hit the 15 minute mark which is when the built-in slug compiler throws a time out error.

By building the slug on our own dedicated server, we could optimise that server for the build at hand and give it more firepower so it can do its crunching. The build that was taking ~15 minutes now takes ~3.5 minutes.

Build Once, deploy many times

Every time we wanted to deploy to one of our pipeline servers (dev, qa, staging) each one would have to do its own build. Now, we build the slug once on our server and release it to each heroku target. This saves alot of time as each release takes ~1.5 minutes.

How

You need to do 3 things to set this up:

1. Build the slug yourself

2. Host the slug at a url on the interweb

3. Tell the heroku server to download the slug at the url provided

Everything you need to set this up is available on github.

Build the slug yourself

For building the slug we use the slug-compiler gem.

Host the slug at a url on the interweb

Once the slug is compiled we boot up a web server to temporarily host the slug at a randomized url.

Tell the heroku server to download the slug at the url provided

To notify heroku you make a REST call. You can use  heroku-anvil to do this for you. In our case we created our own script to make the call so it would fit in with our url generator. The REST call in  heroku-anvil is here. Note: if you’re making the rest call you’ll need your api key.

Cleanup

Once the release has been downloaded, we then kill the web server so the slug is no longer available. Obviously this is optional and depends on how you want to set up your script.

Thats it – if you’re suffering from slow heroku builds this provides you with an avenue to optimise your build process.

Addendum:

It’s worth mentioning that heroku has a new beta feature called ‘pipelines’ which allows you to send your slug upstream from one heroku server to another. We’re not using this yet, but looks like it’ll be a great addition. This means we’ll only need to build once and release once, then use the pipeline feature from there. More info here:  https://blog.heroku.com/archives/2013/7/10/heroku-pipelines-beta

April 27, 2013 / edeustace

New SBT plugin to facilitate seeding your mongo database

I’ve created a new sbt plugin to simplify seeding your mongo db in your SBT projects.

All the information needed is here: https://github.com/edeustace/mongo-db-seeder.

Once you install it you simply call:

seed-dev
seed-test
seed-prod

It will then seed a database based on the seed files you have within your project. Enjoy!

April 10, 2013 / edeustace

Assets Loading plugin/library for Play 2.0.4

On our current project, we were looking for a way of abstracting out how we load our javascript from our html templates. Our wish was that depending on the environment that the application runs in, it’ll render the scripts in different ways.

For Example:

– In dev mode: load all scripts as individual files, not gzipped and not minified.

– in production mode: load all scripts as one single, gzipped and minified file.

This is the plugin we came up with

https://github.com/edeustace/assets-loader

its at an early stage and has only been tested with Play 2.0.4, but thought I’d share.

March 30, 2013 / edeustace

Migrating mongo db

On my current project, we are at a stage where we want to set up a deployment pipeline, from a developer machine to a production environment (with all the steps in between). We are using Heroku for our deployments and a hosted mongo service.

One part of the challenge in setting up this pipeline is migrating the database so that the updated application will work with its data set and schema.

To that end we started looking at mechanisms for migrating Mongo databases. The main requirement was that we could migrate and rollback a database to a target version.

There are some interesting options on github (see below), but they didn’t quite fit into what we wanted. So we’ve created our own library:

https://github.com/corespring/mongo-migrator

We’ve been using it now for the last few months and its working good.

The flow is as follows:

Write a migration/rollback mongo script that has the following format:

var x = "y";
//Up is run on a migration
function up(){ 
  db.item.insert({name: "Ed", surname: "Eustace"})
}
//Down is run on a rolback
function down(){ 
  db.item.remove({ name: "Ed", surname: "Eustace});
}

Save the script to a folder, eg 001_migrate.js

Run mongo-migrator:

mongo-migrator migrate [version_id] mongo_uri script_path_1
//eg
//java -jar mongo-migrator.jar migrate 1 mongodb://localhost/my-db migrations/

The migrator will check what scripts have been already run in this folder and only run ones that haven’t been run yet (it stores these in a collection called ‘mongo_migrator_versions’).

Rollback is the opposite – it runs the down() function in reverse order from the existing scripts in the db.

There’s more information on the project’s readme. Hopefully people find it useful.

Other notable mongo migration projects:

December 8, 2012 / edeustace

new webapp: corestandards.org xml ~> json converter

On my current project we make use of the standards defined by corestandards.org. This website provides its data as html or xml.

This new webapp does 2 things:

  1. Converts the xml to json
  2. Adds category and subCategory (which needs to be pulled from the html site)

The site is here: http://core-standards-json.herokuapp.com/

The source code is here: http://github.com/edeustace/core-standards-extra

We can now use this site as our reference point, maybe you can too.

 

Follow

Get every new post delivered to your Inbox.

Join 93 other followers