A while ago I wrote a mongo seed utility for SBT.
Well I’ve written another one, this time as a node library as we have a few node projects that can benefit from a bit of db seeding from time to time. It’s incomplete, but follows a similar format as the sbt plugin, but adds the ability to write .ejs files that will first get preprocessed. Good if you need to add lots of text to a json property.
The new lib is here https://github.com/corespring/mongo-seeder.
My last post was about self building heroku slugs on your own machine.
This has been our approach until the last month or so, but now we have changed.
We have created a new command line tool called cs-builder that builds slugs but doesn’t use a buildpack. The reason we do this: speed. The buildpack we use (the scala buildpack) was really inefficient, as it would rebuild the project from scratch even though the project had already been built by the previous continuous integration steps like test and integration test. This tool makes use of any assets from previous build steps for slug generation.
It has a number of commands, but the most interesting ones for this post are:
cs-builder build-from-git --branch=BRANCH --git=GIT --build-assets my-binaries
Which builds your project from your git repo and archives the build-assets with a commit hash.
cs-builder git-slug --branch=BRANCH --git=GIT --template=the_slug_template
Which merges your archived build assets with whichever slug template you’re using. It comes with 2 preinstalled templates for node and the jdk 1.7 and it’s easy to add your own.
cs-builder heroku-deploy-slug --branch=BRANCH --git=GIT --heroku-app=HEROKU_APP
Which deploys the slug using the heroku release api.
Using this tool has greatly reduced our build/slug times. We were averaging 6 minutes using the heroku buildpack – now it takes 1 minute.
The full details are in the project: https://github.com/corespring/cs-builder. Also this command line tool is inspired by the following article: https://devcenter.heroku.com/articles/platform-api-deploying-slugs.
heroku.com is a great tool for simplifying deployments of your webapps. You just add the heroku git repository to your app and push your source and it’ll build your app for you.
On my current project we’ve been using this mechanism up until recently, but we have since changed our set up. Instead of pushing our source to the heroku git repository, we build the heroku slug ourselves then ‘release’ the slug to the target server.
What’s a slug?
A slug is an archive of resources that make up your web app. Here’s the heroku definition.
Why?
Build time
As our application grew in size, the build time grew also (we are compiling a Play! scala app – so some compilation work needs to be done). The build time on the target servers was starting to hit the 15 minute mark which is when the built-in slug compiler throws a time out error.
By building the slug on our own dedicated server, we could optimise that server for the build at hand and give it more firepower so it can do its crunching. The build that was taking ~15 minutes now takes ~3.5 minutes.
Build Once, deploy many times
Every time we wanted to deploy to one of our pipeline servers (dev, qa, staging) each one would have to do its own build. Now, we build the slug once on our server and release it to each heroku target. This saves alot of time as each release takes ~1.5 minutes.
How
You need to do 3 things to set this up:
1. Build the slug yourself
2. Host the slug at a url on the interweb
3. Tell the heroku server to download the slug at the url provided
Everything you need to set this up is available on github.
Build the slug yourself
For building the slug we use the slug-compiler gem.
Host the slug at a url on the interweb
Once the slug is compiled we boot up a web server to temporarily host the slug at a randomized url.
Tell the heroku server to download the slug at the url provided
To notify heroku you make a REST call. You can use heroku-anvil to do this for you. In our case we created our own script to make the call so it would fit in with our url generator. The REST call in heroku-anvil is here. Note: if you’re making the rest call you’ll need your api key.
Cleanup
Once the release has been downloaded, we then kill the web server so the slug is no longer available. Obviously this is optional and depends on how you want to set up your script.
Thats it – if you’re suffering from slow heroku builds this provides you with an avenue to optimise your build process.
Addendum:
It’s worth mentioning that heroku has a new beta feature called ‘pipelines’ which allows you to send your slug upstream from one heroku server to another. We’re not using this yet, but looks like it’ll be a great addition. This means we’ll only need to build once and release once, then use the pipeline feature from there. More info here: https://blog.heroku.com/archives/2013/7/10/heroku-pipelines-beta
I’ve created a new sbt plugin to simplify seeding your mongo db in your SBT projects.
All the information needed is here: https://github.com/edeustace/mongo-db-seeder.
Once you install it you simply call:
seed-dev seed-test seed-prod
It will then seed a database based on the seed files you have within your project. Enjoy!
On our current project, we were looking for a way of abstracting out how we load our javascript from our html templates. Our wish was that depending on the environment that the application runs in, it’ll render the scripts in different ways.
For Example:
– In dev mode: load all scripts as individual files, not gzipped and not minified.
– in production mode: load all scripts as one single, gzipped and minified file.
This is the plugin we came up with
https://github.com/edeustace/assets-loader
its at an early stage and has only been tested with Play 2.0.4, but thought I’d share.
On my current project, we are at a stage where we want to set up a deployment pipeline, from a developer machine to a production environment (with all the steps in between). We are using Heroku for our deployments and a hosted mongo service.
One part of the challenge in setting up this pipeline is migrating the database so that the updated application will work with its data set and schema.
To that end we started looking at mechanisms for migrating Mongo databases. The main requirement was that we could migrate and rollback a database to a target version.
There are some interesting options on github (see below), but they didn’t quite fit into what we wanted. So we’ve created our own library:
https://github.com/corespring/mongo-migrator
We’ve been using it now for the last few months and its working good.
The flow is as follows:
Write a migration/rollback mongo script that has the following format:
var x = "y";
//Up is run on a migration
function up(){
db.item.insert({name: "Ed", surname: "Eustace"})
}
//Down is run on a rolback
function down(){
db.item.remove({ name: "Ed", surname: "Eustace});
}Save the script to a folder, eg 001_migrate.js
Run mongo-migrator:
mongo-migrator migrate [version_id] mongo_uri script_path_1
//eg
//java -jar mongo-migrator.jar migrate 1 mongodb://localhost/my-db migrations/The migrator will check what scripts have been already run in this folder and only run ones that haven’t been run yet (it stores these in a collection called ‘mongo_migrator_versions’).
Rollback is the opposite – it runs the down() function in reverse order from the existing scripts in the db.
There’s more information on the project’s readme. Hopefully people find it useful.
Other notable mongo migration projects:
- https://github.com/secondmarket/mongeez
- https://github.com/srohde/MongoUpgrade
- https://github.com/afloyd/mongo-migrate (Note: this is new and similar to what we’ve written).
On my current project we make use of the standards defined by corestandards.org. This website provides its data as html or xml.
This new webapp does 2 things:
- Converts the xml to json
- Adds category and subCategory (which needs to be pulled from the html site)
The site is here: http://core-standards-json.herokuapp.com/
The source code is here: http://github.com/edeustace/core-standards-extra
We can now use this site as our reference point, maybe you can too.
I’ve been working on an internal project where we needed a quick and easy way to interact with a mongo db over rest. Mongolab.com has an existing rest layer that does this, but we wanted to route it through our Play! 2 app.
So I created a Play! plugin called Mongo Rest Layer. Its a simple library that adds REST capabilities to your play app with little setup. You just need to pass in a mongo uri and override the onRouteRequest method of GlobalSettings. The REST behaviour is modelled on the mongolab.com api, but you can add your own flavour by implementing: brokers.RequestBroker.
May be of use to others.
I’ve recently had to perform some bulk imports into our project database. The data was sent to us as spreadsheets and we need to insert this into our mongo db.
The steps I took were:
1. Convert xls to csv
2. Convert csv to json using: http://www.cparker15.com/code/utilities/csv-to-json/
3. Process json keys to match our data model keys
4. Run mongo js script to bulk insert the json
For step 3, I couldn’t find a tool that was suitable for the task of mapping the json, so I put together a little library that did it for me.
The usage is fairly straightforward:
java jar json-processor-fat.jar source.json map.json output.jsonWhere the source.json is the json with the data that you wish to change, the map.json is a set of mapping rules to use and output.json is the name to use for the generated json.
As an example if you had this as your source:
[ { "fruit" : "Apple" } ]
And a map with the following:
{ "fruit" : "favouriteFruits" }The output would be:
[ { "favouriteFruits" : "Apple" } ]In this example, the map defines a new key for a key in the source aka fruit -> favouriteFruits
There are some additions commands you can add in the map that perform various actions:
- !ignore – suppresses that item from the output
- !camelCase – convert the existing key to camelCase eg: “First Name” to “firstName”
- !merge-> merge a couple of the keys into one keyed value. With merge you have to specify a new key name and also provide a template for that new key. So if you add “!merge->newKey”, “newKey” will be the name of the new keyed item, which will use a template called “newKey” that adds the other items into it.
- !insert – insert key/value into output.
There is more documentation on github.
May be of use to someone.
On my current project we are using mongo as our db. This means alot of backing up, copying and restoring dbs. mongo has great tools for this namely mongodump and mongorestore, but I was getting tired of having to add my configurations for different dbs into different shell scripts.
Also I didn’t really have a back up system in place.
To make my life easier – I’ve put together a little gem called *mongo-db-utils*. Its a wrapper on top of mongodump and mongorestore.
The nice thing about it is that you can save your configuration, so the next time you launch it it’ll remember the db’s or amazon buckets you work with.
Also it backs up the dbs using its own convention so you’ll always know where the latest db is. The config file and the backups are stored in ~/.mongo-db-utils.
Its on github, it may be a little rough around the edges (let me know if you’ve any issues), but I’ve been using it daily here for the last week and its working good. Unfortunately it’ll only run on Linux/OSX at the moment.
odds'n'sods 