Skip to content
August 20, 2012 / edeustace

A little Json processing library

I’ve recently had to perform some bulk imports into our project database. The data was sent to us as spreadsheets and we need to insert this into our mongo db.

The steps I took were:

1. Convert xls to csv

2. Convert csv to json using: http://www.cparker15.com/code/utilities/csv-to-json/

3. Process json keys to match our data model keys

4. Run mongo js script to bulk insert the json

For step 3, I couldn’t find a tool that was suitable for the task of mapping the json, so I put together a little library that did it for me.

The usage is fairly straightforward:

java jar json-processor-fat.jar source.json map.json output.json

Where the source.json is the json with the data that you wish to change, the map.json is a set of mapping rules to use and output.json is the name to use for the generated json.

As an example if you had this as your source:

[ { "fruit" : "Apple" } ]

And a map with the following:

{ "fruit" : "favouriteFruits" }

The output would be:

[ { "favouriteFruits" : "Apple" } ]

In this example, the map defines a new key for a key in the source aka fruit -> favouriteFruits

There are some additions commands you can add in the map that perform various actions:

  • !ignore – suppresses that item from the output
  • !camelCase – convert the existing key to camelCase eg: “First Name” to “firstName”
  • !merge-> merge a couple of the keys into one keyed value. With merge you have to specify a new key name and also provide a template for that new key. So if you add “!merge->newKey”, “newKey” will be the name of the new keyed item, which will use a template called “newKey” that adds the other items into it.
  • !insert – insert key/value into output.

There is more documentation on github.

May be of use to someone.




Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: