Tag Archives: pymongo

More pymongo

13 Sep


I am back in MongoDB mode. I grabbed some pollen data from the City of Albuquerque. It has a date, location, pollen type and the count from 2004 to 2013. I loaded this in to MongoDB from a CSV with this script:
from pymongo import MongoClient


for x in file.readlines():

I can then query for elm data sorted by date.
from pymongo import MongoClient



for entry in x:

Great! But I want all elm data on the east side of ABQ sorted by date and plotted. Easy! Matplotlib and Pandas help out here:

from pandas import Series
import matplotlib.pyplot as plt
from pymongo import MongoClient


for w in x:




D3.js and MongoDB

28 Nov

I have not been shy in my love of MongoDB. The honeymoon is not over. Now I want to graph and visualize my data from MongoDB. I just started looking at D3 – I’m coming to the party a bit late – and it is perfect for this task. I have thrown together a super simple, absurd even, example using a DB I had already populated and some left over CherryPy code. Let me say that reusing code is a good idea – when I can find what I’m looking for. It allows me to throw something together quickly. These are not production samples, but just for me to see that I can get it to work. Enough already. Here is a bar chart using D3 and MongoDB.


A Bar Chart, that should be in SVG, from MongoDB data in D3.js

I have not put this on OpenShift yet, but may.  This bar chart is a simple python script using CherryPy and committing my favorite sin – passing HTML as a variable in a return. One reason to put it on OpenShift is so I can template it in Jinja.  Here is the code:

import cherrypy
from pymongo import Connection

class mongocherry(object):

def index(self):
output =[]
output.append(‘<HTML><HEAD><TITLE>D3 and MONGODB</TITLE><script type=”text/javascript” src=”http://d3js.org/d3.v2.min.js”></script><style type=”text/css”>div.bar {display: inline-block;width: 20px;height: 75px;margin-right: 2px;background-color: teal;}</style></HEAD><BODY><h1>D3 and MongoDB</h1>’)
output.append(‘<script type=”text/javascript”>var dataset=[‘)

for x in db.places.find():
output.append(‘0];’+”\n”+’d3.select(“body”).selectAll(“div”).data(dataset).enter().append(“div”).attr(“class”, “bar”).style(“height”, function(d) {var barHeight = d * 5;return barHeight + “px”;});</script></body></html>’)

while i<len(output):

return html

index.exposed = True

cherrypy.config.update({‘server.socket_host’: ‘’,
‘server.socket_port’: 8000,


The Python code prints out HTML that looks like this:

<script type="text/javascript" src="http://d3js.org/d3.v2.min.js"></script>
<style type="text/css">
div.bar {
display: inline-block;
width: 20px;
height: 75px;
margin-right: 2px;
background-color: teal;}
<h1>D3 and MongoDB</h1>
<script type="text/javascript">
var dataset=[35,35.8,38,39,30,31,31,31,33,25,33,0];
.attr("class", "bar")
.style("height", function(d) {var barHeight = d * 5;return barHeight + "px";});

Not much going on here, just a simple D3.js bar chart – not even done in SVG. The MongoDB part is in the variable dataset[..]. While writing the HTML, the python code loops through my mongodb with for x in db.places.find(): and it grabs the latitude of the data I have with x[“loc”][0] prints it out in the JavaScript variable dataset[]. I add a 0 at the end because I get a trailing comma. Sloppy, but oh well.

I return all the HTML and you get the page displayed at the top of this post. The cool part of marrying D3.js and MongoDB is JSON. I have JSON in my DB and D3 takes JSON.

OpenShift: Leaflet.js, MongoDB, and Flask

28 Nov
Leaflet Map

My simple application on OpenShift. Pulling data from MongoDB and presented using Leaflet.js

PaaS: Platform as a Service

I write a lot of small applications and scripts to improve my programming skills and to hopefully use in my work. Unfortunately, I cannot share most of them because  I don’t have access to a server – almost all of my apps run on a WAMP stack using localhost:8080.  Recenlty, I saw a tweet about a service by Red Hat – I remembered them from my Linux club days in college – called OpenShift. OpenShift is a Platform as a Service (PaaS) that “manages the stack so you can focus on your code.” It promised a way to for me to run all the new technologies my web host won’t allow. I had to try it out.

What I Want to Build

I make a lot of maps. Just simple point maps with a polygon here or there. No real geoprocessing or analysis. Just for displaying data. My application needed to display a map. My front-end of choice is Leaflet.js – a JavaScript library- and I have to be able to use it.

I am not a programmer but dabble in a few languages. Python is the one I am most comfortable with and I am more fluent in it than any other language. It is also a language I enjoy and having an interactive shell makes my coding go much faster. I can test solutions before they ever go in my program. If I am going to have a chance at success, I have to use Python.

All of my Python programming on the web has been using CherryPy. It is a minimal framework that was easy to install and to learn. I only need to do simple things like route and pass data so why go Django or Pylons – Pyramid now?

I really like  MongoDB. You may think spatial is special and get all PostGIS on me, but for what I need and what I like to do, I am all about MongoDB right now. So I need to be able to install it.

To summarize, I need – or want:

  1. Leaflet.js
  2. Python
  3. CherryPy
  4. MongoDB

Getting Started

Go to OpenShift and setup an account.

I am not a reader of manuals – OpenShift has a good one – but more of an experimenter and skimmer.  I used the OpenShift getting started and a blog post – REST web services with Python, MongoDB, and Spatial data in the Cloud – to get up and running. The blog used Flask, which I have never used before, but it does the same thing I was using CherryPy for, so I used it. In reading more about Flask, I found that it has a templating engine called Jinja2 available. When pushing my code to GIT, I saw that Jinja was already on OpenShift. I will admit, I am bad about templating and put all my HTML in a variable and have a function ‘return HTML.’ With this application, I wanted to break that bad habit.

After getting the Ruby Installer and Git, I followed all the instructions in the getting started and the blog and was left looking at a folder on my computer with some stuff inside of it. Now it’s time to play.

Coding the Application

The stuff in my folder is an application using Flask that I grabbed from GIT.  The important part is in the WSGI folder. The MyFlaskApp file has code that grabs data from a MongoDB and dumps it out to a webpage. This is the file I will edit to make my app.

Before I do anything else, I want to write my webpage. It will be nothing more than a simple Leaflet map with one marker at a point retrieved from MongoDB. It will also be a Jinja Template.

Here is the code for my webpage.

<!doctype html>
<head><title>OpenShift: Leaflet, MongoDB, Python. Jinja2</title>
<link rel=’stylesheet’ href=’http://cdn.leafletjs.com/leaflet-0.4/leaflet.css&#8217; />
<div id=’map’ style=’width: 900px; height: 350px’></div>
<script src=’http://cdn.leafletjs.com/leaflet-0.4/leaflet.js’></script&gt;
var map = L.map(‘map’).setView([40.71367, -73.99364 ], 13);
L.tileLayer(‘http://{s}.tile.cloudmade.com/API-KEY/997/256/{z}/{x}/{y}.png’,{attribution:’Paul Crickard’, maxZoom: 18 }).addTo(map);
L.marker({{ coord[“pos”] }}).addTo(map)

This file needs to be put in a folder in WSGI called TEMPLATES. You will notice another folder in the directory called STATIC. This is where you can put CSS and JS files.

The code is straight HTML and JavaScript with the exception of   {{ coord[“pos’ }}. This is the Jinja part of the template. In the Flask Application, I will pass coord to the template.

Let’s take a look at a simple Flask application that will grab the point from MongoDB and send it to my Jinja Template. Here it is:

from flask import render_template

def albuquerque():
conn = pymongo.Connection(os.environ[‘OPENSHIFT_MONGODB_DB_URL’])
db = conn.parks
for z in db.location.find():
return render_template(‘map2.html’,coord=q)

The file I modified already has several import statements but you will need to add render_template to use Jinja. I changed the route to /albuquerque – I was going to use some local data but decided for a test I would use the parks JSON file so now the Albuquerque doesn’t make much sense. Then I grab a cursor and pass a point to q. Lastly, I call the render_template and hand it q as coord. Now in my HTML/Jinja Template, I can call coord. For the future, I would run a loop in the Template that grabbed all the points. A loop in Jinja looks like this:

{% for x in json %}
   Do something with X: 
   L.marker({{ x }}).addTo(map) 
{% endfor %}

Now go to my RHCloud.com and see it in action. It seems rather simple – I need to add some popups – but there is some powerful technology behind this. MongoDB has some spatial features like $near, $box or $within $center of a circle. This should somewhat please the spatial is special crowd. One of the things I love is that I can have an object {“pos”:[35,-106],”name”:”ABQ”} and I can have another {“pos”:[33,-106],”name”:”LL”, “visitors”: 50}. In a table, every record has the same fields – I don’t have to do that in MongoDB. I see MongoDB as much more flexible for my needs.

Next Steps

What functionality would I like to add to this?

  1. A form that allows the user to select either all the data or a subset using simple radio buttons or combo boxes.
  2. A way to download a report of the data selected.
  3. A form for users to enter new data.
  4. Popups with additional data. This should only require adding .bindPopup( {{coord["popupContent']}} )

One last thing: It runs on mobile. My HTML is not very good, but a mobile Leaflet app is simple. I have one that uses your phone’s current location on my website.

My App Running on Mobile

Flask App Running on iPhone 4S

MongoDB and Shapely

18 Nov

I wanted to connect my mongoDB to Shapely so that I could create buffers and do some other things, like spit out my points and buffers as WKT. Here is what I came up with.

from pymongo import Connection,GEO2D
from shapely.geometry import asShape,Point


for q in db.b.find({“loc”:{“$near”:[-106,35]}}).limit(1):
point=Point(q[“loc”][0],q[“loc”][1])  #this is passing mongoDB to Shapely

point.x    #just checking I didn’t make a mistake


t.area  #checking. Result is 12.a lot of number



Now I can use mongoDB and find all the points in a bounding box or near another point or near a point where type=Coffee Shop, then send that to Shapely for a buffer or to measure distance or whatever, then spit it out as WKT. Saw a tutorial on using shapely with GDAL/OGR and Fiona. Goodle that and see how to send it all back to a shapefile.

MongoDB with a slice of CherryPy: Running a Query and Displaying Results

15 Nov

As you may have noticed, I love CherryPy. Recently, I have been playing with MongoDB and pymongo. In my last post I showed how to create a shapefile from a mongoDB. In this post, I will show how to connect to mongoDB through Python using CherryPy and spitting out the results of a find() in an HTML table.

Here is what the webpage will look like:

And here is the code – I’m sure there is a cleaner way to write this instead of looping twice but i’m writing this on my lunch break and don’t have a lot of time to do more than make it run.

import cherrypy
from pymongo import Connection,GEO2D
class mongocherry(object):
def index(self):
output =[]
output.append(‘<HTML><HEAD><TITLE>QUERY MONGODB</TITLE></HEAD><BODY><h1>Query MongoDB</h1>’)
output.append(‘<table border=”1″><tr><td>ID</td><td>LAT</td><td>LONG</td></tr>’)

for x in db.places.find():

while i<len(output):

return html

index.exposed = True

cherrypy.config.update({‘server.socket_host’: ‘’,
‘server.socket_port’: 8000,


I have servers running on some ports so I needed to set a config to change the port. This site creates a front end to data that will always be current. You can make your index page a form and allow the user to select the data they want, then in another def returnResults(self, user selected data): Spit out all the HTML that is in the index now, passing ‘user selected data’ to mongoDB.

I have really enjoyed playing with mongoDB and am curious about importing Revit schedule. Not sure what I would do with it once it’s loaded. But still thinking…… Any ideas?