Tag Archives: mongoDB

More pymongo

13 Sep

20130913-134209.jpg

I am back in MongoDB mode. I grabbed some pollen data from the City of Albuquerque. It has a date, location, pollen type and the count from 2004 to 2013. I loaded this in to MongoDB from a CSV with this script:
from pymongo import MongoClient

client=MongoClient()
db=client.abq
c=db.pollen

file=open(“pollen.csv”)
for x in file.readlines():
temp=x.split(“,”)
tempdate=temp[0].split(“T”)
date=tempdate[0]
location=temp[1]
type=temp[2]
count=temp[3]
c.insert({“date”:date,”location”:location,”type”:type,”count”:count})

I can then query for elm data sorted by date.
from pymongo import MongoClient

client=MongoClient()
db=client.abq
c=db.pollen

file=open(“elm.txt”,”w+”)

x=c.find({“type”:”Elm”}).sort(“date”)
for entry in x:
s=entry[“count”]
file.write(s+”\n”)

Great! But I want all elm data on the east side of ABQ sorted by date and plotted. Easy! Matplotlib and Pandas help out here:

from pandas import Series
import matplotlib.pyplot as plt
from pymongo import MongoClient
client=MongoClient()
db=client.abq
c=db.pollen
x=c.find({“$and”:[{“type”:”Elm”},{“location”:”EASTSIDE”}]}).sort(“date”)

b=[]
labels=[]

for w in x:
s=w[“count”]
i=int(s)
b.append(i)
t=w[“date”]
labels.append(t)

a=Series(b,index=labels)
Series.plot(a,kind=’bar’)

plt.show()

Advertisements

MongoDB, pymongo and GridFS

11 Sep

It has been a while since I’ve done anything with MongoDB-I changed jobs and don’t get to code much anymore. I had the urge to learn more and was interested in storing files in MongoDB using GridFS. I googled, read StackOverflow and MongoDB in Action. The problem was most info was storing text as a file or even when storing a file, the code got the _id when executing the put. So of course get is easy, you have the id. Figuring how to get after the fact was where I got stuck, also had to switch file operations to binary. Here is what I have for putting a file in MongoDB and how to retrieve it later.

from pymongo import MongoClient
import gridfs

client=MongoClient()
db=client.mytest
data=open(“image.png”,”rb”)
fs=gridfs.GridFS(db)
thedata=data.read()
stored=fs.put(thedata,filename=”inmongoimage”)

To get it back in the same code you call:
out=fs.get(stored).read()

This works because stored has the _id of the put operation. But what if I need to retrieve in different code from the inserts? Here is how I got it out with some extra code for info:
imports….
client=MongoClient()
db=client.mytest
fs=gridfs.GridFS(db)

filelist=fs.list()
#returns the file names stored

fileone=filelist[0].encode(“ascii”,”ignore”)
#returns a string of the first file name

outdata=fs.get_version(fileone,”rb”).read()
output=open(“somefile.png”,”wb”)
output.write(outdata)
output.close()

Put allows for more metadata than just the filename:
fs.put(thedata,filename=”file.jpg”,field=”string of text”,anumberfield=52)

To find on a different field:
fs.get_version(anumberfield=52)

MongoDB in QGIS

28 Dec
Displaying my MongoDB Data

Displaying my MongoDB Data

I recently installed the MongoDB Plugin for QGIS. The plugin allows you to connect to a MongoDB and load your data. I stored point data – long and lat – for all the public art in Albuquerque. Using the plugin, I connected to my MongoDB and loaded them.

I had trouble getting the plugin to install. My fix was to install pymongo and BSON to my local Python2.7 then copy it from the site-libs to the QGIS directory (C:\Program Files\Quantum GIS Lisboa\apps\Python27\Lib\site-packages). then I had to turn on the plugin under manage plugins.

This did the trick. Then I fired up my mongod, loaded QGIS and connected using the defaults. It prompted me for a Database and then a Collection. Looks like it requires a GEO2D. You can see my data in the image above.

Can’t wait to start playing….

Albuquerque Public Art

10 Dec

Albuquerque has released a bunch of data as part of their transparency initiative. The best of the data is in KML, which I am not thrilled about. Wish that open initiatives would use formats like CSV or JSON instead of SHP and KML.

Oh well, I have moved on. Their public art file is cool. It is a KML of points where you can find art in ABQ with a table of information and an image. I scrubbed the KML through some python scripts I wrote and got the Long, Lat, and a URL to an image. Let me add, they have a website with this map already, but it is an embedded ArcGIS Online map.   I am giving away the data in a python script that creates a MongoDB with geospatial indexing enabled(GEO2D).

The code is a lot of insert statements:

from pymongo import Connection
from pymongo import GEO2D

db=Connection().albuquerque
#db.drop_collection(“publicart”) you shouldn’t need this.
db.publicart.create_index([(“loc”,GEO2D)])

db.publicart.insert({“loc”:[35.1555,-106.591838],”name”:”Almond Blossom/Astronomy”,”popup”:”http://farm8.staticflickr.com/7153/6831137393_fa38634fd7_m.jpg”})
db.publicart.insert({“loc”:[35.0931,-106.664177],”name”:”Formas Esperando Palabra de Otros Mundos”,”popup”:”http://farm3.staticflickr.com/2167/2479129916_0d861b2600.jpg”})
db.publicart.insert({“loc”:[35.1143,-106.577991],”name”:”Sumer #24″,”popup”:”http://farm2.staticflickr.com/1416/908720823_e390a242f4.jpg”})
db.publicart.insert({“loc”:[35.0682,-106.570886],”name”:”Images”,”popup”:”http://farm8.staticflickr.com/7012/6831253349_52c0a680ba.jpg”})
……….

The full python file is on my website.

 

D3.js and MongoDB

28 Nov

I have not been shy in my love of MongoDB. The honeymoon is not over. Now I want to graph and visualize my data from MongoDB. I just started looking at D3 – I’m coming to the party a bit late – and it is perfect for this task. I have thrown together a super simple, absurd even, example using a DB I had already populated and some left over CherryPy code. Let me say that reusing code is a good idea – when I can find what I’m looking for. It allows me to throw something together quickly. These are not production samples, but just for me to see that I can get it to work. Enough already. Here is a bar chart using D3 and MongoDB.

d3

A Bar Chart, that should be in SVG, from MongoDB data in D3.js

I have not put this on OpenShift yet, but may.  This bar chart is a simple python script using CherryPy and committing my favorite sin – passing HTML as a variable in a return. One reason to put it on OpenShift is so I can template it in Jinja.  Here is the code:

import cherrypy
from pymongo import Connection

class mongocherry(object):

def index(self):
db=Connection().geo
output =[]
output.append(‘<HTML><HEAD><TITLE>D3 and MONGODB</TITLE><script type=”text/javascript” src=”http://d3js.org/d3.v2.min.js”></script><style type=”text/css”>div.bar {display: inline-block;width: 20px;height: 75px;margin-right: 2px;background-color: teal;}</style></HEAD><BODY><h1>D3 and MongoDB</h1>’)
output.append(‘<script type=”text/javascript”>var dataset=[‘)

for x in db.places.find():
output.append(str(x[“loc”][0])+’,’)
output.append(‘0];’+”\n”+’d3.select(“body”).selectAll(“div”).data(dataset).enter().append(“div”).attr(“class”, “bar”).style(“height”, function(d) {var barHeight = d * 5;return barHeight + “px”;});</script></body></html>’)

i=0
html=””
while i<len(output):
html+=str(output[i])
i+=1

return html

index.exposed = True

cherrypy.config.update({‘server.socket_host’: ‘127.0.0.1’,
‘server.socket_port’: 8000,
})

cherrypy.quickstart(mongocherry())

The Python code prints out HTML that looks like this:

<HTML><HEAD><TITLE>D3 and MONGODB</TITLE>
<script type="text/javascript" src="http://d3js.org/d3.v2.min.js"></script>
<style type="text/css">
div.bar {
display: inline-block;
width: 20px;
height: 75px;
margin-right: 2px;
background-color: teal;}
</style>
</HEAD><BODY>
<h1>D3 and MongoDB</h1>
<script type="text/javascript">
var dataset=[35,35.8,38,39,30,31,31,31,33,25,33,0];
d3.select("body")
.selectAll("div")
.data(dataset)
.enter()
.append("div")
.attr("class", "bar")
.style("height", function(d) {var barHeight = d * 5;return barHeight + "px";});
</script></body></html>

Not much going on here, just a simple D3.js bar chart – not even done in SVG. The MongoDB part is in the variable dataset[..]. While writing the HTML, the python code loops through my mongodb with for x in db.places.find(): and it grabs the latitude of the data I have with x[“loc”][0] prints it out in the JavaScript variable dataset[]. I add a 0 at the end because I get a trailing comma. Sloppy, but oh well.

I return all the HTML and you get the page displayed at the top of this post. The cool part of marrying D3.js and MongoDB is JSON. I have JSON in my DB and D3 takes JSON.

OpenShift: Leaflet.js, MongoDB, and Flask

28 Nov
Leaflet Map

My simple application on OpenShift. Pulling data from MongoDB and presented using Leaflet.js

PaaS: Platform as a Service

I write a lot of small applications and scripts to improve my programming skills and to hopefully use in my work. Unfortunately, I cannot share most of them because  I don’t have access to a server – almost all of my apps run on a WAMP stack using localhost:8080.  Recenlty, I saw a tweet about a service by Red Hat – I remembered them from my Linux club days in college – called OpenShift. OpenShift is a Platform as a Service (PaaS) that “manages the stack so you can focus on your code.” It promised a way to for me to run all the new technologies my web host won’t allow. I had to try it out.

What I Want to Build

I make a lot of maps. Just simple point maps with a polygon here or there. No real geoprocessing or analysis. Just for displaying data. My application needed to display a map. My front-end of choice is Leaflet.js – a JavaScript library- and I have to be able to use it.

I am not a programmer but dabble in a few languages. Python is the one I am most comfortable with and I am more fluent in it than any other language. It is also a language I enjoy and having an interactive shell makes my coding go much faster. I can test solutions before they ever go in my program. If I am going to have a chance at success, I have to use Python.

All of my Python programming on the web has been using CherryPy. It is a minimal framework that was easy to install and to learn. I only need to do simple things like route and pass data so why go Django or Pylons – Pyramid now?

I really like  MongoDB. You may think spatial is special and get all PostGIS on me, but for what I need and what I like to do, I am all about MongoDB right now. So I need to be able to install it.

To summarize, I need – or want:

  1. Leaflet.js
  2. Python
  3. CherryPy
  4. MongoDB

Getting Started

Go to OpenShift and setup an account.

I am not a reader of manuals – OpenShift has a good one – but more of an experimenter and skimmer.  I used the OpenShift getting started and a blog post – REST web services with Python, MongoDB, and Spatial data in the Cloud – to get up and running. The blog used Flask, which I have never used before, but it does the same thing I was using CherryPy for, so I used it. In reading more about Flask, I found that it has a templating engine called Jinja2 available. When pushing my code to GIT, I saw that Jinja was already on OpenShift. I will admit, I am bad about templating and put all my HTML in a variable and have a function ‘return HTML.’ With this application, I wanted to break that bad habit.

After getting the Ruby Installer and Git, I followed all the instructions in the getting started and the blog and was left looking at a folder on my computer with some stuff inside of it. Now it’s time to play.

Coding the Application

The stuff in my folder is an application using Flask that I grabbed from GIT.  The important part is in the WSGI folder. The MyFlaskApp file has code that grabs data from a MongoDB and dumps it out to a webpage. This is the file I will edit to make my app.

Before I do anything else, I want to write my webpage. It will be nothing more than a simple Leaflet map with one marker at a point retrieved from MongoDB. It will also be a Jinja Template.

Here is the code for my webpage.

<!doctype html>
<head><title>OpenShift: Leaflet, MongoDB, Python. Jinja2</title>
<link rel=’stylesheet’ href=’http://cdn.leafletjs.com/leaflet-0.4/leaflet.css&#8217; />
</head>
<div id=’map’ style=’width: 900px; height: 350px’></div>
<script src=’http://cdn.leafletjs.com/leaflet-0.4/leaflet.js’></script&gt;
<script>
var map = L.map(‘map’).setView([40.71367, -73.99364 ], 13);
L.tileLayer(‘http://{s}.tile.cloudmade.com/API-KEY/997/256/{z}/{x}/{y}.png’,{attribution:’Paul Crickard’, maxZoom: 18 }).addTo(map);
L.marker({{ coord[“pos”] }}).addTo(map)
</script>

This file needs to be put in a folder in WSGI called TEMPLATES. You will notice another folder in the directory called STATIC. This is where you can put CSS and JS files.

The code is straight HTML and JavaScript with the exception of   {{ coord[“pos’ }}. This is the Jinja part of the template. In the Flask Application, I will pass coord to the template.

Let’s take a look at a simple Flask application that will grab the point from MongoDB and send it to my Jinja Template. Here it is:

from flask import render_template

@app.route(“/ws/albuquerque”)
def albuquerque():
conn = pymongo.Connection(os.environ[‘OPENSHIFT_MONGODB_DB_URL’])
db = conn.parks
for z in db.location.find():
q=z
return render_template(‘map2.html’,coord=q)

The file I modified already has several import statements but you will need to add render_template to use Jinja. I changed the route to /albuquerque – I was going to use some local data but decided for a test I would use the parks JSON file so now the Albuquerque doesn’t make much sense. Then I grab a cursor and pass a point to q. Lastly, I call the render_template and hand it q as coord. Now in my HTML/Jinja Template, I can call coord. For the future, I would run a loop in the Template that grabbed all the points. A loop in Jinja looks like this:

{% for x in json %}
   Do something with X: 
   L.marker({{ x }}).addTo(map) 
{% endfor %}

Now go to my RHCloud.com and see it in action. It seems rather simple – I need to add some popups – but there is some powerful technology behind this. MongoDB has some spatial features like $near, $box or $within $center of a circle. This should somewhat please the spatial is special crowd. One of the things I love is that I can have an object {“pos”:[35,-106],”name”:”ABQ”} and I can have another {“pos”:[33,-106],”name”:”LL”, “visitors”: 50}. In a table, every record has the same fields – I don’t have to do that in MongoDB. I see MongoDB as much more flexible for my needs.

Next Steps

What functionality would I like to add to this?

  1. A form that allows the user to select either all the data or a subset using simple radio buttons or combo boxes.
  2. A way to download a report of the data selected.
  3. A form for users to enter new data.
  4. Popups with additional data. This should only require adding .bindPopup( {{coord["popupContent']}} )

One last thing: It runs on mobile. My HTML is not very good, but a mobile Leaflet app is simple. I have one that uses your phone’s current location on my website.

My App Running on Mobile

Flask App Running on iPhone 4S

MongoDB and Shapely

18 Nov

I wanted to connect my mongoDB to Shapely so that I could create buffers and do some other things, like spit out my points and buffers as WKT. Here is what I came up with.

from pymongo import Connection,GEO2D
from shapely.geometry import asShape,Point
db=Connection().a
db.b.create_index([(“loc”,GEO2D)])

db.b.insert({“loc”:[-106,35]})
db.b.insert({“loc”:[-109,39]})

for q in db.b.find({“loc”:{“$near”:[-106,35]}}).limit(1):
point=Point(q[“loc”][0],q[“loc”][1])  #this is passing mongoDB to Shapely

point.x    #just checking I didn’t make a mistake

point.y

t=point.buffer(2)
t.area  #checking. Result is 12.a lot of number

point.wkt

t.wkt

Now I can use mongoDB and find all the points in a bounding box or near another point or near a point where type=Coffee Shop, then send that to Shapely for a buffer or to measure distance or whatever, then spit it out as WKT. Saw a tutorial on using shapely with GDAL/OGR and Fiona. Goodle that and see how to send it all back to a shapefile.