Archive | December, 2012

MongoDB in QGIS

28 Dec
Displaying my MongoDB Data

Displaying my MongoDB Data

I recently installed the MongoDB Plugin for QGIS. The plugin allows you to connect to a MongoDB and load your data. I stored point data – long and lat – for all the public art in Albuquerque. Using the plugin, I connected to my MongoDB and loaded them.

I had trouble getting the plugin to install. My fix was to install pymongo and BSON to my local Python2.7 then copy it from the site-libs to the QGIS directory (C:\Program Files\Quantum GIS Lisboa\apps\Python27\Lib\site-packages). then I had to turn on the plugin under manage plugins.

This did the trick. Then I fired up my mongod, loaded QGIS and connected using the defaults. It prompted me for a Database and then a Collection. Looks like it requires a GEO2D. You can see my data in the image above.

Can’t wait to start playing….

Advertisements

D3 and Ajax: Updating a Bar Chart With New Data

28 Dec

This is the site I will build.

I have been showing several examples of graphing – mostly using Matplolib. In this example, I will show a bar chart in a webpage using D3 and updated via Ajax requesting a text file with the data. I also have a simple python file that randomly generates 20 data points and feeds it to the text file – my host doesn’t allow me to run the python, but you can get the idea.

I took the code from examples by Scott Murray, and some standard Ajax code block off W3. Making a few changes I was able to get it to work. I like to modify existing code to get a feel for what affects what and how things work. This really helps it all sink in.

This draws the first bar chart when the page loads.

var dataset=[12, 10, 13, 19, 21, 25, 22, 18, 15, 13, 11, 12, 15, 20, 18, 17, 16, 18, 23, 25];
var w = 500;
var h = 100;
var barPadding = 1;

var svg = d3.select(“body”)
.append(“svg”)
.attr(“width”, w)
.attr(“height”, h);

svg.selectAll(“rect”)
.data(dataset)
.enter()
.append(“rect”)
.attr(“x”, function(d, i) {
return i * (w / dataset.length);
})
.attr(“y”, function(d) {
return h – (d * 4);
})
.attr(“width”, w / dataset.length – barPadding)
.attr(“height”, function(d) {
return d * 4;
})
.attr(“fill”, “teal”);

This is my onclick function that uses Ajax and updates the chart:

function clicked(url)
{
if (window.XMLHttpRequest)
{// code for IE7+, Firefox, Chrome, Opera, Safari
xmlhttp=new XMLHttpRequest();
}
else
{// code for IE6, IE5
xmlhttp=new ActiveXObject(“Microsoft.XMLHTTP”);
}
xmlhttp.open(“GET”,url,false);
xmlhttp.send(null);
var newd=xmlhttp.responseText;
var da=newd.split(‘,’);
for (var i in da){
da[i] = parseInt(da[i], 10)
}
document.getElementById(‘test’).innerHTML=xmlhttp.responseText;
svg.selectAll(“rect”)
.data(da)
.attr(“x”, function(d, i) {
return i * (w / da.length);
})
.attr(“y”, function(d) {
return h – (d * 4);
})
.attr(“width”, w / da.length – barPadding)
.attr(“height”, function(d) {
return d * 4;
})
.attr(“fill”, “teal”);
}

All I am doing is requesting a text file of data values (12,34,65,67,32,21,10…), then I split it in to numbers and convert to integers. I also print out the values, but that was just a check for me to see what was being returned. The rest is from an example by Scott Murray on clicking a paragraph to change the data – but with the variable hard coded in the JavaScript. I just passed the textfile to the variable and viola!

Connect the function to a button: <button type=”button” onclick=”clicked(‘data.txt’)”>Click
Me</button>.

My goal was to see how I could have a python file write out a data file and then load it in a webpage, but be able to update the data without refreshing everything on the page.  Could have done it all in Python, with Flask and Jinja but I don’t have a production setup for this. This was the most reasonable way to connect what I may be able to run in production.  Here is a python file that feeds random integers from 5 to 25 to a text file – the one that we will load in.

import random
data=open(“data2.txt”,”w+”)
x=0
for x in range(20):
i=random.randint(5,25)
if x==19:
data.write(str(i))
break
else:
data.write(str(i)+”,”)

data.close()

This could connect to MongoDB, SQLite, or generate data on the fly like it’s doing now. The principle is the same – do something and write it out to text.

IPython Notebook

27 Dec
Image

My IPython Notebook

I love IPython Notebook. It is an interactive web environment for executing code. You can also save and share your notebook. Don’t have IPython? you can view notebooks on the web using the website http://nbviewer.ipython.org. I have a notebook for my Pandas Bar Chart Example. On the viewer site, enter the URL:

http://educationalfacilityplanning.com/MySamplePandasBarChart.ipynb

Or, click http://nbviewer.ipython.org/url/educationalfacilityplanning.com/MySamplePandasBarChart.ipynb

Matplotlib Pie Chart

21 Dec
A Pie Chart

A Pie Chart

Not much to say here. A simple pie chart.

import matplotlib.pylab as plt

fig = plt.figure()
labels = [‘Paul’, ‘Joe’, ‘Bill’, ‘Tom’]
fracs = [40, 10, 25, 25]
plt.pie(fracs, labels=labels)
plt.title(‘People Lorem Ipsum’)
fig.savefig(“pie.jpg”)

Matplotlib Bar Chart

21 Dec
Simple Bar Chart

Simple Bar Chart

I did a post on Pandas and plotting. In that post, Matplotlib was used from Pandas. In this post, I will show a simple bar chart in Matplotlib- sans Pandas.

Here is the code:

import matplotlib.pylab as plt

fig = plt.figure()            #create a figure
ax = fig.add_subplot(1,1,1)    # add plot
x = [2,4,6,8]         #x axis value
y = [4,8,12,16]   #Y axis: height
ax.bar(x,y, align=’center’, color=’green’) #required ax.bar(x,y)
fig.savefig(‘paul2.jpg’)      #This saves a file
#plt.show()       This shows it on the screen and gives you the option to save.

That’s it. I like centering my bars, but leave the options out and it will align the left corner with the value of X.  Don’t forget a title – I did. Put fig.suptitle(‘Sample With YERR=2′, fontsize=14, fontweight=’bold’) before adding the subplot. And lastly I like to add the error bars using ax.bar(x,y, yerr=2, align=’center’, color=’green’). The result looks like this:

With a Title.

With a Title.

UPDATE: I hate that the bars go to the top of the chart. fix that by setting the Y Limit. Drop this line in your code: p.ylim(2,20)

Y Limit Set: Much Nicer

Y Limit Set: Much Nicer

Pandas and Python…..Oh My.

21 Dec

Crunching data and rearranging data in Python is cool, but I really need to visualize it. Nothing fancy, just a bar or line chart. I recently saw a D3 implementation in Python – awesome, but for now I just want to stick to Matplotlib. I grabbed a few books on scientific python and data in python. They seem to love IPython – the web notebook is pretty cool. Another tool that is often mention is Pandas. This is what I want to use – it uses Matplotlib for plotting. The one feature that caught my attention right away was the DataFrame. Think of it as a Excel spreadsheet, then if anyone asks about it, tell them it’s like data.frame() in R.  When I learn something, I like to start bare bones, then build it up with extra options and variations. I have put together some very minimal examples of plotting DataFrames and a Series in Pandas. From here you should have a good grasp of how to do more.

Plotting a Series.

Plotting a Series.

Plotting a Series requires a Series and a type of chart. Here is my code:

from pandas import Series
import matplotlib.pyplot as plt

b=[2,4,6,8,10]
a=Series(b,index=[‘a’,’b’,’c’,’d’,’e’])
Series.plot(a, kind=’bar’) #change to ‘barh’ for horizontal. Can also declare kind=’line’
plt.show()

Plotting a DataFrame

Plotting a DataFrame

 

Plotting a DataFrame is what I need the most in my work. Earlier I compared a DataFrame to an Excel spreadsheet. Here is what a DataFrame looks like:

Simple DataFrame

Simple DataFrame

Looking at the DataFrame and the Chart,  notice that each row plots as a group labeled by the index and columns. The DataFrame is created by passing a Numpy Array.:

a=np.array([[3,6,8,9,6],[2,3,4,5,6],[4,5,6,7,8],[3,6,5,8,6],[5,8,8,6,5]])
df=DataFrame(a, columns=[‘a’,’b’,’c’,’d’,’e’], index=[2,4,6,8,10])

To plot the chart, just call plot and pass a type. Here is the complete code:

from pandas import DataFrame
import matplotlib.pyplot as plt
import numpy as np

a=np.array([[3,6,8,9,6],[2,3,4,5,6],[4,5,6,7,8],[3,6,5,8,6],[5,8,8,6,5]])
df=DataFrame(a, columns=[‘a’,’b’,’c’,’d’,’e’], index=[2,4,6,8,10])

df.plot(kind=’bar’)
plt.show()

This is how I learned to use Pandas DataFrame and to plot my data. Knowing this, I felt much more comfortable looking at more advanced examples online.

The Zen of Python, by Tim Peters

20 Dec

Almost forgot about this. Type ‘import this’ at a python command line.

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren’t special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one– and preferably only one –obvious way to do it.
Although that way may not be obvious at first unless you’re Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it’s a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea — let’s do more of those!