Fetch and plot data from the Google Ngram Viewer using Python

Take a look at this Github repo for a Python script that can be used to fetch data from the Google Ngram Viewer. See the README file in the repo for instructions on how to use the script, as well as the PLOTTING file for instructions on how to plot the data in Python using pandas.

The Python script is a modified version of the Python script from the culturomics.org website.

To see an example of the type of research that's being done using the Google Ngram Viewer, check out this TED Talk by the creators of culturomics.org at Harvard.

NOTE: Be nice to the Google servers and don't beat them to death with this script.

Use Python to scrape data from Newegg and store it in an SQLite database

Check out this repo of mine on Github for a set of Python scripts for scraping various data from Newegg.com and storing it in an SQLite database.

The scripts use the mobile Newegg.com site to retrieve the list of all the products in a category, then uses the product ID for each product in that category to fetch and parse data from the Newegg JSON API before transforming it into a pandas DataFrame and dumping it into a table in the SQLite database.

The reason for using the mobile Newegg.com site is because it is lighter weight than the desktop version of the site.

As of right now, I have scripts in the Github repo setup to collect data on the following:

• Desktop CPUs
• Desktop Memory
• Hard Drives
• Laptops
• LCD/LED/Plasma TVs
• PS3 Games
• XBox 360 Games

Each script dumps the data into a separate table in the SQLite database file. Feel free to tweak these scripts however you like, perhaps to retrieve different data from the Newegg JSON API for each product or even to change it to grab data from another set of products on Newegg. Enjoy!

Estimate an Econometric Model from Scraped Data in Real-Time using Python

Here's a link to a fairly lengthy tutorial I put together titled "Estimate an Econometric Model from Scraped Data in Real-Time using Python" which is posted at another one of my websites:  live.economics.io

The application that's analyzed in the tutorial involves determining the factors that influence the price of a desktop CPU, using the prices and features of processors that are currently listed at newegg.com as the data set for the model.

Lastly, if you haven't been to econpy.org in a while, check out the recently updated interface which now includes an IPython terminal right in the browser -- special thanks to PythonAnywhere on that one!

Retrieve Stock Price Data from Yahoo! Finance using Pandas

Here is a simple example of how to use the Python package pandas to retrieve stock price data from Yahoo! Finance (hereby YF). Assuming pandas is already installed on your system, open up a Python shell and make the following imports:

from pandas.io.data import DataReader
from datetime import datetime

DataReader is what we'll use to retrieve YF stock price data. The DataReader class can also retrieve economic data from two other remote sources, namely the St. Louis Federal Reserve's FRED database and Kenneth French's Data Library, both of which will be the topic of future posts.

For now, let's say we want to grab historical stock price data on Microsoft (MSFT). We'll do so by defining an object msft that contains daily prices for Microsoft as so:

msft = DataReader("MSFT",  "yahoo")

The first input of DataReader is the name of the dataset (e.g. the stock ticker for YF) and the second is the location of the dataset ("yahoo" for YF). The msft object is a DataFrame object whose rows are days of the year and columns are the prices for each day, labelled Open, High, Low, Close, Volume and Adj Close, respectively. 

By default, the data contains observations from the past year, but that can be changed by providing a datetime object as the third input to DataReader:

msft = DataReader("MSFT",  "yahoo", datetime(2009,1,1))

The msft object now contains daily price data from the start of 2009 up to today's date. To print a particular column of msft, such as the stock's daily volume, enter:

print msft["Volume"]

As another example, to print the adjusted closing price of MSFT, but only for the last 100 days, enter:

print msft["Adj Close"][-100:]

That's it for now. Expect a follow-up post soon that'll include a more involved example using the FRED database.

Grab BibTeX citations from Google Scholar using Python

1. Download this file: GitHub: gscholar.py
2. Put the gscholar.py file in your '/bin' folder. (Linux)
3. Open a terminal and type: gscholar.py "enter search terms here"
4. Copy the citation into your .bib file.

Using Python for Econometrics and Linear Algebra

The following links contain Python code for various tasks in econometrics and linear algebra. They come from the owner of the blog, Digital Explorations. As a PhD student in economics, and a Python enthusiast myself (see: econpy.org), I am more than happy to see Python code like this being created.

• Cholesky decomposition and inverse of positive definite matrices
• Solving an AR(p) time series model using least squares
• Solving Ax = b where A is symmetric positive definite
• White's test for heteroscedasticity
• Breusch-Godfrey test for serial correlation up to order p
• Nonlinear regression, fitting of linearizable nonlinear equations to data
• Performing one iteration of Cochrane-Orcutt procedure
• Iterated Cochrane-Orcutt procedure
• Ljung-Box test for autocorrelation
• Statistics: Computing quantiles
• Gramm Schmidt orthogonalization algorithm in Python
• QR decomposition with Gramm-Schmidt orthogonalization Python
• Gauss elimination in Python
• Computing determinants via diagonalization