Davis Vantage Pro weather station: software for Python and Linux

This page will describe the software I'm writing to pull information from the Vantage Pro to Linux and then put it on the Web. Currently it - and the software - is under construction. The code isn't in CVS or even in a tar ball on the Web, but I'm can tar up the latest on request.

Linux, Python, and MySQL

These three products are great, flexible, and easy to work with. (Well, those of us who think Unix is user-friendly admit that Unix is picky in choosing its friends.) I use Linux because I've used Unix since 1980 or so. I use MySQL because I had to convert the NH Mensa mail list from a flaky, mostly Asthon-Tate compatible dbase II system to something that had a future. Unfortunately, MySQL and similar "modern" databases don't come with a semi-decent scripting language, so I had to rewrite a big slew of scripts. That brought me to Python, a language that works well on Windows too. If you're considering learning Perl or Java, learn Python first and you may discover you don't need the others.

One of my reasons to buy a weather station was to have an excuse to write more Python code. The Davis Vantage Pro has a fairly simple protocol and is publically documented. All in all, the Vantage Pro is a good system to access from a home computer. The things I knew I needed to do were:

Provide WWW pages with current and historical weather data.
Davis's WeatherLink software and others do this, I didn't find a good starting point for Linux software. I didn't look very hard, as I wanted to write code anyway. Gnuplot makes it easy to come up with graphs to put on the WWW.
Maintain long term computerized records.
I still haven't decided if I want to join a NOAA affiliated program.
Automatically update other archives.
Todd Gross, a TV met at Boston's WHDH has worked long and hard to develop a spotter network, a process that is ongoing. WHDH now accepts weather data by phone, a private Yahoo group's Email list (WHDHWX), and a WWW form that updates an interactive area data page. I have code that runs via cron to submit the post to update the WHDH page. Another effort at http://newxspotters.net/ is used by WHDH and is available to their weather spotters and others beyond the area WHDH covers. My software is behind the data logging parts of that site but is not described on the web yet.
Join the Weather Underground.
I thought I'd be more involved with this by now, but the above wound up getting priority.

In addition to the end goals above, there's a fair amount of infrastructure that includes its own goals and small projects:

Fixing gnuplot's graphs.
I mentioned above that it's easy to make graphs for the Web with gnuplot. However, coming up with graphs that look good requires a bit of hackery. I first ran into that with .GIF files from Gnuplot and wrote a C program that changes colors and transparency. I bashed that into a new verion that does the same thing with .PNG files. It's still in C.
A MySQL registry-like table.
Anyone who started working with Unix in PDP-11 days appreciates the ASCII configuration files it has. Being able to work with them with the rich tool set that comes with Unix opened the door to all sorts of administrative scripting tools. That still holds true today. However, I discovered that in creating the "kludge tower" of programs to do everything I needed, I was creating a number of small files. For example, the WHDH site needs current temperature and wind speed. The program that updates my WWW site saved those in two small files. A cron entry picked them up and passed them to another program that sent the form to WHDH. Trying to keep several pieces of changing data in a file became a tad messy. Since everything is working with a database, I wrote a Python module to maintain a key/value registry and updating each takes just a line of code. Of course, there are functions to export and import files with colon separated fields so you can still work in classic Unix style.
Distractions (mostly welcome).
I've learned a bit more about weather than I expected while working on this code. The learning and writing code based on that took some time I wasn't planning to do right away. However, it's time well spent. One nice thing about a sizable personal project like this is I'm not constrained by customer-centric schedules. Well, almost. The first pass of the WWW update code didn't use MySQL at all because I wanted data on the web ASAP without having to boot Windows and run WeatherLink there.
The Vantage Pro protocol doesn't report all derived data. For example, dew point is derived from temperature and relative humidity. In hunting down the formulae and theory behind that, I came across formulae to compute the frost point, something that makes more sense in subfreezing weather here at ground level with condensation nuclei everywhere I look. So I compute that too.
Todd Gross has a home snow making system and lamented that you need wet bulb temperatures around 29°F or below to make snow, but that few (none?) weather stations report wet bulb. Computing it turns out to be an iterative process, so I wrote a program to make a colorized table to convert air temperature and dewpoint to wet bulb.
The new windchill formula is much, much more sensible than the old one. The old one wasn't worth computing, the new one is. I'm not fond of a couple of the policies behind the new one, namely that wind chill is a dimensionless integer and is undefined at wind speeds 3 mph and less. However, they got a couple things right, especially trying to get the wind chill math to make wind chill nearly equal air temperature at walking speed.

Database and tables

I'm only using a single MySQL database (wx) with these tables:

raw
This holds most of the data the weather station logs each 10 minutes. I'll refer to it as "wx.raw".
daily
This has one entry per day for things like high and low temperatures, snow depth, etc.
registry
This holds data that is passed between programs. Each record has fields for the key, values, and type (string, integer, or real).

Program Flows

I want to come up with a matrix that shows the Python programs used, the MySQL tables affected, and other files affected. Collecting that will take a bit of time, so I'll just write some of the supporting text for now.

curupdate.py - read recent data from console and save in registry.

This reads the high/low data and saves the day's high and low temperature date in wx.registry. It also reads the current data via the LOOP request and saves the interesting data in the registry. This is used by several other programs, notably webupdate.py.

daily_update.py - bring wx.daily up-to-date

This reads recent data in wx.raw and creates a summary for each day in wx.daily.

rawupdate.py - copies Vantage Pro data to wx.raw

I have my Vantage Pro console logging data every 10 minutes. Cron runs a make command every 20 minutes to update the WWW site, and this is the first step in the process. Rawupdate.py gets the latest data from the console and adds it to the wx.raw table. It remembers the date and time of the last sample in a text file, soon to become an entry or two in wx.registry. The basic flow is to connect to the console, recover the timestamp, read new data from the console (via the "dmpaft" command), check each sample for reasonableness and add them to wx.raw. Samples are generally rejected if the wireless link between the sensor suite and console dropped a lot of data. Currently I'm just a few feet from the edge of the range and rain or snow plays havoc with reception. It's easier to just drop the sample than to keep the good data (e.g. barometric pressure) and handle the missing data.

webupdate.py - create gnuplot and sed scripts

This fetches the last 24 hours worth of data from wx.raw and creates a gnuplot script and a data file to create graphs for the Web site. It gets current data from the console via a "loop 1" command and creates a sed script to convert a template file into a HTML page text for the current conditions.

pngfix.c - improve .PNG file characteristics.

Obviously the next step is to run gnuplot and sed. However, there are a number things that gnuplot does that I don't particularly like. The .PNG files it creates have a bright white background. That should be saved for highlights in photographs! Also, gnuplot uses light colors for its plotted lines. Yellow on white doesn't work. Finally, lack of control over line colors prevents me from using colors in the order I want.

Instead of doing the sensible thing and fixing gnuplot, I wrote a C program that patches the .PNG files created by gnuplot. The Makefile calls pngfix to dress up all the plots then runs ftp to get the new data on the server.

whdhupdate.py - update WHDH weather spotters' data

Cron runs this four times a day before newscasts to update their spotter data page. This gets the recent temperature and wind speed from registry values written by curupdate.py.

whdh.py - print Email template for WHDH weather Email list

This gets current weather data and yesterday's high and low, then prints a template for an Email post to WHDHWX. I usually only post when I have something to say beyond what the raw data is.

spotupdate.py - update weatherspotters.com

This updates the weatherspotters.com site with the current temperature and high/low data for the day. It also wants snow/ice/rain, but I'm not collecting that yet.

query.html, cgi-bin/wx_fetch

Query.html displays a couple web forms for querying the raw and daily tables. The forms let you select a starting date, a duration, and the variables to display. Clicking Submit sends the data to wx_fetch which extracts the data from the table and has gnuplot create a .png file. wx_fetch then returns HTML code to access the .png file.

mysqlreg.py, reg.py, regtest.py

mysqlreg.py implements a key/value registry that is used by most of the main code. The registry exists as a mysql table named "registry". reg.py is a command-line program that lets you add, delete, view, and change entries. regtest.py is a limited unit test program.

vantage_pro.py - Davis protocol module

This code understands the Vantage Pro protocol. A class for each protocol request holds the data from the replies. This is imported into any program that talks to the Vantage Pro console.

wxmath.py - library of weather related math routines

Routines include:

Convert degrees to compass point. (e.g. 45 to NE).
Convert wind velocity to and temperature to wind chill.
Convert temperature and humidity to dew point and frost point.

Occasional use programs and quick&dirty hacks

daily.py - Create High/Low plots

This reads some data from wx.daily and creates high/low summary for a range of months in www/highlow.png.

depthupdate.py - Probably useless

I wrote this to copy snow depth from a .csv file emitted by gnumeric to wx.daily. It was useful for moving a season's worth of data, it's not so convenient to use for a few days' data.

export.py - convert WeatherLink date/timestamps to Mysql

Davis's WeatherLink program exports uses an American "mm/dd/yy hh:mm{a,p}" style timestamp, whereas MySql use the odometer-style "yyyy-mm-dd hh:mm:ss". This program export.txt (created by WeatherLink) and writes export.mysql. I forget how I got that into MySQL.

waketest.py - early protocol test program

This is unnecessary for normal operation. It sends various protocol requests to the console and reports progress.

Contact Ric Werme or return to his home page.

Last updated 2005 Nov 25.