AccuWeather API Python Code

Here is a quick and dirty snippet of Python code to get you started with the AccuWeather API. Note that it offers no fault tolerance, optimization, or other things you might expect to find in finished software and only extracts a small subset of the data available in the JSON data.

import json
import time
import urllib.request

API = YOUR_API
LOCATION_ID = LOCATION



def get_weather(api, location_id):
    url = 'http://dataservice.accuweather.com/currentconditions/v1/%s?apikey=%s&details=true' % (location_id, api)
    print(url)
    with urllib.request.urlopen(url) as url:
        data = json.loads(url.read().decode())
    print(data)
    return (data[0]['Temperature']['Imperial']['Value'],
            data[0]['RelativeHumidity'],
            data[0]['Wind']['Direction']['Degrees'],
            data[0]['Wind']['Speed']['Imperial']['Value'],
            data[0]['UVIndex'],
            data[0]['CloudCover'],
            data[0]['Pressure']['Metric']['Value'],
            data[0]['Precip1hr']['Metric']['Value'],
            data)


timestamp = time.time()

temperature, humidity, wind_bearing, wind_speed, uv_index, cloud_cover, pressure, precipitation, raw \
    = get_weather(API, LOCATION_ID)

 

Advertisements

Incremental Backup Script Explained

In an earlier post I laid out why I wanted yet another layer of backups for my important documents and I provided the script without much explanation. As an exercise for myself, I decided to post a step by step explanation for why I wrote the script the way I did. This will be educational for me, and hopefully interesting for you.

Alright, lets start at the beginning.

#! /bin/bash
# 

set -ue

 

I am highly influenced by David Pashley’s superb article “Writing Robust Bash Shell Scripts.” Pashley is not only a great programmer, he is a superb writer who can really explain his reasoning in a way that I find appealing.

Since I want this script to be robust, adding “set -ue “on the first line is a must. The “u” breaks the script if a variable is unset. This is critical as human error can easily turn a script into a vicious beast that will consume your data. The “e” breaks the script on any error. This ensures that everything behaves as expected. Unless there is a good reason not to, always “set -ue”

For reference here is the explain shell for the command.

Here is the first function:

function build_vars
{
# Current Date / Time
epoch=$(date +%s)
month=$(date +%m)
year=$(date +%Y)
day=$(date +%d)

# Building Environment: Directories
configdir="$config_dir"
yeardir="$backup_dir"/"$year"
monthdir="$yeardir"/"$month"
daydir="$monthdir"/"$day"
backupdir="$daydir"

# Building Environment: File Names
backupfile="$backupdir"/"$prefix"."$epoch".tar.bz2
logfile="$configdir"/"$prefix".log
lastepochfile="$configdir"/"$prefix".last
}

 

Basically this function sets up all of the variables which will be used in the rest of the script. Putting them in a single function improves readability and makes maintenance easy.

Here we also see the first crumbs of how the backup system works. By finding the current Unix epoch, the script discovers the current time in a non-timezone dependent way. It can then use this as reference point and as a means of ultimately building unique file names.

In addition, this function builds all of the directory names that will be used throughout the remained of the script.

I could probably make this more efficient by only running “date” once, but all of the target systems are multi-Ghz so I can endure a couple of wasted cycles.

Here is the second function:

function check_epoch
{
#Checking Time Since Last Run
if [ -n "$(find "$lastepochfile" -mtime -1)" ]; then 
  echo "Less than 24 hours since last archive"
  exit 0
fi
}

 

This function uses “find” to check if the file “$lastepochfile” is older than 1 day. As “$lastepochfile” is the last thing created in the script this effectively checks to see if the script successfully completed in the last 24 hours. Since I ended up setting a cron job that runs once an hour, this ensures that even if a system is only powered on for short times during the day the job will still get done but not OVER done which would waste cycles and file space.

For reference here is the explain shell for the find command used.

Here is the third function:

function check_env
{
#Checking Configuration Directory
if [ ! -d "$configdir" ]; then
  mkdir "$configdir"
  echo "$epoch" Making "$configdir" >> "$logfile"
fi

#Checking Year Directory
if [ ! -d "$yeardir" ]; then
  mkdir "$yeardir"
  echo "$epoch" Making "$yeardir" >> "$logfile"
fi

#Checking Month Directory
if [ ! -d "$monthdir" ]; then
  mkdir "$monthdir"
  echo "$epoch" Making "$monthdir" >> "$logfile"
fi

#Checking Day Directory
if [ ! -d "$daydir" ]; then
  mkdir "$daydir"
  echo "$epoch" Making "$daydir" >> "$logfile"
fi
}

 

While this covers a lot of lines, it is really doing something very simple, it checks to see if the year / month / day directory structure exists for the backup snap-shot that will ultimately be produced. If it does not exist then it makes it.

Effectively what this means is that each snapshot will be in its own directory which will be easy to browse to and find.

Here is the fourth function:

function write_file
{
#Incremental Tar & Compression
echo "$epoch" Writing "$backupfile" >> "$logfile"
find "$target_dir" -type f ! -name ".*" -newer "$lastepochfile" -print0 | \
tar czvf "$backupfile" --null -T -

#Log Success
echo "$epoch" Success "$backupfile" >> "$logfile"
}

 

This is the payload of the entire script. Everything else has just been leading up to this. First “find” is used to search “$target_dir” for all files that are newer than “$lastepochfile” and do not match the pattern .* . The reasoning for this should be pretty obvious, but I should not that I added the exclusion (by using the !) so that I wouldn’t needlessly copy lockfiles etc.

To ensure proper streaming of the files to tar I used the -printO.

It is important to note that this only searches out changed files and copies only their directory structure. My first approach copied the entire directory structure resulting in mostly empty directories… this resulted in overly large files as all those empty directories eat up space.

The tar line is pretty standard, note I use –null to ensure that it properly deals with null-delimited output and -T sets tar up to receive the filenames.

For reference here is the explain shell for the find and tar commands used.

Success is then logged.

Here is the fifth function

function write_epoch
{
# Unprotect Last Epoch File
chmod 600 "$lastepochfile"

# Write Last Epoch File
echo "$epoch" > "$lastepochfile"

# Protect Last Epoch File
chmod 400 "$lastepochfile"
}

 

In a way this is the most important part. First it unprotects “$lastepochfile” then it writes the epoch to it and finally the script protects “$lastepochfile.” As the entire system depends on the accuracy of “$lastepochfile” the protection should prevent casual deletion. Worst thing comes to worst, the epoch of the last run is saved in the log file and a new “$lastepochfile” could be created with the date.

Writing the epoch to the file isn’t necessary, but I think it provides a nice backup record and isn’t really that different than simply touching the file.

There is, likely, a way of doing this that does not require an external file (instead referencing the last created archive) but I like that this approach is easy to debug and trivial to understand.

Finally the actual program

# Configuration
prefix=important
target_dir=/home/user/Important
backup_dir=/home/user/ImportantArchive
config_dir=/home/user/ImportantArchive/.kb

#Program
build_vars
check_epoch
check_env
write_file
write_epoch

exit 0

 

As you can see, I’m really just using functions as a way of organizing the script. I prefer to think in terms of modules which I can replace as needed and this provides that. Also I like that “forcing” an update would be as simple as commenting out “check_epoch”.

So there you have it, it is no masterwork of programming but this script has now been comfortably running on two different systems automatically using CRON for 4 months without a single glitch. I can’t really imagine a better testament to its stability than that!

So what could I improve about it? I’m no expert at scripting so I’d love any feedback that anyone has.

Incremental Backup Script

I am Paranoid! (When it comes to Backups)

There I said it. I am a paranoid person but thankfully most of my paranoia is restricted to backups of important files. When you think about it, an academic truly is their files… my main document directory represents thousands of hours of time and years of my life. Scary thought, right?

So I pursue a multi-layered approach to backing up my critical files. My current approach utilizes manual backups, p2p backups / synchronization among my systems, and backups to Google Drive. However, even that wasn’t enough for me to be comfortable. What if I overwrote a file by accident? Only the p2p network (which uses Syncthing and includes rudimentary versioning) could save me. But what if the conflict resolution system made an incorrect decision and preserved the wrong copy of the file? What then?

Like I said, I’m paranoid. So I decided that each major work system that I use should make incremental backups on a daily basis. Essentially with the command line you can easily search for changed files, then tar them. This produces a snap-shot style backup which could, in theory, be used to regenerate your files as they were at any given time.  Write a quick cron task and now it is scheduled and automated.

However, just to make things more complicated I had three more requirements:

  1. While I want no more than one incremental, snap-shot style backup per day, I cannot guarantee that each system will be on at a given time.
  2. I want the snap-shots to be easy to navigate.
  3. As this is a script that I intend to run automatically for years, it has to have robust error handling and logging.

So this is now more complex than a single line of BASH, time for some scripting!

I am attaching the solution I came up with but I’ll explain my reasoning for each thing more in another post

Here is the script I came up with. There is nothing innovative about it but it has been running once an hour (it is scheduled with cron) for 4 months without a single hiccup or fault–my definition of success! If it is useful to you feel free to use/modify/abuse it… consider it under the MIT License.

Thanks for reading and I’d love to hear any thoughts that people have on how to improve it

#! /bin/bash
# 

set -ue

function build_vars
{
# Current Date / Time
epoch=$(date +%s)
month=$(date +%m)
year=$(date +%Y)
day=$(date +%d)

# Building Environment: Directories
configdir="$config_dir"
yeardir="$backup_dir"/"$year"
monthdir="$yeardir"/"$month"
daydir="$monthdir"/"$day"
backupdir="$daydir"

# Building Environment: File Names
backupfile="$backupdir"/"$prefix"."$epoch".tar.bz2
logfile="$configdir"/"$prefix".log
lastepochfile="$configdir"/"$prefix".last
}

function check_epoch
{
#Checking Time Since Last Run
if [ -n "$(find "$lastepochfile" -mtime -1)" ]; then 
  echo "Less than 24 hours since last archive"
  exit 0
fi
}

function check_env
{
#Checking Configuration Directory
if [ ! -d "$configdir" ]; then
  mkdir "$configdir"
  echo "$epoch" Making "$configdir" >> "$logfile"
fi

#Checking Year Directory
if [ ! -d "$yeardir" ]; then
  mkdir "$yeardir"
  echo "$epoch" Making "$yeardir" >> "$logfile"
fi

#Checking Month Directory
if [ ! -d "$monthdir" ]; then
  mkdir "$monthdir"
  echo "$epoch" Making "$monthdir" >> "$logfile"
fi

#Checking Day Directory
if [ ! -d "$daydir" ]; then
  mkdir "$daydir"
  echo "$epoch" Making "$daydir" >> "$logfile"
fi
}

function write_file
{
#Incremental Tar & Compression
echo "$epoch" Writing "$backupfile" >> "$logfile"
find "$target_dir" -type f ! -name ".*" -newer "$lastepochfile" -print0 | tar czvf "$backupfile" --null -T -

#Log Success
echo "$epoch" Success "$backupfile" >> "$logfile"
}

function write_epoch
{
# Unprotect Last Epoch File
chmod 600 "$lastepochfile"

# Write Last Epoch File
echo "$epoch" > "$lastepochfile"

# Protect Last Epoch File
chmod 400 "$lastepochfile"
}

# Configuration
prefix=important
target_dir=/home/user/Important
backup_dir=/home/user/ImportantArchive
config_dir=/home/user/ImportantArchive/.kb

#Program
build_vars
check_epoch
check_env
write_file
write_epoch

exit 0

 

How “Functional” Should Arduino Be?

As mentioned in my last post, using Timer.h in my DHT11/ESP8266 project required me to transform much my sketch into functions. My initial experience with programming was in BASH and since then I’ve done some Python, so I am comfortable with using functions. But in Arduino I kind of liked being able to accomplish things without them. Take a look at this snippet that does basically everything I need the board to do:

void loop() {
  for (c = 0; c < 60; c++) {
    h = dht.readHumidity();
    f = dht.readTemperature(true);
    fS = (alpha * fS) + ((1 - alpha) * f);
    hS = (alpha * hS) + ((1 - alpha) * h);
    Serial.print("c="); Serial.print(c); Serial.print(" f="); Serial.print(f); Serial.print(" fS="); Serial.print(fS); Serial.print(" h="); Serial.print(h); Serial.print(" hS="); Serial.println(hS);
    delay(1000);
  }
  Serial.println("Sending to Thingspeak");
  ThingSpeak.setField(1, fS);
  ThingSpeak.setField(2, hS);
  ThingSpeak.writeFields(myChannelNumber, myWriteAPIKey);
  blink(BLUE, 1);
}

That is code I can easily understand without referring to multiple functions. The new approach breaks that up into basically two functions. The first polls the sensor and records a new value:

void sensorUpdate() {
  f = dht.readTemperature(true);
  h = dht.readHumidity();
  fS = (alpha * fS) + ((1 - alpha) * f);
  hS = (alpha * hS) + ((1 - alpha) * h);
  if (isnan(fS) || isnan(hS)) {
    fS = dht.readTemperature(true);
    hS = dht.readHumidity();
    fSo = fS;
  }
}

 

The second sends to thingSpeak

void thingspeakUpdate() {
  if (fSo > fS) {
    analogWrite(RED, 0); analogWrite(GREEN, 0); analogWrite(BLUE, 50);
  }
  else if (fSo == fS) {
    analogWrite(RED, 0); analogWrite(GREEN, 50); analogWrite(BLUE, 0);
  }
  else if (fSo < fS) {
    analogWrite(RED, 50); analogWrite(GREEN, 0); analogWrite(BLUE, 0);
  }
  printVars();
  fSo = fS;
  Serial.println("Sending to Thingspeak");
  ThingSpeak.setField(1, fS);
  ThingSpeak.setField(2, hS);
  ThingSpeak.writeFields(myChannelNumber, myWriteAPIKey);
}

It also does some fancy stuff with the LED on the development board I am using. (Well not that fancy, it just changes the color based on if the temperature is going up or down.)

Though it made sense to do it this way because Timer.h assumes that you have everything broken out into functions, I feel like I almost aesthetically drawn to the simplicity of the non-function version.

What do you think? Is that just a silly opinion?

Arduino, Thingspeak & an ESP8266 (Part VI)

Fourth Revision: Has the Time come for Timer.h?

In my last post I discussed using a low-pass filter to improve readings from my DHT11. One of the major benefits of using this instead of my earlier method is that it freed me from the requirement of going through a set number of read cycles before sending to Thingspeak. Why is that important? Because I don’t really care about the exact number of read cycles, I just want to send accurate data to Thingspeak every minute or so. Another problem is that this often requires the use of long, blocking delays which are annoying and limit the future possibilities of the Sketch.

The solution to this is a small library called Timer.h. Originally developed by Simon Monk and then taken over by Jack Christensen and Damian Philip, Timer.h is a library that radically simplifies the process of timing events.

At its core is a pair of lines:

In voidSetup()

  t.every(timeperiod, function);

and in voidLoop()

  t.update();

How this works, is that it checks every cycle to see if “timeperiod” has passed and if it has it runs “function.” Simple as that!

However, it does require the script to be re-factored with functions. More on that in my next post but click below for the script

Continue reading “Arduino, Thingspeak & an ESP8266 (Part VI)”

Arduino, Thingspeak & an ESP8266 (Part V)

Fourth Revision: Now We Are Getting Somewhere!

In the fourth revision of this project things finally started to click together for me.

The problem with my last Sketch was that it used a relatively simple mechanism for averaging the readings between two successive cycles of sending data to Thingspeak. This did a decent job of leveling out strange readings but suffered from a major problem: it was impossible to tune. Is this a problem? Probably not, but it really, really annoyed me.

Another problem with this approach was that it was cycle dependent.  That means that I had no effective way of measuring the averaged reading between cycles!

The standard approach to solving this problem would be to use a moving average as is implemented here. However, I happened to be reading a copy of Simon Monk’s superb Programming Arduino: Next Steps and realized that his simple low-pass filter would do the trick.

Here is the basic equation of the filter:

(alpha * runningaverage) + ((1 - alpha) * newreading)

One way to think of this is if alpha = 0.95 then on every cycle 5% (0.05) of the new value is added to the running average.

There are a couple of advantages to this approach: 1) it is easy to understand, 2) it is easy to tweak (just change alpha), 3) it does not use arrays so is fairly memory friendly (not that that matters on the ESP8266!)

I’ll show you the actual sketch in the next post as I also changed how timing works and want to put that under a different heading.

Arduino, Thingspeak & an ESP8266 (Part II)

First Revision

I always tell students that revision is the longest part of the writing process. From my (limited) experience the same thing is true when working with Arduino. It only took me an hour to hack something together out of various examples. It works. It isn’t robust or fancy, but there was a kind of elegance to its simplicity. Since then I’ve spent at least another 15 hours making relatively minor changes.

Here is the first draft of my Sketch.

#include <ThingSpeak.h>
#include <ESP8266WiFi.h>
#include <DHT.h>
#define DHTPIN 4
#define DHTTYPE DHT11
DHT dht (DHTPIN, DHTTYPE);

const char* ssid = "ssid";
const char* pass = "password";
const int RED = 15;
const int WHITE = 14;
const int GREEN = 13;
const int BLUE = 12;

int i = 0;

unsigned long myChannelNumber = "channel number";
const char* myWriteAPIKey = "write api key";
WiFiClient client;

void setup() {
  pinMode(RED, OUTPUT);
  pinMode(GREEN, OUTPUT);
  pinMode(BLUE, OUTPUT);
  pinMode(WHITE, OUTPUT);
  Serial.begin(9600);
  delay(10);
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  WiFi.begin(ssid, pass);  
  
  while (WiFi.status() != WL_CONNECTED) {
    analogWrite(RED, 175); 
    delay(500);
    Serial.print(".");
  }
  
  if (WiFi.status() == WL_CONNECTED) {
    analogWrite(RED, 0);
    analogWrite(BLUE, 75);
  }
  
  Serial.println("");
  Serial.println("WiFi connected");  
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  ThingSpeak.begin(client);

}

void loop() {
    for (int i = 10; i < 255; i++){
    analogWrite(WHITE, i);
    delay(20);
  }
 
  while (WiFi.status() != WL_CONNECTED) {
    analogWrite(RED, 175); 
    Serial.print(".");
    delay(1000);
    analogWrite(RED, 0);
  }

  
  if (WiFi.status() == WL_CONNECTED) {
    delay(1000);
    analogWrite(BLUE, 150);
    delay(1000);
    analogWrite(BLUE, 0);
  }

 delay(1000);
  
  float h = dht.readHumidity();
  float f = dht.readTemperature(true); 
  
  if (isnan(h) || isnan(f)) {
    analogWrite(RED, 175);
    delay(1000);
    analogWrite(RED, 0);
  }
  else {
    analogWrite(GREEN, 100);
    delay(1000);
    analogWrite(GREEN, 0);
  }

  Serial.print("h= ");
  Serial.print(h);
  Serial.print(" f= ");
  Serial.println(f);

   for (int i = 255; i > 10; i--){
    analogWrite(WHITE, i);
    delay(20);
  }
    

  delay(60000);


}

The operation of this sketch is pretty simple:

  1. Connect to WiFi
  2. Measure temperature and humidity in to variables f and h.
  3. Send variables f and h to two channels on Thingspeak.
  4. Do nothing for a minute.
  5. Repeat

The biggest problem with this sketch is that the DHT-11 is not exactly a reliable device. Sometimes it simply takes a bad reading–like a random zero.  When that bad measurement is sent to thingspeak it adds significant noise to the graphs

The next revision will attempt to deal with this through a little simple math.

Continue reading “Arduino, Thingspeak & an ESP8266 (Part II)”