Using Google App Script to return a GRC Password

On the weekend a did some experimenting with Google App Script. I first looked at Google App Script  during its brief pre-release phase but aside from following the tutorials I never did something useful with it of my own.

Of course there are lots of useful things Google App Script can help you out with, especially now that Google has added additional APIs for accessing more services. Workplaces using Google Apps (like my company) should especially find a lot of use for the script for automating common tasks.

To get my feet wet again with Google App Script I chose to work on something very simple (single function simple) and marginally useful to me. I wrote a Google Spreadsheet function that would pull a new 64 character high-quality random password from grc.com whenever it runs. Aside: GRC stands for Gibson Research Corporation; I listen to Steve Gibson and the Security Now netcast each week.

See below for the script itself. It is very simple. To try it, open a google spreadsheet, open its script editor and paste in the script. In a spreadsheet cell, use the formula '=getGRCPassword()'. Whenever the cell recalculates, a new password will be retrieved.


var sPasswordPageAddr = 'https://www.grc.com/passwords.htm';
var sPreambleText = 'preceding fragment from grc. Unfortunately '
  + 'blogger could not deal with it';

//This is a function which will return a new alphamnumeric only 
//password from GRC.com.
function getGrcPassword() {

  //get content of password page
  try {
    var response = UrlFetchApp.fetch(sPasswordPageAddr);
  } catch(e) { return e.message; }

  //Examine response and extract password
  //check to see if page received successfully:
  if(response.getResponseCode() == 200) {

    //if received successfully, examine the content and look for our preamble    
    //marker
    responseText = response.getContentText();
    iPasswordPreambleStart = responseText.indexOf(sPreambleText);

    //check to see if preamble marker was found
    if(iPasswordPreambleStart > 0) {

    //preamble marker was found, extract the password
    iPasswordStart = iPasswordPreambleStart + sPreambleText.length;
    sPassword = responseText.substr(iPasswordStart, 63);

    //display the password for now
    return sPassword;

  } else {

    //We could not find the password preamble. Maybe the page has changed?
    return 'Could not find password string. Page structure might have changed';
}

} else {

    //bad response from webserver. Maybe our address is wrong.
    //Maybe their webserver is down, etc
    return 'Problem accessing ' + sPasswordPageAddr
    + ': Response code ' + response.getResponseCode();
  }
}


Note that I pull the password out using basic text searching. A more robust approach would be to use forgiving HTML parser. As it stands now the approach below is easily broken by a minor page update. I did not use a HTML parser since I could not find an appropriate one to use given the environment. I will keep looking.

Book Review: Pragmatic Version Control Using Git

I have not posted a lot on this blog recently so I thought I would add a quick book review. I recently (about a fortnight ago) finished Pragmatic Version Control Using Git by Travis Swicegood, published by the Pragmatic Programmers / Pragmatic Bookshelf.


I come from a Subversion background. In fact, only recently I think I really started to hit my stride with the version control system. But as sometimes happens I was late to the party. An upstart version control system named Git, first popularized as the new version control system for the Linux kernel, is becoming increasingly popular. A key difference between Git and Subversion is that Git is a distributed version control system control. A git repository is designed to be cloned, changes are pulled between repositories and at anyone time multiple developers may be working on their own individual, complete copies of the repositories before refining their commit history and pooling their changes.

So although Git uses many of the same principles I am used to in Subversion, such as branches and tags, it is a fundamentally a different approach to source control, and requires a good book to explain it.

Travis Swicegood's Pragmatic Version Control Using Git has a lot in common with the previously released Pragmatic Version Control Using Subversion. Physically it is exactly the same dimensions in length and breadth and at 224 pages is about 30 pages longer. The pages are thick, high quality acid-free and mostly recycled paper. The print is grey scale and each chapter has a few diagrams or screenshots, but an average of ten or so plain command and output listings.

The first chapter in the book talks about the basic of what version control is and what some of the terminology means. Chapter two talks about how to get Git installed on your system and some configuration you might need to do. Chapter three sets up an example project has a brief overview of all the common actions, like adding files, committing them and creating a release branch. The detail is light, since the topics are covered in later chapters of the book.

Chapters four through nine delves into the details of the common operations you are likely to use in Git: Adding and committing files, using branches, examining the log, working with remote repositories, organisation and repository administration. The focus is on building on the knowledge form chapter three and exploring useful commands and when they might be useful.

Chapters ten and eleven cover migration to Git from other source control versions and you can set up a Git server.

As in Pragmatic Version Control using Subversion, there is a very useful Command Summary and Recipies section as one of the appendices. Long after you have finished the book you might find yourself returning there to get a quick refresher on the keystrokes required for different Git commands you may not commonly use, such as Rebasing.

The book is written in a very conversational style and is consequently very easy to read. There are couple of areas (for example, during the chapter about rebasing) where more detail and discussion would have been helpful in avoiding confusion.

In comparison to the Pragmatic Version Control with Subversion book, less time is spent discussing conventions of use. This may in part be because Git codifies some of the conventions of Subversion - for example tags and branching - into actual functionality. It would have been nice however to cover in more depth some of the other issues of convention that arise from having a distributed source control system. 

The examples in the book are easy to follow, but they are also a bit simplistic. Each chapter starts with instructions on how to clone the repository required for that chapter from the author's Git hosting site.

Overall I rate the book highly for newcomers committed to using Git. Readers already familiar with Subversion will breeze through the book. Those that are completely new to source control may struggle. Was it worth buying for me? Yes.

Let me know if you have any further questions about the book in the comment area.

App Engine: Am I running on the development server?

Sometimes you need to determine within your application whether it is currently running in a development or production environment. I just spotted a tip from Nickolas Daskalou in the Google App Engine Python forums that allows you to do just that:


import os
...
debug = os.environ['SERVER_SOFTWARE'].startswith('Dev')


This isn't particularly exciting, but I know I will come across this situation soon so I thought I would note the solution for myself in the future. Thanks Nick!

The Google App Engine development environment is generally pretty safe for Google specific resources: no emails are actually sent, only the local datastore can be updated and task queues do not process without manual intervention. When working with external resources, such as read/write web services, however, you may want to limit what your application does within a development.

App Engine: Google fails users.is_current_user_admin() test

The way Google App Engine executes cron jobs indicates that not all admin authentication is created equally. If you secure your cron or task queue URL in the app.yaml file, as Google suggests, your Google-automated tasks will be properly secured in such a way only an administrator can access the URL:


- url: /admin/my-cron-url
  script: main.py
  login: admin


However, if you would like to secure one of these Google-executed URLs yourself, you seem to be out of luck.
Yes, you could check the easily spoofed request user-agent, but on first glance the most useful method would be the users.is_current_user_admin() API method. However, this fails for Google cron and taskqueue page requests.

This is inconvenient for me; I do use Google user accounts for my applications (such as My Web Brain) but I like handling security within the confines of an event handler, where I can control the exception raised, HTTP status code, logging and actual response sent to the user.

Hopefully this inconsistency will be resolved in a future App Engine release, but my feeling is that Google jury-rigged the exception for their own services into their own interpretation of app.yaml, and that the inbuilt users API would not know Google's own requests from anyone else's.

Recent Learning: Git and Picasa

I thought I would fill in my loyal readership (as stealthy and undetectable as you are) about what I have been up to recently, or rather, what I have been learning. I believe in lifelong learning, although sometimes, it is hard to find the time to make it happen. There is so much to learn.

(behind my desk, there are literally piles of technical books, waiting to be read. And like all technical books, they are slowly going off.)

Just last night I finished reading about Git in Pragmatic Version Control Using Git by Travis Swicegood. Git is   a distributed version control system mostly used for storing source code. I had been hearing about Git for a year or two now and it is good to finally get an understanding of what it is all about. I have earmarked one of my next projects to use Git (with a remote 'origin' repository at Project Locker) and I am looking forward to puting some of my new found understanding to the test. I currently use Subversion (a centralised as opposed to distributed version control system) exclusively, and I am interested in whether the features and power of Git will make up for a more complex environment it introduces. I will keep you posted.

The other thing I have been doing (and I finished tonight) was going through the Picasa 3.5 training videos on Lynda.com, presented by Dane Howard. My prior experience with Google Picasa has always involved being slightly confused and lacking confidence enough to to use the tool I use to manage my photos. I can safely say this confusion is gone now.

Apart from the whiz-bang features ('easily create a blog post using your photos!'- That explains the photo in this post) some core concepts which I took away from the training that helped me understand Picasa included:

• Photo edits within Picasa are non-desctructive. You can always roll back the edits. This is great because it allows to experiment with the simple but varied tuning and effects that are available.
• Picasa does not 'suck in' photos and store them separately. Each folder in the folder list corresponds to the real file folder that Picasa is watching on your computer. To manage which folders Picasa (and to give Picasa a kick to rescan a folder you have changed from the file system) you can use the Tools > Folder Manager dialog box.
• On the other hand, the import and export operations are designed to work outside the set of folders Picasa is watching. When you import photos, the photos are copied inside one of Picasa's default folders. Export, on the other hand, creates another copy of the files  anywhere on the file system. Photos are rescaled according your requirements and the photo edits made within picasa are made permanent for these exported photos only. 
• Albums are virtual folders. You use an Album to gather together photos around a particular topic without changing their location on the file system. For example, I created a 'Scenic Photos' album and dragged in what I thought were some scenic photos from my other folders. In my casual use of Picasa prior to the training I missed this whole concept of albums that the you, the user can create and manage.

So I am much happier using Picasa now, and keen to use Git in anger just to see how it work out. 

Supporting Timezones in Google App Engine Pt. 3 - Your Timezone Aware App

I am covering how I added Timezone support to my web application My Web Brain in a series of posts.  Hopefully someone will find the discussion useful or even better contribute ways to achieve the same effect.

• Part 0: The Starting Point - What you get out the box from Google App Engine and the Base Python Environment. 
• Part 1: Getting Timezone information with pytz
• Part 2: Making the Google App Datastore more consistent by writing a UtcDatetimeProperty class
• Part 3: Handling timezones throughout your application (this post)

In Part 0 of the series I described how, out of the box, python Dates and Datetimes could be timezone aware, but how the Google App Engine converts these to UTC when persisted and returns naive values when they are retrieved. In Part 1 I looked at the facilities available in pytz to provide timezone information. In Part 2 I showed a simple custom property which ensured the datastore much more explicitly used UTC. In this last part of the series I hope to show you can input, output and list timezones to take you the last mile toward a timezone aware application.

Note: I've scaled this post back, mostly since I was being tardy in completing it and also with the realisation that the extra detail I might add would be too specific to my recent experience.

Converting datetimes on input

Sometimes your application will accept as input a user provided date or datetime. In a timezone aware application the working assumption must be that this date is in the user's local timezone. It therefore needs to be converted to UTC for storage.

A handy feature of the datastore datetime property is that as long as your datetime is marked with a specific timezone, the datastore will automatically convert the value to UTC when it is persisted. Once you have parsed your datetime from the input, simply assign the user's timezone before persisting. For example:

user_datetime = parse_datetime(self.request.get('user_date'))
my_model_entity.user_datetime = user_datetime.replace(tzinfo=pytz.timezone(user_timezone_name))
my_model_entity.put()


Of course if you are accepting input from the system time, you do not need to assign a timezone, it is already in UTC, even thought it is a naive datetime.

my_model_entity.log_datetime = datetime.datetime.now()
my_model_entity.put()


Converting dates (only) on input

This is pretty straightforward. There is a situation though where your application would normally only care about dates, but because of your support for timezones you are persisting datetime objects. The user would select a date, say the 10th of January of this year, but when persisting this as a datetime you have a crucial decision to make - what time on the 10th of January should you store? In all situations you will want to set this time prior to converting to UTC. The decision about which time to use is non-trivial since your application will likely want to use the information in the effective manner.  The answer will vary from application to application, but a couple of examples might illustrate the point.

In my application My Web Brain, the user can enter a Due Date for a next action that they define. In this application we will be asking one critical question on a regular basis concerning the due date - is the next action overdue? Something is overdue when the due date has past, not during that date, so My Web Brain adds 23 hours and 59 minutes to the due date. This gets converted to UTC when it is persisted, so any Next Action which is overdue can be queried easily by using a condition comparing the due date with the system time.

To provide some contrast, My Web Brain also supports Someday Maybe items with 'tickle dates'. A tickle date is provided by the user to indicate when they should be reminded. In this situation the application will want to find all Someday Maybe items that are due to be 'tickled', and this should include any such item on the same local date the user entered. Therefore, we do not add any amount of time when combining to create a tickle date. Again, since this date was converted to UTC prior to persisted this query is easy for the application to do without knowing the user's local timezone.

Other situations will require other time offsets. Perhaps if a user's work day ends at 5pm, the due date should reflect this. The common thread is that right time offset will depend on the application. Once you have a determined this, the code is easy:


due_date = parse_date (self.request.get('user_date'))
due_datetime = due_date.combine(datetime.time(hours=17))
next_action_entity.duedate = due_datetime.replace(tzinfo=pytz.timezone(user_timezone_name))
next_action_entity.put()


Converting datetimes on output

At the other end of your application you will want to show the user dates and datetimes in their local time. From the datastore you will retrieve datetimes in UTC - these values need to be converted to the user's timezone.

local_duedate = next_action_entity.duedate.astimezone(pytz.timezone(user_timezone_name))

There is nothing overwhelmingly difficult about the above line of code, but you could be forgiven for wondering where in your application it should go. You need to decide if the derived local time of the data should belong to the model, view or controller part of your application:

• If make the conversion to local time a part of the view, you need to share the user's timezone information with the view. The view is often implemented in a templating language like Django templates which out of the box does not provide the capability to do this as a view operation. The best way to use the view in this way would be to define a custom tag for the templating engine, which is not overly difficult. 
• You might make the conversion to local time a responsibility of the controller, which after all can access different parts of the model to both find the user's timezone preference and the data with the times to be converted. On the downside, converting the date times and providing them separately (or mixed in to the data) is messy and, in my opinion, increases the coupling between controller and view more than necessary.
• Making the conversion to local time a responsibility of the model is perhaps the easiest approach in the short term. Simple add a method to provide the local time to your entity's class. This has the benefit of making your entities less anemic, but introduces the requirement for one piece of your model to know the user's timezone, which might belong in somewhere else in the model entirely. It can also lead to timezone conversion code duplication across your model. 

For My Web Brain I have at the moment chosen to invest the responsibility of converting times to the user's local time to the model, but I wonder if view-based solution would provide more long-term benefits to performance and model design.

Who knew the single line of code converting a time between timezones could require such thinking? Maybe it is only me who can make it so.

Other Questions

There are some other points to cover about using making your application timezone aware. How do you generate a list of timezones for the user to pick from? I have a solution for this, but it isn't elegant. Another good question is how the best way would be to guess the correct timezone for a user. Those who have seen my frequent calls to pytz.timezone() may wonder what the performance penalty is (from what I have read, it is worth looking into).

But- This entry is getting long and I might leave these topics for another occasion. Remember that if you have something to add or something to ask, you are welcome to go ahead and do so. I still have significant learning to do in this area and I will keep you posted when I do.

Canceling a McAfee Subscription is HARD

McAfee, a provide of Software Antivirus and other security products, makes the process of cancelling a subscription far too onerous and difficult. In an increasingly competitive environment (the new free Microsoft Security Essentials has changed the anti-virus landscape) McAfee can not afford the bad press their ridiculously difficult subscription cancellation mechanism deserves.

I received a courtesy email in my inbox this morning letting me know that my McAfee Anti-virus license would be renewed shortly. This by itself is actually very welcome; annually renewed services and their renewal dates can be hard to remember and consider ahead of time. The email did not say when the actual renewal date was, which I found slightly odd. I had to log into the McAfee user website to see that the renewal date is actually in February. Their courtesy email certainly gives me enough time to make any changes...

... such as cancelling my subscription. I run a small business and I try to take security very seriously. I do not mind paying for my virus protection. The arrival of the free (and much lauded) antiviral solution from Microsoft, however, combined with the increasingly bloated, nagging, and recently technically challenged service from McAfee meant I have already been trialling the Microsoft alternative. And I am ready to leave McAfee behind. If only I can cancel their service.

After visiting the McAfee website, signing in, and having a good look around, there is no way to do this easily. To turn off the auto-renewal (and charging) of my subscription, I need to contact customer service. This is amazing considering the number of fast ways to purchase new products on the website, let alone the explicit advertising (including expensively produced videos) of the auto-renewal feature itself. Turning on automatic renewal is easy.

Do you want to know how to cancel your subscription with McAfee? After extensive searching of their service site I found an FAQ entry. The process, conducted over chat, email or phone is not easy.

It is hard to escape the conclusion that McAfee has intentionally made cancelling their subscription service difficult to protect their subscription revenues.

As a technically literate and competent user this makes me angry and I now think less of McAfee as a brand. However, the real victims of McAfee's approach are the less confident and less technically savvy users. This could be my mother, or your grandparents, or your mechanic friend Bob, who might not have the persistence or understanding to find and then follow the procedure. The other victim is McAfee itself. The decision to cripple their website like this for seemingly unscrupulous reasons can only create bad press and disrepute.

I have since sent off my email customer support request and I am waiting on a reply, which apparently may take 24 hours. I am disappointed (Can you tell?)