Sunday, September 20, 2015

Transferring photos from Flickr to PicasaWeb

I haven't had to transfer photos from my Flickr account to a PicasaWeb account in a while.  This morning, I found out the script no longer works.  I was getting this error when the script attempts to authenticate with PicasaWeb:  "Modification only allowed with api authentication".  Apparently, Google had dropped support for the older authentication method and opted to use OAuth2 instead.  I had to dig around the web for some readily available code to cobble together a solution.

My solution was derived from the following two sources:

Make sure you read the first.

Well, here it is in its entirety:

#! /usr/bin/python
# requires flickrapi, gdata, and oauth2client
# It's a little ugly, but it is heavily tested and works!
# Sources:

import flickrapi, StringIO
import gdata
from getpass import getpass
from urllib import urlretrieve
from tempfile import mkstemp
from threadpool import ThreadPool, WorkRequest
import os
import sys, os.path, StringIO
import time
import gdata.service
import gdata
import atom.service
import atom
import getopt
import webbrowser
import httplib2
args_opts, album_title_to_move = getopt.getopt(sys.argv[1], '')
print "Will copy " + album_title_to_move + "..."
from shutil import copyfile

from datetime import datetime, timedelta

from oauth2client.client import flow_from_clientsecrets
from oauth2client.file import Storage


video_too_large_save_location = os.path.join(os.path.sep.join(__file__.split(os.path.sep)[:-1]), 'picasa_videos')

if not os.path.exists(video_too_large_save_location):

class VideoEntry(
    pass = VideoEntry

def InsertVideo(self, album_or_uri, video, filename_or_handle, content_type='image/jpeg'):
    """Copy of InsertPhoto which removes protections since it *should* work"""
        assert(isinstance(video, VideoEntry))
    except AssertionError:
        raise GooglePhotosException({'status':GPHOTOS_INVALID_ARGUMENT,
            'body':'`video` must be a instance',
            'reason':'Found %s, not PhotoEntry' % type(video)
        majtype, mintype = content_type.split('/')
        #assert(mintype in SUPPORTED_UPLOAD_TYPES)
    except (ValueError, AssertionError):
        raise GooglePhotosException({'status':GPHOTOS_INVALID_CONTENT_TYPE,
            'body':'This is not a valid content type: %s' % content_type,
            'reason':'Accepted content types:'
    if isinstance(filename_or_handle, (str, unicode)) and \
        os.path.exists(filename_or_handle): # it's a file name
        mediasource = gdata.MediaSource()
        mediasource.setFile(filename_or_handle, content_type)
    elif hasattr(filename_or_handle, 'read'):# it's a file-like resource
        if hasattr(filename_or_handle, 'seek'):
   # rewind pointer to the start of the file
        # gdata.MediaSource needs the content length, so read the whole image 
        file_handle = StringIO.StringIO( 
        name = 'image'
        if hasattr(filename_or_handle, 'name'):
            name =
        mediasource = gdata.MediaSource(file_handle, content_type,
            content_length=file_handle.len, file_name=name)
    else: #filename_or_handle is not valid
        raise GooglePhotosException({'status':GPHOTOS_INVALID_ARGUMENT,
            'body':'`filename_or_handle` must be a path name or a file-like object',
            'reason':'Found %s, not path name or object with a .read() method' % \

    if isinstance(album_or_uri, (str, unicode)): # it's a uri
        feed_uri = album_or_uri
    elif hasattr(album_or_uri, 'GetFeedLink'): # it's a AlbumFeed object
        feed_uri = album_or_uri.GetFeedLink().href

        return self.Post(video, uri=feed_uri, media_source=mediasource,
    except gdata.service.RequestError, e:
        raise GooglePhotosException(e.args[0]) = InsertVideo

def clear_input_retriever(setting):
    return raw_input( + ":")

def passwd_input_retriever(setting):
    return getpass( + ':')

class Setting(object):
    def __init__(self, name, default=None, input_retriever=clear_input_retriever, empty_value=None): = name
        self._value = default
        self.input_retriever = input_retriever
        self.empty_value = empty_value
    def value(self):
        while self._value == self.empty_value:
            self._value = self.input_retriever(self)
        return self._value

picasa_username = Setting('Picasa Username(complete email)')
picasa_username._value = ""
picasa_password = Setting('Picasa Password', input_retriever=passwd_input_retriever)
picasa_password._value = ""
picasa_oauth_client_secrets_filename = Setting('Picasa OAuth Client Secrets')
picasa_oauth_client_secrets_filename._value = 'migrate-flickr-to-picasa.secrets'

flickr_api_key = Setting('Flickr API Key')
flickr_api_key._value = ""
flickr_api_secret = Setting('Flickr API Secret')
flickr_api_secret._value = ""

flickr_usernsid = None

def flickr_token_retriever(setting):
    global FLICKR
    global flickr_usernsid
    if FLICKR is None:
        FLICKR = flickrapi.FlickrAPI(flickr_api_key.value, flickr_api_secret.value)
    (token, frob) = FLICKR.get_token_part_one(perms='write')
    if not token: raw_input("Press ENTER after you authorized this program")
    FLICKR.get_token_part_two((token, frob))
    flickr_usernsid = FLICKR.auth_checkToken(auth_token=token).find('auth').find('user').get('nsid')
    return True

def get_gd_client():

    gd_client = = picasa_username.value
    gd_client.password = picasa_password.value
    gd_client.source = ''

    return gd_client

# Source:
def OAuth2Login(client_secrets, credential_store, email):

    storage = Storage(credential_store)
    credentials = storage.get()
    if credentials is None or credentials.invalid:
        flow = flow_from_clientsecrets(client_secrets, scope=scope, redirect_uri='urn:ietf:wg:oauth:2.0:oob')
        uri = flow.step1_get_authorize_url()
        code = raw_input('Enter the authentication code: ').strip()
        credentials = flow.step2_exchange(code)

    if (credentials.token_expiry - datetime.utcnow()) < timedelta(minutes=5):
        http = httplib2.Http()
        http = credentials.authorize(http)

    gd_client =,
                                               additional_headers={'Authorization' : 'Bearer %s' % credentials.access_token})

    return gd_client

def do_migration(threadpoolsize=7):

    print 'Authenticating with Picasa...'
    #gd_client = get_gd_client()
    gd_client = OAuth2Login(picasa_oauth_client_secrets_filename.value, '', picasa_username.value)

    print 'Authenticating with Flickr..'
    flickr_token = Setting('Flickr Token', input_retriever=flickr_token_retriever)
    token = flickr_token.value # force retrieval of authentication information...

    tmp_sets = FLICKR.photosets_getList().find('photosets').getchildren()
    sets = []
    for aset_id in range(len(tmp_sets)): # go through each flickr set
        aset = tmp_sets[aset_id]
        set_title = aset.find('title').text
        # Transfer only this one photo set ...
 if set_title == album_title_to_move:
            sets = [ aset ]

    print 'Found %i sets to move over to Picasa.' % len(sets)

    def get_picasa_albums(id, aset, num_photos):
        all_picasa_albums = gd_client.GetUserFeed(user=picasa_username.value).entry
        picasa_albums = []
        id = id.strip()
        orig_id = id
        for i in range((num_photos/1000) + 1):
            if i > 0:
                id = orig_id + '-' + str(i)
            picasa_album = None
            for album in all_picasa_albums:
                if album.title.text == id:
                    picasa_album = album
            if picasa_album is not None:
                print '"%s" set already exists as an album in Picasa.' % id
                picasa_album = gd_client.InsertAlbum(title=id, summary=aset.find('description').text, access='protected')
                print 'Created picasa album "%s".' % picasa_album.title.text
        return picasa_albums

    def get_picasa_photos(picasa_albums):
        photos = []
        for album in picasa_albums:
        return photos

    def get_photo_url(photo):
        if photo.get('media') == 'video':
            return "" % (flickr_usernsid, photo.get('id'), photo.get('originalsecret'))
            return photo.get('url_o')

    def move_photo(flickr_photo, picasa_album):
        def download_callback(count, blocksize, totalsize):
            download_stat_print = set((0.0, .25, .5, 1.0))
            downloaded = float(count*blocksize)
            res = int((downloaded/totalsize)*100.0)
            for st in download_stat_print:
                dl = totalsize*st
                diff = downloaded - dl
                if diff >= -(blocksize/2) and diff <= (blocksize/2):
                    downloaded_so_far = float(count*blocksize)/1024.0/1024.0
                    total_size_in_mb = float(totalsize)/1024.0/1024.0
                    print "photo: %s, album: %s --- %i%% - %.1f/%.1fmb" % (flickr_photo.get('title'), picasa_album.title.text, res, downloaded_so_far, total_size_in_mb)

        dest = os.path.join(video_too_large_save_location, flickr_photo.get('title'))
        if os.path.exists(dest):
            print 'Video "%s" of "%s" already exists in download cache of files over 100MB. Aborting download.' % (flickr_photo.get('title'), picasa_album.title.text)
        photo_url = get_photo_url(flickr_photo)
        print 'Downloading photo "%s" at url "%s".' % (flickr_photo.get('title'), photo_url)
        (fd, filename) = tmp_file = mkstemp()
        (filename, headers) = urlretrieve(photo_url, filename, download_callback)
        print 'Download Finished of %s for album %s at %s.' % (flickr_photo.get('title'), picasa_album.title.text, photo_url)
        size = os.stat(filename)[6]
        if size >= 100*1024*1024:
            print 'File "%s" of set "%s" larger than 100mb. Moving to download directory for manual handling. ' % (flickr_photo.get('title'), picasa_album.title.text)
            copyfile(filename, dest)
        print 'Uploading photo %s of album %s to Picasa.' % (flickr_photo.get('title'), picasa_album.title.text)

        if flickr_photo.get('media') == 'photo':
            picasa_photo =
            picasa_photo = VideoEntry()

        picasa_photo.title = atom.Title(text=flickr_photo.get('title'))
        picasa_photo.summary = atom.Summary(text=flickr_photo.get('description'), summary_type='text')
        photo_info = FLICKR.photos_getInfo(photo_id=flickr_photo.get('id')).find('photo') = = ', '.join([t.get('raw') for t in photo_info.find('tags').getchildren()])
        picasa_photo.summary.text = photo_info.find('description').text
        if flickr_photo.get('media') == 'photo':
            gd_client.InsertPhoto(picasa_album, picasa_photo, filename, content_type=headers.get('content-type', 'image/jpeg'))
            gd_client.InsertVideo(picasa_album, picasa_photo, filename, content_type=headers.get('content-type', 'video/avi'))

        print 'Upload Finished of %s for album %s.' % (flickr_photo.get('title'), picasa_album.title.text)


    threadpool = ThreadPool(threadpoolsize)

    for aset_id in range(len(sets)): # go through each flickr set
        aset = sets[aset_id]
        set_title = aset.find('title').text
        print 'Moving "%s" set over to a picasa album. %i/%i' % (set_title, aset_id + 1, len(sets))

        print 'Gathering set "%s" information.' % set_title
        num_photos = int(aset.get('photos')) + int(aset.get('videos'))
        all_photos = []
        page = 1
        while len(all_photos) < num_photos:
            page += 1

        print 'Found %i photos and videos in the %s flickr set.' % (num_photos, set_title)
        picasa_albums = get_picasa_albums(set_title, aset, len(all_photos))
        picasa_photos = get_picasa_photos(picasa_albums)
        for photo_id in range(len(all_photos)):
            photo = all_photos[photo_id]
            photo_found = False
            for p_photo in picasa_photos:
                if p_photo.title.text == photo.get('title'):
                    print 'Already have photo "%s", skipping' % photo.get('title')
                    photo_found = True

            if photo_found:
                print 'Queuing photo %i/%i, %s of album %s for moving.' % (photo_id + 1, len(all_photos), photo.get('title'), set_title)

            p_album = None
            for album in picasa_albums:
                if int(album.numphotosremaining.text) > 0:
                    album.numphotosremaining.text = str(int(album.numphotosremaining.text) - 1)
                    p_album = album
            req = WorkRequest(move_photo, [photo, p_album], {})
if __name__ == "__main__":
    print """
    This script will move all the photos and sets from flickr over to picasa. 
    That will require getting authentication information from both services...

I hope this will save someone else a bit of grief.

Saturday, May 30, 2015

Me and rain

There is something about me and the rain.  I love it.  There is a feeling of childhood fun when you go biking in the rain, knowing that when you get home, you'll be able to just wash away all the dirt splashed all over you.  There is something else about it that makes it fun for photography.  I don't know what it is but I tend to go into a shooting frenzy head on in the craziest downpour.  I'd be out there with my D7000 and getting my lens all wet to the point I can't shoot anymore--either because of condensation in the lens or the D7000 starts to behave erratically.  Yeah, I'm crazy with rain.  So today was no exception.  The PanAm game will start in less than a couple of months and the torch has arrived in Toronto.  Unfortunately, today was also the first day of heavy downpour.  The month of May has been so dry that this downpour is a big welcome I think for farmers in the area.  And for me.  It was fun standing in the rain with my motorcycle rain jacket.  It protected me from the rain and the wind.  I got wet anyway but the jacket kept me warm as I rode my bicycle down the streets, following the torch bearers.  It was also fun to use my cell phone to take pictures.  I had a thought that maybe I'd ruin it if I persist shooting it in the rain.  It was clearly drenched in water all over it but I kept shooting.  Stupid me.  I will never learn.  I kept shooting knowing full well the phone will die.  Well, it didn't die but it was starting to behave erratically too and the camera lens inside the phone stared to fog up.  Great, I thought.  Another Nova Scotia.  It managed to clear itself up, but I kept shooting in the rain.  I should have stopped using it and wait for the big moment to take the phone out.  I should have.  I should have done that.  It was a big mistake to chance it.  The biggest moment came when Chris Hadfield arrived at the Distillery District on the last leg of the torch run from Canada Square to the Distillery District.  The video was fuzzy with a damn layer of fog in front of the lens!  What an idiot I was.  Oh well, me and rain, it's a love and hate relationship.

Thursday, January 22, 2015

Tripod or not on trips

To recap, I have two tripods:  a Manfrotto MKC3-P01and an OPUS OT-1104BH.  I love both.  They are not the best and you know you are compromising when the price is affordable.  You know what I mean--you make an adjustment, you let go, and the camera tilts down a bit.  It's a bit annoying but you learn to compensate for it.  Anyhow, these tripods have gone to faraway places with me, but often, they stay in my backpack.

On a recent trip to New York City, I decided not to bring a tripod.  I knew I'd spend most of the time walking around during the day but given that days end about 4:30pm at the time, I also knew I'd spent quite a bit of time walking in the dark.  It would make sense to bring a tripod.  I decided against it however.  First, I was doing a lot of walking.  Any weight I could save meant a world to me toward the end of the day.  I decided instead to rely on the camera's ability to work with high ISO settings, and to work with existing street structures to prop against or rest on, and whatever else I could use.

Many photos turned out a fuzzy but some turned out not too bad.  I had to do some sharpening for night shots:

Monday, April 21, 2014

Sensor cleaning nightmare

When you are in trouble, you know it.  It happened to me yesterday as I was trying to perform yet another wet cleaning operation on my D7000 sensor.  It started with the nasty field of bunnies on the right.

I was taking some pictures at f/14 and can clearly see some bunny spots in the blue sky.  That was annoying, so I had to look at the sensor.  It was dirty alright and for some reason, the bunnies were concentrating in the far right of the sensor as you can see here.  They were visible with the naked eyes.  I did not need to use the SensorKlear Loupe I purchased from the photo show a couple of years ago--btw, I highly recommend this loupe if you are doing any cleaning with your sensor.

Blowing air did nothing to remove the bunnies.  They were bonded to the sensor.  I did not try to Artic Butterfly and instead went straight to wet cleaning.  That's when things went kinda downhill.

On the first pass, the Pec Pad removed most of the bunnies but at the same time left streaks on the sensor.  The VisibleDust cleaning solution left residues behind as it dried.  The streaks were all over the sensor and even without the SensorKlear Loupe, I was able to see the streaks very clearly with my naked eyes.  So, I grabbed another Pec Pad and cleaned the sensor once more.  Again, streaks all over.  I tried yet once more.  Same results.  I knew it was not dust but residues from the solution.  Does the solution have an expiry date?  Maybe the alcohol has evaporated?  I really don't know but I must have tried cleaning the sensor five more times before I could get a clear sensor.

BTW, my sensor has this halo ring in the lower left corner.  It looks permanent to me as I see it in the other shots I took a couple of years ago.  Darn it.

Sunday, March 23, 2014

Roaming around Toronto on my bike

I find myself using my LG P500h phone camera more these days than before.  It would seem to be available all the time as I need to carry it with me on my belt wherever I go.  It would also seem to be heck more convenient than carrying a large DSLR.  Many of the times, it would be fine.  I may not need to zoom in or may not need to deal with high contrast lighting conditions.

This afternoon, I went to cash in my lottery ticket--a whopping $125!  I decided afterwards to go find the graffiti alley.  It's somewhere in Toronto but I do not know where it is yet, and don't want to look it up and don't want anyone to tell me where it is.  One day, I will run into it and that's how it will keep my biking more interesting these days.  Anyway, I went biking around searching for the graffity alley so I started covering some new territories of downtown Toronto.  I knew the alley is not where I went but I decided to hit a street I have never been to--Mercer Street on the north side of Gretsky's and The Second City.  There, I found behind a building this refuse storage, overflowing with bags of garbage, the doors bashed open.  Yuck!  I cannot imagine how it will be like in the summer heat of Toronto.  That will be disgusting.

Saturday, November 23, 2013

My backyard

My backyard has changed from the lush greens in the summer and couple of feet of snow in the winter to steel, glass, and concrete year round.

I did not think much about this when I was looking for a place in Toronto, but since I was a kid, I always thought it would be cool to live in a high-rise in a downtown area with the glitters of city lights poking through the glass windows.

My backyard.
Well, I am now fifteen floors up and the view is not too bad.  Facing south, I can see some parts of the Toronto Islands and parts of Lake Ontario.  On a normal day, there is enough light out there I could walk around my place without turning on my lights.  On a foggy night, the view could be quite interesting.  I will see how winter will shape the scene--we just had the first snow fall this year in this area and temperature has fallen down to -1 degree this morning.

Wednesday, November 20, 2013

The street reporter

I have not posted anything for a while but I have been out there shooting.  Sometimes, it's lugging my two DSLRs around, and sometimes, it's just--wait a second, something just dawned on me ... did I miss the Henry's Exposure photo show this fall or maybe they cut the show down to only one per year--about running into something interesting while biking around.  And for that, I need something light, like my Panasonic Lumix DMC-FZ30 that I purchased a while ago for $70.  The built-in telephoto lens on this camera is perfect for the ride-about that I do from time to time or the ride to work that I do everyday.

Just yesterday, I ran across an area cordoned off by police.  It was on my usual, daily bike route.  Some car was pulled over earlier in the day (apparently not by random chance) and some small explosive was found in a gym bag in the trunk.  The event had already unfolded by the time I got there but I got a glimpse of the scene.  It would have been more interesting had I witnessed the detonation of the explosive that the police had set off--well, maybe they would have cordoned off a bigger area so I might not have been able to capture anything then.  Anyways, I feel like an amateur street reporter with a cheap camera on hand.

It's almost 8am.  I gotta start preparing to go to work.

Friday, July 5, 2013

Amazon sucks

Ok, so last year, I ordered this studio umbrella set for about $80.  That was last November.  Today is July and it has not been shipped yet!

I thought I would try to wait longer and see how this story unfolds but it's been long enough and I will be moving soon, so in case the order actually goes through and I won't be at my current place, I decided to cancel the order, eight months after placing the order.

Pathetic, Amazon!  The couple of times I contacted Amazon, they say they will try to fulfill the order as soon as possible.  Well, I don't know which earth they live on but on this earth, that means a few weeks at most, not months!


Sunday, June 2, 2013

Reality check

Every now and then, I have the urge to buy a professional lens because my compromise lens works up an itch hard to get rid of.  I actually went on Kijiji and Craigslist to find a used Nikon 17-55mm f/.8.  A used copy could go for about $800 or a bit less if one is lucky.  Anyways, at the Henry's Exposure show this weekend, I had  the opportunity to test out a Nikon 17-55mm and as I feared, I could not make the best use of it.  Hand-held, it's not too easy to get a nice, sharp picture, and when I managed to get a sharp picture at f/2.8 there were some purple fringing (see below).  It's not too much but there is definitely purple fringing.  The Nikkor 18-200mm VR lens gets me a sharper picture, but not the image effects of an f/2.8 aperture.

Nikkor 17-55mm at f/2.8, centre crop

Nikkor 18-200mm at f/6.3, centre crop

Saturday, May 4, 2013

Nature in the backyard

Sharing a meal?  Not so quick!
Quite rare of a sight of two birds taking care of their nest.  I thought the male bird was going to share his grub, but no, it ended up having it all for himself.  Such is nature.

I realized this weekend that the Tokina 80-400mm is darn sharp at f/14 with manual focus!

Maybe I will get to use my 500mm lens when the eggs start to hatch.