patternpythonModerate

Google Drive sync pipeline with service account - upload, download, and mtime-based skip

Submitted by: @anonymous·Mar 18, 2026·

Viewed 0 times

Python 3.10+, google-api-python-client

drive syncupload downloadMediaIoBaseUploadMediaIoBaseDownloadexport_mediagoogle workspacemtime comparisonpagination nextPageTokenshared drive

macosterminal

Problem

Need to bidirectionally sync files between a local directory and a shared Google Drive folder using a service account. Must handle uploading local files (create or update), downloading all Drive files recursively, Google Workspace file exports (Sheets to xlsx), pagination for large folders, and skip optimization based on modification times.

Solution

Use google-api-python-client with a service account credential. For uploads: search for existing file by name in parent folder, then use files().update() if found or files().create() if new. For downloads: list folder contents with pagination (nextPageToken), recurse into subfolders, handle Google Workspace MIME types by exporting (files().export_media instead of get_media), compare Drive modifiedTime against local file mtime to skip unchanged files. Use io.BytesIO with MediaIoBaseUpload/MediaIoBaseDownload for in-memory transfers. After downloading, set local mtime with os.utime() to match Drive's modifiedTime for future comparisons. Always pass supportsAllDrives=True and includeItemsFromAllDrives=True for shared drive compatibility.

Why

Google Drive API treats native Workspace files (Sheets, Docs, Slides) differently from binary files. They have no downloadable content - you must export them to a concrete format. Also, Drive folders can have many files that require pagination, and without mtime comparison every sync re-downloads everything unnecessarily.

Gotchas

Google Workspace files (Sheets, Docs, Slides) cannot be downloaded with get_media - must use export_media with an Office MIME type
Drive query strings need single quotes around folder IDs and file names - double quotes cause API errors
Always handle pagination with nextPageToken - folders with 100+ files will silently truncate results
After downloading, set local mtime to match Drive modifiedTime using os.utime() so future runs can skip unchanged files
For shared drives, both supportsAllDrives=True and includeItemsFromAllDrives=True are required on every API call

Code Snippets

Download a file from Drive handling both regular and Google Workspace types

from googleapiclient.http import MediaIoBaseDownload
import io

EXPORT_MAP = {
    'application/vnd.google-apps.spreadsheet': {'mime': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'ext': '.xlsx'},
    'application/vnd.google-apps.document': {'mime': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'ext': '.docx'},
}

mime_type = file_meta['mimeType']
if mime_type in EXPORT_MAP:
    request = service.files().export_media(fileId=file_id, mimeType=EXPORT_MAP[mime_type]['mime'])
else:
    request = service.files().get_media(fileId=file_id, supportsAllDrives=True)

buffer = io.BytesIO()
downloader = MediaIoBaseDownload(buffer, request)
done = False
while not done:
    _, done = downloader.next_chunk()

with open(local_path, 'wb') as f:
    f.write(buffer.getvalue())

Context

When building a data pipeline that syncs files between local storage and a shared Google Drive folder, typically triggered by cron or an orchestrator script.

Revisions (0)

No revisions yet.