If your knowledge base is built from PDFs in a Google Drive folder, converting the folder once is the easy part. The documents keep changing: a manual gets a new version, a spec is retired, a report is added. The job that feeds your index has to convert only what changed and keep each document’s Markdown at a path that doesn’t move.
This guide sets up a Python script that does this for one folder. It downloads new and changed PDFs with your Google credentials and uploads them to the Parsebridge API, which converts them with Docling. The Markdown goes to a local directory that serves as your source corpus. Chunking, embeddings, the vector index and the chat layer are later steps that the script doesn’t do.
What the script covers
It handles PDFs that sit directly in one folder in My Drive. Subfolders, shortcuts, and Google Docs, Sheets and Slides are skipped. Before listing anything, the script checks the folder itself. It stops if the ID isn’t a folder, the folder is trashed or in a shared drive, or your account can’t open it.
Each Drive folder needs its own output directory. The script stores the folder ID in a state file, .drive-state.json, and stops if you reuse the directory for another folder.
How it decides what to convert
Each run lists the folder with Drive’s files.list and follows nextPageToken to the last page. If a listing request fails, or Google marks the result incompleteSearch, the run stops before it converts or deletes anything.
A document’s identity is its Drive file ID. Names in Drive aren’t unique and can change, so each PDF’s Markdown is written to <file-id>.md. Two PDFs called Handbook.pdf get separate files, and a rename doesn’t move anything. The state file records each ID’s current Drive name and modified time, which your indexer can use for titles. The link is https://drive.google.com/file/d/<file-id>/view.
For content changes, the script compares Drive’s md5Checksum with the checksum it stored after the last successful conversion. If they match and the Markdown exists, the PDF is skipped without a download, so a rename or a sharing change costs nothing. If the checksum differs, the script downloads the PDF and verifies the bytes against that checksum. A file that changed mid-download fails for that run. Otherwise the bytes go to Parsebridge, the Markdown replaces the old file through an atomic rename, and only then does the state file change.
Conversions cost one page of your allowance per PDF page, so the script never retries a request automatically. A conversion can succeed on the server and still fail locally, for example if the connection drops. Those pages are used, and the next run converts the file again. When a file fails, the script prints Failed <file-id> with the reason and keeps that file’s previous Markdown and state. It carries on with the other files, exits with status 1, and the next run tries again. PDFs over the API’s 50 MB limit fail on every run, as do PDFs whose owner has disabled downloads. An exhausted allowance shows up as Parsebridge HTTP 402.
Removing documents that left the folder
The script deletes nothing by default. With --prune, it removes the Markdown for each recorded file ID that the listing no longer contains: PDFs that were deleted, trashed, moved out or hidden from your account. Pruning needs a complete listing, and it touches only files the script recorded. A PDF that is still in the folder but fails to convert doesn’t block pruning, and it isn’t pruned itself, so its previous Markdown stays. The run still exits with status 1 for that failure.
An empty folder is a valid listing, so if someone empties the folder, the next --prune run removes all of its Markdown. If the folder itself is trashed or becomes inaccessible, the run stops at the folder check and deletes nothing.
What your indexer still has to do
Overwriting a Markdown file doesn’t update vectors already built from it. Your indexer should read only the *.md files and key documents by file ID, which is the filename without .md. When a file’s content changes, delete every chunk with its key and index the new version. When an indexed key has no file anymore, delete its chunks. A hash per Markdown file in the indexer’s own records catches both cases. The RAG pipelines page covers the rest of the pipeline.
Drive sharing doesn’t carry over. Once a document is indexed, anyone who can query the index can retrieve it. If different people should see different documents, enforce that in your application or use separate folders and indexes. The script also creates its files readable only by the user running it, so run the indexer as that user.
Set up Google Drive access
The script uses a desktop OAuth client, as in Google’s Drive API Python quickstart.
- In the Google Cloud console, create or pick a project and enable the Google Drive API.
- In Google Auth Platform, fill in Branding. Under Audience, choose Internal only if the Cloud project belongs to a Google Workspace or Cloud Identity organization and your Google account is a user in it (Google’s audience rules). Otherwise choose External, leave it in Testing and add your Google account as a test user.
- Under Clients, create an OAuth client of type Desktop app. Download its JSON file as
credentials.json. - Copy the folder ID from its Drive URL, after
/folders/.
The script requests drive.readonly. Google has no Drive scope for a single existing folder, so the token can read every file your account can access. --folder-id limits what the script processes, not what the token allows. Keep token.json, credentials.json and your API key private and out of repositories.
External apps in Testing get refresh tokens that expire after seven days. An Internal app doesn’t have that limit. Neither does an External app moved to production, though Google can require verification for a restricted scope like drive.readonly. In Testing, plan to authorize again every week. If a Testing app prints Google token refresh failed about a week after you authorized, the token has most likely expired. The same error can mean the token was revoked or an admin policy blocks the app, so check every run’s result.
Download and run the script
You can open or download sync_drive_pdfs.py and requirements.txt in your browser, or fetch them with the commands below.
You need Python 3.10 or newer and a Parsebridge API key from the dashboard. On macOS, python3 can be the system’s Python 3.9. If python3 --version shows that, install a newer Python and create the virtual environment with it, for example python3.12 -m venv .venv.
mkdir drive-sync && cd drive-sync
curl -fsSLO https://parsebridge.com/downloads/workflows/google-drive-pdf-to-markdown/sync_drive_pdfs.py
curl -fsSLO https://parsebridge.com/downloads/workflows/google-drive-pdf-to-markdown/requirements.txt
python3 --version # needs 3.10 or newer
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt # requests, google-auth-oauthlib
# Paste the key at the prompt; it stays out of your shell history
read -rs PARSEBRIDGE_API_KEY && export PARSEBRIDGE_API_KEYMove credentials.json into the directory and run the first sync on a computer with a browser:
.venv/bin/python sync_drive_pdfs.py \
--folder-id YOUR_FOLDER_ID \
--output ./knowledge-source \
--credentials credentials.json \
--token token.json \
--authorize--authorize opens Google’s consent page and saves token.json, then the script syncs. That first sync converts every PDF in the folder, using as many pages as the folder holds, and prints Converted <file-id> for each. Leave --authorize off for regular runs. Without it the script never opens a browser. If the token is missing or can’t be refreshed, it stops and asks you to authorize again.
Run it again without --authorize and with nothing changed, and each PDF prints Skipped <file-id>, with no downloads and no pages used. If you upload a new version of a PDF through Drive’s Manage versions, the next run converts only that file, at the same path. A separately uploaded file gets a new ID and its own Markdown. The old Markdown stays until you remove the old PDF and run with --prune.
Run it on a schedule
Google’s consent flow redirects the browser to a port on the machine running the script, so --authorize needs a browser on that machine. The simplest setup is a Linux computer with a browser that runs both the first sync and the schedule, from the same directory. This example uses /srv/drive-sync; replace it with your directory.
For a headless server, authorize on your own computer instead. Then repeat the download and virtual environment steps on the server, and copy token.json and the whole knowledge-source directory to it, for example with rsync -a, which keeps the files private. Without .drive-state.json, the server would convert every PDF again. The server doesn’t need credentials.json, since only --authorize reads it.
Cron doesn’t inherit your shell’s variables, so put the key in a private env file and start the job from a wrapper. On the machine that runs the schedule, after read -rs as above:
cd /srv/drive-sync
(umask 077; printf 'PARSEBRIDGE_API_KEY=%s\n' "$PARSEBRIDGE_API_KEY" > parsebridge.env)
cat > run.sh <<'EOF'
#!/bin/sh
set -eu
cd /srv/drive-sync
set -a; . ./parsebridge.env; set +a
exec .venv/bin/python sync_drive_pdfs.py --folder-id YOUR_FOLDER_ID \
--output knowledge-source --token token.json --prune
EOF
chmod 700 run.shAdd this line with crontab -e to run it at 30 minutes past each hour:
30 * * * * flock -n /srv/drive-sync/sync.lock /srv/drive-sync/run.sh >> /srv/drive-sync/sync.log 2>&1Don’t let two runs overlap on the same directory. On Linux, flock -n skips a run while the previous one holds the lock. The log records one line per file, never the key or token. Watch it for Failed and Stopped lines.
An --authorize run also syncs, so renew the token with the cron line commented out and no run in progress. On a single machine, rerun the --authorize command from /srv/drive-sync in a shell with PARSEBRIDGE_API_KEY set, then restore the cron line. With a headless server, the server’s output has to come to your computer before that run and go back after it. On your computer, in drive-sync, with the key set:
rsync -a --delete SERVER:/srv/drive-sync/knowledge-source/ knowledge-source/
.venv/bin/python sync_drive_pdfs.py --folder-id YOUR_FOLDER_ID \
--output ./knowledge-source --credentials credentials.json --token token.json --authorize
rsync -a --delete knowledge-source/ SERVER:/srv/drive-sync/knowledge-source/
rsync -a token.json SERVER:/srv/drive-sync/Then restore the cron line on the server. Skip the first copy and your computer reconverts every PDF that changed since it last synced. Copy back only the token and the server reconverts everything your computer just converted. --delete makes each copy replace the directory rather than merge into it, so Markdown the server already pruned doesn’t come back.
To see how Docling handles one of your documents first, try the free PDF to Markdown converter. It converts the first page without an account. Tables come out as Markdown tables and scanned pages get OCR, but check complex tables like ones with merged cells. The sync script needs an API key. New accounts get 50 free pages once, with no credit card, and plans start at $17/month for 5,000 pages. The API docs cover requests and error codes.