Large libraries on removable storage

How to organise 100,000+ photos on an external drive.

If your photo library lives on an external drive, most photo software is fighting you. It wants to import your files into a library folder, or upload them to a service, or build a database in your home directory that holds paths to files it assumes will never move. Point it at four terabytes and it either refuses, spends a day copying, or quietly duplicates everything.

What you actually want is simpler: leave the files exactly where they are, and build the knowledge about them — dates, places, albums, duplicates — somewhere that does not evaporate when you unplug the disk.

The three things that go wrong

1. The tool wants to import. Import means copying. Copying 100 GB takes hours and doubles your storage. The only reason most applications do it is that they assume they own the filesystem — an assumption inherited from phone-based photo apps. You are looking for software that indexes in place: reads your files where they are and writes down what it learned.

2. The database lives somewhere else. Even tools that index in place usually write their catalog to your home directory — ~/.local/share/something/library.db — with absolute paths to the drive. Move the drive, change its mount point, plug it into another machine, and the library is orphaned. You get an empty timeline and a lot of re-indexing.

3. Drive letters move. On Windows especially, E:\ becomes F:\ and every stored path breaks. A tool that keeps the catalog on the drive and refers to files relatively does not have this problem at all.

What to look for

Requirement	Why it matters
Indexes in place	No copying, no duplicate storage, no waiting
Catalog stored on the photo drive	The library survives unplugging, moving and changing machines
Relative paths internally	Drive letters and mount points can change freely
Handles 100k+ photos	Most consumer tools fall over well before that
Works offline	An external drive is often used where there is no fast internet
Duplicate detection	A large pile accumulated over years always has them

The first two are the ones almost nobody does together, and they are the two that decide whether your library is a durable thing or a fragile one.

A concrete walkthrough

Smriti is built around exactly this use case — free, Apache-2.0, Linux/Windows/macOS — so it makes a useful worked example. It is tested at 250,000 photographs on a single drive.

Install it. A .deb, .rpm or AppImage on Linux; an installer on Windows and macOS.
Point it at the root of the drive, not at a subfolder. The library is defined by the folder you choose.
Press "Scan now" on the timeline. Nothing happens automatically — this catches everyone out. The scanner reads EXIF from each file, generates thumbnails and records everything in the catalog.
Leave the drive alone afterwards. The catalog now lives at <your drive>/.photovault/photovault.db. Do not delete that folder unless you are happy to re-index — it holds the albums, tags, face groupings and thumbnails. It is the library, not a cache.
Try the portability test. Close the app, unplug the drive, plug it into a different computer with Smriti installed, and open the same folder. The timeline, albums and faces are all there, because the knowledge about the photos was on the disk the whole time.

Rough timings for the first index, on a USB 3 drive with a mid-range CPU:

Photos	First index (estimate)
10,000	5–20 minutes (documented)
100,000	a few hours
250,000	most of a night

It runs in the background while you use the app, and it is resumable if you close it.

Cleaning up the mess

The genuinely useful part of indexing a large library is finding out what is actually in it. Two things to run early:

Duplicates. Fifteen years of imports and re-imports produce a lot of identical files. Exact-duplicate detection is safe and usually reclaims surprising amounts of space; perceptual detection finds the same photo saved at different sizes.
Bursts. Photographs taken seconds apart are grouped so you can keep the best frame instead of all eleven.

Neither is possible until the library has been indexed, which is why the index is worth the wait.

Practical notes for big drives

Back up the drive, not just the photos. Because the catalog sits with the files, a normal folder copy backs up your albums and tags as well. Skip .photovault in your backup and you keep the photographs but lose the work.
USB 3 or better. Indexing is I/O-bound; a slow enclosure is the single biggest factor in how long the first scan takes.
Index once, then scan incrementally. Later scans only look at what changed, so they take seconds to minutes.
One drive, one library. Several drives can each carry their own library; you switch between them rather than merging them.

For a large library on removable storage, the deciding question is not how many features a tool has. It is where the catalog lives. If it lives with the photos, your organising survives unplugging, moving and changing computers. If it lives in your home directory, it does not.

Download Smriti Server or no server?

Apache-2.0 · Free · No account, no server