I run a car sales website for a client. They are constantly adding and removing cars. When a new one comes in, they add a batch of images and the website generates a thumbnail for each. The site stores the base filename (through which I can access both thumbnail and original). Here's an example:
5e1adcf7c9c1bcf8842c24f3bacbf169.jpg
5e1adcf7c9c1bcf8842c24f3bacbf169_tn.jpg
5e1de0c86e45f84b6d01af9066581e84.jpg
5e1de0c86e45f84b6d01af9066581e84_tn.jpg
5e2497180424aa0d5a61c42162b03fef.jpg
5e2497180424aa0d5a61c42162b03fef_tn.jpg
5e2728ac5eff260f20d4890fcafb1373.jpg
5e2728ac5eff260f20d4890fcafb1373_tn.jpg
The problem comes after a product is removed. In my existing workflow, there isn't a simple way to remove old images. In a period of a few months we end up with 10,000 images, where only 10% are live.
I can search the database and generate a list of live image stubs:
5e1adcf7c9c1bcf8842c24f3bacbf169
5e2497180424aa0d5a61c42162b03fef
I want to delete the images that don't begin with these stubs.
Note that time/space performance is an issue here too. There are ~500+ stubs at any one time. I have tried grepping ls like:
ls | grep -vf <(
sqlite3 database.sqlite3 'select replace(images, CHAR(124), CHAR(10)) from cars_car'
)
This works but it's critically slow (and you shouldn't parse ls
). The query is fast so it's the grep
bit that bogs it all down. I'd like better solutions. Bash isn't necessary but it's what I do most of my maintenance scripting in.