patternbashMinor
Shell script to archive files to AWS S3
Viewed 0 times
scriptshellfilesawsarchive
Problem
This is a simple script that moves files to S3 if they are > 14 days old. It ignores files that are < 14 days old. If the file is successfully synced to AWS S3 then we can remove it from the server. If not, we leave the file where it is. The echo commands in the script are for debugging. I am aware that this could be a one liner, so interested to see what people suggest.
#!/bin/bash
fortnight=$(python -c "from datetime import datetime, timedelta; print (datetime.now()-timedelta(days=14)).strftime('%Y%m%d')")
for f in /var/mail/catchall-????????.gz
do
if [[ "$f" < "/var/mail/catchall-$fortnight.gz" ]] ; then
mv "$f" ./archive
echo "$f"
echo "$(basename $f)"
cd ./archive
if s3cmd sync "$(basename $f)" s3://vpcgwmail ; then
echo "S3 sync successful, removing file"
rm "$(basename $f)"
else
echo "s3 sync not succesful"
fi
fi
doneSolution
It's a fine script, but you have a bug: executing
Perhaps you did not notice it because you tested with only a single matching file. If there are multiple matching files, then
One solution is to wrap the
Another solution is to not
Whichever alternative is fine.
The indentation was also a bit off, from about the middle of the for-loop body.
cd ./archive in a loop.Perhaps you did not notice it because you tested with only a single matching file. If there are multiple matching files, then
cd ./archive will not work, unless you have multiple nested archive/archive/archive/... directories, which would be silly.One solution is to wrap the
cd command and the rest of the loop body in a (...) subshell:if [[ "$f" < "/var/mail/catchall-$fortnight.gz" ]] ; then
mv "$f" ./archive
echo "$f"
echo "$(basename $f)"
(
cd ./archive
if s3cmd sync "$(basename $f)" s3://vpcgwmail ; then
echo "S3 sync successful, removing file"
rm "$(basename $f)"
else
echo "s3 sync not succesful"
fi
)
fiAnother solution is to not
cd ./archive at all:if [[ "$f" < "/var/mail/catchall-$fortnight.gz" ]] ; then
mv "$f" ./archive
echo "$f"
echo "$(basename $f)"
if s3cmd sync ./archive/"$(basename $f)" s3://vpcgwmail ; then
echo "S3 sync successful, removing file"
rm ./archive/"$(basename $f)"
else
echo "s3 sync not succesful"
fi
fiWhichever alternative is fine.
The indentation was also a bit off, from about the middle of the for-loop body.
Code Snippets
if [[ "$f" < "/var/mail/catchall-$fortnight.gz" ]] ; then
mv "$f" ./archive
echo "$f"
echo "$(basename $f)"
(
cd ./archive
if s3cmd sync "$(basename $f)" s3://vpcgwmail ; then
echo "S3 sync successful, removing file"
rm "$(basename $f)"
else
echo "s3 sync not succesful"
fi
)
fiif [[ "$f" < "/var/mail/catchall-$fortnight.gz" ]] ; then
mv "$f" ./archive
echo "$f"
echo "$(basename $f)"
if s3cmd sync ./archive/"$(basename $f)" s3://vpcgwmail ; then
echo "S3 sync successful, removing file"
rm ./archive/"$(basename $f)"
else
echo "s3 sync not succesful"
fi
fiContext
StackExchange Code Review Q#138268, answer score: 3
Revisions (0)
No revisions yet.