Better s3cmd put

posted in code with 0 comments at work

I’ve been using s3cmd to back up the contents of an external disk to Amazon’s S3 service, and honestly, s3cmd is kinda crap. It’s the most accessible tool for S3, and it’s free, but it’s kinda crap. The biggest problem I have stems from when I start, stop, and then resume an upload; already-uploaded files get uploaded again, which means I have to start from scratch, or manually exclude (painfully) already-archived files. sync should skip existing files, but in practice, it simply does not. put also randomly skips files that I can upload fine individually.

This script takes the bucket and folder(s), descends recursively into each given folder, and individually validates that each file exists before it is uploaded. This isn’t the best solution if you want to put files into a deep folder on the bucket, because in my case I want all of the passed folders to wind up in the root of the bucket.

Update: I’ve had this running for about three days now, and given all of the problems I had in the past, my wrapper works amazingly well. s3cmd put skipped every other folder, and s3cmd sync stoically restarted and uploaded existing files every time I ran it.

# Call with command <bucket> <director{y,ies}>
# Email for help. 

print_time() {
    # Appends timestamp to start of next line.
    echo -n "$(date +%H:%M:%S) "

check_deps() {
    # Check for missing dependencies and exit if they aren't found.

    for n in $(seq ${#dep[@]}); do
        if [ ! -x ${dep[$n]} ]; then
            echo "ERROR: '${dep[$n]}' not found"

    if [[ $dep_missing == 1 ]]; then
        exit 1

check_bucket_exists() {
    # Test for the existence of the given bucket, and exit if it isn't found.
    s3cmd ls | grep "$1"
    if [[ $? == 1 ]]; then
        echo "ERROR: Bucket '$1' not found"
        exit 1

s3put() {
    # Test for the existence of $file in the remote bucket, and upload it if it
    # doesn't exist. s3cmd's sync/put command continued to upload 
    # already-existing files. 
    temp="/tmp/s3_$(date +%Y%m%d%H%M%S%N)"

    cd "$1" && cur_dir=$(basename -a $PWD)
    s3cmd ls -r $2/$cur_dir | sed -e 's/^.*  //g' > $temp 
    for file in $(find . -type f -iname "*" | sed -e 's/^.\\///g'); do
        grep "$file" < $temp > /dev/null 2>&1

        if [[ $? == 1 ]]; then
            # s3cmd generates excessively verbose output, so I parse it.
            s3cmd --disable-multipart put "$file" $2/"$cur_dir"/"$file" 2<&1 \\
            | grep -v "WARNING" | sed -e 's/ \\[.*\\]//g' 
            echo "File '$file' is already stored as '$2/$cur_dir/$file'"

    rm $temp > /dev/null 2>&1

check_bucket_exists $1

if [[ $# == 0 ]]; then
    echo "ERROR: No directories provided"
    exit 1

for file in "$@"; do
    if [ -d "$1" ]; then
        s3put "$1" $bucket 


exit 0

by Mark -

Is the hackerspace open?

posted in code with 0 comments

This started as my own tongue-in-cheek take on 091 Labs’s Lo-lo project, an open/closed state indicator for the hackerspace. It has the dangerous potential to become something moderately more obsessive, because the four lines of jQuery at its core don’t fucking work in Internet Explorer (shocker). The page, the CSS? Validates perfectly. The JavaScript I am less sure about, although I am reassured by the fact that none of the browsers throw up any errors.

Incidentally, Firefox has the smoothest font rendering.

by Mark -

Ubuntu died

posted in technology with 0 comments

I did an in-place do-release-upgrade yesterday. The laptop was not completely plugged in, and the battery went out when the upgrade process was about 75% complete. Since then, things have been…very interesting due to the lack of 64-bit recovery media in the hackerspace.

by Mark -

C# annoyance

posted in code with 0 comments

C# code in Sublime Text



The next time I need to calculate compound interest daily, I should probably check my sample numbers with a *daily* compound interest calculator. Before, you know, I spend four hours writing a check-check-checker (check3) and proverbially pulling my hair out.

Turns out that Google Docs and several other online calculators calculate it yearly.

by Mark -

Why a television show and an ICBM are the same thing

posted in technology with 0 comments

I’ll open this up with a disclaimer: I am a nitpicky geek.


Producing a television show, weirdly, has a lot in common with designing a ballistic missile. In the minds of every missile designer exists the perfect missile. This perfect ballistic munition can carry an infinitely large warhead, at an infinite speed, over an infinite distance. This perfect missile will never miss its target, ever.

However, when the missile leaves the mind of its designer, this veritable Apollo’s Arrow becomes subject to the physical laws of our universe. Anything infinite is right out. Travelling at more than a few tens of thousands of kilometres an hour, or delivering more than a hundred petajoules of energy to your target is unlikely. Actually, the further you delve into missile design the more it becomes a game of compromise. Throw-weight is a measure how much payload a missile can carry. Maximizing throw-weight is the goal of every missile designer and the objective of any compromise.

Let’s say that you are a missile designer. You have been asked to increase the throw-weight of a ballistic missile by your superior. You think “aha, easy!” and you come back to your supervisor with a proposal to just put a bigger engine on the missile to increase its throw weight. Let’s consider your engine proposal:

You need to be able to fuel this bigger engine. You can (1.) increase the quality of your fuel, although this in turn requires superior distillation, chilling and transportation infrastructure. This might not be economical or practical; quality control at every stage of missile construction is a nontrivial issue. Or you can (2.) add more fuel. Adding more fuel means more weight in both fuel tanks and plumbing. Ultimately you might not eke out any performance gain at all.*

“So what’s the connection with television?” you ask.

The creator of a television show has to make similar choices: He or she wants to increase the quality of their show, maximize the audience and minimize costs. Achieving any of those three goals requires compromising in the other two areas. If you increase quality you increase budget. If you increase the audience you (may) decrease quality. If you lower the budget you lower the quality.


I’m confident that someone smarter than me has expressed this in an elegant algorithm before going onto author a successful pop-culture book which relates this in a laid-back anecdotal manner to yonder lay audience.

So let’s take Fringe, a procedural science fiction show, spiritual successor to The X-Files, currently well into its fourth season on America prime time television.

Fringe has made a point of altering its dynamic from season to season:

  • In season one we were introduced to Fringe Division and the Pattern.
  • In season two we were shown that a war between universes was brewing, and that a parallel Earth was out to get us.
  • In season three, all hell broke loose, and we crossed back and forth between universes on a seemingly daily basis.
  • And now in season four, we’ve shunted into an alternate timeline where one of the main characters died as a child.

Beneath this is a threefold formula that hasn’t really changed: You have your random monster of the week story, your random mad scientist of the week story and the odd, interspersed episode that advances the overarching storyline.

It’s become a bit stale.

At its heart, you have a cadre of fantastic actors who are really comfortable in their roles. It’s worth tuning just to watch the dynamics between the three leads, Anna Torv, Joshua Jackson and John Noble – and they’ve rightly won awards for these roles. There’s solid quality there. The writing? Not so much. They have to wrap up the monster/scientist of the week story in time for a vignette advancing that season’s storyline. This is getting old. And it gets worse. Very occasionally some of the technical details are flat-out wrong (this is the nitpicker speaking). I’m five episodes into the seasons and we’ve had two mad scientists and a monster. Really, how many damn unethical scientists are out there? Should we just go and pack them into concents already?

However, for the reasons given above, I can’t complain too much. I don’t so much understand the difficulties, as I understand that there are difficulties in balancing a show. But hey. Maybe we could change the formula up more. Show us more of the supposed brewing global crisis. Set dressing permitting, visit somewhere else in America or the world.


Eight hundred words. I’m spent.

(If you want to read more on this topic, then I urge you to purchase or download (free PDF) a copy of Space Weapons, Earth Wars. This excellent monograph on space weapons includes a lengthy meditation on throw-weight.

by Mark -

Resorting to the nuclear solution is getting old

posted in technology with 0 comments


Fuck, it works, but I’d have thought that more computer know-how would somehow let me finesse my solutions a bit more. As opposed to, you know, fucking beating them with a hammer. So, the PC I’m constructing and the problem I had getting an operating system onto it. I did wind up travelling into Sligo, and did venture inside Tesco to price blank DVDs: The price of blank media stopped me dead. I need one disc. A pack of fifty for fifteen euros, I just couldn’t justify, so I put it back on the shelf, left the store and went on with my Saturday shopping.

When I came home a whiles ago, I pulled the hard disk out of the tower and plugged it into my laptop. There I partitioned it with gparted, and then cloned my notebook’s Windows partitions with dd:

[root][mark] # dd if=/dev/sda2 of=/dev/sdb1 bs=4096
640000+0 records in
640000+0 records out
2621440000 bytes (2.6 GB) copied, 106.572 s, 24.6 MB/s

[root][~] # dd if=/dev/sda1 of=/dev/sdb2 bs=4096
10752000+0 records in
10752000+0 records out
44040192000 bytes (44 GB) copied, 1652.62 s, 26.6 MB/s

If the tower doesn’t boot after this then I may have to beat it to death with a stick.

by Mark -