2

I'm having a huge csv file that is url encoded.

I would like to decode all lines, and I thought sed could do the trick for me, but I can't make it work.

Here's my script:

#!/bin/bash

function urldecode() {
    # urldecode <string>
    # from https://gist.github.com/cdown/1163649

    local url_encoded="${1//+/ }"
    printf '%b' "${url_encoded//%/\\x}"
}

export -f urldecode

sed -e 's/.*/urldecode &/e' big_file.csv

This produces a repeating error message of sh: 1: urldecode: not found

EDIT: Somehow, this seems to work in one shell, but not in a different shell. It works in Git Bash on Windows - but not in the Ubuntu 18.04 on Windows. Both are running GNU bash 4.4.19, but obviously slightly different builds.

  • 2
    It will possibly work on systems where /bin/sh is symlinked to bash, but otherwise I don't think it's possible to export functions to arbitrary subshells in the way that you expect. See for example Can I “export” functions in bash? – steeldriver Oct 21 at 12:17
1

As @steeldriver pointed out, sed will spawn /bin/sh which is symlinked to /bin/dash in current Ubuntu releases and doesn't support functions. This is because sed internally uses popen, which always spawns /bin/sh (see man popen).

If you can't or don't want to make bash the default shell and need to use bash functions in sed, you can use the following workaround.

In order to make /bin/sh point to /bin/bash, we first use unshare to spawn a new bash with a private mount namespace, bindmount /bin/bash on /bin/dash and then execute the sed command:

unshare -m -r bash -c "mount --bind /bin/bash /bin/dash && sed -e 's/.*/urldecode &/e' big_file.csv"

That way, all exported functions are preserved. You can also write a function so you don't have to write the whole unshare... part all the time, e.g.:

#!/bin/bash

function mysed() {
    sedcommand=sed
    # restore quotes around each script
    while test $# -gt 0; do
        [[ "$1" == "-e" ]] && { shift; sedcommand="$sedcommand -e '$1'"; } || sedcommand="$sedcommand $1"; shift
    done
    unshare -m -r bash -c "mount --bind /bin/bash /bin/dash && $sedcommand"
}

function urldecode() {
    local url_encoded="${1//+/ }"
    printf '%b' "${url_encoded//%/\\x}"
}

export -f urldecode

mysed -e 's|.*|urldecode &|e' big_file.csv

Be aware, though, that the -r option of unshare, which is needed to be able to bindmount, creates a kind of virtual environment in which you are root. Read/write permissions are the same as the user who called unshare, but uid and gid will be 0. For example, if you call whoami inside urldecode, it will print root.

You could also simply run the whole script using unshare:

unshare -m -r bash -c "mount --bind /bin/bash /bin/dash && ./script.sh"

...but the restrictions from the previous paragraph apply.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.