I think you need a heavier programming language for this. Python is my language of choice so here's a simple script with a simple example of a test:
tests = [
lambda a, b, c, d, e: a+1==b and b+1==c and c+1==d and d+1==e,
with open(sys.argv) as f:
for line in f:
if any(t(*map(int, line.split(','))) for t in tests):
sys.stderr.write('Line removed: %s\n' % line)
That's obviously only a skeleton example of tests but it should be usable. Run normally it will output just the lines that don't match to STDOUT and the ones that do to STDERR. This makes it useful for redirecting into a new file.
Here it is in action:
$ python patterns.py <(echo -n 1,2,3,4,5)
Line removed: 1,2,3,4,5
$ python patterns.py <(echo -n 1,2,4,4,5)
Once you've loaded it up with patterns, you can just pass it the csv:
python patterns.py input.csv
In terms of performance, Python isn't always the fastest. I use it because it's more than fast enough for web development and the time to write is much faster (which is what costs me time/money).
You can speed things up with PyPy. This is an alternative Python runtime that benchmarks amazingly well. You might not need the PPA version (Trusty ships 2.2, PPA is 2.3.1) but here's how you would:
sudo add-apt-repository ppa:pypy/ppa
sudo apt-get update
sudo apt-get install pypy
You'd then launch your script with
pypy script.py or if you're executing it directly change the opening shebang to
#!/usr/bin/env pypy. I've done some very simple testing on a 350000-line input file (your example repeated 50000 times) with the above script.
python2 ran it in 1.417s and
pypy ran it in
0.645s. In my experience, you're likely going to see an even bigger improvement with more complicated algos.
... But yeah, none of this is going to beat the C/C++ equivalent. If the time it takes to run is money, spend some time reimplementing it in a faster language.