7

I cloned with git the source code of a project, and I used grep to find a word in the entire project, SchemaHandler. grep failed to find the word in this strange code file:

https://github.com/yawlfoundation/yawl/blob/master/src/org/yawlfoundation/yawl/schema/YDataValidator.java

Yes, it's not a standard text file, I admit, but grep is able to find strings also in binary file. Indeed it was able to found the word in two jar files in the project. Why it failed with this text one?

The command I used was grep -R SchemaHandler . in the root of the cloned project.

I use Lubuntu 18.0.4 and grep 3.1

  • 2
    Have you also tried git grep SchemaHandler? This greps in all files known to git automatically. – PerlDuck Jul 1 at 16:24
  • 1
    Ok, git grep works, thanks. It's a good workaround. But the fact grep does not work puzzle me. – Marco Sulla Jul 1 at 16:26
  • That is a perfectly normal text file apart from the embedded \r\r. – terdon Jul 1 at 17:29
8

It does find it. The problem is that the file has a carriage return (\r) which causes the file name to be hidden by the terminal. To illustrate what I mean, compare these two commands:

printf "foobar\n"

and

printf "foo\rbar\n"

This is what each should print:

$ printf "foobar\n"
foobar
$ printf "foo\rbar\n"
bar

The second command is still printing the foo, but the \r causes the terminal to go back to the beginning of the line, so it is being overwritten. You can see it if you use od though:

$ printf "foo\rbar\n" | od -c
0000000   f   o   o  \r   b   a   r  \n
0000010

A similar thing is happening when you run your grep -R:

$ git clone https://github.com/yawlfoundation/yawl
$ cd yawl
$ grep -R  SchemaHandler . | grep YDataValidator
 }   }   return schema ;    .replaceAll("type=\"", "type=\"xs:"); xmlns:xs");pes, soonn) e;

At first glance, that results seems strange. Howe come a line that doesn't seem to contain the string YDataValidator is being returned by the grep YDataValidator? The answer is the \r. That line is actually from the YDataValidator.java file, but the file name is being obscured by the \r. You can see this more clearly if you tell grep to only print the matching string and a few characters around it:

$ grep -PRo '.{10}SchemaHandler.{10}' . | grep YDataVal
 * @a/org/yawlfoundation/yawl/schema/YDataValidator.java:d using a SchemaHandler.
./src/org/yawlfoundation/yawl/schema/YDataValidator.java:  private SchemaHandler handler;
./src/org/yawlfoundation/yawl/schema/YDataValidator.java:ler = new SchemaHandler(schema);
./src/org/yawlfoundation/yawl/schema/YDataValidator.java:          SchemaHandler handler =
./src/org/yawlfoundation/yawl/schema/YDataValidator.java:      new SchemaHandler(DOMUtil.g

Here, since there is no \r in the 10 characters immediately around any of the 5 instances of SchemaHandler in YDataValidator.java, the name is displayed correctly. Your original command returned the entire file (the whole file is just one line split by \r which don't define newlines in Linux), so you were seeing one long line which was being garbled because of the \r.

You can verify by running this:

$ grep SchemaHandler ./src/org/yawlfoundation/yawl/schema/YDataValidator.java 
}   }   return schema ;    .replaceAll("type=\"", "type=\"xs:"); xmlns:xs");pes, soonn) e;

And then comparing the output to what you get if you replace all \r with newlines:

$ grep SchemaHandler ./src/org/yawlfoundation/yawl/schema/YDataValidator.java | tr '\r' '\n'
/*
 * Copyright (c) 2004-2012 The YAWL Foundation. All rights reserved.
 * The YAWL Foundation is a collaboration of individuals and
 * organisations who are committed to improving workflow technology.
 *
 * This file is part of YAWL. YAWL is free software: you can
 * redistribute it and/or modify it under the terms of the GNU Lesser
 * General Public License as published by the Free Software Foundation.
 *
 * YAWL is distributed in the hope that it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
 * or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General
 * Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with YAWL. If not, see <http://www.gnu.org/licenses/>.
 */

package org.yawlfoundation.yawl.schema;

import org.jdom2.Element;
import org.yawlfoundation.yawl.elements.data.YVariable;
import org.yawlfoundation.yawl.exceptions.YDataValidationException;
import org.yawlfoundation.yawl.schema.internal.YInternalType;
import org.yawlfoundation.yawl.util.DOMUtil;
import org.yawlfoundation.yawl.util.JDOMUtil;
import org.yawlfoundation.yawl.util.StringUtil;

import javax.xml.XMLConstants;
import java.util.*;

/**
 * This class serves as a validation mechanism for the specification specific
 * schema and the instance data from either the net or a task. This is performed
 * by taking the data available at the various validation points and converting
 * it into conventional XML which is then validated using a SchemaHandler.
 *
 * @author Mike Fowler
 *         Date: 05-Jul-2006
 */
public class YDataValidator {

    // Object that performs the real validation on XML documents
    private SchemaHandler handler;

    /**
     * Constructs a new validator and handler. The
     * handler is not ready for use until validateSchema
     * has been called.
     * @param schema a W3C XML Schema
     */
    public YDataValidator(String schema) {
        this.handler = new SchemaHandler(schema);
    }

    /**
     * Compiles and determines the validity of the current schema
     * @return true if the schema compiled without error.
     */
    public boolean validateSchema() {
        return handler.compileSchema();
    }

    /**
     * Validates a single data variable
     *
     * @param variable to be validated
     * @param data XML representation of variable to be validated
     * @param source
     * @throws YDataValidationException if the data is not valid
     */
    public void validate(YVariable variable, Element data, String source)
            throws YDataValidationException {
        List<YVariable> vars = new ArrayList<YVariable>(1);
        vars.add(variable);
        validate(vars, data, source);
    }

    /**
     * Validates a collection of variables against the schema. This is achieved by
     * temporarily adding a schema element declaration for the data. This avoids
     * attempting to create a new schema containing only the relevant data types.
     *
     * @param vars variables to be validated
     * @param data XML representation fo the variables to be validated
     * @param source
     * @throws YDataValidationException if the data is not valid
     */
    public void validate(Collection vars, Element data, String source)
            throws YDataValidationException {
        try {
            String schema = ensurePrefixedSchema(handler.getSchema());
            org.w3c.dom.Document xsd = DOMUtil.getDocumentFromString(schema);
            String ns = XMLConstants.W3C_XML_SCHEMA_NS_URI;

            //need to determine the prefix for the schema namespace
            String prefix = ensureValidPrefix(xsd.lookupPrefix(ns));

            org.w3c.dom.Element element = xsd.createElementNS(ns, prefix + "element");
            element.setAttribute("name", data.getName());

            org.w3c.dom.Element complex = xsd.createElementNS(ns, prefix + "complexType");
            org.w3c.dom.Element sequence = xsd.createElementNS(ns, prefix + "sequence");

            ArrayList varList = new ArrayList(vars);
            Collections.sort(varList);               // sort on YParameter ordering value

            for (Object obj : varList) {
                YVariable var = (YVariable) obj;
                org.w3c.dom.Element child = xsd.createElementNS(ns, prefix + "element");
                child.setAttribute("name", var.getName());

                String type = var.getDataTypeName();
                if (XSDType.isBuiltInType(type)) {
                    type = prefix + type;
                }
                else if (YInternalType.isType(type)) {
                    type = prefix + type;
                    xsd.getDocumentElement().appendChild(DOMUtil.getDocumentFromString(
                            YInternalType.valueOf(type).getSchemaString()).getDocumentElement());
                }
                child.setAttribute("type", type);

                if (var.isOptional()) {
                    child.setAttribute("minOccurs", "0");
                }

                sequence.appendChild(child);
            }

            complex.appendChild(sequence);
            element.appendChild(complex);
            xsd.getDocumentElement().appendChild(element);

            SchemaHandler handler =
                          new SchemaHandler(DOMUtil.getXMLStringFragmentFromNode(xsd));

            if (! handler.compileSchema()) {
                throw new YDataValidationException(
                    handler.getSchema(),
                    data,
                    handler.getConcatenatedMessage(),
                    source,
                    "Problem with process model.  Failed to compile schema");
            }

            if (! handler.validate(JDOMUtil.elementToString(data))) {
                throw new YDataValidationException(
                    handler.getSchema(),
                    data,
                    handler.getConcatenatedMessage(),
                    source,
                    "Problem with process model.  Schema validation failed");
            }
        }
        catch (Exception e) {
            if (e instanceof YDataValidationException) throw (YDataValidationException) e;
        }
    }

    /**
     * @return String representation of the schema
     */
    public String getSchema() {
        return handler.getSchema();
    }

    /**
     * @return All error/warning messages relating to the last validation/compilation
     */
    public List<String> getMessages() {
        return handler.getMessages();
    }

    /**
     * @return the set of (first-level) type names defined in this schema
     */
    public Set<String> getPrimaryTypeNames() {
        return handler.getPrimaryTypeNames();
    }

    /**
     * Utility method to ensure the prefix is valid (enforces : and
     * defaults to xs:)
     *
     * @param prefix to validate
     * @return validated prefix
     */
    private String ensureValidPrefix(String prefix) {
        if (StringUtil.isNullOrEmpty(prefix)) {
            return "xs:";
        }
        else if (! prefix.endsWith(":")) {
            return prefix + ":";
        }
        return prefix;
    }


    /**
     * A schema may not have a valid prefix if a spec contains no complex types, so
     * this makes sure it gets one in that case
     * @param schema the schema string to check
     * @return a correctly (or defaultly) prefixed schema string
     */
    private String ensurePrefixedSchema(String schema) {
        if (!schema.contains(":schema")) {
            schema = schema.replaceFirst("schema xmlns", "schema xmlns:xs");
            schema = schema.replaceAll("<", "<xs:")
                           .replaceAll("<xs:/", "</xs:")
                           .replaceAll("type=\"", "type=\"xs:");
        }    
        return schema ;
    }
}

Alternatively, you can pipe the output to a pager like less which will show the \r as ^M and can also help you identify the line you're after.

  • Thank you for the explanation. I found I simple "workaround": grep -R SchemaHandler . | less. If you'll add it I'll accept your answer. – Marco Sulla Jul 1 at 21:51
  • @MarcoSulla you're welcome. And I added a line about less. – terdon Jul 1 at 22:02
0

Perhaps more of a comment but I had no problem running grep on your file. I clicked your link, then clicked raw then right-click and Select All then saved the text to a file and ran:

$ grep -R SchemaHandler

greptext: * it into conventional XML which is then validated using a SchemaHandler.
greptext:    private SchemaHandler handler;
greptext:        this.handler = new SchemaHandler(schema);
greptext:            SchemaHandler handler =
greptext:                          new SchemaHandler(DOMUtil.getXMLStringFragmentFromNode(xsd));

Hopefully someone who uses github frequently can describe why raw text works for me and why cloning doesn't work for others.

  • 1
    Probably in raw text your browser, or the text editor where you pasted, converted the \r to \n. – Marco Sulla Jul 5 at 6:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.