To get pip, install python-pip:
  $ sudo apt-get install -y python-pip

Put ocse in python modules with
  $ (cd ocse; sudo pip install -e .)

To get lxml
  $ pip install lxml


Extract sites from Juliet code for various warnings.

To rebuild all site files, run
$ make all
Caution: each one takes over 30 minutes to execute.

To rebuild one site file, use it as the target, e.g.
$ make write_buffer_sites.xml


CODE STRUCTURE

Each weakness class has its own front end-code in this directory.  All
of them invoke extractor with a string indicating what class to look
for.

The main, shared body of the extractor is in extractor.py within
ocse/ocse.  It invokes a piece of code to extract all sites for the
appropriate class.  Code to extract all sites is in
ocse/ocse/node_visitor in one file per weakness class, for example,
buffer_read.py.

Then the extractor invokes buggy_sites() to pick buggy locations.
Picking is a set of heuristics based on proximity to strings like
POTENTIAL FLAW: Possible divide by zero.  The heuristics are tuned for
Juliet 1.2.  Using set_buggy_attr(), it adds a buggy attribute to
sites matching a buggy location.  Using to_xml(), it prints the sites
in an XML format, indicating "buggy" when they have the buggy
attribute.

We add code for exceptions and special cases in various places
depending on the nature of the exception, for instance, a set of cases
that are not buggy or a set of sites to be excluded in one class.


LIMITATION (BUG) IN EXTRACTOR
There are cases where one buffer read site on a line is buggy and
another buffer read site is good.  They are in CWE126_Buffer_Overread/s01.  
An example file is CWE126_Buffer_Overread__char_alloca_memcpy_01.c
It is line 40.  Here is pertinent code:
26    char * dataBadBuffer = (char *)ALLOCA(50*sizeof(char));
28    memset(dataBadBuffer, 'A', 50-1); /* fill with 'A's */
29    dataBadBuffer[50-1] = '\0'; /* null terminate */
32    /* FLAW: Set data pointer to a small buffer */
33    data = dataBadBuffer;
35        char dest[100];
36        memset(dest, 'C', 100-1);
37        dest[100-1] = '\0'; /* null terminate */
38        /* POTENTIAL FLAW: using memcpy with the length of the dest where data
39         * could be smaller than dest causing buffer overread */
40        memcpy(dest, data, strlen(dest)*sizeof(char));
The read from dest, within strlen(), is fine.  The read from data is
buggy; the code reads too much data.  So we have two buffer reads on
one line.

However, the extractor only matches buggy sites by file, line, and
type.  Here's the code in extractor.py:
439        if(buggy_site.filename == site.filename and buggy_site.line == site.line and buggy_site.site_type == site.site_type):
440          site.buggy = True

There is no way to have one type of site on a line be good, and
another site of the same type be buggy.  The variable name (info, in
the output file) has the variable name, so some of the information is
available.

We compensate by a special hard-code check in to_xml(), which writes
the list of sites in XML.


OTHER CODE

sitesxml2csv.py reads the xml file produced by get_*_sites.py, selects
buggy sites, non-buggy sites, or all sites, and write them in a
comma-separated value (CSV) format.  This is invoked in the Makefile
of each class, which provides the name of the xml site file and the
warning name.
