Also adds some tests for this

By using static variables inside helper functions,
these regexes will only get compiled/allocated once.
Since this only happens once, we don't need to worry about freeing the
memory after use.

This greatly simplifies init/cleanup
of the context struct.

While I was at it, make the helper functions
more readable at what they are doing.

Also:
- remove passing "extension" around excessively in fec.c
- stop passing around the full FEC_CONTEXT excessively when you only
need access to a single line.
- Capitalize CSV_EXTENSION since it's a constant

After I did all that refactoring, I now
realize we don't even use this info at all.
As we read each line, we see if it contains
any asci28. If so, then we treat that as the delimiter.
Otherwise default to comma.

It really is a private method of PARSE_CONTEXT,
we don't care about the implementation in fec.c

This is an implementation detail of the csv parser,
we don't want to have to deal with it.

Another implementation detail of the csv reader.

We never need to share one FIELD_INFO between multiple places,
and the lifetimes make sense to be tied to each other. This makes the
API and lifecycle management simpler.

If I add "-Werror" build flag, then I found a few errors.

Ideally I think we should turn on this flag by
default to help catch more bugs.

It was weird to store these regexes in the persistentMemory. We
only have one version of the mappings
per compilation (and runtime) so
we can just store them statically.

Next up would be to refactor the lookupMappings()
function from fec.c
to mappings.c

While I'm at it, clean up the regex compilation to share
the helper function we already have.
This involved creating
regex.h to hold that shared regexCompile()

tmpdir is deprecated in pytest.

We never used the load kwarg, and the
helper function did not need to be a fixture itself.

We want the reader thread to be stopped
if the parent thread stops.
Before, my pytest runs sometime hung.

Before, the testing was a bit half-baked.
We didn't actually check the contents of the output files.
We also weren't very thorough, and only checked some APIs on
some examples. Now we test all APIs on all examples.

Some example .fec files that were tracked in git weren't actually
getting tested at all. For example,
the 13360.fec file actually produces a segfault!
We need to track down the issue and fix it.
It is skipped for now.

These changes also revealed an inconsistency in  howform types
are normalized. See `python/tests/filings

We didn't need all the info that PARSE_CONTEXT contained.

So to run the "trailing_commas" test, you can do:
  pytest -m trailing_commas

Now there is a single understandable
`FORM_SCHEMA *lookupSchema(const char *version, int versionLength, const char *form, int formLength);`
function.

The stuff in fec.c just wraps it with logic specific to FEC_CONTEXT

and similarly,
freeSchema() to
formSchemaFree()

This doesn't have to do with csv fields, it's just a raw string.

When I first read this code I was confused as to this structs role.

endOfField() is only used in csv.c so it doesn't need to be declared.

Now the call sites don't have to do the test, and the loop logic is
much simpler.

Before, it set the version string based on the
implicit
ctx->persistentMemory->line->str,
which was not at all obvious from the call sites.

The testing for whitespace was wrong, which
meant we never found the version key,
so we never setVersion(), so in later steps
we got a null pointer for ctx->version

It is only used in one place, so remove
one level of indirection

First of all this was broken:
- it always returned os.path.join(PARENT_DIR, files[0]), not
  where we actually found files.
- The zig build directory wasn't right.
- Really we want to warn if there are multiple DLLs for a certain dir,
  not for a certain pattern.

Before, I had to run
- zig build
- python setup.py
- pytest

to do a testing cycle.
Now I don't need the `python setup.py` step.

We haven't yet set ctx->types yet, so it will always be NULL at
this point. But make this explicit.

We set ctx->types once we start parsing lines. It would be
irrelevant to even try to pass something here, because
there is no entry in the `types` lookup
table for the `hdr` form.

in the docstring we say it is, but it wasn't!

growStringTo() expects an arg for the total bytes
needed including the null terminator.
STRING->n represents exactly this, we don't need
to compensate with an extra +1 like we do when
we are reading from a raw `char *`

…rt() etc

just pass the actual line we are operating on. Alos make them static
"private" with a leading underscore.

This DOES slightly change the behavior by no longer using
ctx->currentLineLength, but I think that
is vestigal and unused anyway, and I'm about
to remove it.

These were mistakenly used before, when really we should have been using
ctx->persistentMemory->line->n
Now I've switched us to that everywhere, so it's safe to remove this.

This also has the cascading effect of allowing us to simplify the API
of decodeLine() so it no longer returns anything,
it just decodes the line.

This also fixes a bug in iso_8859_1_to_utf_8()
where we didn't increment length when we should have.
I'm amazed this didn't crash before?
It makes me nervous that we're not using it correctly
elsewhere.

Also continue removing passing FEC_CONTEXT around willy nilly

Now we can pass around this object that contains the parsed
value, instead of having to deal with
the pointers in the parser.

I think this API makes more sense. Now
the parser actis like an iterator. You
call nextField() to get hte next field and advance.
No need to do the additional followup step of calling advanceField().

Renaming to numFieldsRead also is more intuitive for the users

Make it a special case to pull out the first column as the form type.
Now we don't need an if() inside the loop, which is more understandable

We don't need the full STRING, so don't ask for it.

We only were ever using the ascii28 flag from the result,
so don't pass around the rest of it.

We get a void * back, so the casts just add clutter.

try to put the related things next to each other

put the parts in order

We might want to write to a different dir and want control
of that outside of WRITE_CONTEXT

This is way more readable I think

It was really hard to tell what each represented

This can be inferred from whether or not filingID is NULL,
and now that the WRITE_CONTEXT doesn't depend on the filingId,
all the filingId processing can happen
in the callers of newFecContext()

These "options" should all be near each other.

Because I'm about to add another that has extra fields
and I want to make the distinction

when you hit an assert, pytest will show the values of the args
of the encolsing function.
Before, you only saw the values to dir1 and dir2,
but now you can see file1 and file2.

I thought this test indicated a bug in the lib, but
actually I just wasn't testing it correctly.

Now if the supplied value can't be coerced to
a Path, you get the error when you first call the function, not when
the file is opened int he callback.

make them shorter to conform to line legnth. improve wording a bit.
dont repeat defaults that are in the funciton
signature.

Now we can see the line of which file.

If we run into a field that is supposed to be a float, but it is an
empty string, this isn't actually a problem.
Before, we were throwing a warning.

It was hard to tell where the stirng ended before, like
if it was an empty string.

Fixes washingtonpost#24

Before, we printed exactly what was in the .fec file. If a row had
more fields or fewer fields than we expected from the schema, we
just printed it as is. This broke downstream
tooling, such as loading the .csv's into
databases.

Now, by default we
- pad short rows with empty fields
- truncate long rows
You can get back to the old behavior by setting the `raw` flag to True.
By default it is False.

Note that this could be BREAKING.

This also adjusts the warnings a bit:
BEfore, you got a warning for every extra field in a row.
Now you only get one warning per row, and we print out the full row
(even though that row is a bit mangled by the csv parser as
it removes quotes and delimiters)

The tests only currently test the default behavior. A follow up should
adjust how we define test cases. Currently, we expect a 1:1
correspondence between an input .fec file and an output. But really we
want a 1:N relationship, where one .fec file
can generate multiple outputs depending on the
options passed. That will require updating our test definition
format.