For many of the more difficult operations in building Toolkit, involving sometimes hundreds of lines of data, sometimes literally thousands, I used what is known as a grep parser.  This is basically a very fancy, programmable text search-and-replace tool.  I use a product called Abacre Advanced Find and Replace, which is not freeware (but, compared to any of its serious competitors, is very cheap).  It's an awesome tool for coders or anybody else working with huge amounts of text, where you may want to change a whole bunch of the text... very, very selectively.  

It can also perform mathematical operations on numbers it finds, and evaluate them as numbers!  An extremely useful tool- I could never have fixed as many of the float errors, etc. in Freelancer's INI code without it- the number of problems is mind-bogglingly large.  I highly recommend this tool, which is supported by the company that makes it (I once asked a question via email about a tricky mathematical problem, and they replied that they would put that particular ability into the next build of the software- and DID IT).  At any rate, I usually don't recommend or push anything, but I had a good experience with this software and the company, and I think that others might find it useful.



How to do Advanced Find and Replace operations on strings:

Here's an example of how to get particular values and return a new one, for things that have various names mixed together in a bunch of lines/files.  This may even work with some freeware parsers, but probably not- advanced Booleans and string storage aren't usually available in the freebies.  But you can try it out.

In this example, I had to change the following text from this:

{
UNITNAME=ROMAN;
    UNITNAME=CANDLE;
}

To this:

{
UNITMENU=ROMAN;
     UNITNAME=CANDLE;
}

As you can see, this is not possible using a normal literal replace method- if you just replace UNITNAME with UNITMENU, you will screw up the second instance, which I wanted to leave alone.  Worst yet, I had to do this in (literally) several hundred text files.  Imagine opening hundreds of text files, by hand, and then having to replace this one bit of text!

Obviously, this is a bad situation.  And it's the sort of bad situation that comes all too frequently for modders who are working with reeeeeally huge data sets, so I thought I'd share my solution.  I'm sorry it doesn't involve freeware, but sometimes you have to pay to get the "good stuff" ;-)

At any rate, I first wrote out my Find grep:

UNITNAME=(\w+);\r\n(\s+)UNITNAME=(\w+);  

This says "get the first part of this string (literal), then get any alphanumeric combination (greedy), then get a semicolon, a newline-carriage-return (\r\n), then any number of spaces (greedy), then another literal string, then another alphanumeric combination (greedy) then a semicolon."

To replace it with a new value for the literal strings we've found, but keep the text that we don't know the values of intact, we need to store those variables as strings.  We use $1, $2, etc. to designate these strings- whatever (\w+) "finds" will be stored as a string value and regurgitated at runtime.  Here's the output:

UNITMENU=$1;\r\n$2UNITNAME=$3;

This changes the first UNITNAME to UNITMENU, but preserves all of the previous string data.  Giving it a two-line context, in this case, prevented it from changing the second UNITNAME instance, which is why I did this in this particular case.

I hope that this random bit of knowledge is useful.  I know that if it weren't for the power of Grep, I probably would've given up on Toolkit ages ago- the amount of data that I've had to modify has been tremendous, and doing it all by hand would've taken forever.