feat/data: update HIFLD data to April 2022 version, refactor line assignment

[danielolsen]

Pull Request doc

Purpose

• Update the data we're pulling to use the latest available (April 2022 for both transmission lines and substations)
• Refactor the logic to account for the fact that many lines don't have substations at their endpoints
• Update our set of line/substation interconnect assumptions in light of the above.

What the code is doing

• In load.py, we're a bit more accommodating of the inconsistent naming of the Shape Length columns that we want to drop.
• In topology.py, we make sure that we tag all of the separated/reconnected lines with their appropriate interconnects.
• In transmission.py:
  • We modify the behavior of the map_lines_to_substations_using_coords function so that if there are no substations 'close' to a line's endpoint (as defined by our new optional parameter), we create a new proxy substation for it. We do the KDTree/query process twice, where the first time just collects a list of transmission line endpoints with no substation, then we add proxy substations, and the second time we actually re-map all the lines.
  • We avoid filtering out substations which are tagged as having LINES = 0, since this information seems to be very noisy.

Testing

Tested manually to generate static grid components. The resulting static grid components haven't been tested for network feasibility yet, so we may also need to update some of the line assumptions that were introduced in #276. EDIT: update: the results of this new code have been tested alongside the new code in #286 and new line/substation assumptions have been added which seem to ensure generator-side feasibility.

Usage Example/Visuals

The call signature to map_lines_to_substations_using_coords remains the same, we just add one more optional parameter that controls how far away from any substation a line's endpoint can be before we create a new proxy substation for it (instead of connecting it to an existing one).

Time estimate

30 minutes? The code changes are fairly light, most of the changes are in the constants which required a tedious guess-and-check process. This is a draft PR for now, since the functionality depends partially on #283, and since we may want to make some modifications to the underlying logic, but the code that's here creates the interconnections as we expect. EDIT: #283 is merged, so this is ready to go.

[jenhagg]

Just realized we require python 3.8 now, so we can add the equals sign to print name=value, e.g.

In [1]: x = [1,2]

In [2]: f"wow so cool {x=}"
Out[2]: 'wow so cool x=[1, 2]'

[rouille]

Wow so cool!

[BainanXia]

Fancy!

[rouille]

That is bad

[BainanXia]

Indeed. And this looks familiar to me which reminds me the days playing with country names from different datasets.

[rouille]

Looks good

[BainanXia]

LGTM

[danielolsen]

I'm going to wait to finish/merge this until #286 is merged, since the updates for feasibility don't work by themselves without the #286 logic. EDIT feasibility updates also depend on #292, which depends on #286, so I think the preferred merge order is: #286, #292, #289.