Segments creation is dropping stops. Speeds missing segments

[JabunchWork]

Two issues I am having are shown in the attached powerpoint.
1st: When segments are created, it seems that stops are being dropped on some routes.
an example is from Harrisonburg Va's GTFS.
The pink line stop times file shows for all trips
Stop sequence Stop
9 88895
10 88751
11 88771
But the segment created goes for stop sequence 9 goes from 88895 - 88771.
A similar example is for the green line.

Also: for trips going from 88871 to 88753 a segment is properly created, and the frequencies by segment correctly calculates the frequency. However, the segment doesn't exist in the speeds by segment output.

Do you have any thoughts on what is going on and how to fix these issues?

JAB
pythonfortransitissues.pptx
Passeo_GTFS_wstoptimes.zip

[JabunchWork]

So upon further investigation it seems that the dropping stops is related to routes that divert off a main road and loop down a local road to a neighborhood (one or more stops) then circle back along the same road to the diversion point and continue on the main road. While there are different shape points in the shape file for each direction, they are either on top of one another or very very close to the same path. Is this solveable?

[lrosenfield-uta]

After a little bit of work, I think I have a functional algorithm for fixing this. The problem is that the segments tool uses the shapely project function, which will always find the point along the line nearest to each stop. However, if a line loops on itself, there will always be one point along the line that's closer both times the loop passes the stop, so we end up with at least one stop out of sequence that either has to be dropped or fixed.

It's pretty easy to detect when this is occurring, but harder to fix it. solution I've found it recursing out from wherever a segment discontinuity is detected, incrementing by one in both directions until you find a fix. I've tested this on UTA's latest GTFS feed, as well as a tester feed I built custom, and it works in both cases, but there may be some corner cases in other feeds where it still causes problems. For that reason, I'd recommend issuing a warning when loops are detected, but otherwise putting this code behind an optional flag so that users can hammer it with their feed and let us know if it's causing more problems than it fixes.

I've included the snippet because I don't know when I'll have time to fully integrate into the main tool and upstream:

First, groupby route patterns, then apply the following function chain to each route pattern dataframe

def perform_secondary_parse(df: pd.DataFrame):
df['substring'] = np.nan
df.loc[df.index[0], 'cut_distance_stop_point]=0 # Technically you can do this before GroupBy, but it's nice to maintain parity
df.loc[df.index[-1], 'cut_distance_stop_point'] = 1
df = diff_and_reproject(df, 1)

def diff_and_reproject(df: pd.DataFrame, offset, recurse = True):
df = df.sort_index(ascending=True)
df['cdsp_prev_stop'] = df['cut_distance_stop_point'].shift(offset).fillna(0)
df['cdsp_next_stop'] = df['cut_distance_stop_point'].shift(-offset).fillna(1)
df['prev_wrong'] = (df['cdsp_prev_stop'] > df['cut_distance_stop_point'])
df['next_wrong'] = (df['cdsp_next_stop'] < df['cut_distance_stop_point'])
#df['cdsp_next_ordered']
if df['next_wrong'].any() or df['prev_wrong'].any():
df['prev_cdsp']=df['cut_distance_stop_point']
fix_pattern(df)
tester = df['prev_cdsp'] != df['cut_distance_stop_point']
if tester.any():
print("Made Corrections, re-setting to offset 1")
offset = 1
else:
print("no corrections made, increasing offset")
offset = offset + 1
if recurse:
df = diff_and_reproject(df, offset)

def fix_pattern(df: pd.DataFrame):
print(f'Fixing pattern {df['shape_id'].iloc[1]}')
df['substring'] = df[df['next_wrong'] | df['prev_wrong']].apply(
substring_general, axis=1)
try:
df['substproj'] = df['substring'].project(
df['geometry_stop'], normalized = True)
except BaseException as e:
print(df['substring'])
raise e
df['substproj'] = (
df['substproj']*
(df['cdsp_next_stop']-df['cdsp_prev_stop'])+
df['cdsp_prev_stop'])
update = (df['substproj']<df['cdsp_next_stop']) & (df['substproj']>df['cdsp_prev_stop'])
df['cut_distance_stop_point'][update] = df['substproj']

[JabunchWork]

Thanks so much James (Jim) Allday Bunch, JABunch Transportation Consulting 411 Penwood Road, Silver Spring Maryland, 20901 240-271-3534 ***@***.***

…

On Thu, Aug 14, 2025, 12:04 PM Lilah Rosenfield ***@***.***> wrote: *lrosenfield-uta* left a comment (Bondify/gtfs_functions#49) <#49 (comment)> After a little bit of work, I think I have a functional algorithm for fixing this. The problem is that the segments tool uses the shapely project function, which will always find the point along the line *nearest* to each stop. However, if a line loops on itself, there will always be one point along the line that's closer both times the loop passes the stop, so we end up with at least one stop out of sequence that either has to be dropped or fixed. It's pretty easy to *detect* when this is occurring, but harder to fix it. solution I've found it recursing *out* from wherever a segment discontinuity is detected, incrementing by one in both directions until you find a fix. I've tested this on UTA's latest GTFS feed, as well as a tester feed I built custom, and it works in both cases, but there may be some corner cases in other feeds where it still causes problems. For that reason, I'd recommend issuing a warning when loops are detected, but otherwise putting this code behind an optional flag so that users can hammer it with their feed and let us know if it's causing more problems than it fixes. I've included the snippet because I don't know when I'll have time to fully integrate into the main tool and upstream: First, groupby route patterns, then apply the following function chain to each route pattern dataframe def perform_secondary_parse(df: pd.DataFrame): df['substring'] = np.nan df.loc[df.index[1, 'cut_distance_stop_point]=0 # Technically you can do this before GroupBy, but it's nice to maintain parity df.loc[df.index[-1], 'cut_distance_stop_point'] = 1 df = diff_and_reproject(df, 1) def diff_and_reproject(df: pd.DataFrame, offset, recurse = True): df = df.sort_index(ascending=True) df['cdsp_prev_stop'] = df['cut_distance_stop_point'].shift(offset).fillna(0) df['cdsp_next_stop'] = df['cut_distance_stop_point'].shift(-offset).fillna(1) df['prev_wrong'] = (df['cdsp_prev_stop'] > df['cut_distance_stop_point']) df['next_wrong'] = (df['cdsp_next_stop'] < df['cut_distance_stop_point']) #df['cdsp_next_ordered'] if df['next_wrong'].any() or df['prev_wrong'].any(): df['prev_cdsp']=df['cut_distance_stop_point'] fix_pattern(df) tester = df['prev_cdsp'] != df['cut_distance_stop_point'] if tester.any(): print("Made Corrections, re-setting to offset 1") offset = 1 else: print("no corrections made, increasing offset") offset = offset + 1 if recurse: df = diff_and_reproject(df, offset) def fix_pattern(df: pd.DataFrame): print(f'Fixing pattern {df['shape_id'].iloc[1]}') df['substring'] = df[df['next_wrong'] | df['prev_wrong']].apply( substring_general, axis=1) try: df['substproj'] = df['substring'].project( df['geometry_stop'], normalized = True) except BaseException as e: print(df['substring']) raise e df['substproj'] = ( df['substproj']* (df['cdsp_next_stop']-df['cdsp_prev_stop'])+ df['cdsp_prev_stop']) update = (df['substproj']<df['cdsp_next_stop']) & (df['substproj']>df['cdsp_prev_stop']) df['cut_distance_stop_point'][update] = df['substproj'] — Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATM2Z7BTMY3ITVZRFOBUBF33NSXRLAVCNFSM6AAAAACD5GBVL2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCOBZGAYDCMBUHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>