Muni Metro Meltdown analysis

Muni Metro Meltdown Analysis

This analysis of the disaster of Muni Metro rail operations was made by Rescue Muni member Richard Mlynarik in late September 1998, expanding on a "usenet" computer network discussion message (briefly cited here) which identified some problems contributing to the August Metro Meltdown.

Note also detailed comments from operators and riders on their experiences.

(Unlike Booz-Allen and Hamilton, who have have been paid over $30 million, and counting, in order to bring Metro operation to its knees, Richard provides this analysis free of charge.)

Reading between the lines, I have some guesses about what might be behind the current MUNI train control problem:

Failure to define the requirements. [...]
No organized test plan. [...]
Design Malfunction. [...]
Inadequate Training. [...]

While I agree with most of these points, there are three things missing from this finger-pointing:

Most fundamental is the question of why Muni was installing a multi-million dollar, newish-technology, moving-block signalling system in a small-scale, medium-traffic, medium-headway railway system in the first place.

The second is the basic engineering management failure to allow for failure and arrange for fallbacks: Muni not only unwisely undertook a whole slew of poorly-planned, capital-intensive, poorly-justified, poorly-procured and poorly-managed projects simultaneously, but made them all almost complete interdependent. The inevitable failure(s) of any element of the capital program lead to system-wide meltdown with no chance for graceful degradation to Plan B.

The third is the question of why an engineering fiasco continues to be inflicted upon the Muni customers and taxpayers and residents of San Francisco when decommissioning the system would increase the level of public transportation service in the city.

For the first part, the delivery of rail service in Muni's Market Street subway has been constrained by the following (in rough order):

Unreliability of equipment:
- Primarily, Muni's inability to provide anywhere near the number of rail vehicles required to provide advertised and timetabled service to customers
- Secondarily, the effects of rail vehicle failure while in service.
Lack of line supervision; any delay on any part of the rail system has been unmonitored and uncorrected, leading to huge service gaps which individual train operators are unable to unilaterally act to fill.
Unreliability of operator staffing Muni hasn't been employing enough train operators to provide the timetabled service, nor rigorously ensuring that those on the payroll work when customer demand demands it.
Traffic congestion at the terminal station. Trains on circa two-minute headways converging and terminating at a single pair of terminal platforms and reversing back via a single scissors crossover (see this news posting for a longer description) can lead to contention for paths across the cross-over and backups for incoming trains.
Counter-productive work rules. Until recently it was the case that a two-car train required two operators, one of whom spent half his or her time with no responsibilities other than fare collection. It is still the case that reassignment of a vehicle and its operator from its scheduled run to any other service (i.e. reallocating service to where they are most needed because of delays or loadings) requires a small premium to be paid to the operator.

Note that, due to an inability to provide trains to provide service, line capacity in the Market Street subway has never been an issue, and will not be an issue for many years, if ever. The new signalling system, advertised as expanding the line capacity of the Market Street subway by reducing headways between trains, is simply unnecessary.

Note also that line capacity measured in trains per hour is a quite different matter from what most effects customers of the system, which is reliable headways on each of the lines combined with adequate passenger capacity measured in vehicles per hour.

The moving-block signalling system and automatic train control is not only overkill for a small-scale operation with few trains on undemanding headways, but it completely fails to address the real problems.

Now to the second matter, that of the unnecessary inter-connectedness of capital projects.

Muni has recently spent over $700 million on capital projects of extraordinarily dubious efficacy.

Discussed above is the ATCS signalling system. It does not solve any operational problem, was budgeted below $30 million, was contracted to Alcatel at around $50 million, and presently represents a waste of nearly $80 million (including nearly $30 million in "facilititation" fees paid to the Booz-Hamilton consultancy.) Functioning of the ATCS system is entirely dependent upon installation of on-board equipment on each of the vehicles which will use it. Muni "planned" that it would have replaced its existing fleet of unreliable and aging Boeing LRV streetcars with a new fleet of vehicles before ATCS was introduced, and so would not need to require backward compatibility of ATCS with existing cars. So ATCS functioning at all depended upon successful delivery of a complete fleet of...
New streetcars. Muni specified a custom, globally-unique design and was only able to attract a single bid, from Breda Ferroviarie, to build it. The resulting cars are grossly over-weight, are grossly over-cost by world metrics (now over $3 million per vehicle!) and have proven extremely unreliable. It is fashionable within Muni and within San Francisco politics to lay all blame for the complete disintegration of the rail system at the feet of the (admittedly) "obsolete", "unmaintainable" "aging" and "mis-designed" Boeing LRV cars; what is conveniently not mentioned is that the fleet availability of the Breda LRV2's is only "approaching" that of the much-maligned Boeings, according to testimony from Muni's own director. It is quite extraordinary that such huge sums of money could be spent on such not addressing the problem.
When it became apparent that the years-late ATCS system would be placed in operation years before the more-years-late fleet replacement program, it was decided to outfit a portion of the Boeing LRV fleet with ATCS equipment. The present-day result is a fleet of 52 ATCS-compliant Bredas, 58 ATCS-compliant Boeings, and 20 non-ATCS Boeings. (Note that, in another spectacular example of mis-planning, Breda and Boeing cars cannot couple together to form trains.) This is down from a peak fleet of nearly 130 Boeings. The formerly-scheduled peak hour demand was for 99 cars, more recently reduced (though un-announced service cutbacks) to 91. Before ATCS introduction, Muni was consistently fielding fewer than 65 vehicles; this figure has in fact dropped since ATCS introduction. Nobody appears to bat an eye when it is reported that Muni is running far less than 70% of its advertised service.
Muni Metro Turnback. The addition of this track, which forms a small subterranean yard beyond the former end of the system, is the only part of Muni's planning which made operational sense.
By adding a new track (combining storage and reversing functions) and adding new cross-overs, this facility allows for Muni to increase the predictability of service in the subway, to add some redundancy to one of the more problematic of the multiple single points of system-wide failure (the scissors crossing at Embarcadero) and to add some operational flexibility (by providing locations to store reserve or disabled vehicles.)

Unfortunately, and professionally inexcusably, operation of the MMT was made entire dependent upon ATCS, so this useful, on-time and under-budget project was held hostage to a wasteful and unnecessary signalling system which in turn depended upon a wasteful and poorly-procured complete vehicle fleet replacement program.
Muni Metro Extension. This line serves no transportation use, but came about for a combination of reasons.
The first reason is that an extraordinarily large property development (Mission Bay, owned by Catellus Development, bankrolled by the City of San Francisco) is about to get under way once again. A gift of a $50+ million dollar rail line (the eventual cost will be closer to $200 million as MMX is extended further into Mission Bay under the guise of Muni's "Third Street Light Rail Line") to a developer when cheaper, faster bus service could have provided better service is about par for transportation politics in San Francisco.

To add insult to injury, the design of the development was turned on its after MMX had been planned and nearly constructed, and so the present terminal station of the constructed system will be isolated and useless when the Third Street project starts extending further along a new and different alignment.

To add a final slap in the face, the entire MMX alignment will be largely redundant if the planned Third-Stockton Street subway -- a further $450 million gift to Mission Bay masquerading as transportation to Chinatown -- is built to bypass it in a decade or so.

The reason second is that Muni management is itching -- as the ATCS procurement fiasco amply demonstrates -- to get its hands on the wads of money and transit-professional prestige which appears to come with rail extensions in this country. Building MMX gets them a toe-hold in the federal New Rail Starts pork-barrel stakes. Improving the chronic reliability problems of the existing rail system, improving the chronic reliability problems of the much larger and under-capitalized bus and trolley-bus system, planning for and making needed service changes to respond to evolving demand, advocating for exclusive vehicle rights of way on existing streets, improving efficiency and safety, providing a more pleasant experience for riders and so forth and so on are extremely unglamorous compared to planning and building and attending the opening of spanky new, multi-hundred-million-dollar rail extensions... even when, or especially when, the same transportation need could have been filled less than one quarter of the cost by significantly improving bus service (trolley bus electrification, exclusive right of way, signal preemption, low-floor vehicles.)

Note that operation of the MMX was dependent upon completion of the MMT (through which it runs) and hence on ATCS and Breda procurement.

The overall third question is: "Why is this fiasco is being inflicted on San Francisco?"

Inception of the ATCS system was an unmitigated disaster. Despite over three years of evening and weekend closures of the subway to revenue service, borne by Muni's riders through unreliable and inadequate "replacement" bus shuttle service, the ATCS system completely failed. Trains were delayed for hours. Weeks later [now months later], the problems are not fully resolved.

Rather than reconsider whether operation of this expensive boondoggle -- which solves no pressing or even medium-term problem of the system -- should be suspended, Muni's management, out of some combination of political directive and professional embarrassment, has persisted in operation of the system. Unfortunately, this has happened to the direct cost of the customers of the Muni system.

In order to improve "reliability" of ATCS, the 20 available non-ATCS Boeing cars have been removed from service. Muni's already broken-down, under-sized, chronically unreliable streetcar fleet was thus immediately reduced by 15%.
In order to prove some sort of political point, the MMX line -- which carries only around 3000 riders per day, hardly enough to justify a bus line -- is being served by through service from what was Muni's most heavily-used and what is now Muni's most chronically-unreliable rail line: the N-Judah. Since the additional round-trip made by N trains from Embarcadero to the MMX terminus on King Street and back again takes approximately an additional 20 minutes on top of the former N line's approximately 80-minute schedule, this represents a 20% decrease in the level of service along the entire N line, and approximately a 5% decrease in the level of service in the congested Embarcadero-Civic Center section of the rail system.
Partly in order to appear to be using the ATCS "investment", and partly because chronic and partly-ATCS-induced vehicle shortages offer no alternative, Muni has cut down the number of multi-car trains in the subway and dramatically increased the number of uncoupled single cars which are run. This enables its management to grand-stand about how many "trains per hour" are being run in the new regime without mentioning either that overall system passenger-carrying capacity has been grossly reduced or that those trains are still not dispatched in any rational, demand-reflecting manner. Lots of "trains per hour" does one little good if the headways on a particular line are not maintained and if, when a car for the desired line finally appears, it is so crowded that boarding is physically impossible.
Presumably in order to justify the MMT, no trains are being turned at the Embarcadero scissors crossing. (In a rational world one would have expected the MMT and the existing crossover to function together as a system to improve overall system throughput.) Running trains out through Embarcadero, into MMT and back is an operation that requires from three to nine minutes more than reversing at the platform. This represents approximately a further 5% degradation in overall revenue service provision.

So, the result is: Muni planning started a large number of expensive projects. The cost-effectiveness and indeed the usefulness at any price, of a number of those projects is extraordinarily poor.

Muni planning designed those projects so that they success of any one of them depends on most of the others.

ATCS: depended on Breda
MMT: depended on ATCS
MMX: depended on MMT
MMX: depended on ATCS

There were no contingency plans for what to do if any component failed; say, late Breda delivery or non-functioning ATCS. Individual failures are seldom predictable in any engineering undertaking, but the possibility must always be allowed for. Failure to do so was an engineering management and agency management disaster of the first rank, and no amount of finger-pointing at other agencies or at contractors or at past administrations can obscure this fact.

The result of this lack of fallback is a $700 million "investment" which has to be "made" to "work" because it has become a matter of all or nothing.

The result of that is that service on the street to actual tax-paying, fare-paying, San Francisco residents, workers and visitors has gone to hell so boasts can be made about increasing the number of "trains per hour" and photos can be taken at ceremonies inaugurating service on new rail lines.

Another result is that the appalling state of the much larger and much more patronized remainder of the Muni system -- the diesel and electric buses lines -- has been been further swept under the carpet.

Muni Metro rail service -- even in its present disastrous and reduced state -- is in many ways superior to that on many bus lines, and is certainly receiving a disproportionate amount of attention, and will continue to receive a disproportionate amount of investment.