Columbia Accident Investigation Board Public Hearing Thursday, June 12, 2003 9:00 a.m. National Transportation Safety Board Conference Center 429 L' Enfant Plaza, SW Washington, D.C. ADMIRAL HAROLD GEHMAN: Good morning. This public hearing of the Columbia Accident Investigation Board is in session. I'd like first of all to thank Administrator – Chairman Ellen Engleman for allowing the Columbia Accident Investigation Board to use the NTSB conference and briefing facilities. They're magnificent, and we're very appreciative of them. Today, we're going to review issues having to do with resources and management, and we have two panels of two very distinguished experts who are going to help us understand that. The first panel is made up of Mr. Allen Li and Marcia Smith. Allen Li is the Director of Acquisition Sourcing Management at the General Accounting Office. He is responsible in his position for several accounts, which include NASA as well as several Department of Defense accounts, like tactical aircraft. Previous to this, his duties included such things as energy and science, nuclear safety and the Department of Energy management issues, which turns out to have been something that we looked at, also. So, we very much appreciate the richness of your background and you're willing to help us here. Mr. Li has been selected to the Senior Executive Service in the GAO, and is a senior member of the American Institute of Aeronautics and Astronauts. Marcia Smith is a senior level specialist in Aerospace and Telecommunication policy at the Congressional Research Service of the Library of Congress and, at that, of course, she serves as a policy analyst for all the members and all the committees of the Congress on matters concerning U.S. and foreign military and civilian space activities, and on telecommunications issues. Previously, she held the position of section head for Space and Defense Technologies, as well as Energy, Aerospace and Transportation Technologies in that division, which again bears directly on what we have looked at in this area. She is a Fellow of the American Institute of Aeronautics and Astronautics and the British Interplanetary Society, as well as the American Astronautical Society. Before we start, I would like to request that the two panel members affirm to the Board that the information you are providing to the board today will be accurate and complete to the best of your current knowledge and belief. MS. MARCIA SMITH: I do. MR. ALLEN LI: I affirm so. ADMIRAL GEHMAN: Thank you very much. I would ask the panel members to introduce themselves and add anything to their biographical sketch that I may have underplayed or gotten wrong, and if you would – first of all, I would like you both to introduce yourselves, and then we will get an opening statement. Marcia, you want to go first, please? MS. SMITH: Well, thanks for the very nice introduction that you gave me already. I would like to explain briefly what CRS is and does. CRS is a department of the Library of Congress but, unlike the rest of the library, which works for both the public and Congress, CRS works exclusively for the members and committees of Congress, providing them with objective, non-partisan research and analysis. We do not take positions on issues. We don't make recommendations. Our job is to help the members and their staffs sort out the issues, look at the options that they have available to them, and help them understand the pros and cons of those actions. So, we don't have opinions. People often ask me what my opinion is, but I'm afraid that only my teddy bear knows what my opinion is. Everybody else gets pros and cons, and I apologize for that if you were hoping for some opinions this morning. And I have been a policy analyst at CRS since 1975, except for one year from 1985 to '86 when I served as Executive Director of the National Commission on Space that developed a long-term, 50-year plan for the space program. ADMIRAL GEHMAN: Thank you very much. Mr. Li? MR. ALLEN LI: Thank you, Admiral. My name is Allen Li. I've been with GAO now for almost 23 years. Been working on NASA issues for over five years. Have had the opportunity during that time to look at a lot of the programs that NASA's had, and had the opportunity to work with their top management in that regard. The only thing I would like to add, similar to what Ms. Smith was talking about in terms of what CRS does, GAO does provide recommendations. When we do our particular reviews, we also are part of the legislative branch, and provide advice and information to the Congress. But, we do provide recommendations, as we do in different programs. If we see there are certain management issues that need to be brought to their attention, we do so. The statement that I will provide in a few minutes is largely based on a report that we had provided, called the ÒPerformance and Accountability SeriesÓ that we provide to the Congress every two years, and it's our snapshot of what is happening at the agency and what are some of the challenges that that agency faces. Thank you. ADMIRAL GEHMAN: Thank you very much, and I – we understand the caveats, but they don't – to me, that – they don't seem to be very inhibiting to what we need to get at, and we're sure that your report will be very, very useful, and we appreciate you – your willingness to help us with this investigation. Which one of you is ready to go first, and the floor is yours, and if it's all right with you, we'd like to be able to dialogue and ask questions as you go along, if that's all right. You may have been told about the Board's tradition, that we – our tradition is that the briefer never gets past the first few graphs, so let's – if that's all right with you, we'd like to ask the questions as the issue comes up, because it's both fresh in your minds and fresh in our minds. Thank you very much. The floor is yours. MS. SMITH: Absolutely. I've watched all of your hearings, so I'm familiar with your tradition, and I'm hoping that I've left the most interesting slides till last so that I can get through the first few. If the folks in the slide room could bring up my presentation? There we go, and we can go to the next slide. You asked that I speak to you today about the NASA and the space shuttle budgets over the past 10 years. I thought that it would help to first put the NASA budget into context because, of course, budgets have to do with setting priorities. And so, I think it's interesting and important to understand where NASA fits in the total federal budget. So, this shows you, for fiscal 2002, the last completed fiscal year, which I'm using as my benchmark for this 10-year look-back. This is how the funding was split up in the $2.2 trillion federal budget. Mandatory spending was 56 percent, discretionary spending, which includes NASA, was 36 percent, and the interest on the national debt was 8 percent. And you can see on the slide where NASA fits into the non-defense discretionary account, which is 19 percent of the total federal budget. Next slide, please. This shows how that 19 percent gets broken up, and how much of that ends up at NASA. The defense discretionary is on the right-hand side, and the non-defense discretionary is on the left-hand side. The agencies that are in the other category, by and large, were smaller in terms of dollar amounts than NASA, so these are sort of the largest of the various agencies that get funded in that account. NASA is part of the Veteran's Affairs, Housing, Urban Development independent agency's appropriations bill, and I'm sure that everyone on the panel is very familiar with the federal budget process, but it might help if I just gave a 30-second review of how budgets happen in Washington. ADMIRAL GEHMAN: Are you or Allen – are you going to talk about whether that 2 percent is going up or going down, or what's the historical trend there? MS. SMITH: I have some trend charts in here. ADMIRAL GEHMAN: Thank you. MS. SMITH: In Washington, the way budgets happen is that agencies develop budgets through internal processes. They're submitted to the Office of Management and Budget, which is part of the White House, and so they're looking at various agencies' requests within the total context of the federal budget. That comes to Congress usually in February of each year as the President's request to Congress. It is Congress – under the Constitution, it has the responsibility to decide how this money is going to be spent. Congress passes a budget resolution that sets the parameters within which the various Appropriations Committees have to decide how to spend the money. And these agencies are all divided up into 13 different Appropriations Committees on Capitol Hill, and NASA is part of the one that's on Veterans, HUD, the Environmental Protection Agency. It used to fund FEMA, the Federal Emergency Management Agency, although that's now been shifted into the Department of Homeland Security. And so, there are a number of different agencies in the appropriations bill that funds NASA. ADMIRAL GEHMAN: Now, if my understanding of the process is correct, if you take – if you take something like education, for example, that actually rolls up a whole lot of agencies and things like that into an education budget. MS. SMITH: This education, 7 percent, is what OMB shows in its tables as the amount dedicated to, I believe, it's the Department of Education. ADMIRAL GEHMAN: Okay. Well, let me rephrase that. In most of those other categories, you have up there a cabinet-level agency rolls up a whole lot of agencies and subordinate budgets into one submission. But, in the case of NASA, they don't – their budget – they don't have a cabinet officer, and they're not in a department. They're an independent agency. So, they're – so, there's a little difference there, isn't there? MS. SMITH: Well, you may have a department like the Department of Commerce, and within the Department of Commerce you have the National Oceanic and Atmospheric Administration, and you have the Bureau of Export Administration. So, you have different offices within a cabinet department. NASA is a stand-alone independent agency, like EPA is. ADMIRAL GEHMAN: That's what I was referring to. There's a step that NASA doesn't have to go through, in the sense that – take the FAA, for example. They submit a budget, but they submit a budget to a cabinet agency, which put – which does things with it, and – whereas NASA's being an independent agency doesn't have to go through that. MS. SMITH: NASA has an internal process through which the Administrator of NASA submits a budget request to the White House, whereas FAA would submit a budget to the Department of Transportation, and then it would go to the White House, correct. ADMIRAL GEHMAN: That's correct. Thank you. MS. SMITH: So, the Appropriations Committees look at how much money they've been allocated, and they divide it up amongst the agencies within their jurisdiction. A budget gets passed. It goes back to the agency. The agency looks at what Congress did with their budget. They then decide if they're going to try and make a little bit of change here and there, and they notify Congress of those changes through something called an operating plan. There are initial operating plans, intermediate operating plans, final operating plans. Congress also, after the initial appropriation, can pass supplemental appropriations. And so, throughout any given budget year, there are a number of steps that are going on that are deciding exactly what the final amount is that any agency is getting for any particular activity. UNIDENTIFIED MAN: Marcia, one more question. Are there cabinet agencies with smaller budgets that NASA? MS. SMITH: I don't know that off the top of my head. Do you know, Allan? MR. LI: No, I don't know. I wouldn't think so. I don't think so. UNIDENTIFIED MAN: There are a lot of agencies with smaller budgets that NASA, but not cabinet agencies. MS. SMITH: Next slide. So, this is the NASA funding trend line over the history of the agency. The top line number is in 2003 dollars, the bottom line number is in current year dollars, and the first big spike you see, of course, is the Apollo program funding, and the next spike you see around 1987 is the funding for the replacement orbiter after Challenger. So, those are the various trend lines. Next slide, please. For the past 10 years, fiscal '93 to fiscal 2002, this shows how much the President requested for the NASA budget, and it shows how much Congress appropriated for it, and what the change was. Many people are interested to see how NASA's budget as a whole has changed over these years, which have been very difficult years, as President Bush – the first President Bush, President Clinton and Congress all were striving to reduce the federal deficit. And agencies, including NASA, were working under very austere conditions. So, this is how the NASA budget fares overall. Next slide, please. For fiscal 2002 – again, that's the last fiscal year that's been completed – NASA's budget authority was $14.9 billion. And within NASA, these are the different activities that NASA is engaged in, so when you try to look at the shuttle budget, which I'm getting to, this is what the shuttle must compete with, if you will, in terms of resources, the resources that the head of NASA has to deal with when he's allocating them amongst the various activities. And you can see that the space shuttle was 23 percent of the NASA budget in fiscal 2002, which is the largest single percentage for any of these activities. ADMIRAL GEHMAN: And manned space flight is just under half. MS. SMITH: Human space flight is just under half. And there's been a lot of discussion about the replacement programs for the shuttle. Those are funded from a different account. That's in the Office of Aerospace Technology, the X-33 program, X-34, the Space Launch Initiative. So, they are not, if you will, in direct competition with the human space flight side of the bar here. They're being funded within that account. But, of course, within the total NASA budget, there are always competing demands upon the total amount that's available to the agency. Next slide, please. This shows just the shuttle budget, and I decided to use as my base year – my benchmark year for this as 1981, the year of the first shuttle flight. I thought it would be more useful to see the total trend line over that period of time rather than just the past 10 years initially. MR. STEVEN WALLACE: May I interrupt, Marcia? We – unless you're going to describe it later, could you perhaps describe a little bit of the division between the space station budget and the space shuttle budget idea? In other words –. ADMIRAL GEHMAN: – Go back one. MS. SMITH: Go back one slide, please. MR. WALLACE: Shuttle, you know, basically, how – are they funded entirely independently, since the programs are so intertwined and sort of rely on each other? MS. SMITH: They are very interdependent. That was not true historically, but it has been true at least through most of the 1990s as the primary purpose of the space shuttle is to assemble and service the space station program. So, they are very closely intertwined. You can see changes throughout the years in how NASA has been organized – NASA headquarters has been organized in terms of managing those programs and bringing them under the rubric of the Office of Space Flight, and how the Administrator for Space Flight has handled those programs and bringing them much more closely together. And as you'll see in some of the subsequent slides about space shuttle funding, some of the funding from the space shuttle program has moved over into the space station account as the space station has encountered funding difficulties. MR. WALLACE: There's sort of a continuing debate, I would assume, about who pays which parts of the bill on this between the programs? MS. SMITH: Well, in one sense. When NASA talks about the costs of the space shuttle – of the space station program, for example, they do not include the cost of the space shuttle flights. So, when you see budget estimates for space station, that it's going to cost 17.4 billion or 25 billion or whatever it is, that does not include the cost of the space shuttle flights, even though you obviously can't assemble and operate the station without the shuttle. So, in that sense, the space station total funding is not taking into account the amount of funding required for the shuttle launches. MR. WALLACE: So, if you say there's – the shuttle budget for 2002 is 3.3 billion, if we just – you might say that, what, three-quarters of that is more or less, or three-quarters of the program, or some percentage is in support directly of the space station. MS. SMITH: Yes, it is. Next slide. So again, this is the shuttle budget since the first flight, again seeing a spike there for the replacement orbiter. ADMIRAL GEHMAN: Let me ask a question about graphsmanship or chartsmanship. I understand this, and I appreciate you putting it both in current year dollars and in any year dollars. A lot of times, I've seen this same chart in which, instead of using 2003 dollars, they use 1981 dollars. And, of course, if you did that, the yellow line would show, in real-term spending, shuttle funding is going down. MS. SMITH: Well, you can choose whatever base you're – you wanted to choose. I put it into the current year dollars because I thought that that would be, you know, most helpful to you. But, one can calculate these numbers in a variety of different ways. ADMIRAL GEHMAN: That's right, okay. But, what I mean is, would you agree that, if the yellow year line were to be 19 – then-year dollars, 1981, then the yellow line would trend down? MS. SMITH: I haven't done the calculations, so I couldn't presume what it would show. ADMIRAL GEHMAN: Okay, all right. MS. SMITH: Next slide, please. This is getting back to the 10-year time frame that you were interested in, and it's important to notice that this is the President's request up here. This is the request to Congress, what Congress did with it, what happened to it after that in terms of operating plan changes or supplementals that were done to it. What this does not include is the other step in the process, which is going from NASA to the White House, to the Office of Management and Budget. It doesn't show changes that were made from the agency's request to the White House. They are also, obviously, a very important player in the whole budget ballet that goes on, the NASA number, the White House number, and the Congressional number. What comes to Congress is a White House number, and what happens prior to that process, the negotiations that go on between NASA and the White House, are very closely held, and I do not have data on the so-called pass-backs between OMB and NASA as they formulated these budgets. ADMIRAL GEHMAN: Excuse me, go ahead, Jim. DR. JOHN LOGSDON: First, a comment, Marcia. Neither do we. I think the observation that we should look at that chart is that Congress may re-shuffle, as you're gonna show in a while, the money within the shuttle budget slightly, but that Congress has not made major changes in what the president has requested for the shuttle, that the key decisions are what the White House requests. MS. SMITH: There were some substantial decisions in the early 1990s regarding the advanced solid-rocket motor program –. DR. LOGSDON: – Right, but in recent – in the more recent years –. MS. SMITH: – But in the more recent years, a lot of the changes, if there were changes, would have happened prior to the President's request coming to Congress. But, I don't know whether there were changes or not. ADMIRAL GEHMAN: Let me – would you help me with what the definition of Òfinal amountÓ is? Is that –? MS. SMITH: – That is what's in NASA's final operating plan. It's the amount at the end of the fiscal year as to how much actually ended up being allocated to the shuttle after all the puts and takes throughout the year. DR. LOGSDON: This is not the appropriation? ADMIRAL GEHMAN: No, no, this is how – what actually was spent, obligated. MS. SMITH: And the subsequent slides will show you the changes that were made to it, both by Congress and by NASA. DR. LOGSDON: So – but this –? MS. SMITH: – I have the other data, but I thought that I would be overwhelming you with too many budget charts, so I didn't put in all the tiny little numbers that showed you every step of the way. But, the – it was NASA that developed these numbers. They were developed in advance of the February 12th hearing, the joint Senate/House hearing, and these are NASA's numbers, and they do show a greater level of detail. So, if you need that, I can provide you with an estimate. ADMIRAL GEHMAN: No, I just want to make sure I understand that –. MS. SMITH: – This is a final amount that is a final final amount. It's not the Congressional appropriation. ADMIRAL GEHMAN: No, no, I understand, after all the puts and takes and pushes and actions. But, when I read this chart then, at the yellow line, I should assume, then, that NASA actually spent, at each year, less than the President's budget? MS. SMITH: They would have had the authority to spend less. This is budget authority. I don't believe it's outlay. ADMIRAL GEHMAN: Well, that's why I was asking what Òfinal amountÓ means, whether or not final amount – I got the impression that this was the final amount at the end of the year after – based on their operating budget. MS. SMITH: Of budget authority. ADMIRAL GEHMAN: Of budget authority, yes. That's pretty close to saying that this is what they actually spent. MS. SMITH: I do not know that these are outlay numbers. ADMIRAL GEHMAN: Okay, they may not be outlay numbers. MS. SMITH: But then, sometimes money can shift from one fiscal year to another fiscal year, so it would have been spent eventually. But, whether it was spent in this fiscal year, I don't know. I think this is budget authority. MR. LI: They have things like carryovers that, when they don't, even though you obligate the funds and we don't spend them, then they carry that. DR. DOUGLAS OSHEROFF: But, what's true is, from '93 through '98, or something like that, there is, in fact, a constant offset between those – not constant offset, but, obviously, the amount that – the final amount is always lower. So, you're – I mean, you can't carry that over from year to year. You could have a whole pile of money left over. MR. LI: And we had testified on that actually a few years ago, and some of the issues in what causes a carryover are things like, on the space station, when construction was – did not go as planned, and things were not put up in space and scheduled as they thought, then that expenditure would not have been made during that year, and that causes carryover. MS. SMITH: And I think the next slide is going to answer some of these questions, as well, because they go into the detail year-by-year as to what the puts and takes were as time applies. If I could have the next slide? And I emphasize this is NASA's explanation. This all comes from NASA data. This is not something that CRS derived itself. And I think I have five slides that show these. I don't know if you want me to go through all of them. In the interest of time, if you want, I can just show you a couple, because I think what you're interested in is mostly the trend as opposed to specific cuts or additions that were made. ADMIRAL GEHMAN: Right. Well, why don't you go through the first one, and then we'll see if we get the idea. We're slow learners, but we are solid learners in this. MS. SMITH: Well, you can see that, in the appropriations process, Congress cut $50 million. They cut that because NASA had terminated one of the upgrade projects, the electric auxiliary power unit. They also added 20 million for upgrades, they added 25 million for repairs to the vehicle assembly building at Kennedy Space Center. So, that's what happened in the appropriations process. Subsequent to that, NASA transferred $7.6 million to fund other agency requirements, and they cut 1.2 million for a rescission requirement. So, all of that would have happened after the normal appropriations process, bringing the shuttle budget to a net change of minus 13.8 million in that year. ADMIRAL GEHMAN: Once again, we have that top line up there, where the President's budget requested 3.283 billion, and then we have that final number over there again. MS. SMITH: Right. It's the final operating plan. ADMIRAL GEHMAN: It's from their operating plan? MS. SMITH: Final NASA operating plan. ADMIRAL GEHMAN: Right. So, their operating plan, again – once again, there were a lot of changes in between there, all kinds of puts and takes in between. MS. SMITH: These are the changes. ADMIRAL GEHMAN: Yeah, I know, but they're – okay. MS. SMITH: This is – these five bullets are what get you from the three – 3.8 to the 3270. That is –. ADMIRAL GEHMAN: – Some of it was done by Congress, some of it was done by NASA in the execution of their operating plan? They moved money –. MS. SMITH: – Right, and some was done by Congress because of NASA actions or at NASA requests, and that's the trend that you're gonna see throughout all of these changes, is that, you know, Congress is making changes, NASA was making changes. It – the only part we don't know is what changes the White House might have been making prior to the budget submittal. And so, for 2001, Congress cut 40 million at NASA's request for a Mars Initiative, and my recollection of that is that there was a joint project between the Human Space Flight part of NASA and the Space Science part of NASA on the Mars program, and the Human Space Flight part decided not to pursue the program, and they didn't want the Space Science side to take the hit – the budget hit, and so they moved the money over. So, this was cut for that reason, and NASA also cut 6.9 million because of a rescission. So that, again, takes you from the 3165.7 to the 3118.8. DR. OSHEROFF: These are really kind of changes that's a margin. I mean, there's, what, less than 1 percent of the total budget, right? MS. SMITH: Right, in these years. DR. LOGSDON: Marcia, what's a rescission requirement? MS. SMITH: A rescission – Congress can, in the actual appropriation bill or in a supplemental appropriation bill, take back money that they had originally appropriated. DR. LOGSDON: Tax. It's a tax. DR. OSHEROFF: But that's not agency-specific. It's throughout the government, isn't it? MS. SMITH: Very often – this is – I do not believe that this is a tax that various parts of an agency are sometimes required to pay. I know this happens a lot in DOD, that each program gets a certain tax amount to it. This, I believe, is in response to a Congressional rescission where Congress has said all the agencies are gonna take a .0065 percent reduction in order to balance whatever budget amounts they had available to them. DR. LOGSDON: Rescissions are also congressionally mandated? MS. SMITH: Yes, Congress can mandate rescissions. We just had a rescission in the fiscal '03 budget. There was a .0065 percent rescission across all the agencies except Defense. ADMIRAL GEHMAN: Okay. Would you – let's look at the next couple, please, 2000. MS. SMITH: Next slide. So, you can see these – the things on here that you might find of interest are, under 2000, the two bullets that are in italics do not affect the total amount available to the shuttle, but they do change how the money is being spent within the shuttle program. And the last one on there was $40 million that was identified for what was called the R2 mission. The STS-107 mission was actually undertaken because of Congress's interest in continuing the ability for scientists to have access to orbit during the phase of space station assembly. The original idea was that NASA would fly shuttle science missions, the space station would get ready, and the science would continue on the station. But, as the station schedule slipped, there was gonna be a long hiatus where scientists would not be able to conduct research. So, first they allocated money for the STS-107 mission, and then they subsequently said they wanted a second dedicated science flight and, for that, they specified $40 million. They called it R2, for Research Two. Now, in actuality, in 2000 was when NASA was looking at whether or not they had pushed too far on the shuttle budget. They had had the problems with STS-93, they had the McDonald Report, and NASA decided that they needed to put more money back into personnel. And so, this 40 million, as far as I know, was ultimately spent on plussing up the personnel accounts in the shuttle program, and on shuttle upgrades. And the R2 mission has been canceled. There is no R2 mission any more. ADMIRAL GEHMAN: It's interesting. When you look at that net change, which is essentially zero, but then you look at all these 25 and $40 million chunks of money moving around, it's kind of fascinating. Congress adds 25 million for upgrades, and then 26 million gets taken away by – for one thing or another, so you wonder about whatever happened to the upgrades. And then, they transferred 25 million for upgrades from operations, so that's not new money, that's just moving money from one account to another, and so you wonder what happened there. And then, Congress specifies how 40 million is gonna get spent –. MS. SMITH: – But did not add the 40 million. ADMIRAL GEHMAN: But didn't add any money, but they specified how 40 million was gonna get spent, which restricts NASA's ability to use that. MS. SMITH: Except that they ended up using it for the shuttle, anyway, for upgrades and personnel. ADMIRAL GEHMAN: And they used it anyway. So, yeah – all right, thank you. Now, in '99, the 60 million, that is a pretty – that's a pretty healthy cut. MS. SMITH: Yes, and you can see that Congress cut 31 of that at NASA's request to fund other requirements that I think that were in the mission support category at the time. ADMIRAL GEHMAN: Fund other NASA requirements? MS. SMITH: Right. And NASA cut 32 million itself, transferring the money to the space station. ADMIRAL GEHMAN: To the space station. MS. SMITH: But, they also added back in two million that they had for space station, so it ended up a net of 30, basically. MAJOR GENERAL JOHN BARRY: Marcia, I know you're gonna talk a little bit on the remaining slides here, but since '94, when they combined space station and the shuttle in the Office of Space Flight, could you give us an indication of the dance of monies that have been moving between space station and the shuttle? Can you characterize – ? MS. SMITH: – Well, according to this NASA table from which this is all extracted, between – in that time frame, there's 330 million that was transferred from shuttle to station. MAJOR GENERAL BARRY: Over what years? MS. SMITH: From '96 to 2000, I think. ADMIRAL GEHMAN: And no flow the other direction? MS. SMITH: Correct. MAJOR GENERAL BARRY: And would it be safe to characterize also that the increase in the space station has resulted in funding, but the space shuttle has been held back to an inflation level growth pattern? We have one character reference that made reference to that, and I just want to see if you share the same point of view. MS. SMITH: The goal for the shuttle program, especially since the initiation of the space flight operations contract, was to hold the line on shuttle spending while not compromising safety. That was the goal. MAJOR GENERAL BARRY: And that goal was stated when? MS. SMITH: Well, it's been a goal for the shuttle program through the 1990s. And when they signed the Space Flight Operations Contract, SFOC, that – it was clear that that was the point that they were trying to make by getting more contractor workforce involved in the program. So, in essence, if you see a level budget for the shuttle, that is exactly what they were shooting for, as long as it did not compromise safety. They were always careful about this. And during these years when the shuttle budget was very constrained, there was a lot of discussion about the fact that the budget was very constrained. There were hearings about it almost every year. I mean, there are hearings on the NASA budget every year. But, in addition to that, there were separate hearings on the shuttle program and on shuttle safety almost every year during this 10-year time frame. There were reports from the Aerospace Safety Advisory Panel. The reports – you know, the McDonald Report that came out, GAO reports. So, it was well known that there were stresses on the shuttle budget during this period of time. ADMIRAL GEHMAN: We better let Marcia move on. MS. SMITH: Okay, why don't you go to the next slide? These get shorter and shorter. Why don't we just go to the next one? Here, you start seeing bigger cuts. Why don't you go to the last one? ADMIRAL GEHMAN: Wait a minute, go back one. We want to see where those cuts are coming from. MS. SMITH: In '96, Congress cut 53 million for the Yellow Creek facility. This was a facility that was being built to support the Advanced Solid Rocket Motor program, the ASRM. The ASRM program was canceled, which you see on the next two slides, and there was interest, when they first canceled ASRM, in transferring some of the other SRB work to Yellow Creek. So, they didn't cancel the facility at the same time they canceled the rest of the program. But, when they got to '96, they did finally cancel that facility, as well. So, that's all part of the decision to terminate the ASRM program. ADMIRAL GEHMAN: In the – '96, there was a transfer to – of shuttle – from shuttle to the space station. MS. SMITH: That's right. ADMIRAL GEHMAN: In '95, there – that's just a cut. MS. SMITH: That was just an across the – that was a cut, and you'll see a note on there. My colleagues, Dan Morgan and Amanda Jacobs, went back through appropriations reports to try and look at all of these numbers, and we have our own report about what the House did and what the Senate did, and what the final appropriation was. And we couldn't find this one in the appropriations reports. It's not that we doubt that this is what happened, because the NASA people are very meticulous about these things. But, we just put a note on there that we couldn't find that. We did find 141 million in cuts in the appropriations conference report. ADMIRAL GEHMAN: Okay, thank you. MS. SMITH: Next slide. And here, you see the termination of ASRM in '94. In '93, Congress had actually tried saving ASRM. The last year of the first Bush Administration had decided to terminate the program, but Congress thought that it still should proceed. And so, in '93, they were saving ASRM, but by '94 they were convinced that it was no longer necessary, considering the slips to the space station assembly schedule. And part of the reason for ASRM was to increase safety, and they were feeling comfortable that the Re-Designed Solid Rocket Motor, the RSRM, had demonstrated sufficient safety that they didn't need to go to the ASRM for that. And that is my last slide –. ADMIRAL GEHMAN: – Thank you. MS. SMITH: I think the overall message of all this is that, you know, people want to know who cut what, and the answer is we know that Congress made cuts and NASA made cuts, and we don't know whether or not the White House made cuts. And so, there have – it's been a give and take, and that's what the budget process is, by and large. And one can argue that, and there's certainly people that argue that the shuttle budget has been cut too deeply, and that there may have been negative impacts on the shuttle program because of those budget cuts. It's very difficult, I think, to, perhaps, tie this into a specific situation like the Columbia tragedy and trying to say that, because of budget cuts, the Columbia tragedy happened. I know that you still remain flexible as to what the actual cause of the Columbia tragedy was. You haven't come out and made a statement but, obviously, a lot of people are thinking that it was foam hitting the orbiter. And if NASA did not fully appreciate the dangers associated with foam hitting the orbiter, it's not clear that an increased budget would have helped that situation. So, everyone's, you know, looking to budget issues, trying to determine whether or not budget cuts are responsible, and it may well be that you'll conclude, as others have before you, that the shuttle budget was cut too far, but it will be interesting to see whether or not you can tie that directly to this catastrophe. ADMIRAL GEHMAN: One last question, then we'll let Allen get on stage here. But, if you look at '93, '94, '95, my – the big numbers were back in the mid-90s. If you look at '98, '99, 2000, the – usually the cuts or the transfers are little numbers, 13 million, one million, 40 million. But, when you get up to the 400 millions and the 180 millions and the – things like that, 190 millions, those are back in the '95, '96, '97. So, my – what I take away is that the really big transfers or cuts were in the late 90s and not so much recently. MS. SMITH: Yes. They were back when the focus was on cutting the deficit, and all the federal agencies, including NASA, were suffering cutbacks in order to reduce the deficit. ADMIRAL GEHMAN: And also, there were – this was the first couple of years of the SFOC contract. MS. SMITH: No, it was '96. ADMIRAL GEHMAN: That's what I said, '95, '96, '97, '98, and so there were perceived savings there. Whether they were real or not, we don't – we'll get to. Okay. MS. SMITH: And NASA had metrics to show that the restraints on shuttle funding were not affecting safety. They had charts showing that there were fewer in-flight anomalies despite the cutbacks in personnel. ADMIRAL GEHMAN: Last question, Mr. Wallace. MR. WALLACE: Well, just as sort of a process question, we've heard a lot in that – the history of the program about various compromises in the design of the shuttle, that – sort of part of making the whole thing go, you know, military requirements or cross-range requirements or payload base size requirements, and things like that. And I'm curious. So, we hear about compromises which may have resulted in designing a vehicle which was not optimized for the mission to end up flying, or perhaps not even optimized for safety, and I'm wondering – I have sort of a two part question – is are those compromises typically before the submission comes to Congress in the White House/OMB/NASA part of the process, or are they after the submission to Congress? That's the first part of my question, and the second part is, does Congress have a process to sort of technically vet these things, experts they rely on to, sort of, understand the technical consequences of these decisions? MS. SMITH: In terms of the original design of the shuttle and decisions on how much money was going to be spent on the shuttle, and how they finally got down to that $5.15 billion for research and development, that was all done before the President approved the program. So, that was what started the program, which then led to the annual budget request to fund it. So, those negotiations as to how big it was gonna be and whether it was gonna have – be fully reusable or partially reusable, those were discussions that happened prior to President Nixon's 1972 approval of the program. And Congress does have a mechanism to vet any agency request. They have a hearing process where they will call upon a variety of witnesses. GAO often does studies for Congress and testifies about them. They can always come to CRS so – but we don't have the investigative powers that GAO has. And they rely on outside experts in industry, they – the Aerospace Safety Advisory Panel testifies to Congress, so they have a number of people that they can rely on in formal settings, and they also can discuss with people in informal settings if they're concerned. ADMIRAL GEHMAN: Thank you very much, Ms. Smith, and Mr. Li? MR. LI: Before I start my summary of my presentation, I wanted to address one of those issues, because I think it's important. A few years ago, when I testified before the Senate, we were talking about the issue of upgrades, and this is an issue that I think permeates some of the discussion here. It's very important to know what your end state and goal is before you make whatever decisions you have to make. And the thing that, Mr. Wallace, I wanted to bring to your attention, because I know you're familiar with the commercial side of the aviation industry and not so much on the space side. The analogy that I used that I thought was effective in conveying my feelings was I was talking about my 1986 Volvo, and I had to make a decision – it's like making a decision whether or not you want to replace the – what components, are there some things that are less expensive? Is the water pump worth fixing this year, or do I want to do a full-blown ring change for the engine? That – my decision is based upon whether I'm gonna keep it for one year or five years. And the issue that I would like to raise to the Board is that I believe that, at the time that Ms. Smith was talking about those cuts and whatever, that was never well-focused in terms of how long are we going to keep the shuttle. And I think that that uncertainty has a lot to do with, well, how much money should we spend? It's a lot easier to make an argument to OMB or to somebody else and say, ÒLook, I'm gonna keep this vehicle for X amount of time and, therefore, I need to make this investment.Ó When that changes from year to year – and luckily now, I think we have – or, at least before the tragedy – we had a good idea that it was going to be now 2020, but at least they put the line in the sand. They drew it. We knew what we had to do and, therefore, they came up with the – a sort of slight extension program. But, before that time, that particular line in the sand had not been drawn. So, I just wanted to raise that to your attention, that's a consideration that they should have. Okay, let me move onto my prepared statement. Chairman Gehman and members of the Columbia Accident Investigation Board, thank you for asking GAO to highlight its work at NASA. We recognize the Board's daunting task of not only establishing the cause of the Columbia accident, but also in understanding the agency's environment in which management decisions are made. We believe our body of work at NASA can help the Board in this area. In January of this year, we identified four challenges facing NASA, namely one; strengthening strategic human capital management; two, correcting weaknesses in contract management; three, controlling the costs of the international space station; and four, reducing launch costs. I will highlight these four challenges, and then provide an observation on the effectiveness of knowledge sharing at NASA, an issue I understand is of high interest to the Board. The first challenge is for NASA to strengthen strategic human capital management. It may sound like a clichŽ, but an agency's most important asset is its people. NASA, like many federal agencies, faces ongoing difficulty in attracting and retaining a highly skilled workforce. This was no more evident than when we reviewed the shuttle workforce. In January of 2001, we reported that the shuttle workforce had declined significantly to the point of reducing NASA's ability to safely support the shuttle program. Recognizing the need to revitalize the shuttle program's workforce, NASA discontinued downsizing plans and initiated efforts to hire new staff. As we reported in January of 2003, this problem has yet to be fully resolved. Staffing shortages in many key skill areas of the shuttle program, such as electrical engineering, remain a problem despite the new hires. New shortage areas have recently emerged, such as subsystems engineering and flight software engineering. NASA believes that similar workforce problems affect the entire agency. Currently, the average age of NASA's workforce is over 45, and 15 percent of NASA's science and engineering employees are eligible to retire. Looking down the road, 25 percent will be eligible to retire in five years. At the same time, the agency is finding it difficult to hire people with science, engineering and information technology skills. Several bills have been introduced in this Congress to provide NASA with requested flexibilities for attracting, retaining and developing its skilled workforce. NASA's second challenge is to correct weaknesses in contract management. Much of NASA's success depends on the success of its contractors. I'm sure you've heard that. These contractors receive more than 85 percent of NASA's funding in fiscal year '02. However, this reliance does not come without problems. ADMIRAL GEHMAN: Excuse me for interrupting. Could we ask – I think if we want to ask a workforce question, this is probably – you're about to go onto contracting. MR. LI: Yes. Yeah, right. ADMIRAL GEHMAN: In your statement, you said that, in your January 2001 report, that the report – and I've read all these – the report states that NASA's workforce has declined to the point of reducing NASA's ability to safely support the shuttle program. MR. LI: Yes, and I wanted – I'm glad you mentioned that, and the issue and the point that we were making was not that it had declined to the point where it was unsafe to fly. It was within the context of what was happening in the near future, that increased flight rates were about to happen to support the space station. And what our concern was was that, if this trend of downsizing were to continue, and with the increase in the number of shuttle flights that was to happen, then we saw some problems. But, you're absolutely right, Mr. Chairman. We were not saying that it was unsafe to fly. ADMIRAL GEHMAN: Right, but it was declining? MR. LI: That's correct. ADMIRAL GEHMAN: In the January of 2003 report, you say that the challenges have not been mitigated. MR. LI: Yes, and by that, I'm saying that all the new hires, in terms of having a critical skill that had, like, a backup, that that had not been fixed, that people are still very thin as far as expertise. ADMIRAL GEHMAN: Right. Good. Thank you very much. Now – and we are also reviewing those things, and will come to our own conclusion on that, but we thank you for that. One of the areas that we are focused on – and you listed a whole number of areas, training programs not attended because the people are working to hard, leave – not – annual leave not taken, the amount of overtime, advanced degrees not pursued because there's no time to give sabbaticals, and – I mean, all kind. One of the areas that we have been looking at is the area of – and maybe I'm – this kind of balances toward your next section of contracting, but one of the areas that I'm concerned is the area of cases in which NASA no longer manages subsystems. In other words, the subsystem manager is a contractor. Did you – do you have any insights into that, or maybe – and I don't – it may be that – it may be not so much a workforce problem as it is a – the level at which the line between contractor and government oversight is maybe moving up and down. And can you –? MR. LI: – I think that, you know, rather than just talking specifically about shuttle, I think you can extrapolate that to the entire government. The issue is that there is always pressure to reduce the number of government employees associated with any program. That said, the terminology in NASA that is often used to describe the situation that you were describing, Admiral, is one of oversight and insight. And that really came – it was really clear when I was looking at the X-33 program. In the X-33 program, it was one of those instances where they decided a very minimal government participation was going to occur. It was primarily one in which the contractor was developing the X-33 demonstrator. The feeling was that the government insight, which is working alongside with the contractor, was going to provide them as good information as doing oversight, which is checking things, that they had a certain milestone, certain product delivery that they had to provide, and then they would check that. There is a lot of controversy associated with that insight and oversight. As a matter of fact, when we brought that issue up, I believe that when they restructured after X-33 and they did the strategic launch initiative, there was additional government oversight associated with a lot of those contents. ADMIRAL GEHMAN: Yes, Dr. Osheroff? DR. OSHEROFF: Well, I'd just like to bring up one specific example. The constant shedding of foam from the external tank, whether it caused the accident or not, is irrelevant. It cost NASA a tremendous amount of money in maintenance costs for the orbiters. And yet, it seemed like rather little was being done to understand the properties of the foam and why it shed. Now, the question is, whose responsibility was it to actually do this work? MR. LI: I'm afraid I can't answer that particular – your question, sir. I believe that, had they known that that was a problem, I think that NASA would have stepped up to the plate and said we need to do something about this. DR. OSHEROFF: I beg your pardon. I think that they knew that it was costing them a lot of money. That's not an issue. I think that was very clear. And yet, my statement still stands. ADMIRAL GEHMAN: Let me rephrase the question, or let me add my question onto Dr. Osheroff's question. In the manpower review that you did, did you analyze, or did you consider the issue of whether or not, in a unique technical enterprise like human space flight, which nobody else does this, whether or not a rich and robust U.S. government technology workforce is necessary for a whole number of reasons, including career progression, including for the government to exercise property fiduciary contract management, and if – I could name seven or eight reasons why a robust, rich technology workforce should be paid for by the government. One of the reasons is to be able to have the technical competence to answer Dr. Osheroff's question, but I can think of seven or eight other reasons. To kind of boil my question down to the issue of did you consider, or did you look at, whether or not it should be just a cost of doing business to fund a rich technological workforce as just a cost of doing business? MR. LI: No, we did not, but I understand. I think it's a matter of philosophy. When I was talking to some of the engineers, and this is an important issue that, I think, in looking at the future, when I talked to engineers at NASA, they say, ÒHey, I came to NASA to design aircraft, to design spaceships. I did not come to be a project manager/contract overseer,Ó and I think that gets to your point. ADMIRAL GEHMAN: Well, I think that – what we're concerned about – and I don't want to speak for Dr. Osheroff, but we're concerned about – there are a number of reasons why the government should have government technologists and government expertise. There's other reasons – there's other places where the government doesn't need to do this, where if it's duplicated in industry or academia, fine, go hire them. But, we are wondering whether or not, if you're gonna send men into space and nobody else does this, whether or not the government should just have to bear this cost as part of doing business. MR. LI: I think there's some merit to that philosophy. One side of me, in terms of being – having had the engineering background, tells me yes, in order for me to be able to oversee something, I need to understand that process, and I need to be able to understand whether or not somebody is doing something wrong. That is correct. But, I also am reminded of a saying, and when I'm asked the question of my own staff at GAO, when people are saying, ÒWell, how can you go ahead and review these programs when you're not engineers?Ó And I'm reminded of a saying that says, ÒYou don't have to be a chicken to smell a rotten egg.Ó DR. OSHEROFF: I think the full issue is very complicated. Let me make one more point if I can, and that is that the people that produce the external tanks, that apply the foam, had rather little to gain by investing in research to solve a problem which NASA was not complaining about. USA was repairing the tiles repeatedly and, presumably, they had every reason to do that, but it wasn't in their venue. The one organization that would profit by solving this problem was the parent organization. ADMIRAL GEHMAN: Dr. Ride? DR. SALLY RIDE: I wanted to ask a question related to the workforce, as well. In your 2001 report, as you said, you identified constraints on the workforce, that you didn't identify as safety of flight problems at the time, but as something that needed attention, and rather desperately. And I was wondering if you could comment on how you related that to the flight rate, and to the work that was foreseen to be coming in the next few years, because I think that that – you know, the level of people related to the level of anticipated work, I suspect, was a major issue. MR. LI: My recollection, Dr. Ride, was that, when the flight rate increases above the eight per year, that's when you – things are getting really dicey in terms of that workforce and how much they're going to be stretched. But, I believe it was in that general area between eight and 10, and there was talk at that point in time, as you perhaps recall, that at the peak of construction, they were planning to – almost a dozen flights were being planned out in the future. So, that was of concern, and I – to NASA's credit, they recognized that particular weakness and, as I said, they did stop their downsizing and start hiring again. DR. RIDE: Did you look at that same issue in your 2003 – January 2003 report, in relation to the projected flight rate? MR. LI: We updated – no, no, that part we did not. But, however, as you probably know, at that point in time, the decision had already been made to completely change the space station. When we did that original work in 2001, there was no talk about core complete, and things like that. Now, we're in the situation where the space station is truly not an apples and oranges type of thing. The – as you know, the original space station was one where we were supposed to have seven crew, and now we're talking about something much smaller and, as a result, the number of flights would probably be more in the four per year, four to five per year. DR. RIDE: I think it was set to be around six left to core complete. ADMIRAL GEHMAN: Dr. Logsdon? DR. JOHN LOGSDON: Allen, in your testimony, you say the agency is finding it difficult to hire people. Why? Have you done any reflection – I mean, is it not competitive with other federal high technology agencies, or is it not competitive with private sector opportunity? MR. LI: I've had – I've had many conversations, actually, with the agency, and with the – NASA's chief Human Capital Officer, Vicky Novak, and they have some aggressive things that they are planning right now. The issue is one – and this is in their justification for the legislative relief that they're asking for, and has translated into those three bills that I mentioned. The issue is that, yes, there is difficulty throughout the country in terms of hiring science and engineering. The Aerospace Commission recently mentioned the same thing. So, NASA's not alone. That said, it's incumbent, and the responsible thing for the administrator of NASA, is to find ways in which his particular agency can weather this particular issue and, as a result, they have made those proposals. The types of flexibilities that they have asked for, Dr. Logsdon, include things like retention bonuses for the people that are there already, but in terms of recruiting, they are trying to go now at the – even the base level, at the – from the kindergarten on up, they're trying to enhance their participation in many programs such that there would be greater interest in NASA. ADMIRAL GEHMAN: I think we better move on. I actually have a couple more questions, but let's move on. I'll save them for later. MR. LI: Since 1990, we have identified NASA's contract management function as an area of high risk due to ineffective systems and processes for overseeing contract – contractor activities. I think that rings a bell there. Specifically, NASA has lacked accurate and reliable information on contract spending, and has placed little emphasis on end results, product performance and cost control. While NASA has addressed several of its acquisition-related weaknesses, key tasks remain, including completing the design and implementation of its planned financial management system. As the agency implements this system, it will need to ensure that its systems and processes provide the right data to oversee its program and contractors. Data must allow comparisons of actual costs to estimates, and provide an early warning of cost overruns or other related difficulties. NASA's third challenge is to control the costs of the international space station. We had a perfect example of that when Ms. Smith was talking about some of those changes. As the Board knows, the estimated cost of the station has mushroomed, and expected completion has been pushed out several years. These weaknesses have affected the utility of the station with substantial cutbacks in construction, the number of crewmembers and scientific research. The grounding of the shuttle fleet has a significant impact on the continued assembly and operation of the station. The station is not only the primary vehicle for transferring crew and equipment to and from the station, but it is also used to periodically re-boost the station into a higher orbit. While controlling costs and schedule and retaining proper workforce levels has been difficult in the past, the grounding of the shuttle fleet will likely exacerbate those challenges. Because the return to flight date for the shuttle fleet is unknown, and manifest changes are likely, the final cost and schedule impact on the station is undefined at this time. The fourth challenge is –. ADMIRAL GEHMAN: – Let me – let's stop here a second. General Barry? MAJOR GENERAL BARRY: I'd like to ask a question about contracts. Let me run this by you. The Board is looking at the issue whether the space shuttle is an operational vehicle or a flight test vehicle, and we're debating that rather vigorously. When you look at contracts, right now, NASA, particularly for this shuttle, when you look at the SFOC contracts and the other contracts for the other components of the space shuttle system, NASA uses primarily the cost plus award fee contract formula, with the incentive fees, performance fees and so forth. Is it your opinion that that focus on that kind of a contract, rather than maybe fix the – or as the SPC used to be before the SFOC before 1996, would it be fair to say that maybe this award fee/performance fee focus does not encourage technical competence? Is there any issue there in your mind, insofar as what the motivations are insofar as a contractor? You said 85 percent of the budget goes –. MR. LI: – Right, contractors. MAJOR GENERAL BARRY: For contractors. So, can you give us some impression – some of your views on whether this award fee/performance fee focus is the right one for a flight test vehicle as opposed to an operational vehicle? MR. LI: Let me – let me answer this this way, and I preface this by saying that we have not done a thorough review of the SFOC contract, and so I am not as familiar with that contract as the Board is. However, in – with regards to your question as to what type of contract would be applicable for a vehicle that was either not in full operational use and one that's in an experimental, I don't believe that the contract – there is that sort of relationship where I would change a contract in order to reflect what state the aircraft or spacecraft was. I don't believe that that is the salient point. I also don't believe that the incentives, or some of the discussion that I've read in the media about what the Board has been asking questions about with regards to whether or not USA had the proper incentives, and perhaps disincentives, to, you know – with regards to trying to meet a schedule as opposed to ensuring safety. I don't believe – and I've had many, many interactions with the USA folks – and, regardless of whether or not they're contractors or government people, some of those people at one time used to be NASA people. And I think that, while I understand that – and I heard some comment in one of your hearings where they said the heart is there, but that does not necessarily mean that safety was – could be ensured. I really believe that their heart was there. I have had many interactions with USA staff up and down, and I don't believe that they ever had any such malicious intent. ADMIRAL GEHMAN: Okay, thank you very much. Let me – I'm sorry, but let me – you said that you had not looked in depth at the SFOC contract itself. Really, this section on contracting is really a section on financial management and –. MR. LI: – It's on financial management, but it's – also reflects the work that – and the cost overruns problems that we have found. For example, I mentioned earlier that the shuttle is being used to re-boost the station. That was not the original intent. The original intent a few years ago was that they would have a propulsion module that was attached to the space station to do that. That propulsion module was canceled, and part of the problem was this fantastic cost overrun associated with the propulsion module. They've had cost overruns on many, many other things, some of the things on the upgrades that were canceled. For example, the CLCS that the Board is well aware of had that problem. And the issue that we're raising here, Admiral Gehman, is they did – they do not have that financial management system that provides them the real-time, accurate information that they can project this is where I am and, therefore, the next few months we're gonna be in trouble. ADMIRAL GEHMAN: Right. MR. LI: But that was the issue. DR. LOGSDON: Could I – quick follow-up? ADMIRAL GEHMAN: Yes, absolutely. DR. LOGSDON: You, GAO, had been looking at the almost billion-dollar investment that NASA's making in new financial management systems. What level of confidence have you developed about the success of this, since it's the third time – third try? MR. LI: I think the issue – the – right now, the feeling is a mixed one. We just issued a report that was released just last week, and the issue there was that we do believe that the current core financial module, which is one of the components of the IFMP, is providing NASA, for the first time, with information that's consistent across all centers. Up until this time, they've had separate accounting systems pretty much throughout all their centers, and that's the reason why they were never able to provide their top management with accurate information. They had to go through heroics in terms of manual spreadsheets in bringing that all together. So, from that standpoint, that's positive. We, however, as we identified in our report, we have several concerns associated with how they are testing the core financial module. We're worried about – that some of the issues associated with providing the project managers and cost estimators with information, that that is not going to be provided just yet because they had not consulted with them early on in the program. So, we had – to answer you, Dr. Logsdon, it's mixed. The fourth challenge is for NASA to reduce launch cost. In our earlier identification of cost to build the station, we listed shuttle launch cost as being a substantial cost component, almost $50 billion. NASA recognized the need to reduce such costs as it considered alternatives to the shuttle. A key goal of the agency's earlier effort to develop a shuttle replacement vehicle was to reduce launch costs from $10,000 per pound on the shuttle to $1,000 per pound by using such a vehicle. As we testified in June of 2001, NASA's X-33 program, an unsuccessful attempt to develop and demonstrate advanced technologies needed for future vehicles, ended when the agency chose not to fund continued development of the demonstrator. Subsequently, NASA initiated a five-year, $4.8 billion program to build a new generation of space vehicles to replace the shuttle. In September of 2002, we reported that NASA was incurring a high level of risk in pursuing its plans to select potential designs for the new vehicle without first making other critical decisions such as identifying the overall direction of its integrated space transportation plan. NASA agreed with our findings. In November of 2002, the administration submitted to Congress an amendment to NASA's fiscal year 2003 budget request to implement a new plan. The new plan makes investments to extend the shuttle's operational life for continued safe operation, and refocuses the earlier effort to develop an orbital space plane and conduct development of next-generation launch technology. As I indicated at the onset, I will now comment on the effectiveness of knowledge sharing at NASA. In January of 2002, we reported on the results of a survey we conducted of NASA program and project managers. The survey revealed that lessons are not routinely identified, collected or shared. Respondents reported that they were unfamiliar with lessons generated by other centers or programs. Many indicated that they were dissatisfied with NASA's lessons-learning processes and systems. Managers identified challenges or cultural barriers to the sharing of lessons learned, such as the lack of time to capture or submit lessons, and their perception of intolerance for mistakes. They further offered suggestions for areas of improvement, including implementing mentoring, storytelling and after-action reviews as additional mechanisms for lessons learning. In closing, I will conclude with the following observation: to successfully implement its programs, NASA will need sustained commitment from its top management to focus attention on strategic planning, organizational alignment, the human capital strategy, performance management and other elements necessary for transformation success. The challenge ahead for NASA is to impart top management's commitment and vision to the rest of NASA by establishing the framework necessary for its successful implementation. Chairman Gehman, this concludes my verbal statement. Thank you again for requesting my participation in today's public hearing. Be happy to answer any questions that you or other members may have at this time. ADMIRAL GEHMAN: Thank you very much, Mr. Li. MAJOR GENERAL BARRY: Just one real quick question. Do you think NASA as an agency is platform-centric, or does it have, in your view, a focus on a strategic plan and where it wants to go? MR. LI: That is a very difficult question. I think that the strategic plan that the administrator put forth establishes that the agency is no longer one in which it is platform-centric. It is the science, and there's no longer a destination-specific mission. It's one in which there are certain goals that have to be achieved. So, to answer you from that perspective, I believe that they are not platform-centric. MAJOR GENERAL BARRY: Thank you. MR. LI: To use a DOD term, kind of remembering that platform-centric versus network-centric. ADMIRAL GEHMAN: Mr. Li, in your – you have a tremendous amount of experience in this area. I'm talking about reducing launch costs and launch reliability and things like that, which you've done at least two studies on this. If you look at the space station, for example, you have reports in here, and I won't quote them, and I'm not gonna get hung up on statistics here. But, you submit a report on this ISS that it's behind schedule and over budget, and then a year later, you submit another report in which everything is doubled. I mean, just in one year, the costs double and you get another year behind, and then another year goes by and you submit another report, and, you know – and costs have gone up, and it's behind again. Which – I'm not being – I mean, it may sound like I'm being critical, but I'm not really being critical. This is the nature of exploration and doing things that mankind has never done before. To me, we should expect that. Now, maybe we could do a better job of cost accounting and things like that, but I don't find that the process of going places and building things that mankind has never done before, I don't find that there's a lot of slop and error in there, and a lot of unexpected kinds of things in there, but that's my own view. Okay. So now, we talk about space launch initiatives, and we set a set of requirements like reducing the – you know, you used the number that cost to launch a pound is $10,000. We actually calculate it to be way higher than that, but that depends on how you calculate it. And so, now we have – let's get it down to $1,000 and have a fully reusable vehicle that doesn't take six or 700 man-years in between each flight, etc., etc., and all that kind of stuff. And then, we spend $1 billion and we find out we can't do it. And then – so then, we launch another initiative, and – do you find that – do you agree with me, or would you characterize in your own words whether or not we continually fall into the same trap of setting goals which are unachievable, underestimating their costs, and then not following through? And we seem to repeat this – we as a nation. I'm not talking about NASA here, because there are a lot of parties involved in this. We seem to repeat this pattern, and then punish ourselves when we realize that space exploration is so hard. And consequently, we find ourselves today without a replacement vehicle for the shuttle. Am I way off base here, or could you – could you relate that in your own terms, based on your experiences? MR. LI: As I've faced General Barry, I have to be careful because, you know, I just recently testified on the FA22, and that is not a spacecraft, and that has also had many cost increases associated with it. The issue is that, yes, technology is making it very difficult for some things to happen, and people do underestimate the complexity. Again, the – like, on the FA22, the software complexity of integrating all different avionics into this aircraft is causing tremendous problems, and you would think that by now our technology would be such that we can do that, but it doesn't happen that way. On the issue of the reusable launch vehicles, and especially on the international space station, one thing that perhaps you have found in your analysis is that NASA has been trying to force-fit projects within their budget. And one of the criticisms of the international space station and why we've had these overruns and why does it – suddenly somebody says, ÒHey, by the way, we have a $5 billion overrun.Ó How does that happen? Well, it happens because the focus is primarily on the budget year and trying to fit everything within that budget year. They are not looking at the cost to complete a project. If they had that particular perspective, and I believe that they are now, we perhaps would not be achieving those – and seeing those sorts of cost overruns. ADMIRAL GEHMAN: Do you believe that this tendency, or this characteristic – we're not just talking about NASA here. We're talking – I mean, this is shared among several agencies and branches of the U.S. government. I mean, NASA has to work in a certain environment, you know. NASA has several bosses, and they get this characteristic of focusing on the budget year with some help. MR. LI: And you're right, and just not to – kind of tooting GAO's horn, but one of the issues, in terms of when we're talking about weapons systems development, and one way to control cost overruns is to make sure that you have mature technologies before you go to production. Now, I understand that, from the standpoint of NASA, that is not a good similarity. But, the issue still is that, in the budget process and trying to get that particular political support for something, there is a tendency to try to establish a cost that everybody is going to be agreeable to supporting. And unfortunately, as more things are known and as technologies are found to not be as mature as they are, then cost overruns will happen. I don't have a solution to that. I think, in terms of the X-33 that I spoke of, and I think you were implying with – talking about the $1 billion, that wasn't an instance where it was hoped that, even if there was increases, Admiral Gehman, that private industry would have been willing to pick up that slack because of that brass ring that was going to happen at the end, which is the Venture Star, which Lockheed Martin thought that they were going to build and capture all that for our – from a commercial perspective. ADMIRAL GEHMAN: Except in that particular case, we have a built-in set of checks and balances that, once a commercial entity realizes that there is no brass ring out there, they back away. I mean, there's an automatic check and balance here, whereas in space exploration sponsored by the U.S. government, sometimes you really want to get it done. You just have to keep – you just have to – you have to overpower the problems, and I don't know a better solution, but it does seem to be – it seems that the process has left us here with a space shuttle program which is entering its third decade. People are talking about it having to enter a fourth decade, and we do not have a viable replacement vehicle because of a couple of false starts and things like that. We seem to be repeating this process. Dr. Logsdon? DR. LOGSDON: I have a question for Marcia Smith. If we could get her presentation back to chart number seven, I want to try to ask you, Marcia, to talk a little bit about what was going on in the program, see whether that works. MS. SMITH: Could you bring up my slide number seven? DR. LOGSDON: If you look at that and look at the current dollars, you see that, between fiscal '92 and fiscal '95 that the shuttle budget went rather dramatically down, and it's been more or less level since then. What was going on in the program in those three years? How much of that is ASRM cancellation? How much is –? MS. SMITH: – And budget deficit reduction, you know, the general reductions that were made at the agency's discretion, which are some of those larger numbers that you saw on the later slides. But, during the early 1990s, there were still plans to build the ASRM, the orbital maneuvering vehicle was still being planned at that time, another project that was ultimately cancelled. And so, there were funds being spent on ways to upgrade the shuttle, basically, and the decision was made not to proceed with those, and that, coupled with the struggle to reduce the federal budget deficit, brought the numbers down by the mid-1990s. Also, at the time after they'd had the Chris Kraft report in 1995 that suggested going to something like SFOC, and then in '96 they went to SFOC to try and level out those shuttle budget numbers so that it was not consuming such a large percentage of the NASA budget. DR. LOGSDON: But this chart shows that, from SFOC on, the budget hasn't – there hasn't been big cost savings because of SFOC, or maybe there have been savings that have been offset by upgrade investments. I mean, you know, what –? MS. SMITH: – Well, I think it's that, if you had not had SFOC, then the line would not have been able to stay stable. That's my understanding of it, that although, you know, it goes down to a number and it stayed pretty level, that if you hadn't had SFOC in there, it wouldn't have gotten down there and it might have kept going up. But, that – SFOC has saved significant money for NASA compared to where the program would be had there never been an SFOC. ADMIRAL GEHMAN: Or cost avoidance systems. I mean, it's avoided having the program go up in cost. MS. SMITH: Right. ADMIRAL GEHMAN: All right. Dr. Ride? DR. RIDE: Let me just make a point that's going back to, I think, a point that Mr. Li made right at the very beginning of your discussion, in fact, before your prepared remarks, in that is related to this discussion of the repeated tendencies to start an initiative to replace the space shuttle or to develop a new vehicle which then overruns in cost, turns out to be harder than everyone thought it was going to be, and is ultimately canceled. One of the – one of the ramifications of that continued tendency has been that we're always ten years or less away from what we think is going to be the replacement to the shuttle. As a result, we don't put a lot of investment into upgrading the shuttle, and giving it the ability to last beyond those 10 years. So, we've been kind of trading off those investments, investing in new vehicles versus the upgrades to the shuttle. Could you just comment on that? MR. LI: What you're saying is absolutely true, and the starts and stops have affected it, and with the hope that the shuttle would not have to go beyond the 2012 at one time frame. Whether or not these were, in hindsight, not the right things to do, the X-33 was a technologically complex program. It was – they had decided to do a single stage to orbit as opposed to a two stage. They were trade-offs. It was going to be less costly, and that's the other aspect, Dr. Ride, that we should remember is that, not only were they looking for something that was going to replace the shuttle, but they wanted something that was going to reduce launch costs of significant magnitude, and that's a very difficult nut to crack. ADMIRAL GEHMAN: Mr. Turcotte? REAR ADMIRAL STEPHEN TURCOTTE: I might piggyback up on that a little bit, but let's talk a little bit about the effects of budget and indecision, I guess, on infrastructure. Looking at a lot of the facilities that – specifically, the Cape comes to mind. A lot of indecision on where the program is going over the years has caused us a lot of delays, and there literally are a lot of facilities that are crumbling. Could you comment on that? MR. LI: Yeah. When I was at Kennedy just a few months ago, I did notice that, and you're right. As far as the investment in those particular structures, many of those structures were built for the Apollo projects and, as a result, things are starting to fall apart. I was there when the crawler had that problem, which is the transport mechanism that takes the shuttle to the pad. And for the want of a giant $10,000 shock absorber, that crawler was immobile there for a while. And you're right. It's that sort of investment, but I am not prepared to criticize NASA management for not having made those investments, because I recognize the fact that they had a lot of other priorities. And just as I have to manage my home budget, I realize how difficult it is, and you make trade-offs. And I think the trade-offs were made, and when those particular problems were not one that was immediately on the screen, they did not make those. But, in retrospect, they should have, and I'm hopeful, Admiral, that the current extension program and the monies that they're going to be putting in the shuttle is going to also address that, because I understand they're going to put money in the infrastructure, as far as the shuttle expansion program. MS. SMITH: If I could just add one data point that you might find interesting, I didn't put in a slide showing how much money had been spent on upgrades over the past few years. But, since upgrades have been separately identified in the budget, which began in 1995, NASA spent $4 billion on upgrades from '95 to '02. So, there was an investment being put into upgrades. It was starting to tail off there towards the end, I think a lot, because of the uncertainty as to how long it was they were going to be keeping the shuttle around. But, they did invest 4 billion in upgrades during that period of time. ADMIRAL GEHMAN: Mr. Li, no one's asked any questions on the last part of your report, which is the knowledge sharing kind of a section of the report. And I have not – I actually have not read that particular report, or that particular work by the GAO. But, in your statement, you make some comments about cultural resistance and the requirement for various centers and stovepipes and things like that to work better together on lessons learned and knowledge sharing, and things like that. You have some relatively strong words in there. Do you feel that this is a relatively significant issue that NASA needs to address? MR. LI: Absolutely. ADMIRAL GEHMAN: Stovepipes and –? MR. LI: – Absolutely, and I think that, to Administrator O'Keefe's credit, he recognizes that his program of one NASA is one that addresses that particular issue. Some of the infrastructure sorts of things that they're doing, I mentioned the integrated financial management program is one that I believe is going to provide that sort of consistency. But, I – more important than that is this issue of, yes, the individual centers at one time were very competitive with one another and trying to bring them together and make them feel that this is a team effort is something that's very difficult. And lessons learning is one in which right now I think you're asking me do I think it's important. Yes, I think it's important because people like myself are gonna be retiring pretty soon. We're gonna take away a lot of knowledge that our institution should have. Now is the time for NASA to be investing in that and ensuring that that knowledge is transferred to the younger people. And if I can say so, you know, one of the things that has concerned me through this whole process of – and especially in reading in the media about the bureaucracy that perhaps NASA's had, and the slowness of its decision-making, I want to – if there's anything that I wanted to convey to the young people of America is that – please don't look at this as additional vindication that government service and public service is not one that's important. If any time we need good, young people to come into the workforce, now is the time because, you know, I remember when I wanted to be in the federal service, after I got my degree in Aerospace Engineering. I remembered those words of our President. That said, we don't do things because they're easy – we do them because they're hard. And with what's happening with NASA right now, we need young people to come in. So, I think it's extremely important that people be able to disassociate the fact that yes, NASA has had problems, but this is an opportunity for them to make a difference with us. ADMIRAL GEHMAN: I certainly – speaking for myself, I certainly agree with your statement, that almost everything NASA attempts to do is very hard. I know that for myself, until I began to understand a little bit more about this, I didn't realize how difficult it is to put an object in orbit in space. It was difficult when we first did it in the 60s, and it hasn't gotten any easier. I mean, we've still got a few laws of physics that are not going to change, no matter how hard we try to circumvent them. That's what we're trying to do here. So, this is still very, very hard, challenging work. And I agree with your comments that this is an exciting and worthwhile, national endeavor, that anybody should be excited to join. My last question gets back to this personal capital – this personnel business. And in your report, you mentioned NASA hiring initiatives and special pay initiatives, and special initiatives and things like that. Did you have an opportunity to look very deeply at a different mechanism? And that is, internal career development, promotions processes that – were you able to – and I understand (inaudible) hiring and things like that and it's all a good idea. But, is hiring necessarily the fix to this? Or are there – did you look at the internal promotion and upward mobility kinds of aspects –. MR. LI: Not in detail. But the thing that is important to understand, is that NASA's human capital strategy is not only for hiring. It's for hiring and retaining and being able to secure the expertise that they need. Some of the initiatives that they're proposing or that are being proposed in legislation have to do with securing people that have the expertise and being able – have more flexibility in being able to get them to come into NASA. And the retention bonuses are for people that have that expertise, but are on their – would like to retire. And we're trying to convince them that hey, please don't retire just yet, we need your expertise. There is one aspect that you're mentioning, that I think it's the mentoring issues that really have to come to fruition at NASA, that there has to be some innovative ways in which we can provide that transfer of information from the experienced people to those that are coming into the workforce. ADMIRAL GEHMAN: Thank you very much. MR. LI: Thank you. ADMIRAL GEHMAN: One more. MAJOR GENERAL BARRY: This is not unique to NASA, as you well know – that we are short on scientists and engineers and in the military and all that other stuff. Did you find any of the benchmarking being done to figure out any transfer of lessons learned between, like DOD and NASA or any other parts of the federal government? MR. LI: No, we have not. But, however, as you're probably aware, GAO has identified human capital management as a high risk area for the entire government. And I know that at DOD, Secretary Rumsfeld has introduced and has been wanting to make specific changes associated with that. I think that is the trend. Everybody's recognizing now that with the impending loss of a lot of knowledge on the part of people, like my age, and they just need to do something and they need to be able to manage that workforce better than they have in the past. ADMIRAL GEHMAN: I would like to thank you both, Ms. Smith, and Mr. Li, on the behalf of the Board, for your candor and your willingness to dialogue with us and work with us as we try to understand things that you're experts on and we're not. We all have the same goal here, which is to return – safely return man to traveling in and out of space. And we want to do it as quickly as possible. And we are hoping that our work will help do that. And your willingness to help us along that line is greatly appreciated. We will take about a 10 minute break here to set the next panel. Please, not more than 10 minutes and we'll go back to work. (Inaudible). Okay, alright, we're ready to resume. I'd like everybody in the hall here to take a seat and stop talking, so we can proceed, and with the second half of our public hearing. We might call this, moving from the way it ought to be, to the way it really is, or something like that. We are though, very happy and pleased to have two very, very experienced managers and directors that really know how things run. Russ Turner. Russ Turner, until just May, was the CEO of the United Space Alliance and he was – he had a very, very long tour as the CEO of the United Space Alliance. Probably longer than he planned to when he got there. We'll let him tell that story himself. But Russ is a business man and has a long and rich history in the space shuttle business, going back, I believe, to Rocketdyne and has been in this business a long time. Tom Young, is a former NASA Center Director, former space industry CEO and he serves, it seems like, a professional task force, a professional board, a professional advisor, on all matters of space and shuttle programs, not only to NASA, but to the US government at large. As is the process of the Board, before we begin, let me ask both of you gentlemen to affirm the information you provide the Board today, will be accurate and complete, to the best of your current knowledge and belief. PANELISTS: I do so. ADMIRAL GEHMAN: Thank you very much. I'd like, starting with Mr. Turner, if you would introduce yourself and say something to either amend my remarks or say anything you want about your background and expertise. And then we'll ask Mr. Young to do the same thing. And then I'll ask (inaudible) and comments. MR. RUSSELL TURNER: Yes, I'm Russ Turner. For the last five years, until recently, I was the Chief Executive Officer and President of United Space Alliance. I was also involved at the very beginning of the formation of the United Space Alliance. I was on assignment for about six months, with Kent Black, who was the originating CEO. So, I have a perspective, both from the start, and where it is today. ADMIRAL GEHMAN: Thank you very much. Tom. MR. TOM YOUNG: My name is Tom Young. I'm an engineer. And as the Chairman mentioned, the first 20 years of my professional life I worked for NASA, concluding with being Director of Goddard Space Flight Center, almost totally in the automated side of the NASA activity. The next 13 years for Martin Marietta, where I was President and Chief Operating Officer, retired in '95, currently on several boards of directors, and as mentioned, involved in various advisory activities. I might mention just two or three of those. I did chair an independent review of the Mars program, after the Mars '98 failures. I did chair International Space Station Management and Cost Task Force. And I'm currently chairing a review of the DOD Space program, looking at cost and schedule related issues. Thank you. ADMIRAL GEHMAN: Thank you. Thank you very much. And I believe, Mr. Turner, are you prepared to go first? MR. TURNER: Yes, sir, I am. If we could put my first chart up, please. I'm here to give the contractor's perspective on the Space Flight Operation Contract or SFOC, as we call it. And I'm going to just spend a chart on the origins, to give you sort of a common basis of understanding that the SFOC. I'm going to talk about what was different about SFOC and what had gone before, talk about how that affected performance and what the performance has been under SFOC. Talk about the things that SFOC achieved that ought to be retained, and then draw some conclusions. Next chart. As you know, the SFOC was created by NASA in 1996, and it was viewed to being the next logical step in improving shuttle contractor performance. NASA had done internal reviews and they had brought Chris Kraft and a team of folks in, to provide an independent external review of where the shuttle was and what was possible for the next step. And the consensus was, the existing approach, which I think you saw in Marcia Smith's testimony, was a series of budget reductions over a period of time. I think if you look over the period about 1992 to 2002, it was more than a 40 percent decrease in the number of contractor employees and the number of government employees supporting the shuttle program and mostly driven through budget reductions. And so, the consensus was that if we're to continue to reduce costs and do so safely, we needed a different approach, and the SFOC created a prime contractor for shuttle, where there had been to that point, lots of separate contracts. Not only separate contractors, but even within a contractor, multiple contracts, creating a very complex situation that I'll come back and talk about a little bit more. So, the SFOC created this prime contractor. I have noted, in the media there's been some confusion over the fact that SFOC was not an outsourcing initiative. The contractors were already performing the vast majority of all this work. We'll talk about the accountability shift that took place. But this was not taking folks who had been performing work as government employees and shifting them into being contract employees. This was simply reorganizing how the contractors approached their work. The initial work scope of SFOC was 100 percent existing –. ADMIRAL GEHMAN: – Mr. Turner, if it's okay with you, we should ask questions as we go along because – so you don't have to jump back and forth. And I – my understanding of that last comment that you made, that the SFOC contract was a collection of independent contracts and subcontracts, and was not a privatization or an outsourcing – I mean, that is my understanding. Never the less, during the same time frame, there was a significant reduction in government employees at NASA. MR. TURNER: Well, as I indicated actually, that reduction in employees started back around 1992. The total reduction – I have it in front of me – was 56 percent. And if you look at the curve and you look at the beginning of SFOC, I think you conclude there really wasn't much of a change, in terms of the rate of decline in government employees that could be attributable to SFOC. There were some shifting accountabilities that enabled NASA to move some folks around – absolutely. ADMIRAL GEHMAN: That's correct. Now, I think we're saying the same thing. I am not attributing any of the government personnel cuts to SFOC. However, at the time this was going on, there was a steady – it started before this and it continued after this – a reduction – a pretty – as you say, over a number of years, 50 percent reduction in the number of government employees. MR. TURNER: Agreed. And, by the way, the same reduction was going on, on the contractor side – almost the same percentage. And if you look at the slope of this chart, it actually accelerates a bit around the time of SFOC, and that was because of SFOC and I'll talk about that a little bit. ADMIRAL GEHMAN: And also at this time, there was some pushing and pulling of responsibilities and functions, between the government and the contractors. And I don't know if you're going to get into that or not. MR. TURNER: Yes, we will. We will talk about that. ADMIRAL GEHMAN: Thank you. MR. TURNER: Although, I prefer to think of it as a planned transition, as opposed to pushing and pulling, but we'll talk about that. ADMIRAL GEHMAN: I'll use your terms. MR. TURNER: The initial scope of the SFOC was 100 percent existing Rockwell and Lockheed Martin contracts. There were subcontractors to those contracts, but on day one, that was the work-scope that was included. The United Space Alliance was created by Rockwell and now that, of course, is owned by Boeing, and Lockheed Martin, specifically to compete for the SFOC. When we understood the government's intent, we looked at what the best way to respond to that was, and concluded that a joint company that had the best skills from both, would best serve the government. And NASA ended up sole sourcing then the SFOC to USA, after evaluating industry capability statements. They held a suppliers conference, where they talked about what SFOC was going to be and asked for 25-page capability statements to be submitted by interested parties. We submitted such a statement and ultimately, were awarded the contract, sole source. Next chart please. ADMIRAL GEHMAN: So your understanding is, that this was a sole source of award? MR. TURNER: Yes. ADMIRAL GEHMAN: And if this is not an appropriate question, you just tell me if it's not appropriate. But, can you tell me what the contractor investment in the new entity was, dollars? MR. TURNER: I can't tell you dollars off the top of my head. That number's available. We can provide it. Each of the companies, Rockwell and Lockheed Martin, contributed a certain amount of capital. And then they contributed a significant amount of human capital in their key engineering and manufacturing organizations. And so, we can provide you with that total investment amount. ADMIRAL GEHMAN: I have read in some sources that the investment capital provided by each of the two entities was a nominal amount, I mean, like a million bucks or something like that. MR. TURNER: No. ADMIRAL GEHMAN: Okay. MR. TURNER: It would be substantially more than that. Because, they each did have, when they put the companies together, existing capital that went into the companies. And so, machinery, computers, facilities, anything that belonged to the two companies that were associated with operating the system, went in. I think the million dollar reference –. ADMIRAL GEHMAN: – But that wasn't new. MR. TURNER: No, but it was property they owned that they gave up to USA, that otherwise they could have retained for their shareholders. I think the million dollar reference is talking about money they put up, day one, to just be able to form a company –. ADMIRAL GEHMAN: – Probably right. MR. TURNER: Right. But in order to do SFOC, they had to put all that other capital in first. As you know, the SFOC emphasized contractor performance accountability and it was a shift in approach and a gradual shift in actual accountability. NASA's accountable for establishing the goals and objectives as it has always been. And really very important here is that NASA continued on the requirements. There was no shift of requirements ownership. In the shuttle program, requirements determines how you process the vehicle and how you fly the vehicle, and they retained authority of that and have it still today. The contractor's mostly accountable for what and how you achieve those goals and objectives, with those requirements. Certainly accountable for the technical performance, for the scope of its contract. Certainly responsible for its own systems and processes, except where they're controlled. So, for example, a non-conformant system which would be used to track the hardware performance and any discrepancies with it, that would be a controlled information system. If there were going to be any changes to that, that would end up having to be approved by NASA. But, if we wanted to make a change to an internal risk management tracking system or a human resource system, those are within our purview to make changes as appropriate. And of course, the contractor was accountable for – is accountable for total cost. And I'll come back to what the implication of that total cost accountability is. DR. LOGSDON: Russ, just to make this clear in my own mind. What function did NASA perform before SFOC, like safety admission assurance, that they either have many fewer people or no people performing after SFOC, where the function and responsibility was transferred to the company? MR. TURNER: Well, actually, you just asked two different questions. In instances where there was a person doing a task at NASA and USA picked it up, and therefore NASA no longer had any function to perform for that person, related to shuttle and reassigned them, I suspect, John, it's a relatively low number. And I'll get that number for you. What happened most often, was a very conscious shifting of accountability. So, Admiral, I believe you brought up the issue of subsystem managers. The subsystem manager before SFOC was a NASA person. That NASA person would have the accountable contractors, who in many cases probably had most of the technical expertise, and they would meet in boards and panels, and the NASA person would chair the boards and panels, because they're a subsystem manager. After SFOC, the subsystem manager role transitioned to the contractor, where the technical expertise was, but the NASA folks would still participate in the panels and boards, and therefore, there wasn't a shifting of a – while there's a shifting of accountability, there wasn't a shifting of a job. It was a change in how the process was done. DR. LOGSDON: But USA would chair this board? MR. TURNER: Absolutely. USA would take over chairing those boards. And I have a list, which we can also provide you, which gives you top view of where the shifts took in accountabilities on various boards and panels. So, as I'll get to on a chart that's coming, this is a significant change in accountability. Prior to this contract, these were more like level of effort activities, where the contractors provided all the actual hands and feet and technical expertise, but there would be a lot of day to day direction from the government. And the transition to saying, you're providing the technical expertise, you're accountable for your performance. We're still going to participate with you. You still need our approval for any changes. But we're going to give you more end to end accountability. That was the change. ADMIRAL GEHMAN: One of the – this is not a quiz and so I'll ask you a specific question. But, from my reading on this subject, one of those subsystems that shifted from government responsibility or government oversight, to contract oversight, was the Thermal Protection System. MR. TURNER: Absolutely. ADMIRAL GEHMAN: And now we have a contractor who is the subsystem manager –. MR. TURNER: – Correct. ADMIRAL GEHMAN: – Of a Thermal Protection System. MR. TURNER: We might want to expand then, how the process works. For the various elements of the shuttle system, NASA provides the TMR, Technical Management Representative. And the TMR has ultimate authority over that system. And that accountability did not shift to the contractor. So, the way the process would work – and we use TPS as an example, is if there's an issue relative to TPS, the technical teams would do the detailed work and the chairperson of that activity, the accountable person for making sure that the right people are on that team and they're answering the right question and they're doing a good technical job, is the subsystem manager. That team result is then taken up through NASA to the TMR, to the TMR's board for reviewing whether or not that, indeed, is adequate. Now, that's what it looks like on paper. In reality, this is a day to day communication activity that goes on. And so, the TMR would have been very connected to what, in fact, probably was the origin of the request. And then, if there was an issue that needed to go further, then the TMR would be able to take it forward to, for example, the PRCB, which would be the shuttle program manager, NASA position, that ultimately would approve any changes of a certain level. ADMIRAL GEHMAN: Thank you. MR. TURNER: So, a change that takes place is that NASA now is in the position of evaluating contractor performance, relative to this change in accountability. Dr. Li talked about this phrase, oversight versus insight, and the discussion we had around accountability just now probably helps illustrate what that was. NASA was responsible for watching USA, how we performed, what our processes were, the robustness of our processes, and the quality of our products. But the accountability for actual execution of all that resided with USA. And that was a shift. There was and is in SFOC, an increase in objective performance – management criteria. I submitted to the Board, a set of the 200-plus metrics that NASA tracked as part of their insight activity, to validate our performance. And then the contract shifted, from what in most cases had been cost plus award fee, to a much more sophisticated cost plus award fee, performance fee, and cost incentive. And associated with that, were safety gates, to insure the proper focus. And I heard this come up earlier. The way this was structured, NASA did a traditional NASA award fee evaluation of USA, how are we doing against a set of criteria, and that would have been rated by each of the TMRs and then summarized up to the Shuttle Program Manager. A separate rating would be given to USA for its safety performance. And that was led by the Quality TMR and how well we're responding to NASA's safety priorities. A separate evaluation was done on how well we achieved the performance criteria for properly processing the vehicle and properly launching it and returning it. If the safety score were good or lower, we would lose the opportunity to enter in any of the cost incentive during that same period. So, the safety performance, added a minimum threshold, very good. So it had to be in the very good to excellent range, or there was no cost incentive. MR. WALLACE: May I ask you a question, Mr. Turner? MR. TURNER: Sure. MR. WALLACE: Has that happened in the history of your contract? MR. TURNER: No. MR. WALLACE: So, you've never fallen below that threshold? MR. TURNER: We've never fallen below very good, and we're at the excellent level – USA is at the excellent level currently. MR. WALLACE: This presents an issue that we wrestle with a lot within NASA, and also in terms of these contract award criteria. So, philosophically, how do you write a contract where since there really is a high objective, maybe a top objective, without ever creating an incentive to under-report safety problems? I mean, we see, occasionally, proudly displayed reductions in, let's say, in-flight anomalies, in some phase the program, yet we also see, in parts we look at really closely, certain discrepancies, which seem to IFAs before, that later on aren't IFAed. What are your thoughts on that? MR. TURNER: First, the process you just described is a NASA process. So, that isn't an opportunity for the contractor to define what is and isn't an IFA. So, I wanted to put that aside, because you're asking a more general question, which is around, does the government have enough insight into our metrics to know that the performance we're reporting is the accurate performance. They do. And from a contractor's perspective, and I'll come back to this, it is very clear that all of the contingencies are around safety performance. Yet there are scheduled contingencies, there are issues around making sure that we form well the budget, but there isn't a business, if you don't have the ultimate level of safety. And in fact, I'll go to a side now – one I was going to make later. When you evaluate this and make recommendations, I encourage you to be empiricist about it. That is, in addition to the philosophy issues you've raised, you need to spend some time on the facts and ask yourself, how did the contractor behave under this arrangement? And what you're going to find out is that USA spent a lot of money on safety activities that they did not have to spend, by the nature of the scope of the contract, and which meant that they did not get cost incentives, that they could have gotten, by not spending the money. And I'll give you an example. We initiated a bonus for the employees – an annual bonus, based on the company's safety performance. Every employee in the company could get a check at the end of the year for $750, if we met all the objective performance goals around safety. And the employees did very well against that. I don't know if they maxed-out every year, but they were –. MR. WALLACE: But was a lot of that occupational safety, missed work –? MR. TURNER: – It's a whole bunch of things; occupational safety, in-flight anomalies, processing escapes, damage to hardware, it's the full range of things related to safety. And we negotiated that set of categories with NASA in advance, so that they could agree that that constituted a good measure of how the organization was performing. So, 10,000, $750 an employee, that's about $7.5 million a year, in money that USA committed to spend, that it didn't need to, in order to respond to the contract. That's one example. Second example, a big investment in a new safety system, risk associated trouble spots, which was put in place to get employees to identify more issues, and rewards employees for identifying more issues. And it's a closing system, so once identified, a manager is assigned to it and the manager has to close it out. Now that cost money to put in place, not only to administer it, but then you have to respond to all the issues. Not required to be done under the contract, but done because safety is the primary focus of the contract. Trained every manager in the company a several day training program on how to lead for safety. Integrated – implemented VPP ÒStarÓ status at every facility. Implemented the new quality system. So you go through this list of things, and what you'll learn as an empiricist is, this list of things didn't have to be done. The total amount of money USA spend on activities that it was not required to do under the contract, over this first six years, is about $190 million. If instead of spending that money, they had just counted that as savings, they would have gotten 35 percent of that. So, the nature of this safety gates and the nature of the business we're in and the nature of the culture we're in, had a result. And I encourage you to look at the result in terms of the systems and the money that USA spent, and the result is what NASA was trying to achieve. Which was a balance between, first and foremost, a focus on safety, and then given that, how do you reduce cost and how do you make sure you meet the manifest and meet the mission objectives. And I'll show you some performance figures later on. I also will later tell you, from a contractor's perspective, it doesn't matter whether there's a cost incentive or not, it's based on what the government wants to achieve. So, I'm not lobbying you to keep a cost incentive in, I'm just encouraging you, before you draw conclusions about its effect, to make sure you look at what the effect actually was in terms of contractor behavior. Next chart. I didn't mention on the safety gates, in addition to having to have at least a level of very good in the safety score, you had to have at least a level of good in the overall award fee score. So there are actually two safety gates, and you had to hit both of them in order for the cost incentive to be available. ADMIRAL GEHMAN: While we're on that subject, this is not exactly on the subject, but what does the contract say about loss of vehicle and loss of crew? MR. TURNER: Loss of vehicle, loss of crew, we lose all the cost incentive, the performance incentive, a bunch of other stuff. In the instance of the Columbia tragedy, I think it ends up being a $70 million impact, to USA. And some of that – a bunch of that, is money that we actually have to pay back to the government – money that we had earned in the prior periods, and based on the – if the loss of the vehicle were a result of a USA action or a failure to act by USA, we would need to refund that money. MR. WALLACE: You sort of anticipated my question. If this is an element you would lose certain awards, and then as far as there being sort of a penalty would require some kind of finding by who, I don't know –. MR. TURNER: – Well, the way it's worked, it's actually very straight forward in that the contracting officer for the SFOC contract makes these calls. Now, how NASA behind the scenes works, I wouldn't have visibility. But the contracting officer would send us a letter saying we've made the determination that USA's accountable for the following, and then the terms and conditions of the contract dictate not only loss of the opportunity to earn, but also refunding money earned in prior periods. ADMIRAL GEHMAN: Yes, sir, go ahead. DR. OSHEROFF: Well, I just – let me ask a question, slightly different, that I asked in the last round. Which is, issues that were not necessarily safety issues, but maintenance issues, such as the impacts of foam shedded from the external tank, whose responsibility was it for dealing with those? And I mean, I guess I really want to know why there was no concerted effort put forward to understand the problem and to eliminate it? MR. TURNER: That's very clearly, a NASA space shuttle program call. USA was getting the brunt of that. And you commented earlier that it cost a lot of money. It cost USA that money. We didn't get any relief as a result of having to make those repairs and I'm on a cost incentivized contract. So, the additional work necessary to repair that tile was coming out of my funding. And so I was motivated to find a way to have that foam not come off anymore, because it was creating a turnaround issue for us. But, the SFOC is only a limited part of the total shuttle program. So, the external tank, the space shuttle main engines, and the solid rocket motors are not in the scope of SFOC. If the tank had been in the scope of SFOC, the question you just asked me would be an appropriate question for me, why didn't you get after it. And we would have been motivated to do that, but the external tank isn't part of the Space Flight Operations Contract. DR. OSHEROFF: But still, if you felt – how much was it costing, by the way? Do you know, roughly speaking? MR. TURNER: Not off the top of my head. It, you know, every one of those tiles has to be inspected. There are rigid standards for how you repair or replace it. And clearly, like during the period when we were getting the popcorning, there were more tile damage and those tiles had to be replaced. We certainly – with feedback through the system, what those costs were and expressed the desire to see where we reduce the damage to the tile, but that's the extent of our ability to influence the outcome. DR. OSHEROFF: But, if you had thought that you could save money by undertaking a research program to understand the problem, you would have done so? MR. TURNER: Well, not with the external tank being – there is no mechanism by which I could undertake a research program on the tank program. And that's what you'd have to do. The research would be on how would you change the foam so it performs differently, and since I'm not the contractor for the external tank, I don't have a mechanism to insert myself and say, hey guys, I'm gonna do a research project, give me your experts, let me have access to your hardware, and I'll let you know that outcome. DR. OSHEROFF: Who is – who does have the contract for the external tank? MR. TURNER: It's a Marshall contract, so it would be through that Marshall chain of command. DR. OSHEROFF: What is the subcontractor? MR. TURNER: The contractor is Lockheed Martin on the external –. DR. OSHEROFF: – Okay. Lockheed Martin certainly has a part – I mean, this is very funny. I mean, obviously, you are – USA is partly owned by Lockheed Martin, correct? MR. TURNER: USA, yes, USA is a limited liability company. And we have two shareholders, 50-50, Lockheed Martin and Boeing. But the nature of a limited liability company is there is a very, very limited governance relationship. And for all intensive purposes, the Lockheed Martin external tank people are the same to us as Thiokol or any other non-USA company. There isn't any relationship. I can talk to people, but I can talk to people at Thiokol as well. The shuttle community, the contractors, talk to folks. And so I can tell them that it'd sure be better if the tank performed differently, but I'm not in a position to direct them to do anything. ADMIRAL GEHMAN: Even though you can't quote a dollar amount, would you give us a subjective evaluation of whether or not TPS repair was kind of the driving factor in turnarounds or frequently was the driving factor? MR. TURNER: For a long time, TPS was the – in terms of cycles, was the long pole. Now, that's, as you know, TPS damage comes from a lot of sources, not just foam. In fact, in prior years, it wasn't foam from the tank at all. The most damage, I think, in the early years, was from the solid rocket boosters. You get damage on liftoff from stuff around the pad. You get a lot of damage when you land, because of stuff that gets kicked up when the vehicle touches down. And so, the impact of all of those things together, had for some time – and I think in recent years it was less of a long pole, but it was still a significant element. And partly because of the process to repair and replace tile. ADMIRAL GEHMAN: Anything else on that? Go ahead, General Barry. MAJOR GENERAL BARRY: To go back to the trend of keeping on one slide here, but the bottom line is on this cost plus award fee performance fee cost incentive. Let me ask you a couple of quick questions. Is the shuttle an operational vehicle? MR. TURNER: I heard you talking about that earlier. I think you need to – I think we need to do some definition of terms. There is – I think what folks mean by operational – there's the way you fly a 737 airplane. That's what an operational vehicle is. That's one end of a spectrum, I think. The shuttle is not like operating a 737. It's much more demanding that – and Admiral Gehman referred to how hard it is to go to space. There's some real physics limits to what we're doing. So it's not like a 737. It is however, operational in the sense that the purpose of the vehicle is not testing. It's not a test vehicle. The purpose of the vehicle is to perform a mission that's independent of the vehicle itself. And most recently, of course, deploying space station. So, it has an operational role. And there really is very little that is going on in a planned way, around collecting more data and doing experiments about how does the vehicle perform. The vehicle performs in an envelope that's well-defined and understood, in order to perform those operational missions. Having said that, it is not a 737, and there are two kinds of issues here. One kind of issue is finding out things that don't perform as well as expected, because of design limitations, we'll say. And that's like, I had a relatively new car and the water pump went out on it recently. And that's not – there's no new science in that. Water pumps go out. Turns out the design, probably has the Serpentine belt putting too much side load on the water pump and the water pump failed. We have failures like that in the shuttle, for example, the wiring problems that we had. What I think brings people to talk about being a test vehicle is there's a second category of issues, which we have in the shuttle, which I call new science. And an example of that, I think, is the cracks in the flow liners. Nobody else operates reusable cryogenic systems. And in operating it, we discovered that there's some kind of back pressure phenomenon from the SSMEs that creates some kind of environment with those ultra-cold temperatures that causes these micro-cracks. That's new information about how such a thing operates. You're not getting that with 737s. That happened earlier, you know, in the development of the jet aircraft and you learned a whole bunch of that stuff. But because shuttle is at the cutting edge and nobody else is doing it, we are still getting those kinds of new findings that affect our ability to operate. MAJOR GENERAL BARRY: I know I'm jumping ahead a little bit on your slides, but with that understanding, is the incentive part of the SFOC conducive with this kind of an operation, in your opinion? MR. TURNER: Yeah, I'll go back to my empiricism, I think yes. Because if you look at the behavior of the contractor, we're focused on the things you want us to be focused on. We're spending money where you want the focus on. I think it's more appropriate to ask, was the program properly funded. As Marcia was discussing, given that it's that kind of vehicle. Should you be putting more into studying those new things that you're finding out, out of the vehicle? Should you invest more in the vehicle? Should you have a better (inaudible) planned improvement program for the vehicle, because you know it's not as mature as the 737 and therefore, you plan that you're going to be operating it that way. MAJOR GENERAL BARRY: Final question. If you were going to redesign this contract again, would you have it lean more towards the fixed fee elements or the incentive side? MR. TURNER: Will you let me wait till my last chart, where I make recommendations? Because I do want to talk about that, but I have a little ground work I'd like to lay before I get to it. This is the dreaded 15 minute chart that we all think about when we do briefings. It's been up there a long time. The SFOC – we're talking about what changed with SFOC. The SFOC contract made a significant simplification in NASA interfaces. The scope of work we're talking about were nine separate contractors, I think, 28 separate contracts, all with interfaces in technical business contracts – fairly complicated. Under the SFOC, we had a single contract, a single contract manager. The first thing that enabled us to do, which was probably the biggest single element of savings on the program, was to eliminate duplicate business organizations, duplicate information technology organizations, duplicate human resources organizations, and eliminate tiering of corporate flow-downs and corporate fee. So, if you go back and look at the numbers, there was a huge savings as soon as the contract was signed, out of just eliminating what was completely non-value added redundancy, that was a result of the prior contracting mechanisms. This is not reducing a single engineer on the program. This is simply getting rid of the support structure that was unnecessarily burdensome. Second thing we got out of the simplification was more unified technical requirements flow down. And I view that as an important safety and quality issue. When you had complicated flow-downs that were different by contract, inside the same contract or Rockwell, you would have a different way and philosophy around technical requirements, then the contractor was forced to figure out how to integrate those. And the nature of this contract is a single employee in the old days, would be charging to two or three different contracts. And therefore, as they shifted work, would be shifting where their technical flow-down came from. And under the SFOC, we had a single flow-down for the requirements. Now, the sub-bullet there that however, under the SFOC, the NASA Center differences were still in place. An example of that that I use I think that makes it very clear, is in Florida, there's a Marshall facility. There used to be a fence around it. And that's the solid rocket booster processing facility. If a USA employee is inside that facility, now they're responding to a technical requirements flow-down to come from Marshall, with differences in the quality system. And a real concrete example I use, is how often do you certify the calibration of a torque wrench? And let's say, at that facility, it's 30 days. That same employee, working over on the KSC side of the facility, with that same piece of hardware, but now as part of the integrated shuttle staff, will be under the KSC technical requirements flow-down, and that same type of torque wrench might have a 90-day re-calibration certification requirement. So there are two issues here, one is that ends up being complicated for the workforce. And the other is one of those two numbers is probably wrong. It's probably either 30 or it's 90. If it's 30, then we're not doing it as well as we could. If it's 90, then we're spending a lot of time doing calibration that isn't adding any value. So, that didn't change under the SFOC. And we still have the TMR, the Technical Management Representative structure aligned to the historical NASA structure, which I think is best characterized as around how the vehicle was originally developed. And so, you'd have a TMR for orbiter, a TMR for the external tank, a TMR –. And, and so, that also didn't simplify it as much as you might have. But still, much simpler structure that really helped us to address both cost and technical issues. The contract had a broad written scope – and still does – to avoid continual change traffic. I've come back to that as one of the benefits that enable the contractor to do what was right, without having to get into a contracts loop that drove a lot of time. And then the Center accountability change in the contract to be primarily base support, not program performance. The exception, until recently, being JSC, which was designated as the lead Center for Human Space Flight. So, JSC did have a very strong program performance accountability. But that's now gone. And the Centers now have this support role that the program is aligned within USA, to the shuttle program. Next chart. Now, a number of those changes enabled significant improvement. Let me talk, how do you measure improvement? Everybody's improving, so I can talk about improvement in a number of different ways. The SFOC enabled safety, quality, and cost performance improvements. And I'll talk more about why, but first, what improved and what can we compare it to. There is a very clear absolute improvement in performance over historical levels. So that's very straight forward to look at. I'll show you some numbers and you, I think, have access to more detailed data. It also improved, compared to a contemporaneous heritage company performance. There are still elements of Rockwell now boiling in Lockheed Martin. And if you compare the USA's performance to what they've done over the same time period, and you look at these various metrics, you see that USA's performance is improved over that. Where we could, we've done formal benchmarking. And this is very limited. Because it's hard to find an operational system – you can look at the Concorde, which we've looked at. We've benchmarked with Delta and some other airlines folks, to look for points where you can – how do we compare to best in industry? Places it's easiest to compare are in places like information technology, procurement, things that all companies do. We do those comparisons and USA's performance is in the top quartile, compared to the best comparable companies in the country. We also compare ourselves to Thiokol and other non-SFOC NASA shuttle contractors. And in doing those comparisons, our safety and quality at USA is as good or better. And we have a lower cost structure. We didn't talk about this earlier, but one of the advantages to the government and USA is of a single purpose company, so it doesn't have the flow-downs of a larger corporate entity, it has a very low G and A rate, it's focused on the single purpose and it's not a design and development organization, so it doesn't have the IR and D budgets, it doesn't have the Science Center costs, and so it ends up being a very low cost provider. Now, how do we get those improvements? The focused contractor accountability allowed us to go after optimizing the system, instead of the subsets of the system, and putting in integrated information systems, common training approaches, common certifications, those kinds of things. That was a help. The simplified structure is probably the most important things here, because it enabled us to eliminate a lot of handoffs. And any of you that are interested in the science of variation know, that every time you have a handoff, every time there's a communication, every time there's a task that moves from one organization to another, there's an opportunity for a mistake. So when you simplify that structure and decrease the number of handoffs, you get a system that's much more robust and that accounts for a bunch of the improvement. Much improved communication with one organization and we have one set of goals and objectives. We have one set of policies and procedures and it makes it much easier to work together. And then I mentioned, we have the advantage, therefore, of being able to look at the system. And so, if we're going to implement a better way of doing work authorization documents, we don't do it one way for flight crew equipment and another way for another element of the company that doesn't come together and doesn't enable us to, in fact, achieve all the benefits. We're able to implement it across the company. Next one. Now, I didn't – you have the book of the several-couple of hundred metrics. And I only put these up to be illustrative. We track a lot of detailed metrics and it may roll them up for me. We look at hardware inspection yield and that's sort of the classic, are parts that you're making, meeting the quality requirements. It's high and continuing high and I want you to note this starts in '99. The reason it starts in '99 is, in late '96, early '97, when USA was created, there weren't any common ways to measure any of these quality of performance. Because these were nine different companies and 28 different contracts. So we now have a unified set of measures around our quality and safety performance that we can report. We measure our overall product quality across all the different kinds of products we produce, software and paper products. We've implemented a first-time quality approach that we call process surveillance, where we go out to a site where an activity is going to be performed, like a tile installation. And an independent auditor evaluates everything related to that operation. Is the tech certified, are the right parts there, is the right work authorization document there? Are the proper lock-out, tag-out things in place, is it FOD-free? So they look at an entire list of things. And the goal is to have the answer to be yes, to 100 percent of what's on there. And that's measuring the percentage of everything being correct, and how that's gone up over time. And then we also do inspection, of course, of our subcontractor hardware as it come in. And that shows how that's improved over time. Next chart. I've included our industrial safety performance data here, which you're familiar with, and I've included it, because I think it's back to this issue of how does the contract terms and conditions influence the behavior of the organization. Industrial safety performance measures employee behavior. It measures the behavior of the employee on the floor. Are they putting on the right protective equipment when they're supposed to? Are they taking the extra time to do lock-out tag-out, so they don't have a risk of getting electric shock? In general, are they paying very close attention to what they're doing? And on all these charts, smaller is better. And if you look at the number of lost time occurrences, it's down to .04 across the organization. This shows that the work force is paying very close attention in performing their work, and similarly in those other measures of industrial safety. Next chart. Another benefit in the SFOC is it that the cost and performance incentives that you've been asking a lot of questions about, motivated stakeholder behavior by USA. And what I mean by stakeholder is, the company felt accountability, responsibility and ownership for the shuttle system. And that was reflected in their behavior in a number of ways. It motivated the company to take a longer-term view. The 10-year horizon on the shuttle is very important to USA. We had a six-year based contract and two, two-year options, and that allowed us to have the perspective to say, I can make investments today, in improving safety, improving the shuttle hardware, and I won't see the benefit of that for three or four years, but that's worth it to me. And I'll give you some examples of that. And that the overall optimization was more important than the year-to-year sub-optimization, because we needed to live with this vehicle, because we were going to be the contractor for this vehicle for a considerable period. I mentioned earlier, when we got into this subject, because of the cost incentive, USA had the discretion to reinvest savings for system improvements. I put re-invest in quotes, because this is NASA money, but it's money that we have saved, and the dollar amount today is $190 million, that we could have declared as savings and then gotten 35 cents on the dollar on. Instead, USA made a decision to change a piece of hardware on the shuttle. In some cases, USA funded hardware changes. Decided to fix, or do an improvement to some ground infrastructure, that could have delayed further, but it was the right thing to do. I talked about the various things in investments in safety training programs, bonuses to employees, new information systems, whatever. Now, why would you make that $190 million investment? Because, if you're the stakeholder, you're going to be operating this for 10 years and making the same kind of investment that anybody would make in order to make sure the system's healthy, and so you're not inheriting problems downstream that are going to cost you money or safety issues. DR. LOGSDON: Russell –. MR. TURNER: – Let me finish. I'm not claiming some new kind of altruism. I'm just saying, because you're a stakeholder, you make those proper, long-term investments. Dr. Logsdon? DR. LOGSDON: At the time of the initiation of SFOC, there was a lot of rhetoric about savings of a billion dollars a year. You say there have been savings declared. What's kind of the sum total over the first six years, of declared savings? MR. TURNER: I'll answer that crisply, but you're going to need a pencil because it's complicated. First, a billion dollars a year was based on the assumption that USA was given, as was originally intended, total prime contractor responsibility, including the external tank, the SSME, and the RSRM. That never happened. It wasn't implemented the way originally intended, so that billion dollars, you know, doesn't track, because we didn't do what was recommended. To talk about what USA saved, I'm going to give you a set of numbers. Because you have to talk about saved, relative to what. So the original base line for savings was the NASA POP, the Program Operating Plan, that showed what it was gonna cost to operate the shuttle without a prime contract. Then a group of smart folks got together and said, what should a prime contractor – we call that the A-line, the starting point. What should a prime contractor save? If you did SFOC, how much money should it save? And they came up with a number. It was a couple of hundred million dollars less than the A-line. Then they got into negotiations with USA and said listen, since we're having to sole source this to you, we're going to give you a big cost challenge, to make sure that the government is getting best value. I think that was another $388 million, they negotiated out, which we call the C-line. And then USA's performance is saving another couple of –. DR. LOGSDON: – Is that per year or over a period –? MR. TURNER: – Total six years. I'm answering you six years. And then, USA has saved another two to 240 million dollars below that C-line. And then the government has tracked some savings in terms of less oversight than was required on their part. The number that we have agreed with NASA, is the total of the difference between the first six year's performance and the A-line pop (?), was about $1 billion. Okay. And last item on stakeholder behavior, is the result of us at USA saying hey, we're really accountable for this thing. We formed a counsel that had never existed before, with all the other contractors, not only the SSME and RSRM and ET, but also with a next-tier contractors that were supporting the major subsystems, to discuss things like quality issues, how do we maintain a healthy supply chain, the common problems that all of us were facing in terms of supporting the shuttle. Next chart. Now, having said there were some good contractual features that supported the intent, there were some contractual features that were not aligned to NASA's goals and culture. And this gets to the part of the SFOC that didn't work as well. This cost incentive provision really didn't benefit the space shuttle program. When we saved that money, if we declared it as saved, it didn't come back to us to spend on upgrades, you know, improving the workforce, or whatever. That money went somewhere and I wouldn't have visibility, but from the perspective of the shuttle program, it didn't necessarily go to the shuttle program to invest in other items. And so, it made it a two-edged sword. Say the money for the shuttle program's a good thing to be able to talk about, but it ultimately, what it was doing was eroding what the total budget was for the shuttle. You had a misalignment between contract terms and conditions. And I'll come back to that. The performance incentives provisions were not entirely consistent with the manifest priorities. The performance incentives motivated USA to get a vehicle ready on time. It'd be ready to launch. It would be launched successfully, and return safely. But the way the contract was set up, if we knew that a sequence of launches, one, two, three, were coming up, and the vehicle on number one were going to be late, this terms and conditions actually encouraged us to quit working on number one and go work on number two, and it wouldn't be late, so I wouldn't have a waterfall effect. We didn't behave that way, and that circumstance actually occurred after the wiring problem. We worked on them in order. But that was despite the fact that the contract would have encouraged us to do otherwise. Something certainly was changed, going forward. I bragged about the fact that we had these objective forward performance measurements, but they were not used to determine our award fee performance. So the award fee performance was a separate evaluation, and our objective performance measurement system was excellent all along, and our award fee might or might not correlate to that, period-to-period. So, again, my only point here is it's a misalignment between what we were doing on the contract and the way the system actually works. And then we talked about the change in the NASA Center role, that was in the contract, but that wasn't aligned to the NASA culture. That was very difficult for the NASA Centers. It did not want that diminishing a role. And that put a variety of stresses on the system, in terms of well then, what is their role and what's their accountability. Next chart. And, in general, NASA was not aligned with its various elements, to the SFOC. We were all on the same page on safety. That's where everybody started. But then there were a lot of differences on what came next. Clearly, the administration wanted savings, as Dr. Logsdon pointed out, the SFOC was sold on the basis of how much money it was going to save safely. And the administration was focused on that. The Office of Space Flight was really much more focused on meeting the manifest. Because if you look what station was all about, you had to look at optimizing the total cost of shuttle and station. And saving money on shuttle didn't necessarily translate into a total cost savings. And so, they were more focused on meet that manifest, and of course, you have to stay within budget. Budget is always a constraint. The Lead Center, when we had it, which was JSC, was very much focused on manifest to meet the budget, but then was also very focused on how can you get the savings spent on upgrades, so that the lead center was very interested in reinvestment in the shuttle. Not reflected necessarily in the levels above the Lead Center. I've already mentioned the Centers wanted to retain contract authority, budget and management accountability, so that put tension on the system. And when we were talking about, bringing, for example, the external tank and other elements in the USA, Marshall Center, as an example, is very concerned about how, if that were done, the money would still flow through the Center, so there wouldn't be sort of a loss of their perceived role in the program. The administration for some considerable time promoted that what we call, phase two, which is bringing those other elements under USA, so that the external tank problem would have been a USA problem. But that was resisted almost uniformly by everybody else except the administration. It's happened partially. We moved a few elements, and then it stopped. And then you've heard some about the NASA budget process. The NASA budget process clearly drives short-term thinking. It's an annual focus. An example I'll give you of that is in '98, we ended up laying of 700 USA employees, not because there was a cost incentive on the contract, because there was a national budget shortfall. And then a year later, NASA was recommending to us that we needed to hire more employees into USA. And the difference between those two years was an issue of budget. They had the money one year and they didn't have it the other. And you would like, if you're going to take a system view, you'd like not to be ratcheting year to year, never mind the implication in terms of the number of folks, just think what it does to the organization to go through and have a layoff and then try to rehire and to do that on a cycle. So, that lack of alignment means, despite the great performance SFOC has had, it could have been better if there had been better alliance within the agency. Next chart. Having said that, what would I recommend would be retained as you folks look at making some kind of conclusion about SFOC. Next chart. The biggest thing to me, is the safety and quality issue and that's retaining the reduced organizational complexity. Going back to nine contractors and 28 contracts for a scope of work that really has some very clearly integrated process flow, is a bad idea. You should retain the alignment to process and system. This is process 101. You organize around what process has to get accomplished and you try to minimize interfaces. We recommend you do not re-separate the aligned work content. And a step that needs to go further across the whole shuttle program is to ensure process commonality for core processes. We talked a little bit about supply chain. There will be a supplier out there – a lot of suppliers out there, who provide products and services to all the contractors on the shuttle program. And we all have separate contracts with them. And our contracts have different provisions. And our contracts will have different technical requirements flow downs and different quality systems flow downs. So that contractor is having to behave in four different ways, all of us as shuttle contractors. Now, if this were a active production program for an airplane, that might not be as big a deal. But the volume is very low for these contractors. And it's very hard to keep a viable supply chain with a low volume. And if we're hard to do business with and if we aren't looking at optimizing their performance and reducing their interfaces, we're putting the supply chain at risk. Shuttle program needs a single, integrated supply chain activity that unifies the support that these sub-tier suppliers are providing. And I encourage some of that has gone on that needs to be retained and expanded. I won't go into as much detail, but it's the same thing in terms of quality system, non-conformance and problem reporting and corrective action. Those should be integrated systems across the shuttle program. They aren't. They're complicated with handoffs now. I think you folks yourself have looked into the bracket issue. And those kind of systems ought to be shuttle systems and applied to this reduced complexity. And for each company, there ought to be single contract instruments for the work scope, again, so there's a unified set of requirements, unified technical flow down, and supports unified requirements. So, keep it simple. Organization will create – complexity creates an upper limit for shuttle safety and quality performance. Please don't recommend 29 handoffs in order to get a task done. Next chart. I think the increased contractor accountability has been very successful. I know you're gonna have thoughts on that, whether it went too far, but please, when you look at that, we do not want to go back to the level of effort where the contractor actually doesn't feel – the employed, individual employee turning the wrench or doing the calculation, doesn't feel completely accountable, because it's what's called a government accountable function. We still have those today. About 10 percent of my work force is government accountable. That's what it's called. Now what does it mean to the employee? It means the government's calling the shots. If it's right or wrong, it's the government's call. We want these very bright people bringing their minds to work. And the contractor accountability makes it clear to each of those folks, they're accountable for the technical correctness of their products. So, there needs to be clear, unambiguous contractor accountability, even if you shift what some of this oversight insight is. This gets, I think, to a question one of you asked that I asked to defer. I think it was maybe you, Dr. Logsdon. The terms and conditions of the contract should be aligned to NASA's goals and priorities. So, if you lay out clearly that NASA's goals and priorities are around safety, meet the manifest, ensure supportability, and improve the system, then yes, redesign the terms and conditions in a way that support that. Now don't assume that a cost incentive is bad. But, the contractor wants terms and conditions that are aligned to the government's priorities. That's the best contract you can have, is one that when you perform well, the customer's happy. The two of you are in alignment. And so, to the extent this contract wasn't aligned – and I had a chart on that, either NASA needs to align itself around the goals and objectives, or you need to change the contract in order to reflect that. This reinvestment that we were able to do under the SFOC was a very positive thing. So, if you eliminate the cost incentives and pull out our contracts people, you need to figure out a provision that allows the contractor to continue to have this accountability for doing the best thing to keep the program going, that doesn't get you into a continual negotiation with the government, which ultimately slows the thing down and prevents progress. I mentioned align the terms and conditions to goals and priorities of NASA, but also make sure that they reward excellent performance. You do want the contractor motivated to really do a great job. And John, I think you asked about fixed fee – I'm sorry, General Barry. Make sure that you don't have in place something in which the contractor's range of achievement is too – is a narrow band. You want the band to be broad. In SFOC the brand was very broad. If we did very, very well, we could do reasonably well. Not as well as a development program, but well for an operations contract. And if we didn't do well, we not only could not earn anything, we could end up owing money. So, you do want to keep the contractor motivated by having enough differentiation around performance, that they put the kind of effort that we did, into performing well. And I hope you continue the emphasis on objective measures of performance. In my career, I had the opportunity to work on DOD programs. And really this SFOC change was moving more in the direction of the way the DOD operates programs, and using CPARS, having really clearly defined what is goodness, how we're going to measure that and having both parties agree to that and then actually track and reward relative to those objective measures, is very positive for the system. It's hard work, because you have to know what is good. But you ought to know what's good going in. Next chart. So, to finish up. After working on it for – at the very beginning, and then for five years, it was the next logical step in changing how NASA operated the shuttle. And for the goal towards improving contractor performance, contractor performance improved under the SFOC, whether it was optimum or not, it definitely improved, relative to prior arrangements, and it improved in the important areas; safety, quality, and in cost. I talked about the key structural and accountability features that ought to be retained. You notice I didn't give you that long a list of what ought to be retained. There's lots of degrees of freedom to change this thing, if you retain those core elements. And then finally there really needs to be an alignment process. And I've got to compliment Sean O'Keefe. He caught on to this alignment issue early on, and he has this one NASA initiative going inside NASA right now, that is doing exactly what I'm recommending here. Which is, getting NASA more culturally aligned to be a single organization. If he's successful in that, that will really help to address the fine line. ADMIRAL GEHMAN: Thank you very much. Mr. Young. MR. YOUNG: Mr. Chairman, I do not have any charts. I have a couple of comments I would like to make, that were stimulated largely from the previous discussion. And it really has on the role of government and the role of the contractor and insight versus oversight. I have enormous respect for the capabilities of the Aerospace industry. I have enormous respect for the difficulties of space flight, whether it's human or automated. And I am a firm believer that the government has a significant value-added function involved in the execution of space programs. And let me see if I can say a little bit about what I'm trying to imply there. First thing, if you'll allow me the discussion of oversight, versus insight for space activities. If I could figure out how to remove that from the space dictionary, I would do so. It applies to defense electronics. It applies, probably, to the Hum-V. It probably applies to some of NASA facilities activities, but in my view, not to space flight. And the reason I say that is, that space flight, again, whether it's automated or whether it's human involved, is really a one strike and youÔre out business. And there are not many things in the world that are that way. But, it really is a circumstance where thousands of people can do things perfectly, and one well-intended individual can make a human mistake, be it in workmanship or judgment or analysis. And so, in my view, the reason that we achieve the degree of success that we achieve, is because we have a system of independent verification. And independent verification in my mind, starts, the best is testing and that is, if you can test as you fly and then fly as you test, things usually do well. In some instances you can't do that, like some software systems. And in that instance, I believe that the technique of IV and D is an important element. And in some instances, you can do neither. Such as for the rocket engine, or solid rocket, which is – you can verify the design, you can do analysis, but you – many of the functions you can't really verify on the system. And in that regard, I think the technique that we use is inspection. And that is that somebody watches what somebody else does. So, what I'm really coming around to is, I believe that the government, NASA in this instance – let me say it a little bit differently. I am with Russ. I am an advocate for the contractor having the accountability. I don't know the right choice of the words. Maybe what I'm saying is that NASA has – maybe the contractor has the accountability, but NASA has the responsibility. And what I really mean by that, is that I don't believe that the government is simply in a role of funding and properly executing contracts. I'm a little bit where I heard you describe or ask a question. I do think that there is a government overhead – and I don't mean that in its classical sense, involvement in these programs, because it's a one strike and you're out kind of business. I think that there are every day activities that go on in the execution of a space program, where the government really has to play a significant role. NASA has enormous capabilities at its research centers and those research centers should be used, where they can effectively look into problems that are so complex, that they should not be trusted to a single string kind of a solution. So, I don't know whether I've helped on that, but my observation is that I said space is unique. One strike and you're out. And that says you've really got to have the best of the government and the best of industry, to have a high probability of these succeeding. And I do not believe that the government can pass that on to industry, though I have an enormous respect for the capability of industry. The only observation I would make during some of the review process is that you referenced is, we made some significant changes in the way we do business throughout the acquisition process in the Ô90s. And for a lot of reasons that we could spend a lot of time on. Much of it was moving things from the direction of the government to industry. And I, for one, believe that we went well too far in that regard. That there are functions that only the industry can do and there are functions that only government should do, and we need to work hard to have that common balance and to assure ourselves that we respect the risk associated with space flight, the risk of the human – single human mistake that's going to happen no matter what we do. That can be mission catastrophic, that we've got to have a safety net under that process that minimizes – it won't eliminate, but minimizes those occurrences. I'll stop at that point and be delighted to answer any questions that you might have. ADMIRAL GEHMAN: Thank you very much for your views here. I'll ask the first question and then I'm sure my colleagues are ready to jump in here. You use the term independent verification. As one of the things we might call the government's value-added or the government's proper function. And you said that an example, for example, where it starts would be with testing, for example. And I assume you mean independent, objective testing of things, provided by contractors or suppliers or something like that? MR. YOUNG: Well, I really mean both the – I don't mean to exclude the contractor. I think that the best form of independent verification for industry and government, is a quality test program. And all I'm really saying is that I think it – my observations would be that when we test as we fly and then we fly as we test – if I could use that kind of clichŽ, we maximize the probability of things succeeding. And when we operate outside of that window, again, in my view, we're not taking risks, we're gambling. Because you don't understand how a system is going to perform that hasn't been tested. So, I didn't mean to imply that the government has to duplicate industry's testing. I mean, I have a lot of respect, as I said, for industry's ability to do that, but the test program does have to be as complete as it can be. And when you're operating in a mode where it's not complete, then I said, I think you've moved into the realm of gambling, as opposed to taking risks. Simply because you don't know. ADMIRAL GEHMAN: Right. Thank you for that. And you just used another term that I'd like you to qualify a little bit more and that's understanding. The Board is attempting to agree among itself as to how much of shuttle missions is test or developmental flight – how much is exploration and how much is running a trucking line. And there's – obviously, there's some of each in here. But, we hear and a matter of fact, Mr. Turner used the phrase already – we hear as we talk to people, oh we understand all this. In other words, we're flying in a regime and we repeat the same regime every time and we understand this. And yet, we keep getting bitten by things we ÒunderstandÓ. So, my question is, would you put some kind of a value statement on what is the government's role in constantly attempting to find the unknown unknowns or to really understand things that are happening? MR. YOUNG: Yes, let me, if I might, a couple of comments. And you folded two or three things in there that I probably shouldn't go back to, but I was intrigued hearing twice, General Barry and now you ask, you know, is the shuttle operational. And to be honest, I never quite thought about it that way. So, I was sitting here as you were talking to Russ, thinking about it. Within the context of space, I'd say the shuttle is operational. However, I'd say there's nothing that's involved in space that's operational, as we define it. And I don't know whether that means anything or whether I'm being redundant. But I mean, I don't think we do anything in space today that you can clarify by what is our traditional definition of operational, as being operational. But the shuttle, relatively fits into that, but to the understanding. I, you know, again, I think that there is a constant issue with, you know, with trying to understand the circumstances associated in the systems that we're involved with. So, I don't know if I'm exactly answering your question, but I don't think we, with any space system, and particularly shuttle, we have not reached the point that which we should say we understand it, if that's the answer that warrants your question. I think we should reach the point of saying that we have got to constantly be worried about those items that, you know, can creep into the system that can be mission catastrophic. We probably understand, you know, the performance or the design, but that's a lot different than understanding the performance of the total system, which heavily involves the people who are operation involved with the system. ADMIRAL GEHMAN: Well then, how would you – in your – how would you formulate a government program, which would enable us to – which would enable this program to be inquisitive or dubiously curious? How would you fund – does personal if you think NASA has a robust, rich program, and where does the money come from to kind of ask those tough questions and to understand what's happening? MR. YOUNG: I don't know if I can answer the latter question. Now, let me go back to the question you lead in with. Again, personal observation, I think in the kinds of things that we're mostly involved in with space, be it shuttle or be it Titan or be it a Delta or be it a Mars vehicle, I think that they benefit enormously by what I'm loosely going to call a third set of eyes. And what I mean by a third set of eyes, having watched it practiced and watched it not practiced, I'm not convinced it's a lot of money. I mean, so maybe we're not on the same wavelength there. But, I think that again, the ways of implementing a space program – and Russ went through a lot of it, as I mentioned is, there's got to be the mainstream activity and then there's got to be the independent verification. I'm an advocate of, I don't know what – I'm going to call it a small group. A group of people involved in programs, that don't have a responsibility. And if again, if I could say a little bit more about, you know, what I mean by that – if I go to kind of some personal experience. I know with launch vehicles, you know, I have the experience of having half a dozen or so people, who, you know, were at the Cape. And what they do is they walk around and they talk to people and the – I'll call it the less experienced engineers had the opportunity to go and say look, I saw some troubling aspects in the test we ran last night, and I don't quite understand it, but the data didn't look like it looked like to me, the last three times we've had a vehicle down here processing. It's somebody for them to go talk to. That's this third set of eyes that I'm talking about. My observation is that's not big money. I mean, that's having a few people who have a lot of experience, and who have an intuition for, you know, for functioning in this capacity. That as I said, don't have A-line responsibility, but they probably work, you know, two shifts a day. But they're not filling out forms. They're not running a test. They're available to follow their intellectual and safety curiosity and be responsive to the mass of people who do have these formalized jobs. ADMIRAL GEHMAN: Thank you. Dr. Osheroff. DR. OSHEROFF: Well, I guess I'm, I mean what you're suggesting, I think, is a great thing to do. And I suspect that many organizations, not just Aerospace organizations, should have people that fill this role. But, let's say that something looks wrong when the test gets run. Don't you need a research organization then that can understand this? MR. YOUNG: Yeah. I think the answer to that is yes. And again, if I extrapolate and maybe follow what I think you're asking, I do believe that, you know, this, what I'm going to call third set of eyes, and maybe that's not a good choice of words, but this third set of eyes, I think if they cannot satisfy themselves, I think they would be – I don't mean uniquely. Everybody should view it. But I think they would go stimulate the research organization. My answer is yes, you need the research organization, you know, and NASA has that. I mean, NASA has extraordinary capabilities to go analyze a broad spectrum of problems. You know, much of it is utilized at the Center, such as Langley or Ames or Glenn, you know, not the Space Flight Centers. Not that they don't also have capabilities. So yeah, I am with you in that regard, but I think that what I'm trying to suggest is like the third set of eyes. And I don't want to diminish the responsibility of the program. But I think that, you know, it's a technique for pushing things to root cause maybe. Which is what you're talking about. Which say that you have to turn on a larger group of people. You know, but I again, my observation is that NASA has that capability to do it. DR. OSHEROFF: When you use the term, which I've heard before, test as you fly and fly as you test, that sort of suggest to me that we may – we need to be looking more at the orbiter and in fact, the shuttle system, on particularly launch and return, instrumenting it the way it was instrumented back in the early days, in order to – I mean, I find it amazing that the only ÒdataÓ that existed up until very recent, of the temperature profiles through the foam and in through the bipod ramp, was in fact, a calculation, it wasn't actual data. And so, you can't do that as Langley, of course, it's going to have to involve USA as well, I would guess. MR. YOUNG: Certainly. ADMIRAL GEHMAN: Go ahead, John. MR. LOGSDON: Tom, one of your engagements that hasn't been mentioned is that you chaired the Space Flight Advisory Committee for NASA for some period of time, looking at the whole Human Space Flight activity. And you talked earlier about the need for a good safety net. This is kind of a two-part question. One is, do you have some comments about the current state of that within NASA? And second, you talk about the third eyes as being not very expensive. And what about the function performed for DOD by the Aerospace Corporation? Is that just an extreme version of third eyes? MR. YOUNG: Let me answer the last one and then I'll come back to the Space Flight Advisory Committee. I think Aerospace provides a third set of eyes, but I think they also – my observation of Aerospace is that they augment the DOD capability in assuring that appropriate programmatic reviews are done and assuring that, kind of the right, what I'm using, this independent verification take place. So, I would say part of – they do have – they do, do the third set of eyes, but I'd say most of their activity is probably – I didn't mean to get into this terminology, but it's in the second set of eyes role. Let me come to the Space Flight Advisory Committee. And I need to probably should give just a tad of background into the group. NASA set up, a long time ago, a International Space Station Advisory group. Actually, Academy Engineers did it first, and then it moved to NASA because it was more operational. And I chaired that for several years. The Associate Administrator for Office of Space Flight, about mid 2000, recognized that they did not have an advisory group, as much of the other enterprises in NASA do, such as Space Science or Earth Science. So, he said – Joe Rothenberg is who I'm speaking of, by the way. Joe said, you know I think we should take the Space Station Advisory Group and expand it and – Space Station – and expand it or all of the Office of Space Flight. And would, asked if I would chair it, which I did. And that was done in mid 2000. We operated, effectively for a year, then we kind of stood down while we did the Space Station Review that I talked about earlier. And then we came back and we had another meeting in mid '02 and then NASA abolished the Committee. So it's not currently existence. We really looked at – we spent a bulk of our time – now we looked across everything that was done in the Office of Space Flight, but we did spend time on shuttle. And we really spent a fair amount of time on two issues. One was upgrades. And the second was infrastructure. That was discussed. And then a little bit on the next stage of privatization that Russ mentioned. And I don't know, if you would like, I'd say a little bit about the upgrade process, because at least that illustrates to me, a little bit of the style of operating. Would that be of value to do? ADMIRAL GEHMAN: Yes, go ahead. MR. YOUNG: What we did – there's been various stages of upgrades. The first observation I would make, a little bit following Marcia's discussion, is that NASA's interaction with OMB is, in my view, had a stronger influence on the budget activities than the interaction with the Congress. And a lot of policy is really established by OMB, in that budget process. But NASA, finally was able to get some upgrade funding in mid-2000, or I guess probably it was FY 2000. I think it was $1.6 billion at that time. And OMB required there be an independent look at it, and we did that. So, we spent a fair amount of time on that issue. And if I would – some observations from that, probably irrelevant to what you're doing, when we first looked at it, this was led by Johnson Space Center, though all the Human Flight Centers had been involved, and basically what was presented to us was a collection of items that were: improve the cockpit, an electric APU, an APU for thrust vector control for the solids, and instrumentation for the engines, and then a few smaller items. The problem we had as a group with that was that is was really – it was budget driven, clearly. And it was also – let me make sure I use the right words. It was probabilistic risk assessment driven. And so, what NASA had done, is they basically took – and they have an extensive PRA system for shuttle, and they had looked at it and they had kind of taken those items that had the highest probability of making a positive improvement in the shuttle, that fit within the budget. Okay, our criticism of it was, forget the budget. Show us the list of all of the items. You know, what would have the biggest impact and probably the biggest at the top would be a crew escape capability. Clearly bigger than $1.6 billion, and maybe not a practical item, but what we were interested in was show us the list of items, kind of from top to bottom, and then after we understand that, then you do have to apply the budget restrictions, but make sure everybody knows that the upgrades are not going from the top priority. They're going from what you can afford. We had a terrible time ever making that happen. And I'd say, probably never did. The second thing that we were concerned about in that process was that – a little bit going back to the earlier discussion, my personal view and I think our group's view, was that for mature systems, things again, often fail because of the human interaction. And that's either because of a workmanship error, judgment error, an analysis error. The PRA doesn't have any of that in it, you know. So, we kept trying to say, you know, aren't there items that should be in a shuttle upgrade that are not PRA hardware related? I don't mean – I think PRA is good and the hardware is good. But aren't there items that you could change in shuttle that make it more testable, you know, between flights? Make it easier. And Russ would know a lot more about this than I, to refurbish. My understanding is, as an example, to refurbish the hydraulics system or do something to the hydraulics system, you have to remove the engines, you know. Is there a way that you could, you know, make some changes that you don't have to remove the engines to do that, to minimize even a – we were never able to get that into the process. So I'm sided, a little bit of, you know, of a way of operating that, you know, was – there was no question this process was budget driven. Even though I think the items were good that they had. And I think it's no question that what the scope of what was being considered was not as broad as it could be, to assure that whatever funding we had, we made the maximum impact on shuttle. I don't know whether that was useful or not, John. But it's background. ADMIRAL GEHMAN: Anybody? Nobody else? Yeah, go ahead, Dr. Ride. DR. RIDE: I'll just ask one, you know, in one of your many oversight roles, you spent quite a bit of time reviewing the space station and implementation of the station and I presume, at some level, its interaction with the space shuttle. And the two have really become very closely intertwined over the last few years, so that it's very difficult to evaluate the shuttle program without talking about the station and the station's effect on it. And I just wonder if you could maybe reflect a little bit on the stresses – not so much the budget stresses, because I think we've heard a little bit about that. Marcia touched on that a little bit. But, maybe some of the other stresses that the station and the station manifest, for example, or station requirements might have placed on the shuttle program. And Russ, you can have a view on that too. MR. YOUNG: Well, two or three items. I mean, there's no question about what, you know, the fact if we look forward, I guess, all of the anticipated missions are, as we can see a manifest, for the shuttle, is to support the station, with the exception of two. And one is Hubbell upgrade and two is a Hubbell return mission. So, as you said, they are, you know, highly related. First off, if I just go back and just add something to the budget, even though you kind of excluded it. You know, I don't think there's any question, if you look at – I don't have the visibility that Marcia did. I was really impressed listening to all her activity on some of the budget numbers. But if you look at the station and shuttle together, you know, the combination was inadequately funded. And you know, one measure is that we take station from FY Ô94 to – you take the first six years, I guess, from FY Ô04 to FY – from Ô04 to – from Ô94 to Ô02, I'll get my numbers right. The station actually moved to the right four and a half years, you know. So, that is a measure of, you know, a program that was significantly under funded. So that circumstance existed. There were continuing the station requirements largely for supplies, grew. And so shuttle missions had to be added in that regard. But, I think – and again, Russ would be more knowledgeable than I, but I think that demands of shuttle flights per year, you know, were perfectly reasonable. You know, I don't think they were at all out of bounds. And I don't know that the station demands on shuttle moved it into an area of concern. I would say it was probably – they handled that in quite a responsible way. MR. TURNER: Yeah, I would agree. Actually, as an operator of the system, it's better when we're flying. It's better for the workforce when we're flying. It's better for the processes and the systems when we're using them. And we were comfortable at that higher flight rate and enjoyed flying at that higher flight rate. Given that we have the adequate resources available, it's actually not putting stress on the system. On the contrary, it's keeping the system well-oiled and working well. So, we were comfortable with that. You may not know, Dr. Ride, actually the part of it we all worried about was the EVA component of it, which was a huge load on the astronaut corps. But in terms of operating the shuttle, we were comfortable with that and would be happy flying at that rate. I have a comment about the test program, as long as I've started talking again. I agree with Tom. And in fact, as you look, going forward, I believe Dr. Li brought up this issue, how long are you gonna fly the shuttle? And so, we've had this rolling period, well, we're going to stop flying it in a little while, for the last six years. Which has prevented us, I believe, from making the longer term decision that I talked about when I was talking about being a stake holder. And one of those decisions is robust test program. And I don't mean just in submitting the vehicle. I mean, why don't we have a ground test program? The SSME has a pretty good ground test program. The solid rocket motors is a pretty good ground test program, where they're getting new data about new options by testing it on the ground. But what about all the systems on the orbiter? We talked about the cracks in the flow liner. Why isn't there a main propulsion system test article, where we're really thrashing it out and having it be the fleet leader instead of the vehicle that flies the most to fleet leader. Well, when you don't know how much longer you're going to fly the vehicle, you probably don't make that investment. One of the things that ought to come out of this is, we should be doing more testing. And then that ties into your question about then, research and development activity that can go on and parallel with that within NASA, which is absolutely their role. To get the data from that, use it for helping to make the shuttle more robust that folks are missing here is if we don't get the learning out of the shuttle into the next RLV, it's gonna be different but not better. And so, why wouldn't it have cracks in the flow liners? So, I do think that a lot more can be done with testing and the result of that testing will be a much more robust shuttle program. ADMIRAL GEHMAN: I'll ask the last question. We thank you very much for your patience. Mr. Young, again, I wrote down something you said here about the Center should back up the programs – the Centers should be able to back up the programs. One of the things that this Board has done, in order to understand how high reliability organizations attempt to achieve high reliability, there's a couple of characteristics which we have found in common among them. And one of those characteristics is independent verification. We mentioned for example, in the case of the Department of Defense, almost all of the launch vehicles are contracted for, but then they pay Aerospace to do independent verification – second set of eyes. We find that in other industries too. I won't mention them but we have found a number of cases in which there is a very, very strong, independently funded, not in the program, independently funded –. MR. YOUNG: – Critically important. ADMIRAL GEHMAN: Set of eyes, who don't care anything about schedules, don't have any interest in budgets, and they independently verify what's happening. And every time they have an itch or a scratch, they have the funds to go look at it. And they are not stealing money from the program and they are not slowing the schedule down, unless they raise their hand and say, wait a minute, cease and desist. My understanding of the process that we have here is that that second set of eyes – the first set of eyes is the contractor. The second set of eyes, NASA has decided that they ought to be second set of eyes, because there's nobody else doing it. But what I'm concerned about is that my understanding of the way the process works is that if you go to any one of these Centers, particularly Human Space Flight Centers, that the engineering sections that work on shuttle are funded by the program. And I'm wondering whether or not you agree with me or whether or not you would describe it that way, that what we should really look for, if we believe these characteristics that I talk about, they ought to be independent, independently funded set of people who don't care about the schedule, don't care about the budget, that whatever this entity is, you either hire it or contract for it. You ought to get Aerospace to do it for you, you ought to get somebody else to do it for you, or it ought to be, if you want it to be in the Centers, it can't be funded through the program. It's got to be funded in some other way and whether or not the full cost accounting issues are an issue that we see, well now we've got something butting heads here. Do we have two concepts butting heads here? MR. YOUNG: Let me see if I can really respond to your question. It's my belief, and I might not be close enough to it. But it's my belief right now, if one of the programmatic Centers or Space Flight Centers called up Ames today and said we've got a real issue. You know, you understand plasma flow problems, you know, better than any of our people. I think today, Ames could, you know, turn forth, put a group of people on that and go work that problem. And maybe I'm mistaken. So, as I said, because I'm, you know, a little removed, but I don't think that would be a budget item. And I think that kind of capability. So, I think NASA has a lot of that capability. Now, it does require – one of the things that you're maybe talking about is that the scenario I just went through required somebody to call up and ask. And that's an important consideration. But, and so maybe this third set of eyes I'm talking about, you know, in NASA's instance, maybe has to be broader in that regard. Because I don't know that Ames would recognize the need for that, unless, you know, unless they'd been asked. So, I'm stumbling through, but my experience, which again, may not be today correct, but I think it probably is, is it's not so much a budget issue, as it is whether or not the Ames of this regard would know about the issue. So, if you're trying to say we need to worry about how do we couple this, what I think is enormous capability that NASA has, more closely that their curiosity can be stimulated, as well as the program's, that's probably a valid thing to, you know, thing to pursue. I think the other item – and I'm trying to figure out how to say this without it coming across adversely. My observations – and I want to emphasize, my observations, is that the Human Space Flight area doesn't ask for help very often. And you know, if you interpret that as, you know, being a critical comment, I meant it, you know, that way. I'm not trying to over-emphasize. But again, my observation is that there is a Human Space Flight culture, you know, that exists, is you don't get a lot of – and you all probably have the statistics, which maybe prove me wrong, but my observation is, do not frequently ask for help. Which maybe again, comes back to the importance of what you're saying, is that maybe this third set of eyes has to have some relationship beyond Human Space Flight or that that culture has to be somewhat changed. ADMIRAL GEHMAN: Well, gentlemen, thank you very much on behalf of the entire Board. We could do this for hours, because your depth of knowledge is really impressive and your willingness to share it with us is deeply appreciated. I want you to know that. I wish we could can it and take it with us. But, it's been extraordinarily helpful to us, as I expected it would be. And we all certainly share the same goal here and that's to learn from this tragedy as much as we can, and get back to flying again. And we thank you very much. And we have a short break here and we're going to re-set up for a 1:00 press conference right here. Thank you very much. END Back to Event