Columbia Accident Investigation Board Public Hearing Monday, April 7, 2003 1:00 p.m. Hilton Hotel 3000 NASA Road 1 Houston, Texas Board Members Present: Admiral Hal Gehman Major General Ken Hess Brigadier General Duane Deal Dr. Sally Ride Dr. John Logsdon Mr. G. Scott Hubbard Mr. Steven Wallace Witnesses Testifying: Col. James Halsell, Jr. Mr. Robert Castle, Jr. Mr. J. Scott Sparks Mr. Lee Foster ADM. GEHMAN: Good afternoon, ladies and gentlemen. This public hearing of the Columbia Accident Investigation Board is in session. We're privileged to have with us today two experts to help us see our way through some of the issues that we have to deal with, and we're going to deal with the treatment of anomalies and waivers and certifications and all that sort of stuff today. We have a panel of two -- I don't know if you'd call them experts or not; we'll see at the end of the day whether they're experts or not -- but to help guide us through the first part of this process. The first is Colonel James Halsell, who is an astronaut and has a couple of duties, one of which is, I presume, to command a mission here in the future, I trust; and Robert Castle, who is from the Mission Operations Directorate. Gentlemen, before we begin, let me ask you to first to affirm that the information you provide the board today will be accurate and complete, to the best of your current knowledge and belief. THE WITNESSES: I do affirm. JAMES HALSELL and ROBERT CASTLE, JR. testified as follows: ADM. GEHMAN: Would either one of you start and introduce yourselves and tell us a little bit about your background but also tell us what your duties are today. COL. HALSELL: Okay. I'll start first, sir. It's my privilege to be here to have the opportunity to work toward what certainly anybody at NASA considers to be one of the most important things we'll ever do in our career -- that is, to find out what happened, to fix it, and get back to flying safely. I have a background in the Air Force. I'm an active duty Colonel in the Air Force. My background in aviation was fighter aviation, followed by test aviation, and then an assignment to NASA for the last 13 years, since 1990 as an astronaut. I had the privilege of flying five missions; and at the conclusion of my fifth mission, I was asked to take on, as a career-broadening experience, a management job down at the Kennedy Space Center as a launch integration manager, working directly for the program manager, Mr. Ron Dittemore. I did that from the summer of 2000 until January of this year, when I was relieved of that job in order to take my next assignment, which was to command STS 120, which will be a mission to the International Space Station, taking up Node 2, one of the hardware components that will complete the American initial phase of the construction of the station. If you'd like, at this point in time I can talk to you -- ADM. GEHMAN: Before we do, let me ask. Do you also have a role in the return-to-flight process? COL. HALSELL: Yes, sir. I received word just two weeks ago that I would be requested to head up a return-to-flight planning team. We would be doing a staff planning function, reporting directly to the deputy associate administrator for station and shuttle. That's Retired General Michael Kostelnik. Our job is to be his interface to the shuttle program and, in fact, throughout the NASA system working this issue, to come forward with recommendations and options in response to the Columbia Accident Investigation Board's findings and recommendations. So the way it should work is that once your investigation board wraps up with a report, and hopefully even in the interim before that final phase, we'll have the opportunity to map out a response to your investigation board's findings and recommendations. I'm sure that we'll come down to a set of options that we'll offer up to our leadership and our management and they will make some of the tough choices that have to be made with regard to what has to be done to fly safely again, what needs to be done in the long term to make the system even safer. ADM. GEHMAN: Let's let Mr. Castle introduce himself, and then you can start. MR. CASTLE: Okay. I'm very honored to be here and take part in this, in the return-to-flight effort for the Columbia. A little bit about myself. I'm a full-time career civil servant. I've been working for NASA for 25 years now. I started working one of the mission control sections as a communications officer, did that for about ten years, and then was a mid-level manager for about a year and then was selected for the flight director office in 1988. So I've spent right at 15 years as a NASA flight director, running missions in Mission Control. I have recently left that job to become the Missions Operations Directorate chief engineer and currently working on things like orbital space plane and some upgrades in the control center as well as contributing work on the International Space Station. I should also say I was a shuttle flight director for virtually all of that time. The last two years or so, I've switched over and become mainly a flight director on the International Space Station. That started around the middle of the year 2000 was when I did that much more than I did shuttle flights. So that's my current role to date. ADM. GEHMAN: Thank you very much. Colonel Halsell, if you have a statement or perhaps a presentation, we're ready to listen. COL. HALSELL: Yes, sir. I did come prepared with a presentation package. Certainly I would expect -- and feel free, as I'm sure you will, to ask me questions as we go along in this somewhat lengthy package. It's my understanding that I've been asked here today to give you any information that I might provide with the preflight process. In the shuttle program we call it the Flight Preparation Process, FPP for short. So if I use that acronym, that will be what I'm talking about. And that is the all-encompassing phrase, if you will, for everything that we do to get ready to go fly safely, including a subpart of that is the Certification of Flight Readiness and all the reviews and boards that we go through for that. Before I launch off into the details, it might be helpful if we just review the basics. The basics are basically this. The way the shuttle program is set up -- and I believe correctly and appropriately so -- is we have a set of requirements. It is huge, long list of requirements. It's broken down by the projects and the elements and all the contributing manufacturers, but the space shuttle program is responsible to be the keeper of the list of requirements. It tells us how we're going to build a component, how we're going to use it. It tells us how we train the crews. It tells us how we prepare the vehicles. Everything we do answers back to a requirement; and before we go launch a shuttle mission, it's absolutely required that we know we have lived up to and, in a closed-loop accounting fashion, answered each and every one of those requirements successfully. In a perfect world, you would have your requirements on one hand and before we go to launch, you'd have absolute and utter proof that you met each and every one of your requirements. We do live in that perfect world except there is such a thing as a waiver, in the sense that oftentimes if you can't meet the intent, indeed, the scripture of a requirement, then you have to come forward to the program, and specifically the program manager, and make the case for what you are offering instead is sufficient to allow a complete productive and safe mission. If you can pass that test, then with the waiver we are allowed to go ahead and fly. So it's requirements, closed-loop accounting system, and to the degree to which they don't match up perfectly, we enter into the waiver process. That's the 37,000-foot view of what we do, and almost everything that we talk about from this point on could be tied back to that very simple basic process. I know that after Challenger, it was recognized that these processes were not as disciplined and as rigorous as they should be; and I believe what I hope to tell you today and what I hope comes out of my presentation is that following the Challenger disaster, we went back and did rigorously enforce that discipline. In the degree to which we fell short in the Columbia accident, that's why we're here today and that's what we want to find out. I think it might be helpful just to lay out a couple of other basic thoughts. The shuttle was designed with the philosophy that you should not have a system in which you suffer a failure and you lose your vehicle or your crew. It needs to be fail-safe. Furthermore there was a high operational desire to be fail-operational -- that is, suffer a failure and still complete the mission. The basic requirements are that the vehicle and all of its subsystems will be fail-safe. From the very beginning, there were three of the systems which it was acknowledged we could not achieve that desired goal. The thermal protection system was one. It was recognized as being a Criticality 1 -- that is, if it doesn't work, you're going to lose the vehicle and/or the crew and we don't have a backup system to it. Pressure vessels, whether it's the pressure vessel in which the crew resides or the pressure vessels which holds our fuels and our oxidizers and our cryogens, was another. And finally the primary structure of the vehicle. The vehicle was not built with the intent that you could lose anyone -- you could always guarantee that you could lose one primary load-bearing piece of the structure and still maintain your safety margins. So those are the three areas where the design of the vehicle, it was acknowledged, would not live up to the basic requirement of being fail-safe. On the other hand, in the area of avionics, they designed it with a higher than fail-safe, that is, a fail-operational requirement. In our avionics area, it was designed to be able to suffer any one failure and continue to nominal end of mission. Those are my opening thoughts and maybe background that might help you as we delve down into the flight preparation process in detail. So with that, if I can press on to the next slide, please. This is a flow chart that shows you the program level reviews. Each of these represents a review, a large meeting of all the relevant NASA and contractor personnel; and it's also just a program level. Below each of these program level reviews is a vast array of project level reviews, but let me just briefly go through this and it will give you the outline of what we do and how we do it. Starting in the upper left-hand corner, the Flight Definition Requirements Document. That is the bible that a flight, a mission, in the preparation of a vehicle for that mission, where it all gets laid out. Normally this is presented to the Program Requirements Change Board, which is the program manager's venue for considering these top-level issues, about 16 months prior to flight. You can go from the front of the vehicle to the tail of the vehicle and talk about the level of detail, but basically that first block should be preceded by two or three years of preceding blocks where our customer and flight integration office receives inquiries from our potential customers to understand what payloads they want to fly, what mission requirements they are considering, and that's mapping those against the shuttle capabilities and whether or not we can satisfy those requirements. In a very complete iterative process we go through understanding what do they want to do, what is it that we're able to do, and to the degree that it doesn't match up, let's try to better understand how we might be able to force a match there. When you get to the FDRD, you know the vehicle you're going to fly on, you know the size of the crew, you know how much cryogenic oxygen and hydrogen's going to be on board, because that drives how long the mission can be because, of course, that's breathing oxygen for the crew and that's also what we use to generate electrical power for the payload and for the other systems on board the orbiter. You know exactly what the payload configuration is going to be in the payload bay, down to the keel and the trunnion attachments on the side walls of the vehicle. You know probably the serial numbers of the engines you're going to fly. It baselines everything there is that you really need to start out to do the detailed final preparation for the mission, and that baseline can only be changed from that point on by going back to the Program Requirements Change Board and asking permission. So that's the FDRD, and it's really the first milestone at the program level. The other blocks as we follow along there have names which are fairly self-explanatory of what they do and what we're there to do. The Cargo Integration Review highlights and further refines details with the payload that we're going to be carrying for that mission. The Ascent Flight Design is a program-level review because that is understood to be the most dynamic phase of flight. It's the one where we have to tailor the software the most from flight to flight, given any one of a number of variables, not only the payload you're carrying and the weights involved and the load of propellants that you're going to carry on that particular flight. So we bring that to the program level. The FPSR, the Flight Planning and Storage Review, is the one that's near and dear to most crew members' hearts because that usually happens at about the ten-month-or-so month prior to flight and that's just about the time that the crew has just been named and has started working together as a crew. So that's the first one that the crew normally supports; and the Flight Plan and Storage Review, it really summarizes the issues which are most importance to the crew. The flight plan tells everybody, including the crew, what you're going to be doing every second of every mission; and if you can nail down the flight plan and make it answer back to the requirements of the flight, it's a lot easier on the commander to be able to plan his mission and to plan his training for his crew, which is one of the primary jobs of the commander pre-launch. The other important part is stowage. Living on board the space shuttle and working on board the space shuttle has been likened to a camping trip in a closet in that you have to know exactly where everything is so you can get to it in a timely fashion and you also have to get it back in the right place before you come home. And the degree to which you don't know that or you make it more difficult than it has to be, it directly impacts your ability as a crew to get your work done. So you try very hard after you're first named as a crew to get to the Flight Planning and Storage Review and understand the degree to which we have a high level of fidelity in that planning process, because that's your first clue as to how much work you have in front of you in planning the mission, the details of it. The next three blocks really have to do with the same subject, and that is at the Kennedy Space Center what are they going to have to do after that orbiter lands from its previous mission until you launch it on its upcoming flight. The first block, the Integrated Launch Site Requirements Review, is where you hash out what are the actual requirements. You know you've got to be able to get the payload into the payload bay. What are the requirements before and after and leading up to that event? What are the modifications that you want to do on this vehicle? At any given time in the shuttle program, there is usually a list of modifications which are ready to go to be implemented in any given vehicle, and you have to weigh is now the time to try to insert any of that particular modification to bring the improvements that it does either to the capabilities or to the safety level, or do you have to understand that the manifest at its current state is such that work would be better implemented one flow following this flight. So you make those trade-offs and those kinds of determinations at that time. Then the Kennedy Space Center comes back at the Launch Site Flow Review and they tell you their ability to meet those requirements and that they're going to be able to do it and to the degree that there's a mismatch, we hash it out at that meeting. There's one other meeting, the Delta Launch Site Flow Review. By the way, the timing is 60, , plus 15. That is, it's about two months prior to the landing of the orbiter from its previous mission that you really try to nail down the requirements. It's about one month prior to that landing that you do the flow review and have Kennedy come back and tell you if they are going to be able to accomplish it. After the landing from the previous mission has accomplished and they've been able to roll the vehicle into the processing facility, you understand better the condition and any in-flight anomalies which it had during the previous mission, how that might impact what you had planned to do previously. You bring that back to the program at the Delta Launch Site Flow Review and that's where you make any final determinations and judgments on what we are and are not going to do on this particular flow. If necessary, you adjust the launch dates to meet those requirements. So that's the program level review, starting at 16 months prior, to actually up to two weeks after the landing of that orbiter and you start to process the vehicle. This is what's typically referred to as the flight preparation process. The last block that I'll lead into with the asterisk is called Milestone Reviews, and this is going to be where we now tend toward more of a Certification of Flight Readiness flavor for what we're doing. If I could have the next chart, please. I believe I've talked about all this. So if we could press on to the next chart. The next chart, please. Here we go. Here's the wiring diagram to talk about the milestone reviews and the certification of flight readiness that results from this process. The chart flows from the left to the right. On the left-hand side, you have the different projects and elements, each one responsible for a particular major system on the orbiter. On the far right-hand side, you have our flag -- I'll call it our flagship review, the Flight Readiness Review, which typically happens about two weeks prior to launch, where we present all the information to senior NASA management to determine the final readiness for launch; and everybody's required at that point in time to sign up to the Certificate of Flight Readiness. In between is an incremental improvement at each step in our ability and a refinement in our ability to say, yes, we are headed toward the satisfactory Certification of Flight Readiness. Starting at the left on the project level, their major review would be the Element Acceptance Review. That's where the government project manager will accept from the contractor the piece of hardware. Once again, there's a whole hidden set of pre-reviews that led up to the Element Acceptance Review. I've talked to a number of project managers and I think they'll all tell you it would be totally unacceptable for them to be surprised or to hear an issue at the Element Acceptance Review that they did not previously know about. So it's worked in real time, but we do lead up to the EAR for each major component of the vehicle. Then where I've gotten involved in my job as the launch integration manager are in the two double-bordered boxes that you see there. The ET/SRB Mate Review and the Orbiter Rollout Mate Review. Each of those represents a processing milestone that we want to be very careful and we want to be very studious, if you will, before we go through that milestone, without taking a breath and stopping and pausing and making sure we're ready to go do that. I approach it from the point of view of two aspects. First of all, those mate reviews were my opportunity as the integration manager to actually understand the rationale that was going to be brought forward at the Flight Readiness Review for any of the major waivers, hazards, first-time flight items, changes to processes, in-flight anomalies to be considered up to that point in time. It was my opportunity to hear that in a formal forum and to understanding how they were going to present it to the Flight Readiness Review. Now, let me make it immediately clear that, just as it would have been unsatisfactory for a project manager to come to an Element Acceptance Review that did not know everything that he was going to be told, it would be equally unsatisfactory for me as the launch integration manager to come to a mate review and not know the details of everything that was going to be presented and have had a history of having known the development of all those issues over the prior months. Nevertheless, that's the first time we put it all together in one package. ADM. GEHMAN: Let me interrupt. This is where -- I mean, you mentioned this. I just want to be clear about this. In the Element Acceptance Review, these EARs, as well as at these program reviews, previous waivers and waivers that are currently in existence, disposition of old in-flight anomalies would all be brought up, kicked around the table, and if they had been accepted in the past, the acceptance would be re-agreed? COL. HALSELL: Yes, sir. I believe I understand the intent of the question. There is a requirement both at the project level and at the program level for us to fully understand in-flight anomalies as they apply to that particular piece of hardware and the mission that's about to be flown. There's a requirement to review and understand all the waivers that had been issued and, in particular, concentrate on any change of waivers or any new waivers. If it's a waiver which has previously been approved through the program and through the entire system and there is nothing different about it's applicability or this flight as compared to the previous flights, then it's not necessary that it be brought forward again and again and again; but what is absolutely required is that any new waivers or changes to waivers be highlighted at each of these progressive milestones. ADM. GEHMAN: Just from an administrative point of view, if a system over a period of 20 years is operating under 25 waivers -- which, by the way, that's probably not an outlandish number; it might be more than that in some cases -- how does the system deal with the fact that a waiver's starting to accumulate. COL. HALSELL: I am aware during the time that I was at the Cape that the program approached that exact issue at least on a couple of occasions. Just before I took over as the launch integration manager in the summer of 2000, my immediate predecessor, Mr. Bill Gerstenmaier, under Ron Dittemore's direction, had gone through a review of the waivers. The question was: How many are out there? Are they all still valid? How often do we review this situation so that we're not guilty of unknowingly accumulating waivers? To what degree are we confident that we have good rationale for retaining waivers in place? What we found out from that review is that we do have a good process in place. There's an annual review of the waivers to make sure that it is still appropriate, it's still applicable, it's still necessary. Remember, we should probably back up a step and just talk a little bit about how you go through the process of granting a waiver. What you want to do, to the degree that you can't meet the requirements that you have in place, you want to try to change that and satisfy the requirements. So your first goal would be to try to execute some type of design change that allows you to satisfy that requirement. To the degree that that's not possible, then you look at other mitigating factors, if you're able to put warning devices or safety systems in place or a crew or ground work-around procedures in place which mitigate the risks. Those are the kinds of things that have to be part of the acceptance of the residual risk when you do go forward with a waiver. ADM. GEHMAN: Okay. Thank you very much. That answered my question. So the kind of legacy waivers then are reviewed annually or periodically, depending on what the project manager wants as a kind of bring-up. COL. HALSELL: Right. Once again, we concentrate most directly -- in the Flight Readiness Review process and the Certification of Flight Readiness for a particular flight, what you want to know is what's changed from this mission to the previous missions or those waivers which need to be highlighted due to the operational flavor of this particular flight and maybe being different from recent previous missions. You'll make sure that those differences, those deltas, as we call them, that's what you bring forward. The same would be true for the failure modes and effects analyses, the hazards, the program hazards. So there is a family of processes which we sometimes capture in this one word "waiver," but they're all reviewed and all brought forward as required during the Certification of Flight Readiness process to make sure that we're not guilty of missing a waiver rationale that is in need of review prior to that upcoming flight. MR. WALLACE: You said that it would be unusual at an Element Acceptance Review for something to come up that you hadn't heard of previously. I have to say in the weeks learning about the FRR process and even the Launch Readiness Review just done in the days before the launch at the Cape that this is sort of a recurring message, like the work is kind of done before these meetings. I'm curious is it fair to say that these meetings, then, don't get scheduled until the work is done or is it unusual things get stopped at these meetings? Does the meeting become sort of a sign-off formality? COL. HALSELL: I guess the best way to answer your question would be to talk a little bit about my personal experience in this area. When I stopped flying on a shuttle crew for a while and I went down to be the shuttle launch integration manager, I perceived some of the same flavor that you're talking about. That is, the important work was being done and being done exceptionally well -- so well, in fact, that when we got to some of these milestone reviews, it appeared to me that all of the hard issues had been discussed, all of the hard decisions and trade-offs had been made. So I questioned the value to our senior management of these level of reviews; but after being in the job for a longer period of time and after having discussed this situation with a number of my project managers, they had a different point of view. They didn't disagree with the fact that the way we do business is such that most of these problems, not always, but most of them, have been flattened out prior to the formal review, but it's because of the presence of those formal reviews and the fact that you know that senior NASA management, the people that you answer to and the people who are ultimately responsible for the safety of the upcoming mission, 'cause you know they're going to be there to hear that story, it drives all that outstanding work that happens before. So from the point of view of the projects and the elements, they did not want to change or consider any dramatic changes to the forum or to the agenda of any of these reviews because, from their perspective, they were driving the kind of reaction within the system that was healthy and needed. DR. LOGSDON: If I heard what you just said correctly, then what's presented to the senior managers is the situation after things have been smoothed out. How much visibility do the senior managers have to the process of resolving issues prior to the formal reviews? COL. HALSELL: Let me see if I can say it in a clearer fashion. I believe that the senior management within NASA, since the Challenger disaster, serves a critical role in deciding upon the final readiness to go fly safely, and it's our job as the middle-level managers to provide them with the information that they need to make that determination. I believe that the process we have in place works very well to do that. I believe that absolutely if we get to a Flight Readiness Review where there are any outstanding issues or if there are any issues that need to be discussed to the infinite level of detail for that level of management, we do that; and I can recount a number of instances where a Flight Readiness Review which was marching along according to the agenda and there were no particular issues, we would come upon one that required the next hour of discussion. It would require a number of people to stand up ad hoc and discuss their participation and their rationale. The Flight Readiness Review board, as would my board on the orbiter roll-out and the mate reviews, if there was something fuzzy or something that we did not agree with or something that we needed additional clarification, we would delve into those details at that board, up to and including the flagship review, the FRR. The point I was trying to make earlier was it's knowing that you are subject to that level of review and that level of detailed review, if necessary, that drives all the good work leading up to it. DR. RIDE: This may not be quite the right time to ask this question. Maybe it should be further on in your preparation, but you've now mentioned twice that since the Challenger accident, processes have been improved and put in place. I just wonder whether you could elaborate on that and maybe be a little bit specific about changes that you are aware of. There were, of course, FRRs before 51L, PRCBs before 51L, senior management was pretty heavily involved in the key meetings leading up to a launch. I'd just be interested in your assessment of what changes have actually taken place. COL. HALSELL: Thinking back to some of the Challenger findings and recommendations, I believe there were ten major findings and recommendations and then appendices behind that. I know that NASA responded to each and every one of those. The two that come to mind, one that's particularly important to me because it has certainly affected my life, was the thought that we needed to involve the astronaut corps in more of the middle and, if appropriate, later in their career, senior management jobs because bringing that operational expertise over to the managerial side of the house was value added to the entire system. I do know that, for example, immediately after the Challenger accident, a number of astronauts were consciously moved into management positions and we have retained that priority for astronauts as part of their career progression ever since then. I don't know the degree to which astronauts were involved prior to the Challenger, but I know that, after, the answer has been quite heavily and in numerous occasions. I know that another finding from the Challenger commission had to do with the fact that on the specific decision to go ahead and fly, given the new data that was brought forward the night prior to that launch, that information, that discussion, the dissenting opinions and the method of which it was finally decided that we were going to go fly that day, all that was not brought forward to senior NASA management in a timely fashion; and I truly believe that today, given the processes that we have in place -- and you'll hear more about the Mission Management Team later on -- that would not be the case. That issue would have been elevated to the appropriate level, given the same set of circumstances today. DR. RIDE: I guess I was just curious whether you could point to any specific -- and again, this may not be the time -- but any specific parts of the process that have been added or specifically strengthened in the pre-launch process. COL. HALSELL: I guess I can speak to the strengths of the processes that we have in place. With regard to the details of comparison how it was pre-Challenger, which was prior to my participation, I probably would not be the right person to ask; but when I get to the part about the Mission Management Team and the process that's in place, I would invite anybody who is knowledgeable about being able to compare that specifically to what we did pre-Challenger to help me out there. GEN. HESS: Colonel, before we get too much further in your briefing, which might be in question, I was curious about providing some balance in the discussion with regards to the line responsibilities to the requirements meetings and these various reviews and how that is balanced by the S&MA organization and recalling the Rogers Commission saying you needed an independent safety process. So if you could help us out at these various stages and give us some idea about how safety figures in and whether or not they can actually overturn one of these meetings because of their degree of questioning over any particular portion of the mission as it's going. COL. HALSELL: Let me answer the last element of your question first, and the answer is absolutely yes. On each of the reviews that I've participated in, whether it be the orbiter roll-out review or the mate review, the safety community is represented through several different channels. Also, the pre-launch Mission Management Team review at O minus 2 -- that's launch day minus 2 two days -- and then at the Flight Readiness Review, Safety is always there. They're always represented and they are always polled and they always expected to come forward with a dissenting opinion which would cause everything to stop at that point in time and we not progress to the next review on the right side of that chart until we had it hashed out. So that's the answer I want you to hear is that Safety absolutely has not only the ability but the requirement to step forward if they believe that the engineering community is headed down a wrong path. I believe that's the essential element of one of the strengths of the processes that we put in place. That is, that, in my opinion, a large part of your safety that's built into the system is accomplished through the strength and the viability of your engineering community and their in-house safety work that they do in line. But it's also important -- and I know that Ron Dittemore has always felt very strongly about this -- it's also important that we have an independent over-the-shoulder assessment of how we're doing from the safety community also. And the important aspect that we've always worked hard on is making sure that as we do our job in line, we have that independent assessment looking over our shoulder and then the fact that they are staffed, have the resources, and empowered to give that independent look at what we're doing. That's the fundamental strength, I believe, in the process that we have in place. ADM. GEHMAN: Colonel Halsell, we're using the term "waiver." You already said this. I just want to clear it up. We're using this term "waiver" kind of loosely here because it really characterizes a number of administrative steps that are taken to account for processes. Can you mention what some of those other ones are called? COL. HALSELL: Yes, sir. Some of the other categories that we talk -- for example, hazards. Hazards are a top-down look. You start with a fairly limited number of ways that you can lose a vehicle or crew and then as you drill down deeper and deeper and you spread out farther and farther, you understand the more detailed failures that could cause that hazard to be recognized. The shuttle program is designed to avoid these hazards and, to the degree we are not able to do that, then we try to control them. You control them by looking at your design and implementing changes, if possible, or the safety controls or warning devices or crew operational procedure work-arounds that I talked about earlier. ADM. GEHMAN: Is that what you referred to as a FMEA? COL. HALSELL: Well, a FMEA CIL is actually a different process. It's a bottom up. It's where you talk about, all right, what if that component of that box failed? Then at the box level, what if this avionics box fails or this component within my auxillary power unit hydraulic system fails? What's the worst thing that could happen to me as a result of that? We have requirements within the system, as I explaining at the beginning of the discussion, with regard to our willingness to expose ourselves to risk. We always want to be fail-safe. We desire to be fail-operational. The degree to which we're not able to meet -- and you also use a risk matrix approach, if you will, in analyzing some of those risks associated with the different failures. Basically it boils down to looking at what is the probability of an occurrence of a particular failure and what are the consequences if that happens. Depending upon where you fall in that risk matrix determines whether it's unacceptable, in which case you don't fly and you make a decision to go fix it -- and I can give you examples of those kinds of cases -- or if it's an accepted risk because you believe that the mitigations that you have in place make the combination of probability and consequences a safe situation for you to go fly in. Then a totally controlled risk is where you don't believe there is any significant risk that you're being exposed to. ADM. GEHMAN: If we took a case like the cause celebre of the day, foam hitting the orbiter, if during the course of the years that foam shedding and foam hitting the orbiter had been previously waived and had previously been disposed of, it's likely it would not even have come up at the ET review. Let me rephrase that. That's a question, not a statement. COL. HALSELL: Yeah. And I believe my correct answer to your question is that I don't believe that to be true. We'll use that as an example, if we want to pull on this thread a little bit. I think it's well known that we did liberate a piece of foam on STS 112; and the process by which we went through understanding what had happened, how that related to our previously accepted hazards and FMEA CILs and what was the appropriate course of action from that point on all followed the processes that we had in place to try to ensure that the right decisions and the right trade-offs and risks got made. For example, in the in-flight anomaly situation for STS 112, that did come to a Program Requirements Change Board. It was decided there that an in-flight anomaly designation was not required for this particular item because the previously accepted and documented hazards -- and if I remember correctly, there were two integrated hazards which were violated or which were called into question by this particular instance -- two of them dealing with the external tank liberating foam and creating a hazard to some other vehicle component -- there was nothing about that particular instance which invalidated the rationale for the previously accepted risk. In other words, we didn't move up and to the right on the risk matrix, according to what we knew at that point that time. So the action that was levied at that Program Requirements Change Board was to the external tank project to go back and fully understand what had happened, why it had happened, and what we were going to do to keep it from happening in the future. Also another action was levied to bring that item forward at the Flight Readiness Review to make sure it was discussed prior to STS 113. So using that as my example, I would say that that's an example of how the process worked properly and the item was brought forward to the Flight Readiness Review and it was discussed at some considerable length there. DR. RIDE: How would that have been different if it had been classified as an in-flight anomaly after 112? What would have been different in the disposition process? COL. HALSELL: Nothing. In the sense that whether it's designated in-flight anomaly or not, the important item is that two PRCB directives were issued at that time which directed the project to go back, analyze the problem, find out what it is, and fix it. Another action was issued to make sure this was brought forward to the Flight Readiness Review. So whether it's designated an in-flight anomaly or not, the answer is it would have made no difference. Now, let me jump ahead and make sure that I'm not guilty of not answering the question you meant to ask, which is, if we had designated at the highest level, which is in-flight anomaly with constraint to next launch, then it would have been immediately an issue which had to be not only fully understood but resolved either with an engineering design change or an appropriate rationale for flight and formally documented. So on this particular case, I would maintain that that process was worked, because we did discuss this issue at the STS 113 Flight Readiness Review at some length. The process of making sure we felt comfortable and safe and that we understood the risks and the hazards and that there were no significant changes from those that had been accepted in the past, all that was done, despite the classification that we came forward with at the PRCB. MR. WALLACE: If I could follow up. I understand from reading some of the PRACA documents that all PRACA reportable items must be dispositioned in some way -- I mean, prior to the next. Is that a fair statement? COL. HALSELL: Yes, it is. However, there is sub-documentation that gives you guidance by which projects are allowed to enter into interim disposition as opposed to disposition prior to the very next flight. And it was the consideration of that particular set of guidance, of rules, along with what we thought was an understanding of no significant increase of risks due to the liberation of STS 112, that led the PRCB to decide that the appropriate way to deal with that particular issue was to issue the directive for the external tank project to come back and find it and fix it and tell us what they had done and also discuss it prior to the Flight Readiness Review. In general, yes, all problem resolution reporting and corrective action items have to be dealt with. The level at which they get dealt with depends upon the criticality, Criticality 1 being the most significant and requiring the highest level of managerial insight and concurrence with. On the other end of the spectrum would be Criticality 3, which means you have no risk of loss of vehicle or crew. Those can sometimes, under the guide rules that we have written down, be dealt with at the project level and with different combinations in between going to different levels of management. I would hasten to add that, as a project manager or as a program person, you don't have the right to decide, on any given day, what level it's going to go to. That's all been decided for you, and it's documented for us in our processes. MR. WALLACE: So this item which was a PRACA reportable item but not an in-flight anomaly on 12, there was an interim disposition? COL. HALSELL: Yes. MR. WALLACE: Which then didn't include any hardware changes -- it wasn't an assignment to -- COL. HALSELL: We can read the exact directive; but paraphrasing as I remember, it was: "ET Project, you've got until the 5th of December -- and I think that date was later extended due to some conflicts of scheduling -- but you've got until the th of December to go find out exactly what happened, reinforce for us what you're telling us today, which is you have no reason to believe that it's a generic issue and that we're at any increased risk on the upcoming flights of suffering this problem. We would like your options for engineering design changes which could be implemented to completely alleviate this problem in the future. Come back and report to us what your options are and what your recommended plan is." MR. WALLACE: Could you tell us about the decision-making, I guess it was in the post-112 PRCB, the roles of different elements in the decision-making as regards the classification, in-flight anomaly or not, and the decision to go with an interim deposition, particularly the external tank element and the S&MA office, if could you speak to that. COL. HALSELL: I'm trying to think, Mr. Wallace. What additional information or what avenue are you trying to get me to talk about specifically that I haven't talked about already? MR. WALLACE: Just really focus on who makes the call on that, on the in-flight anomaly decision and on the interim disposition items. COL. HALSELL: You're doing a good job of doing my presentation for me -- and that's fine. That's good. Let me. If I can go to the final two slides, if I remember, in the presentation, prior to the backup. Let's cover the two in-flight anomaly pages. After every flight, or as you're doing the flight, every element, every project, including Mission Operations Directorate, which Bob will have an opportunity to talk about here in a moment, they're compiling their list of things which have happened during this flight. Sometimes you hear it called the funnies list or the action log. It goes by a number of names depending upon which element or project you're talking to. I'll use the name "funnies list." That's everything that happened that was worthy of attention by somebody. In general, that entire list, all the problems, all the elements, all of their funnies get brought to the Program Requirements Change Board. Usually it's the first one following the landing of that vehicle. Sometimes it goes to the second PRCB. The program documentation says we need to do it no later than two weeks after landing, is our general goal. It's a fairly long and detailed PRCB agenda item where you go through each and every problem that you experience, all the engineering information that you know that might have caused it, and the elements first blush on where we need to go from here. As part of that and as we go through each and every one of those items, it's a PRACA reportable item. You never have the option of saying, well, thank you very much but I don't think that's worthy of my attention. Everything gets dispositioned one way or the other, and part of the process that everybody is focusing on appropriately in this discussion is in-flight anomaly or not. What you see before you are the listing of rules by which the funnies can get elevated to an in-flight anomaly. Just to go through them briefly, if it's a Criticality 1 or 2 -- meaning that we threaten the loss of vehicle or the crew, Criticality 1, and Criticality 2 meaning we threaten loss of a normal nominal mission, that's worthy of in-flight anomaly consideration. If it's software, either orbiter flight software or the space shuttle main engines, it could cause Mission Operations Directorate -- and Bob can probably give us examples of these kind of situations where we got the nominal mission accomplished but they had to work extra hard and had to do a lot of work-arounds on orbit to make that happen -- then we don't want that to have to happen again. So we deal with that as an in-flight anomaly. If it caused or if it could have caused a countdown hold or a launch scrub or a launch abort, then we want to deal with that. If it could have affected safety or mission success or caused significant impact on resources, logistics, or schedules for the future, or if it's any anomaly that the designated responsible design element wants to make an in-flight anomaly, they have the final word. So that's a list of things that we use as criteria for consideration as in-flight anomalies. If I could have the next slide, please. As far as interim deposition is concerned, these are some of the items by which it was appropriate for us to give the elements more time to deal with these issues and not call them constraints to the very next flight. Let me run through those. Remember, it's one of the following criteria: If it's not applicable to the flight -- in other words, whatever broke last time, you're not flying next time, that's obvious; if the problem condition is clearly screened during pre-flight checkout or special tests and you know you're not subject to that same problem; if the problem is time/age/cycle related and the flight units will accumulate less than 50 percent of the critical parameters by the end of the upcoming flight; if there's no indication that this is a generic problem or if you have no overall safety-of-flight concern; if the problem is applicable to flights, however, the PRCB agrees that we have sufficient evidence that the system can be flown safely with acceptable risk, then those are the kinds of circumstances under which we would go to an interim disposition. And it's my belief that it was the consideration of these type of issues which led to the determination that the external tank foam, using that as an example, would be an appropriate issue for us to talk about completely at the upcoming FRR but to give the project additional time to come forward with their corrective action. MR. HUBBARD: I'd like to go a little bit to the hand-off between the end of one mission and the beginning of another. You just characterized what you do post-launch. Now, let's go pre-launch to the next mission. What is the process by which the collection of things that have happened over the various missions get put into a data base or some kind of a memory bank, other than just individuals around the table so that, as the missions go forward one after the other, you build up a sense of trends? You know, maybe there's nothing on one specific flight, but maybe there's an accumulation. How does that get brought to the attention of management during the review process? COL. HALSELL: I believe the answer to your question is PCAS, which stands for Program Compliance Assurance System. Lately the new word is web PCAS in the sense that its been upgraded to a web-based system, and previously it had been a mainframe-hosted computer system. Web PCAS is a web-based system which allows any person associated with the program at any level, including senior management all the way down, to access all the sub-data bases. PRACA's been -- the problem resolution reporting and corrective action system, that's one of the sub-data bases which is part of PCAS, for example. The waivers list. The in-flight anomalies list. The FMEA CILs. All of these data bases -- and we could probably go on for quite some period of time to have an exhaustive list -- are part of the web PCAS which the engineering community and the safety community use equally in this type of trend analysis and in what we characterize as the paper close-out that has to happen before we go fly again. Before we fly, we have to be 0 percent sure that we have our requirements and our closed-loop accounting system has sufficiently -- you can't launch if you simply know nobody's elevated a problem. You have to have the reassurance that people have looked and that they have closed out all of the open paper, and it's only upon that positive affirmation that you can go fly. MR. HUBBARD: So just to follow this one step further. This data base is available. Is there anybody who is charged with actually looking at it and as you go around the FRR and these other reviews saying, wait a minute, to take our favorite topic, I see a trend in foam-shedding or something like that? COL. HALSELL: Yes, sir, and there are two somebodies. Every project and element -- and you'll see the participation in the Flight Readiness Review -- every project and element associated with the program has to say that verbally at the Flight Readiness Review. They are signing for that when they sign the Certificate of Flight Readiness that, yes, we have looked at this and we know we have closed out all these issues; and the independent assessment that we were talking about earlier, that's an important part of their function in ensuring safety is they look over our shoulder and they make sure that every project and every element has closed out those issues appropriately also. ADM. GEHMAN: Could I ask you to go back one viewgraph here. I don't want to talk about STS 107 specifically. We're talking generic processes here, but I would like to talk about foam-shedding as a generic process. So if you can go back one viewgraph, please, to the in-flight anomalies, the IFA. Thank you. Okay. So as I understand it -- and I don't know whether this viewgraph comes from NASA regulations or procedures or where it comes from, but I'm going to assume it's accurate for right now -- we, of course, will check that out -- it says there that any one of the following criteria makes it an IFA. I assume that damage to TPS, since it's Crit 1, that Item A there, any problem that affects a Crit 1 system which is damaging TPS, we've got ourselves an IFA. COL. HALSELL: Yes, sir. I mean, reading No. A, that's what it says; and I would once again draw your attention to the second page which we've already covered, which gave further guidance which would allow an interim disposition. ADM. GEHMAN: Now, I want to go to the second page. Once again, I'm not talking about the FRR of STS 107. We're going to go into that in some detail. I'm using this as a generic case. It looks to me like something hitting the thermal protection system or damage to the thermal protection system is a Crit 1 system and therefore anything that hits the TPS ought to be an IFA, looks to me, just using this score card. And if we look through the disposition here, it says that interim disposition is acceptable or a final closure is required if you meet any one of the following criteria. So I look at A, problems not applicable to the flight we're talking about -- that doesn't apply. A problem condition is clearly screened pre-flight -- that doesn't apply because you can't tell what piece of foam is going to fall off. C doesn't apply because it's not age related. D, I would say, doesn't apply because it's a generic problem and can happen anytime and anyplace else. Then we get down to E: There is no safety-of-flight concern. Now, can you tell me how -- or even the last one: The board agrees that sufficient evidence exists that the system can be flown safely. How in the world does the system determine that there's no safety of flight? Do you know what processes there are involved or is it judgment or... COL. HALSELL: I know you say we're not going to discuss and this is not STS 107 related, but it is ET foam related. So continuing with that as our example, as I remember, the particular presentation at that PRCB, the nature of the rationale that was presented in that forum was that the external tank had gone back even at that point in time before they had responded to the following action and they had vigorously tried to understand did we do something different with the tank where we had this problem as compared to all the other tanks which had flown successfully. What came out of that was they felt comfortable that there was no new and generic issue that they could identify, either with changes or weaknesses in their processes in applying the foam or manufacturing or in the vendor that provides the raw material. They had already gone back and looked at all of that and they felt comfortable at that point in time that they had no generic issue that indicted follow-on future tanks that we were going to go fly. Furthermore, I do not know for a fact that it was presented in that form but I do know that as part of the Boeing transport mechanism there was no elevated level of concern that anything liberated from that location would have impacted the orbiter. What all this added up to was the conclusion that we had not moved up and to the right on the risk matrix with respect to the previously accepted hazard, the two hazards that had been accepted and which we had flown for much of the life of the program, I believe, since STS 27. ADM. GEHMAN: Thank you for that. To follow up on Mr. Wallace's question, is it the PRCB that would make that decision that there is no safety of flight or -- I mean, it wouldn't wait for an FRR; you would have settled this some other way, I presume. COL. HALSELL: It isn't the Program Requirements Change Board, that the program manager has the ultimate responsibility for determining what are we going to classify as an IFA, what are we going to classify as an IFA with constraint, and which are we going to classify as an interim disposition with an action assigned to come back at a later point in time. But also it's important to understand that the Flight Readiness Review, upon review of any of those actions, certainly has the ability to upgrade any item that they so deem necessary. ADM. GEHMAN: Absolutely. DR. LOGSDON: I am going to ask a question about STS 107. If the mission had been successfully completed, would the foam shedding have been classified as an in-flight anomaly and, if so, by what criteria, since there was an analysis that said it was not a safety-of-flight issue. It was counter-factual, unfortunately. COL. HALSELL: I want to make sure I answer exactly the question that you're asking, and it's in the context that we have had the foam liberation on STS 112. ADM. GEHMAN: No, what he's saying is Columbia gets struck by foam just like she did but she returns safely. COL. HALSELL: Yes. Absolutely. And given that we have now had a second occurrence -- DR. LOGSDON: Go back to the prior slide. COL. HALSELL: Before you do, just remember "D" there about the generic problem. At that point in time, I have absolutely no doubt that following the STS 112 incident and it happens again on 7, what you now have on your hands is a major issue that has to be dealt with before we consider even rolling out the next vehicle, much less flying the next vehicle. MR. WALLACE: And the fact that on the 7 it struck the orbiter, does this even make it way more clear that this would rise to the level of an IFA? COL. HALSELL: Especially given that the Boeing transport analysis seemed to indicate that we were not at severe risk of having a strike against the orbiter from a piece of foam liberated in this area. Now, to be complete and fair -- and I know you know this -- that same transport analysis also indicated that there were weaknesses in the program that was being used to do this analysis. Perhaps most specifically, they made the assumption that you were dealing with a non-lifting something and that as soon as you implied some lift in a direction, then that would have to undergo further additional analysis that took that into account. ADM. GEHMAN: Why don't we let him move on here. GEN. DEAL: Well, I'll go ahead and ask you an opinion question here, Jim, a little bit. It's based not just on your extensive experience in the shuttle but also your flight test experience. If 1 out of every 25 flights you're flying a test development vehicle and it drops a panel forward of the intake, you know, I would think you would be a little bit concerned. We talked to some test pilots that say the deserts around Edwards are littered with panels out there, but, you know, I equate foam falling off of a bipod and hitting some part down below that's critical to the flight as being something forward of a jet intake. Can you give us any perspective about if we showed the right level of concern with four previous bipod ramp incidents where the foam broke off as compared to what type of precedents we put on it. COL. HALSELL: I understand the context of the question you're asking me. As a test pilot and somebody involved in the job of acquiring the data with which a vehicle that's going to be flown for hundreds of thousands of hours over the fleet and making sure that we vet out all those issues while we're in the test phase, as opposed to in the operational phase, trying to transfer that experience to what we're dealing with here. One of the limitations that we've had over the entire life of the shuttle program is that we've never had the opportunity to accumulate the number of flights and the number of flight hours and the number of occurrences of any particular item to be able to apply the same statistical rigor that we're able to do in flight tests, for example, where you do quickly accumulate that kind of experience. I think trying to draw that analogy or that comparison might be an error on my part. So I would ask that I not be asked to do that because I don't feel comfortable doing so. I will take what I think is the intent of your question, and that is at the point in time when STS 112 occurred, we had not had a loss of ramp foam, if I remember correctly, since approximately STS 50. There might have been some interim problems with ramp foam, but nothing of that size and significance. Following STS 50, they had changed some of the procedures and some of the foams; and we thought that had been an improvement in our processes and in our materials. So when STS 112 happened, whether it was appropriate or not, I think there was a consideration that this was a new occurrence, given a new baseline, and trying to statistically infer that what had happened prior to those changes were applicable to our current configuration was not appropriate. I'm sure that that consideration will be something that the investigation board will feel charged to draw an opinion on. GEN. DEAL: I've got two other questions. Since we're controlling your briefing for you, if we can go back to Slide 10, I've got a question for you because we haven't covered that one yet. We bypassed it. When I look at the FRR, Jim, I see a lot of people in there. Some of them are former astronauts. Is the mission commander involved in this? Are the current astronaut corps involved in the FRR? COL. HALSELL: The Flight Readiness Review, the flight crew is represented to the board or the Flight Readiness Review through several different avenues. The center director for the Johnson Space Center, the astronauts are hired and work for that person. So he represents their interests. The manager of the space shuttle program -- GEN. DEAL: On the three that you commanded, did you attend the FRR? Were you a part of it at all? COL. HALSELL: No, I did not; and, furthermore, I think that that's the right thing to do because sitting right behind the board, not at the board table, as the commander of a shuttle mission, I have my direct and immediate two people I consider to be my reps to the board. That is the chief astronaut, that's currently Kent Rominger; and the director of flight crew operations, currently Bob Cabana. Those two individuals, in my opinion represent the flight crew, the flight crew interests, the flight crew point of view, and that's who I want to be there and to concur with any issues having to do with the Flight Readiness Review. Now, I think there's a page of presenters here; and I forget if it's forward or backward. But very close to here is going to be the agenda. There we go. You should see flight crew and the left-side halfway down, the flight crew operations director will make his presentation to the Flight Readiness Review board as to the readiness of the flight crew to press forward into launch countdown. At that point in time he's certifying that the crew has been fully trained, is ready to go fly, they have all the procedures, they've been trained on all the procedures, they have all the equipment and training on how to use it to accomplish the mission. Bob Cabana, the FCOD director, doesn't just stand up and say that. In preparation for the Flight Readiness Review, he has a pre-FRR at which the commander of the mission does attend; and it's at that meeting here at the Johnson Space Center approximately three to four days prior to the FRR. It's the face-to-face meeting where the FCOD director queries the crew commander and asks him: Are you ready to go fly this mission? Do you have any concerns? Do you have any issues? So I feel 100 percent justified in saying that even though the flight crew is not physically present at the FRR, they are 100 percent represented in terms of their ability to make it known to anybody and everybody if they have a question. I guess I feel like I know something in this particular area that I would like to express. There are about 100 meetings that you don't want the flight crew to go at. Because at this point in time in their training, two weeks prior to launch, that's when their highest task loading is. That's what they're trying their hardest to -- it's actually now in the preceding two or three months they're trying to congeal together as a crew, ingrate all the procedures, all the issues, and at this point in time they're typically involved in the terminal countdown demonstration test where they go to Kennedy Space Center and participate in a full dress rehearsal where from the time you wake up that morning until you do the simulated emergency egress out of the vehicle, every step from waking up, suiting up, going out to the briefings, going out to the pad, getting strapped into the vehicle, going through all the procedures of the last couple of hours of the countdown, that's what you're concentrating on. And I would maintain that as important as it is to make sure that there's a chain of communication from the command to senior NASA management, it's also important that we don't overburden them with an unnecessary requirement to be at certain meetings. We just need to make sure they have that communication path; and I believe certainly for all our reviews, including FRR, we do. ADM. GEHMAN: Go ahead. GEN. DEAL: I've got one more follow-up, but I can wait. COL. HALSELL: Did I miss a question? ADM. GEHMAN: No. Go ahead. COL. HALSELL: With the presentation? I've kind of forgotten where I was. ADM. GEHMAN: Page 6. COL. HALSELL: Okay. Thank you, sir. Let's see we were talking -- the vehicle preparations. Element Acceptance Reviews. And I think I got through the external tank mate reviews. And we got taken down what I -- I said there were two things that as the launch integration manager I tried to concentrate on on the mate reviews. The one we covered in a lot of detail. I called it the paperwork, but it is the close-loop accounting system to make sure that we have positive affirmation, that we have met all the requirements, that the rationale for the waivers that we need to go fly with are in place and still valid. The other part I'll call the practical side. As the launch integration manager, I did not ever want to be guilty of getting caught having gone through a significant milestone such as mating the external tank to the solid rocket boosters or, later, rolling the orbiter out of its protected processing facility and bringing it over to the Vehicle Assembly Building, going vertical and mating it and then finding out that there is something not right, something that I should have known about at the mate review or prior that, in hindsight, would have stopped me from going through that milestone. After you mate the orbiter, for example, you don't have nearly the access that you do in the orbiter processing facility. So there was a practical side to those mate reviews that it was important to make sure we had full understanding of, also. Next slide, please. This slide probably does a better job than I did verbally of answering a question earlier of is there a process by which all the waivers, all the FMEA CILs, all the open hazards, any upgrades in hazards or FMEA CILs, that it's all brought forward, what is that closed-loop accounting process that we make sure we're ready to press forward to the next level of readiness. This slide gives you that, and I think we've touched upon some of the important elements of that. Next slide, please. Now we're talking about Flight Readiness Review, which I think has been done. Let me see if there's anything on this chart that we haven't really talked about. I think the important thing to understand is that the Flight Readiness Review exists at its core for the associate administrator of the Office of Space Flight, Mr. Bill Readdy now, to make a final determination if he feels comfortable that we have done everything that we said we would in our requirements to get ready to go fly safely. Next slide please. This slide should look very similar to the one that I presented two slides ago because it says basically the same thing. We review all the open issues, make sure that our baseline configuration, what we're flying is what we said we were going to go fly and, if it doesn't, that we understand why and that we agree with that. Any significant unresolved problems or resolved problems since the last review and the flight anomalies, any open items on constraints, any and all new waivers and any open actions from the Flight Readiness Review or any of the element reviews that led up to that have to be closed out at this meeting. At the formal end of Flight Readiness Review -- could I have the next chart please. I'll continue my thought in just a moment. Here is the participation of the board. What I might have in the backup charts but, if I don't, I want to make it clear to you, that this is not just a table with these people. It is, rather, a table in the center of a very large room with these people surrounded by literally hundreds of other people. Every project, every mid-level and lower-level manager of each project is represented there, each of the contractors, from the CEO down through every individual that he or she thinks is necessary to provide the necessary support. Literally a couple of hundred people at least are attending these meetings and are right there in the same room. Next slide, please. Some of the logistics are talked about here. We try to hold this review a couple of weeks prior because that's soon enough so that if we identified any issues at that point in time that need to be dealt with, we have some chance of still making a launch date after having satisfactorily resolved those issues. You don't want to do it much earlier than that, though, because you're reviewing a flight for which issues and problems are going to arise in the interim period of time. So that seems to be the right middle ground. We talked about how all the NASA and contractor personnel are there. One important aspect is that we insist that the whole world of the space shuttle program travel to the Kennedy Space Center and be there in person. You do not participate in the Flight Readiness Review by telecon. You will be there and, if you can't, your designated alternate will be there. It's that face-to-face conversation, face-to-face interaction, that allows you to gain so much more information than you can from a telecon and a voice transmitted to you over the telephone. So the face-to-face nature, I think, is something that's important. Also not only do we have minutes but we audio- and video-record the proceedings. I know, for example, in answer to Dr. Ride's previous question, that's one thing in particular I remember was implemented post Challenger that we hadn't done such a good job of previously. Maybe we had been as good at analyzing some of our issues, but the documentation of the way we resolved those issues wasn't as stellar as we would have liked. We made sure that problem was fixed, hopefully, after Challenger. MR. HUBBARD: This is a little bit of a subjective question, but let me start off with just a fact or two. You participated in FRRs as the manager of launch integration, and what you described is a big show. I mean, it's a big deal and it's a big room and a lot of people. Somebody once said if you have more than five people at a table, it's not a meeting; it's a conference. So you've got, as you said, a couple of hundred people, more than a hundred people in the room. What do you feel like when you're in an FRR? What do you think the tone is? You know, people have their antennae quivering, looking for issues? Do they feel like their working their way through a series of boxes? How do you feel when you're going through an FRR? COL. HALSELL: I feel like it is the culmination of a very, very long and involved process. I feel like when we're there in that room, we are putting the important final touches on the work of thousands of people. It is thousands of people. Tens of thousands of people. That filters up at the engineering and manufacturing level, up through the element processes and reviews and the element project managers to what I'll call the mid-level to upper-level management that I participated in in my reviews as the launch integration manager. But it certainly wasn't just me. There are a lot of other mid-level managers doing the same thing in their areas of responsibility. And I feel like the Flight Readiness Review is that flagship review at which we have that last and final opportunity to present our story to senior NASA management. And we know that they've been made aware in an interim basis on everything that we've been doing. But I feel that at the table at the FRR board you have the representatives of the right organizations to lend that final not only senior managerial level but that experience viewpoint and common sense viewpoint and asking the straightforward simple questions: Have you done this? Have you accomplished that? Why do you feel comfortable that your assumptions that you made here allow you to make the conclusions that you're presenting to us? I feel that that's the level of inquiry that we get at the Flight Readiness Review, especially on issues that require that at that point in time. So I feel like it is an appropriate and exhaustive review that culminates an appropriate and exhaustive process. MR. HUBBARD: Just one follow-up on that. People, in general, can feel very comfortable saying things one on one, maybe even in a group of five or ten. I don't know if your average engineer -- and, of course, this is a group of senior managers -- but do you think people feel comfortable raising an issue in a room with a hundred people? COL. HALSELL: I know that in this particular forum there's absolutely no hesitation to raise your hand, even if you're sitting with your back up against the back wall, against the wall of the building -- and it happens every FRR. And I would simply volunteer to bring forward transcripts and also recordings to back up what I'm telling you. It would be highly uncommon for somebody not to interrupt a presenter in the middle of their presentation and say, "Well, now, wait a minute. How can you say that when we had something else happen two years ago which now seems associated. What do you think about that?" At some points in time, as the secretariate, if you will, of this particular presentation, my issue has not been with getting full and free participation but just making sure I get it documented. I've got to stop people. I've got to say, "Please come forward. Make your way to the microphone. We need to get this recorded. We need to understand what you're trying to tell us." So my issue has been just to make sure that those types of input are recorded and documented properly. So I do feel that the Flight Readiness Review is a full and open forum. DR. LOGSDON: If there is that kind of lively interaction at the FRR -- and this is really a question asked out of literal ignorance -- have there been FRRs that have resulted in a decision that the mission was not ready to fly? COL. HALSELL: Yes, sir. We have a way and we have a process to document that. It's called the Exception to the Certificate of Flight Readiness. Next slide, please. I'm trying to see if I have it up here. Next slide, please. Okay. We'll stop right there. What happens at the end of the Flight Readiness Review is that after all the elements have presented, the chair, Mr. Readdy, will typically ask an all-encompassing question. He'll scan the room, try to make eye contact with everybody and say, "Is there anybody in this room who has any information that has not been brought forward that is relevant to making a decision as to flight readiness?" It is rare at that point in time that anybody raises their hand because they should have done it -- and they do do it -- during the element's previous presentation. Nevertheless, Mr. Readdy makes sure he gives that last and final opportunity for anybody to raise a hand and say, "Yeah, there's something here that we haven't talked about yet." Also during the course of the presentation, prior to this point in time, the elements can take an exception to their Certificate of Flight Readiness, which is basically a way of saying: I certify that I did everything that's required by 8117, also the appendix to 8117, which is my element-specific requirements that I'm signing up to, and also the preamble to 8117 which applies to everybody equally. I am signing up that I did everything and I've closed up all the open issues in a closed-loop accounting fashion with the exception of this one following issue; and that's the Exception to the Certificate of Flight Readiness. A last thing we do at the Flight Readiness Review is that Mr. Readdy will poll his board members and contractors and they will have the opportunity to say verbally if they certify to flight readiness. Anybody who has taken an exception to flight readiness will, in addition, at that point in time, verbalize that exception, say something to the nature of, "With the exception of issue of working with shuttle main engine thermocouples" -- I'll just use that as an example -- "we certify that we're ready to go fly the next flight and, furthermore, we will not allow the launch to proceed until we clear this exception to the COFR." You're kind of a good lead-in to the pre-launch MMT because that's going to be the venue at which we clear the exceptions to the Certificate of Flight Readiness, if you'd like me to continue on into that at this time. DR. LOGSDON: As you do that, can you give me a sense of how often you get to a pre-launch MMT with significant open items? COL. HALSELL: Exceptions? I would say that -- I'm going to guess. We can go back and get the exact percentage over the last couple of years, but it is not unusual, somewhere between 25 and 50 percent of the time, I would guess, that at least one exception to the Certificate of Flight Readiness is presented, and it's always presented with the conclusion of Mr. Readdy, "We think we can or cannot clear this exception in time to make the launch date that you're considering and therefore we do or do not recommend that you press forward toward that currently suggested launch date." At that point after the flight readiness poll and everybody's had a chance to say their piece -- and this might play in a little bit to the question that Mr. Wallace had -- it is tradition that Mr. Readdy adjourn to another smaller room with only invited participants. Usually that's going to be the Flight Readiness Review Board, the prime contractor CEOs, the launch director, the manager for launch integration, and a few other selected folks. In that smaller forum, Mr. Readdy makes it clear that if there's anybody who for whatever reason -- and I can't really understand why -- but if there's anybody who wants to say anything there in that smaller forum that they were not willing to come up with in the larger forum, now's the time and place to do that, before we set a launch date. And it is in addition to that information that's made available to the associate administrator at that time that he considers before he presses forward with setting the launch date or not. We can and we do set launch dates with exceptions to the Certificate of Flight Readiness still pending, but only if he has firm understanding and recommendations that we're going to be able to clear them prior to that launch date. If you like, I'll press forward with the next couple of slides. So we've finished the Flight Ready Review process. The members of the board have been polled. We've adjourned. The associate administrator has adjourned and had his opportunity to hear anybody in private and also to decide if he wants to set the launch date. For the purposes of this illustration, we'll say the launch date was set and that we do have some actions and an Exception to the Certificate of Flight Readiness that have to be accepted prior to going to fly. Let's go ahead now to two days prior to launch. Remember, the whole world came to the Kennedy Space Center for the Flight Readiness Review. They now go away and do their business. Two days prior to launch, we require once again that everybody come back to the Kennedy Space Center. We do it two days prior to launch because we want everybody to have a chance to get back, get in place in plenty of time to set their other job duties aside and to concentrate only on the next safe and successful launch. Two days prior to launch, we convene the Mission Management Team. The Mission Management Team -- and I believe if we could go to the next slide, please -- I was thinking that I had a slide that showed the composition of the Mission Management Team. Basically if you go back to the FRR agenda slide, remember all the participants, all the people who participated in presenting the information to the Flight Readiness Review associate administrator, those organizations and their leaders now become the launch integration manager's mission management team. It's totally appropriate to think that we've not had our review by the very senior level of NASA management and they are now handing off to the mid-level management, with their supervision, the job of launching this vehicle safely within the constraints and within the rules that have been set aside for us to work with them.  Columbia Accident Investigation Board Hearing So that Mission Management Team convenes and we go through basically the same agenda that we did for the Flight Readiness Review. Every element, every project gets the opportunity to present any interim issues, anything that has arisen since the Flight Readiness Review. If there are any exceptions to the Certificate of Flight Readiness, the full and complete rationale for that is presented there to the same level of rigor that it would have been presented in the Flight Readiness Review. As the launch integration manager chairing that pre-launch MMT, I felt it was important that I get input verbally and visually and in public from the program manager and from the associate administrator at the MMT that they concurred on that FRR COFR exception. In other words, it wasn't just the middle managers now clearing something that previously wasn't good enough for the senior managers to go with. At the end of that MMT, we, once again, poll all the participants to make sure that they are "go" to press forward with the countdown. From that point on, the Mission Management Team is activated. I know where each of them is. I can convene a meeting in literally an hour's notice if I need to during the launch countdown. The next time we convene will be formally three hours prior to launch, in the Launch Control Center. If I can have the very last slide in the whole package, I believe it's a picture of the Launch Control Center. As he's scrolling forward -- at three hours prior to launch, the Mission Management Team will convene in this room that you see. Next slide, please. Here's another view of it. Up and in the dark to the upper left is where the Mission Management Team resides. The larger room is the Launch Control Team and the Launch Control Center under the direction of the launch director, who stands just about underneath that American flag in the center of the room. It can help you to understand the relationships here as we go through the final hours of the launch countdown. At this point, the Mission Management Team has really done their job and we've handed off responsibility for the successful launch of the mission to the launch director who is directing the Launch Control Team, as long as he or she is able to work within the constraints of the Launch Commit Criteria. That is huge, several-volume book which is the what-if of every launch and represents the corporate history of all the problems that we've either experienced or we've had the opportunity to think through ahead of time that we might experience and our reactive measures that we would take to further clarify the problem and our ability to go launch safely. For practically all the launch commit criteria, when you run through the procedures, it ends up in one or two branches. Either you have resolved the issue as being safe to go fly, clear to launch or, no, we're not sure, you have to stand down that day, unless the Mission Management Team is offered rationale which allows you to press forward and approves it in real time. The Mission Management Team is there to provide guidance if the launch director gets outside the launch commit criteria and needs guidance. GEN. DEAL: Jim, I just want to get back to in-flight anomalies very quickly and get your perspective because you experienced a very serious one on STS 83 personally. What I want to do is get your perspective on, following STS 83, how the process went, did it underscore the strengths in the program, or were there lessons learned by which we improved the in-flight anomaly process following STS 83. COL. HALSELL: Certainly I can lend my experience from STS 83, and I think the question that you're asking about the in-flight anomaly process is one of the reasons that we invited Bob Castle, as one of the representatives of the in-flight MMT team, to comment. So I'll hand off the remainder of that question to him. The issue you're talking about on STS 83 back in 1997 was that after we launched, we experienced an in-flight anomaly concerning some out-of-family and unacceptably divergent fuel cell substack delta volt readings, which is a way of saying there were some increased level of risk that if we were to continue the mission with that fuel cell powered up that you could experience crossover and that could lead to fire and/or explosion. So that was deemed to be an unacceptable risk. It was equally unacceptable to shut down and save that fuel cell and continue the mission to nominal conclusion on just the two remaining fuel cells. So the Mission Management Team came to the conclusion that the only safe and prudent thing to do was to have us close up the lab, prepare to make an early entry back home; and we did so after only four days in space. The conclusion of that story is that between then and STS 84 which, as I remember, wasn't the very next but the one-after-the-one-after flight, on STS 94, they resolved that particular issue, they understood it after they were able to get the fuel cell and do all the testing back at the vendor to understand that, in fact, it had most likely been an indication problem, not an actual issue, and that we could have stayed up on orbit. But there was no way to have known that in real time and I, certainly as the recipient of the safest course of action, I appreciate the action that the MMT took at that time. So I think that is an example of how, when faced with extremely difficult choices, expensive choices both in terms of money, in terms of the manifest having to be replanned for probably several years downstream, but still when confronted with that highly undesirable set of consequences for making the safe decision, the on-orbit Mission Management Team did make that decision. They brought us home and we re-flew that mission a couple of flights later with a full measure of success. ADM. GEHMAN: Okay. Let's let Mr. Castle give his introductory remarks, and we can always ask questions later. MR. CASTLE: Okay. Well, that does lead into what I was going to start talking about a little bit. I don't have any charts. So you can feel free to interrupt me even more freely than you have already. As far as the way the realtime team goes, we pick up the launch. Right after liftoff is when the realtime team picks up and starts conducting the flight. I would call flight director the mid-level management team that Jim referred to. The flight director also has his set of requirements. The specific ones that come to mind are the flight rules and the SODB, which is the Shuttle Operational Data Book. The flight rules is a large book. I didn't bring one around. It's about yea thick for the space shuttle. It's what I call pre-made decisions, decisions you've already done your what-if'ing and you've thought about them and you've thought about the situations and the cases very carefully and you write down what it is that you're going to do for each of these particular cases. In the one that Jim mentioned, the loss of one fuel cell, it says you need to land what's called a minimum duration flight to minimize the length of time we stay in orbit because if you lose another fuel cell, you can land with only one fuel cell but the power-down you have to get into is dramatic and it impacts your avionics in lots of other ways. So we've already gone through that debate. If we lose one fuel cell, we're going to land and we're going to cut the flight short, early. The MMT got involved with his flight because it wasn't really clear from the indications whether we really had a bad fuel cell or not. So that's where we had to call the engineering guys together to look at that. But if it's clear we've lost a fuel cell, the flight control team doesn't have to consult anyone. We'd say, okay, the flight rules say go do this, so this is what we're going to go do. The SODB is the Shuttle Operational Data Book. That is another book that is maintained by the space shuttle program. It's a list of how you operate the shuttle. You can operate the shuttle with the temperatures on this loop, greater than this and below that. This type of information. Kind of like an owner's manual for your car except, again, it's several volumes. It's fairly thick. The flight rules are controlled by the shuttle program. The final version of all of them are taken forward to the PRCBs for approval. There are several lower-level boards chartered by the program that manage those rules. People have asked about the safety process. Any changes to the rules, that's done on what's called a CR form, a change request. The Safety folks review those as well, as all the rest of the disciplines -- engineering, program offices, space and life sciences, FCOD, MOD, all the different areas. There is a mid-level board, what's called the Flight Rules Control Board, which is chaired right now by one the deputy chiefs of the Flight Director Office. Again, all of those same organizations represented and then their approved set of rules come forward in a change package to the PRCB for final approval by the program. A very similar process used for the SODB, the way it's managed. So those are two things that I start off with as my requirements, if you will. There are a couple of other things that are like the flight requirements document which are a mission-specific document. Okay. The other two I just mentioned, that's how you operate the orbiter, how you fly. The FRD says, well, here's what we want you to go do. We want you to conduct a space lab mission. Here's how long we want you to stay in orbit. Here are the priorities of things we'd like you to do. That type of information. There is also a much smaller book of flight rules that are flight specific. In that again, you're writing down rules, mainly a priority list, rules that are specific to the payload or the particular operation you have on that flight. Those are flight specific. Also approved by a very similar process and finally approved by the shuttle program manager at the PRCB. Also I want to say that the flight rules are things that when we train people, we take these things very, very seriously. The simulation folks try to put in failures and various scenarios that will stress people's thinking. Okay? They'll break a piece of instrumentation someplace in the simulator. Well, do people recognize what's just failed? Do they recognize the instrumentation they've lost? Do they understand the implications to the flight rules? Have you just had a flight rule violation because of this failure? Sometimes just loss of instrumentation is no big deal. Sometimes you really have a rule violation because we've thought through if I don't have this measurement, then this thing that's really bad can happen to me and there's nothing I can really do to detect it or I've actually impacted the safety of the vehicle because I can't measure something. Sometimes they don't. Each rule is also annotated. Let me back up. Jim talked about the top-down hazard process and the bottoms-up failure modes and effects process. Anytime that this hazard control process says we need to control this hazard by a certain operational constraint, we want you to always flip this switch before you flip that switch, a flight rule gets written that says always do it in this order. That flight rule gets annotated that it's a hazard control. So anybody reading the rule book knows that this is a control for a hazard that's been identified for the program. That does a couple of things. The main thing it does for you is when somebody comes along and says I'd like to change this rule for whatever reason, it's in black and white, right in front of you, that you've got to run this by the safety community, you've got to look at it carefully, look up that hazard control, make sure you're not undoing what we carefully did. They're also flagged from the bottoms-up review. Anybody in the bottom-up review that comes up with a classification of either a Crit 1, 1R, 1S, and 2, I believe, gets classified on a Critical Items List or a CIL. So we flag those rules, as well. It says, okay, this rule is part of the rationale for saying this critical item is acceptable. Again, you get the same type of things that are controlled operationally. If you have Failure A, then you must take this following action to make sure another problem doesn't sneak up on you. Everybody works really hard to understand those, even though the book is very, very thick. We train them very, very heavily. Our simulation guys are very sneaky. They will put in an instrumentation failure here and a power system failure there and an avionics box failure here and you've got to realize that when you add all those three things up, you've really got a much more serious problem than it seems like. Generally they'll set us up that you need to recognize, hey, one more failure could really be bad. So we work that very, very hard. Again, I'm just going to keep talking until somebody wants to stop and ask me questions. Let's see. The basic rule, again, is the flight rules and the SODB -- when I say the realtime team, let me talk a little bit more about who the realtime team is. There is the Flight Control Team, which is led by the flight director who sits in the middle of the room. I don't have a picture, but you've seen the room. There are flight directors there 24 hours a day during a shuttle mission. We also appoint a lead flight director who is appointed generally at least on the order of a year before the mission. They oversee not only the mission but all the launch preparation, all the preparation times, all the crew training, everything else that goes on for that prior year. That includes the little first chart that Jim put up, all the little boxes. Either the flight director or some member of his team plays in every one of those boxes throughout the preflight process. There are other members of what I call the flight control or the realtime team. A very important team is the MER, the Mission Evaluation Room. That is a room that's down on the first floor of Building 30. It is run by the program office and is staffed mainly by people out of engineering and various contractor support -- Boeing, various subsystem contractors. Their function is evaluation. They watch what's going on on the vehicle. They look for more shuttle trends, things aren't clear black and white but maybe more subtle problems. If there is a problem, of course, they're ready to be activated, ready to go work any details. Things are never quite as crisp and clean as they look like in simulation. So you always like to have the engineering talent there, ready to go. That group in the MER includes a safety console position, again, always watching what's going on as we operate the mission, understanding all the hazard controls and all the things that have been preflight analyzed. There's another room in the building which is called the Customer Support Room and that is a program office room. Representatives who report directly to the program manager staff that room hours a day. Again, they're watching out for programmatic requirements. They're there to be consulted. If we get into a situation where I can't do what their priority list says I need to do, they're there to go rework that. "Okay. This just happened. I can't do your No. 3 item on the priority list. What would you like to do? What options would you like to invoke?" So they're there 24 hours a day, 7 days a week during the mission, ready for consultation; and they pay attention pretty well. There's a formal CHIT system. It's called a CHIT. I don't know what CHIT stands for, but there's a formal paperwork system where if we make a request for information or a request for special analysis, we write down exactly what we want. It is coordinated through the appropriate person who we're requesting this of. Anyone in the building can write such a CHIT. It comes down with an answer, and we don't close that CHIT until the originator agrees that whatever they wanted done has been done and done correctly. Again, it's a very formal process, I think, that works fairly well. ADM. GEHMAN: Let me interrupt. To carry over the discussion, I asked a hypothetical question, as did Dr. Logsdon, that if Columbia had returned safely from this mission, we still would have an IFA of a major foam strike. MR. CASTLE: We would. It's interesting. People have talked about it from the flight director's perspective. That's one that would come in through the program office and not the realtime team, because the realtime team didn't know the foam came off the tank. It was only the photo analysis folks the next day who came in through the MER who knew something had come from the tank. During the realtime ascent, I'm pretty sure the team didn't know anything about it. ADM. GEHMAN: That's right. But a day later or a day and a half later, whenever the photo analysis of the ascent, the launch photography was made available and the MER was informed that there was a strike, is it formally classified as an IFA at that time or does it take more paper and more meetings or something like that? I'm thinking MMT now. Are members of the MMT or the flight team, are they aware now that we have something to deal with? MR. CASTLE: Yes. As soon it came through the MER, it should be made known to the next MMT, whenever that was. MMTs like generally every two or three days. Now, I'm going to have to talk generically here because I had really very little to do with STS 107. I was there for a tiny period of time. ADM. GEHMAN: That's perfectly all right. MR. CASTLE: The realtime team, we probably would hear about it from the MER even before it came to the MMT. I say probably because we talk to those guys a lot. We play in their games a lot. ADM. GEHMAN: Since damage to TPS is a Crit 1 issue, if you had debris striking the TPS and the system was aware of it, I mean, both the flight directors and the MMT personnel, they use the same rules and the same categories and the same processes. MR. CASTLE: Yes, we do. Sometimes the in-flight anomaly list or the funny list will vary. The flight control team may have different items on their list than the MER has on their list and the CSR has on theirs, which I think is a healthy thing. You get together and decide which ones you want to carry forward on a formal programmatic level. ADM. GEHMAN: You mentioned that loss of one of the fuel cells is in the flight rules. MR. CASTLE: Yes, it is. ADM. GEHMAN: What about damage to TPS? Is there a flight rule for damage to TPS? MR. CASTLE: I would have to go look it up. I don't think there is one, mainly because I'm not sure what the flight control team could do about it, is the real gotcha there. If you knew exactly where it was, then maybe you could do a little something about it. But if there are any rules, they just tend to go with -- ADM. GEHMAN: So if it's outside the flight rules, then it would be kicked up to the MMT. MR. CASTLE: I think it's kicked up to the MMT, yes. ADM. GEHMAN: Correct me if I didn't get this right. Did you say that changes to the flight rules are approved by the PRCB? MR. CASTLE: Yes, they are. All changes to the generic rules are approved up at the PRCB level. We don't take individual changes. What we do is we process individual changes at the Flight Rules Control Board, which is one board down. Then when we collect up enough that we need to make an actual page change to the book, we bring that forward to the PRCB. There is a realtime flight rule change process that is in place where the flight director or the mission ops representative, the representative of essentially my boss, John Harpold. Those can be signed off by the flight director or by the mission ops representative; and the actual process allows it to happen without the MMT. That is there so that if there's no time to go have an MMT meeting, you can go do what needs to be done. As a matter of practice, I don't think any of them have ever been signed off without being fully briefed to the MMT; and the number of realtime changes is very, very small. ADM. GEHMAN: Okay. MR. WALLACE: Can you give a rough sort of breakdown of the MER in terms of contractor versus civil servant size? MR. CASTLE: I really don't think I can give a good breakdown because I don't really know. We operate very much badgeless, is the term I like to use around here. Even on the flight control team, the people that I know, I'll tell you their names and their wife's name but I can't tell you whether they're a contractor or a civil servant because it's not really important. So I really don't know about the MER. In the flight control world, simply because I've seen other statistics, it's about 30 percent civil servant and about 70 percent contractor. MR. WALLACE: So when you say that anyone can write a CHIT, then that includes contractors can write CHITs? MR. CASTLE: A contractor write a CHIT. They bring it to the MER manager for forwarding on into the system. MR. WALLACE: Does the CHIT guarantee a certain level of elevation, and what would that be? MR. CASTLE: Well, it guarantees that it goes through a controlled process. They can write a CHIT. So, for example, someone in the MER could write a CHIT to the flight control team saying I would like to go do this or I'd like this particular information retrieved from the vehicle via a data dump or something. It's guaranteed to go to the MER manager; and if the flight control teams has to do anything, then, of course, the flight director will hear about it. That could be as far as it goes and the CHIT gets closed. MR. WALLACE: Does the CHIT go to the MMT in an appropriate case, or is the CHIT something that's with the flight control team? MR. CASTLE: It's within the flight control team, the CSR and the MER. It could certainly go to the MMT if either the missions ops rep or the MER manager or the CSR reps wanted to elevate it to that point as an issue, but CHITs routinely do not go to the MMT. MR. WALLACE: Generally, could you describe the sort of level of contact, day to day, between the MER members and the realtime flight team? MR. CASTLE: Fairly routine contact. Generally, at the system level an electrical power guy will talk to the EGIL electrical power guy on the flight control team probably on a daily or shift-by-shift basis. They will talk to each other on voice loops and just say, "How are things going? Were you working anything?" I know they did that back when I worked in that level. At the flight director level, probably daily we talk to the MER managers to see what's going on, or they will talk to us. MR. WALLACE: Is there a process at shift change, sort of a formal tag-up process, or is that by individual? MR. CASTLE: There's a formal shift change of the flight control team where we hand over to each other. It's all done on a voice loop where we go around the room: "What are the issues that you're working?" The MER is certainly available to listen to those loops, and I know from experience that they often do. MER is not usually, as an entity, polled during the handover for shuttles. Last time I did the shuttle flight was a couple of years ago. Now, on the station program, we do poll the MER if they're there. They're not there nearly as often. MR. WALLACE: I have heard it said that typically the MMT might become involved in a decision if it's sort of outside the book, outside your flight rules. MR. CASTLE: Yes, that is by definition. I look at the flight rules in a couple of ways. It is pre-compiled list of decisions that have been agreed upon. It's also what I consider kind of my contract with the program manager. If it's inside this book, then he's already agreed that this is something that's appropriate for me to do with the vehicle that really is his responsibility. I'm being delegated it during the flight. So if it's inside the rules, then that's perfectly my right or the flight director's right to go operate within the rules, within whatever the program has laid out. If it's outside the rules, it needs to go to the MMT. It needs to go to the MMT for approval of whatever I'm about to do, if there is time. There is a caveat, again, since you're flying, if there's not time, the flight director and the mission commander do what they think needs to be done. If there's time to consult the MMT, then by all means you do and you get your approval before you press forward. MR. WALLACE: Would the sort of realtime flight team expect to be aware of most anything going on between the MER and the MMT as a general practice? MR. CASTLE: As a general fact, yes. We have a representative to the mission. We call him the MOD. It's really, again, a representative of my boss, being the director of MOD, who attends all the MER meetings, all the MMT meetings. So anything that goes on in that meeting, the realtime team has a representative there who comes back and consults, talks with the flight director. So the flight director will be aware of anything that's going on in the MMT; and like I say, we not only come back and talk about that, the rep comes back and writes a little short report: Here's what got discussed; here's what the flight team needs to know about what's going on in the MMT. DR. RIDE: You said that there are no flight rules that cover tile damage. MR. CASTLE: I can't remember any off the top of my head. I'd have to go look it up. DR. RIDE: I'm curious whether that would have been a conscious decision by the program. I know the flight rules are reviewed periodically. They are really the bible that the flight team uses to operate. So I would have thought that at some point someone would have brought up should we have a flight rule on tile damage. So I'm curious about what the discussion around that would have been and why there isn't one. MR. CASTLE: I remember some of that from quite a few years ago. Again, a flight rule is a decision, is the way I like to look at. It's a decision that's been made. So it should be, if you know you have tile damage, then you go do this. If you don't know what to do and there's nothing you can do differently, then there's no point in having a rule. So to my knowledge, we've never had an answer to what you do if you have tile damage, because there's nothing we can do in real time to do much with trajectories or anything else that I'm aware of that would make any difference. DR. RIDE: Let me ask maybe just a little bit of a different way. You know, suppose we're back in time and 107 is in orbit and the crew happens to look out and sees damage to the left wing. Then it would have been reported, essentially, into the flight control team. I just wonder whether you could describe how that situation might have been handled and whether it would have been handled differently, whether the assessment would have been handled differently or whether the flight control team's involvement would have been different than it was. MR. CASTLE: How it would have been handled, the flight control team would have immediately reported that up the chain because we're going to need more resources than the realtime flight control team has to do anything about it. I'm sure we would have turned on all sorts of effort in the mission evaluation room to look at possible repair. The trajectory guys would get turned on yet again to go look at is there any other way, anything we can do to fly the vehicle differently because of the specific damage that we see. We would have worked on it very, very hard. I'm sure we would have pulled out all the stops to try and do anything about it; but, again, there are no flights rules on it right now because, as Jim talked about in the very beginning, there are three areas that are simply Crit 1. If they fail, there's nothing you can do about them. Thermal is one. We do not have a flight rule on structural damage either. If you found a broken member someplace, there's no flight rule that says what to do about that. Pressure vessels -- actually we do have flight rules on that. Because if you have a leak, you know, you can take action before whatever it is all leaks out. But structure and TPS, there really aren't any rules on that. But, yes, if we had known about it, we would have pulled out all the stops and done everything we could to find the answer, I'm sure. The realtime team would not have been able to do much but implement whatever somebody else figured out. ADM. GEHMAN: Mr. Castle, is your reporting chain to the center director? MR. CASTLE: My reporting chain, yes, is to the center director. ADM. GEHMAN: I mean, I understand under the flight rules and when you're flying, you're working as an agent of the program manager; but your reporting chain is to the center director. MR. CASTLE: Yes, my reporting chain is to the director of MOD who reports to the center director. ADM. GEHMAN: I don't know whether or not the slide presentation that Colonel Halsell put up there is retrievable or not. I don't even know where they come from. Could we have Slide No. 12? Let's go back to Slide No. 12, which is the FRR agenda. What does the S&MA organization say when it's his turn to speak? MR. CASTLE: He talks about any program safety paper that is open or any hazards that are open, need to be closed, any new hazards that have recently come into play, even if they've been safely controlled, that type of thing. He gives a report on that and are there any things in the safety reporting system, this anonymous safety reporting system that's been set up, are there any of those that are out there that affect this mission. He talks about and reports on all those areas. COL. HALSELL: In addition, the safety community, prior to the Flight Readiness Review, has their own pre-FRR review. I believe they call it the PAR. Really that stands for Prelaunch Assessment Review. That's done by all the elements in the project safety organization reporting up to Code Q, which is Bryan O'Conner at headquarters, in association with the Johnson Space Center safety space shuttle division. All of these elements come together to review all the issues. In addition, if there have been any increases in hazards -- and I wasn't taking good notes -- but all of the elements that we've talked about that the safety organization is responsible for being the look over our shoulder to make sure that we're doing our closed-loop accounting system. They report that there. Once again, they report it in the affirmative and also the negative. It's not good enough that they say we don't know of anything; they come forward and say we looked and we did not find anything. In the degree to which it's not possible for them to stand up and say that, then we have an exception to the Certification for Flight Readiness. GEN. HESS: Let me talk about the MER just a second. I really have a simple question. During the course of the mission, the MER works for whom? MR. CASTLE: The MER works for the MMT. GEN. HESS: Now, do you have any direct authority, as the flight director, over the MER? MR. CASTLE: In general, I can ask them to go work on things. I can send them CHITs asking for things. Do they absolutely have to do what I tell them to do? No, they don't; but, in general, I think it's been rare that if a flight director really wants something with good rationale that they don't jump in and do their best. GEN. HESS: That's a good lead-in to my other question here. We've heard a lot of characterizations about the preflight FRR process and then the on-mission process that goes; and some would say that one part, the pre-launch part is very, very formal but then it tends toward being a little bit less structured and less formal because you have this book of rules and so the communication is decidedly different. How would you respond to that? MR. CASTLE: I think it is a little less formal during the flight, for a couple of reasons. One, I think since things are moving much more rapidly, I think it needs to be a little less formal. I think we also, unlike the previous meetings and all the other work in the offices, everything that the flight control team and the MER team does with each other, they do it on voice loops. All of that is recorded so we've got records of everything that's happened. We can go back and sort out exactly what's happened. Things do need to move a little faster when you're flying than when you're sitting on the ground deciding whether you should fly or not. And that's what's built in to allow more flexibility and a little more speed in making decisions. We try to have everybody in the building on a voice loop who has got a stake in the situation and can listen and participate in making the decision right then. The MER manager is listening to what the flight director is talking about doing on the flight loop. In my experience, those people have a remarkable lack of shyness. If they feel they need to stand up and be heard from, they will stand up and be heard from. Is it as formal with normal paperwork going back and forth and signatures and all of that? Yes, it is less formal, considerably less formal in that perspective. GEN. HESS: Following on with that, we all have in the back of our mind this perhaps Hollywood picture of Apollo 13 and, you know, failure is not an answer and the flight director was the center of gravity in running that particular event, but what you're describing today is that if it's something that's outside the bounds of the flight rules, it's not the flight director that's the center of gravity, it's the MMT. MR. CASTLE: The MMT is the center of gravity for making all the decisions and deciding which way to go, yes. In terms of actively solving a technical problem, I think you'll find the MER and the flight director are the ones most involved in trying to come up with a solution to a technical problem. I was not in NASA for Apollo 13. I'm not quite old enough for that, but I do know quite a few folks who were here in that time frame. The movie, as all movies do, simplified things. There were a lot more people involved in working on Apollo 13 than the few that you see on the movie. There was a huge number of people in both the MER and the flight control team that did a huge amount of work, pulling all those pieces together. GEN. HESS: So then would your expectation as a flight director be that, in the case of 107 where we had the debris strike we know about and then the visual debrief of the ascent video showed this debris and the engineers were beginning to work and decide whether or not that there was a problem with the orbiter, that the CHIT system and the request for information would have led to a filling in some of the blanks that the engineers were obviously after? MR. CASTLE: I don't know if it would have or not. Again, I was not working 107 specifically during the orbit phase. GEN. HESS: I'm just talking about normally. I mean, if you had been, would you expect that process to formalize itself and get into a formal CHIT if the engineers wanted information. MR. CASTLE: If they wanted information that they felt we could provide, I would expect them to write a CHIT; but again, if they know we can't do anything or know we can't provide the information, they don't spend their time writing a CHIT for it. If they thought we could get it, I would have expected them to do so. MR. HUBBARD: One question about who's "in" box problems end up. You described a very rigorous process with a lot of opportunities for people to speak up and simulations that involve all manner of different processes, things that could go wrong and the evil simulator sitting back there failing things on you and so forth. So that captures a way of doing business that encompasses a whole great raft of problems. Now, looking at the other side, you have damage to the thermal protection system tiles, every single flight. You know, something greater than divots, greater than an inch and more than a hundred total; yet TPS is a Critical 1. It's one of the handful of three things for which is there not a fail-safe and there's no flight rules probably on this. So that problem, whose "in" box does that kind of conundrum, that problem end up in? COL. HALSELL: We'll tag team this one. The short answer is that it's the space shuttle program manager's job to organize the appropriate response to any and all issues when it comes to making the final determination if we can recommend to the associate administrator that we're ready to go fly safely. So if Ron Dittemore were sitting here in front of me, he would he say, "It's my 'in' box" because he's the one who controls the resources and the application of those resources; but at a personal level, I think each and every one of us involved in any way, shape, or form with -- touching the particular example you're talking about here, the TPS, a lot of people have responsibilities which touch upon that, whether it's myself as a launch integration manager and that means that last year the person who runs the interagency imagery working group, the people who took the imagery that first revealed this issue to us, for example, that person reported to me. So that's one area that I'm involved, one of many; or if it's systems integration, the people responsible for grabbing hold of these issues -- and this would be a perfect example of where what one element in project over here is doing or not doing may or may not impact another element over here and we need to make sure we're never guilty of not communicating back and forth. And it's the systems integration group which is responsible of being the accountability hounds to make sure that that kind of conversation takes place. And then you get down to the elements themselves. External tank, if they're shedding foam, it's got to be their primary responsibility for understanding that issue and then dealing with it. If it's the orbiter vehicle who has an issue with the environment within which their thermal protection system is being asked to operate, then they are equally accountable for raising their hand and making sure those issues are brought forward. You can say that the solid rocket booster element could possibly either be the source or recipient of debris also. So everybody has a responsibility in this area, and it all goes uphill to the man who's in charge. ADM. GEHMAN: Well, gentlemen, Mr. Castle and Colonel Halsell, thank you very much for your very, very forthcoming and complete and responsive testimony today. It's very helpful to us. We agree with your opening statements that we're all here for the same reason, to find out what happened to STS 107 and to recommend measures to prevent it from ever happening again. So we all have the same goal here. You've been very responsive, and your answers have been very complete. We appreciate your patience, and we're going to take a short ten-minute break while we seat the next panel. Thank you very much. (Recess taken) ADM. GEHMAN: All right. Board, if we're ready, we'll resume. I'll ask the people in the room to please take your seats and be quiet, please, so we can get back to work. The second half of the afternoon public hearing will be looking more specifically at foam events and debris events. We have with us Mr. Scott Sparks, who is the department lead for external tank issues, and Mr. Lee Foster -- both, I believe, from Marshall, if I'm not mistaken. THE WITNESS: Right. ADM. GEHMAN: Before we start, gentlemen, I would ask you to affirm that the information you provide to the board today will be accurate and complete, to the best of your current knowledge and belief. THE WITNESSES: I will. ADM. GEHMAN: Thank you very much. Would you please introduce yourselves and tell us a little bit about your background and what your current duties are. LEE FOSTER and SCOTT SPARKS testified as follows: MR. FOSTER: My name is Lee Foster. I've been at the Marshall Space Flight Center for over years. Currently I'm with the Space Transportation Directorate. I'm an old technical guy. I've spent many years working aerodynamic design and aerothermal design of the Marshall space shuttle elements, and I've been involved with the aerothermal testing of the TPS. Currently to the external tank I'm kind of a gray beard that they call on occasion. ADM. GEHMAN: Thank you very much. MR. SPARKS: Scotty Sparks. Academic background, a Bachelor's in chemistry, Master's in polymer chemistry. I have been employed with NASA since '89. I been working external tanks since '91. I have worked other composite cryo tankage issues. Just recently I mainly have specialized in the areas of cryo insulation. ADM. GEHMAN: Thank you very much. We're ready for you to begin. If you have a presentation for us and whichever one of you is first, go ahead. MR. SPARKS: Let's go ahead and get the first chart up and we'll start, hopefully. ADM. GEHMAN: We have copies of your presentation. Let's go ahead, and they'll catch up with us when the electrons do. MR. SPARKS: The objectives that Lee and I want to discuss would include cryoinsulation's purposes and its characteristics in the external tank, material development and qualification, flight environments, debris history, and some past issues, some efforts, to try to tell about our efforts to reduce debris, and also some recent observances. ADM. GEHMAN: If you could go through quickly the first two or three. We're really interested in the environment and debris history and efforts to reduce debris. Please proceed. MR. SPARKS: The purpose of cryoinsulation. The main purpose of cryoinsulation pre-launch is to minimize ice formation, but it also maintains the oxygen and hydrogen boil-off rates to acceptable levels. We try to eliminate cryopumping totally and we also try to densify propellant so we can get the maximum mass per the finite volume that we have. Upon ascent, we have to protect the tank from aerodynamic heating as well as plume-induced heating. We minimize effects on the structure of aerodynamic loading, static loads, unsteady aerodynamic load. Also, upon re-entry, we have to maintain a certain breakup altitude window to make sure it doesn't break up too early to scatter some debris over a large area or too late to scatter larger pieces of debris. ADM. GEHMAN: My understanding is in pre-launch, even if ice formation were not a problem, you still would want to insulate in order to slow down the rate of heating of the liquid hydrogen and liquid oxygen. MR. SPARKS: That's correct. There would have to be some level of insulation to control that. ADM. GEHMAN: When you say you try to eliminate cryopumping, are you going to tell us what that is or later? MR. SPARKS: We will tell you about that. ADM. GEHMAN: All right. MR. WALLACE: Mr. Sparks, in following the board's tradition of never letting anybody get through their briefing, can you give just a general sense in terms of the bipod, the whole bipod insulation structure, as to its purpose as between pre-launch, ascent, and reentry? I mean, does it really have an important purpose particularly as regarding re-entry? MR. SPARKS: Upon re-entry? No, it does not. MR. WALLACE: Using this page of criteria -- pre-launch, ascent, and re-entry -- could you sort of speak to the relative importance of those areas? MR. SPARKS: Sure. Upon pre-launch you are going to get some level of possible ice formation in that area, and that's one reason why we do have some cryoinsulation in that area. There is some level of rotation that that structure has to go through. So there is some small areas that do not contain cryoinsulation. That's the reason why we have the heater inserted into that bipod to try the minimize that frosting or that ice formation in that area. As far as ascent, Lee, you might want to talk ascent. There's not an appreciable amount of loading in that area, but you might want to talk to induced -- MR. FOSTER: It's a very complex flow field in that region, which we'll go over in a few charts. We have a ramp on our bipod to lessen the aerodynamic loading on there. So all the TPS works for the ascent part. It's a very massive piece of structure, the bipod fitting itself, and the structure it's on. So during the re-entry part of this, there's really no effect. MR. SPARKS: Going to the next chart, please. One of the questions we are often asked is why don't you just fly one type of cryoinsulation. We're currently flying four types of foams on there, and it's driven mainly because we've got different environments for different locations of the tank. In the areas where we don't have high heating, we'll be flying a polyurethane foam; and the two types of polyurethanes are the BX-250 and a PDL-1034. And on the LOx tank, we'll fly a polyisocyanurate material, which is a little bit higher heat-resistant material; and that's the NCFI series, the 24-124 materials. The thicknesses vary upon the tank also, but the thicknesses are driven primarily to minimize ice formation and if there is additional thickness required because of re-entry, then that's added there upon that design. Next chart, please. Me personally when I'm working a foam issue, I like to think of the issue in four terms as far as structure when it comes to working a foam issue. First is a polymeric structure, and very quickly this is a polyurethane or a modified polyurethane, polyisocyanurate materials that we're talking about. That forms the basic backbone of the polymer and generally determines the strength of the material. It also determines the strain capability at cryogenic temperatures. Polyurethanes are extremely compliant at cryogenic temperatures, and that's the reason why we use these materials. There are very few materials that can take that strain. The next level of structure would be cellular structure. Generally, it's very important to at least understand your cellular structure. We'll look at a few pictures here. As you see the sort of semi sort of random behavior of those cells, certainly they are important in that some of your thermal insulation characteristics are driven by your cell structure. Knitline geometry. This material likes to be sprayed in fairly thin passes. In other words, if you spray it very, very thick, all at one time, it tends to pull away from itself upon cure and forms stresses. So it is better to spray in passes. So what that does is once you spray a pass, it skins over on itself and the subsequent pass forms what's called a knitline. There in that bottom picture is radiograph of some materials that have been sprayed on to a substrate. That's complex geometry in the intertank region. That's a rib geometry. But the radiograph magnifies the appearance of the knitlines, just to show that feature. The strength can change due to that knitline structure. In the region of concern that we've been talking about the past few weeks, the bipod area, especially when you manually spray an area, it's very hard to determine from part to part an organized or a specific structure as far as the knitline geometry. On the automated sprays, the barrel sprays on both the LOx tank and the hydrogen tank, you have more of an order to those knitlines. Finally, geometry. A flat panel with foam on it, that foam's going to react differently if that is sprayed upon, say, just a rib geometry, for example. We found in some in-flight anomalies a couple of years ago taken on a thrust panel that that material would perform nominally on a flat panel but when applied to a ribbed situation that the expansion coefficient pushed up and the stress became great at the tops of the ribs and contributed at least to the loss of that material in that area. If that had been on a flat substrate, that effect probably would not have been demonstrated. Next chart. Again, here's some photographs of some foam blown up. You can see in this picture here the story is mainly the cell structure. You can see the semi sort of random structure, what I call the football nature. The rise direction is going vertically, and you can see that it is preferential to rise direction. You can also see the bar there, being 0 microns, and the picture just a little bit lower is 0 microns. It's the same photograph blown up. Fairly small cells. That is one of the key elements of this foam is that it's close cell and that it does have a very low thermal conductivity gas in those cells. What you're looking at are struts that form on the outside of the cell in what I would call windows that maintain that gas in that cell. Again, polyurethane is a very compliant material. Also foams, all of us here are sitting on polyurethane foams right now that's just not a rigid foam. The chemistry of that material is just a little bit different to make it flexible. So it's a very compliant material if formulated in that fashion. Knitlines. You kind of have to look closely there to see that knitline, and that's a 00-micron bar. So knitlines can vary in thickness, depending upon the spraying conditions and also the time allowed before you spray the next pass. That was just a picture to show you how thin a knitline can be and also how it is knitted, more or less a continuous polymer running through that area. By the way, this is material that has been pulled off just recently from the ET120 dissection that we're doing out at Michoud. This is just a random anomaly that I picked out of the laboratory and showed. We have rollover phenomena; and that phenomena occurs generally when you have, I guess, a complex geometry underneath it that you're spraying. The rollover, when you spray foam, it will push up on itself and start to rise; and if you have a complex geometry, it won't fold over on itself, much like a wave in the ocean will fold over on itself and it forms a small void. Can we hyperlink that? Can we show that video, please? Talking about the relative hardness of the material. This is going at approximately 700 feet per second, which is visco-elastically. You see the foam. That's a 3-inch piece of foam, about an inch in diameter. BX-250, the material used in the bipod. If you can click that again and show that again, please. Maybe it has to quit before you click it again. Undoubtedly, you see the flexibility of those struts and that material able to absorb that energy, and then finally the shock wave does break it apart. We haven't looked at those materials yet or at least I haven't seen the analysis, the electron micrographs of those materials, but we're going to look at that and I conjecture that those windows in that cell that we're looking at are probably burst but the struts may be somewhat maintained. So the material looked like it was still holding together somewhat, even though the pressure in the cells probably were blown out. That was a load cell. I think that was a steel load cell. They were trying to understand the amount of energy in that material. MR. HUBBARD: Two questions here. When it says chilled, how cold is that? MR. SPARKS: I believe they submerged in liquid nitrogen and it was a best effort to take the foam bullet, put it in a sabot, and then fire. I believe it was around -- MR. FOSTER: Minus 38 degrees or something. It was only chilled. It wasn't cryogenic temperatures. MR. HUBBARD: C or F? MR. SPARKS: F. MR. HUBBARD: I mean, minus 38 -- MR. SPARKS: Fahrenheit. Okay. Go back, please. MR. HUBBARD: And the little stripes in what looked like five segments along your column there, are you highlighting the knitlines, or is that something else? MR. SPARKS: That was half-inch gradations, just showing that was a half inch. MR. HUBBARD: Oh, to see the compression. MR. SPARKS: Correct. Next chart please. Very quickly, this is a top-level chemistry view. One of the things that we're talking about is polyurethanes in the form of BX-250. On the side wall we're talking about NCFI materials; and that's a polyisocyanurate, which is a modified polyurethane. The difference between the materials generally can be explained here. You have a general polyurethane reaction occurring between a diisocyanate polyol. It forms a very flexible urethane linkage. On the lower half of the chart, it describes the first reaction for the polyisocyanurate. It's a trimerization reaction that then undergoes urethane reaction with its R components. It's a little bit more ring structured which forges a little bit higher heat resistance. This comes into play when we look at processing conditions. One of the reasons why we use polyurethanes in some locations is that we can spray it out on a floor because the substrate does not have to be heated. For polyisocyanurate processing, the substrate has to be heated. One of the reasons why is because this reaction here is a little bit slow in kicking in. So you have to give it a little bit of help thermally to kick in to start the reaction. Next chart, please. Again, a little cartoon here showing the constituents of NCFI. I just chose NCFI as an example. You have a Component A and Component B. The Component A is the isocyanate, Component B is a polyol and all the other ingredients such as blowing agent, flame-retardant packages, surfactants, and catalyst packages. ADM. GEHMAN: Is this a good time to talk about blowing agents, or are we going to talk about it later? MR. SPARKS: Let's go just a little bit more. Next chart, please. This is really an eye chart, but it is in your package and I wanted to include that so it would be in your package. Maybe what I just want to speak to is the blowing agent issue. I listed the HCFC material on the top, and the CFC material is the second material in the top row, materials that have been transitioned away from. One of the questions we're asked often is, generally, from a material properties perspective, what happens when you transition from an HCFC to a CFC. Generally, what we've seen and what this chart points out fairly well is that at room temperature and elevated temperatures your tensile properties and compression properties went down a little bit only on your NCFI series of materials. The other materials, the PDLs and the BXs and also the cryogenic properties of the NCFI materials seem to be equivalent or superior with the HCFC materials, blowing agents. MR. HUBBARD: One question before you leave this chart here. I think I'm correct in saying that this column here is the bipod ramp material, correct? MR. SPARKS: That's right, Mr. Hubbard. MR. HUBBARD: Specifically, BX-250? MR. SPARKS: That's right. MR. HUBBARD: One of the issues that people have been debating is how heavy a piece it was that fell off the bipod ramp and hit the wing leading edge. I notice that there's a range here and the density which, of course, tells you how heavy it is; but you have a typical number. How typical is the typical number? If you were to go take 15 samples, would they all be very closely grouped around 2.4 or are you going to see this full spread which is something like, you know, a 40 percent spread? MR. SPARKS: Right. If 2.4 was typical in an area, the foam is going to give you variation. It's going to give you variation in mechanical properties. It's going to give you variation in the density. I would presume a 2.2 to 2.6, that much of a spread; but that's just a guess, Mr. Hubbard. It might span that range. I don't think it's going to go down to 1.8 all the way up to 2.6, but it's going to come close probably. I think Lee's got a chart also that might discuss that a little bit also. Next chart, please. Moisture absorption. I did pull some limited information, but I did want to present that. The bottom line of the story is the material is fairly moisture resistant as far as to absorption. This is a study that was done, again, back in '98, I believe, done upon 1-foot-by-1-foot panels that had a substrate. They were sprayed upon a substrate. So they were exposed on top in accelerated exposure chambers, at 7 days for 125 degrees F, percent relative humidity. You can view the amount of moisture gained for the NCFI 24-124 at .12 percent. The BX material's at .16 percent; SS, .42 percent; PDL, .83 percent. Personally again, in working with a lot of foam materials and measuring those foam materials, those essentially are about the same because you're going to see a lot of scatter in the data that you receive lot of times from those materials. It would be hard for me to say that there is a difference here. I tried to go back and find the numbers of samples that each of those numbers were up against and I couldn't find that, but I would guess that the range certainly you couldn't differentiate between any of those as far as moisture gain. MR. HUBBARD: Do you know of any studies done, instead of at 125 degrees, closer to freezing? MR. SPARKS: No. We're looking at that. We've been made aware of that. We're going to look at that and investigate that possibility. We know that possibly that might be linked to the chemical formulation, the ethylene oxide or propylene oxide ratio. We're also going to try to figure that out and see if it's applicable to our cryogenic situation. One of the issues, though, Mr. Hubbard, the tank very rarely would be at 32 degrees, being at Florida. Say, if it was frosty during loading, it would be for a limited amount of time; but still we're going to check into that and make sure we run that down and possibly set up some tests to look at that. Next chart, please. Actually this is a chart that I presented a few years back, just a high-level chart of some of the things that we do when we go off and try to look at qualification. Physical properties, we look at bond tension. In other words, material that's been sprayed on a substrate. We test it all the way down from cryogenic temperatures up to positive 300 degrees F. We do a flat-wise tension, which is blowing ice, just looking straight at the foam material. We do plug pulls, density, and compression on those materials. To give you a rough feel, probably maybe several thousands of those tests in that test series. Mechanical properties. Cryoflex is a very severe strain, checking the ultimate strain capability of that cryogenic temperature material. Monostrain is getting design information as far as modulus, and we do that at cryogenic temperature and elevated temperature. We do some shear and some Poisson ratio. Again, a lot of these pieces of data are feeding into analysis; and we're doing, again, a swag, thousands of those. Thermal properties. Thermal conductivity, we take it down to cryogenic temperatures and measure it all the way up 200 F. We look at the oxygen index. In other words, what percentage of oxygen. Is it flammable. We look at the flammability as far as its flame capability of extinguishing itself. Specific heat and TGA, more or less looking at when the material starts to lose its weights as you increase the temperature. We do aero-recession and hot gas wind tunnel and we do thermal-vac, which is a synchronized radiant heating and vacuum profile. We probably do hundreds of those tests. Then we do major flight acceptance tests that are more or less all at config tests. Of course, you don't do as many of those, but those ultimately receive a little bit more visibility and really have a little bit more fidelity as far as representative of the hardware. Next chart, please. A processing chart. Again, this is for BX-250. The message for this chart is, looking at the two bipods, the ET 93 -Y bipod and the ET 115 -Y, that did shed debris recently. We went back and looked at the processing conditions to see if there was anything outstanding about those. To this date, we haven't seen anything that's really sticking out. I very quickly put in a processing chart here. The white box -- you can barely see it on this chart -- is more or less the invisible processing area that we can conduct our activities. They're grouped in that certain area there because that generally is the temperature and humidity inside the factory at Michoud. Qualification tests have been run at the corners of the box, and you generally get about as much variation from a sample down here and a sample up here as you do if you get two samples in the middle. Again, foam sometimes can be quite frustrating in terms of data analysis because it does have certain variations in the material. Next chart, please. Again, looking at mechanical properties of the past few bipod ramps and looking at the 112 and 107 bipods. Both are falling in the population average, if you will, of those I think being sprayed. Almost going back to ET 106 through ET 116. These two points here, the chart is not very clear on that. Again, this kind of demonstrates the variability sometimes we'll see in the material. These two low values were pull, and requirements are that you pull right next to it to see if it was just a variation of material. I believe on this one it's a , and on this one it's a 60, pulled right next to it. That's one of the issues that you have often with performing plug pulls is that you will get a bad plug pull where the value will be low, but right next to it, it will be just fine. If you dissect the material, it looks just fine. Next chart, please. We have these charts for all the different materials; and this is just kind of walking through, I guess, more or less a day in the life of a person that follows cryoinsulation. It's fairly frustrating as far as obsolescence issues and as far as other issues mandated from other organizations. BX-250 to SS-1171 to BX-265 is a good example. Originally, of course, BX-250 was the original ET material chosen for ramp and closeout applications. In '93, the CFC 11 blowing agent manufacture was discontinued. It was because of the accelerated EPA date. In '95, the SS-1171 material was chosen to replace the BX-250; and we secured the available stock of CFC 11 to use with the remaining BX-250 that we had. In '95, we had a flame retardant issue. We have to obtain some material from overseas to back-fill. In '98, production issues identified with the use of SS-1171 sort of making us scratch our head. This is about the time that we were qualifying all new materials going from CFC to HCFC materials. What was occurring with these processing anomalies were the SS material was processing just fine in component shop, a little bit more control of environment; but on the floor it was not processing as easily. In 1999, again, SS was continuing to have issues; and we discontinued that material in 2000. Mondur Dark was the type of polyisocyanurate used in BX-250. It was phased out of production. In 2001, BX-265 is qualified to replace BX-250. Stepan is the manufacturer of BX-250, and that's the BX-250 material with a HCFC 141b blowing agent. And we implemented in 02, 2003, EPA phase-out of HCFC 141b. A waiver approving that exemption was granted just recently, March the 5th, 2003. That's generally just the life and times of somebody trying to work these issues with the materials sometimes when the raw materials are becoming obsolete. ADM. GEHMAN: BX-265 doesn't appear on your generic tank. It's used in the acreage and replacing BX-250 now. MR. SPARKS: That's right. I didn't really label it very well. The previous tank, that was ET 93 configuration. On that real big eye chart, you'll notice the transition in the upper right-hand corner from BX-250 to SS-1171 to BX-265 did include that material there. So that material will be phased in and used in the areas where BX-250 is used now. ADM. GEHMAN: And the shift of blowing agents back in '93 was done strictly to comply with EPA regulations, not because there was a better blowing agent or your blowing agent wasn't working or anything like that. MR. SPARKS: That's correct. All right. I'm going to hand the ball off to Lee here. MR. FOSTER: Okay. Scotty's first chart said the TPS had to take the flight environments and protect the structure. This is a sketch showing what some of the environments are. External tank, as also the rest of the elements, have to take the aerodynamic loads and the heating. We show this as hot spots, like on the front where you have high aerodynamic heating. On the back end of the tank, you have plume-radiation heating and plume recirculation. You see in front of the orbiter and SRB noses that there are shocks generated that all impinge in the intertank region and even some of those shocks coalesce and they're shown as separated flow and recirculation region right ahead of the orbiter nose shock. As you can see from this, a lot of the areas on the intertank and specifically in front of the bipod are a very complex region. The next chart is a computational fluid dynamics chart that basically we borrowed from JSC, and it is to show the complex flow field. I'm not really going to go too much into that. I'm just going to let you look at the pretty lines and see that the flow is going every which way. ADM. GEHMAN: Can you point out if there are any shock-shock interfaces or reinforcing places in here? MR. FOSTER: Well, yes, I can. The previous chart showed the shock coming off the nose of the orbiter. It's impinging there. The SRB on the other side here has a shock coming through this way. You can see the flow from the nose of the left-hand SRB here. So it all coalesces into this area. You can see that we're getting some vortices formed here and it also has the LOx feed line here that influences the flow. ADM. GEHMAN: And you point out the left bipod ramp. The density of the lines indicates more stress, I guess, or aerodynamic pressure? MR. FOSTER: I apologize for not being able to answer well the CFD. I can barely spell it. I told you I was an old technical guy, and this is a lot of new stuff here. But, yeah, I guess it's like watching the weather. When the lines are close together, it's higher pressure there. I can get back with you with the specific numbers there. DR. LOGSDON: One more question. Foam came off on 107, about 81 seconds into the mission. Is that the Mach speed at 81 seconds? MR. SPARKS: Yes. DR. RIDE: Just one more. From this picture, you know, we're looking more directly at the left bipod. I can't quite tell whether the flow around the right bipod looks the same. Does it, or is it just the perspective? MR. FOSTER: No, it is different because of the presence of the feed line here; and this particular solution did not have real high fidelity geometry upon the right bipod. They're working that. This is a chart that's used for illustration here. DR. RIDE: Okay. So you would expect the flow to be the same around the left bipod and the right? MR. FOSTER: No, it's going to be different. We can get those numbers for you, but what we've shown with our flight history is that if we have good foam and it's not affected by, I'll say, some of the hypothesized failures we have -- and I'll show you later on -- both sides take the environments. We'll get into that in just a little bit. Next chart, please. What we're doing here is looking specifically at foam loss and debris. There are three things that we, on the ET side, look at to quantify the debris for us. One is the ascent photographic coverage. You know we have hundreds of cameras watching the ascent. We have groups at each center that look over those things and try and identify if there is debris coming off at whatever times they can identify it. We also have the separation photos that are in the umbilical well cameras. These, of course, don't come back until the orbiter does. We also have several occasions where the crew has the hand-held cameras. Those are usually not quite as much information that we get from that 'cause it's a while before they can take those. Also, after each flight, there's the orbiter tile damage assessment; and we look at all of those things to try and quantify what kind of debris we're getting from the tank. There were some additional methods lately. We had several SRB cameras to look at the intertank region. That was a result of IFA 87, which I'll talk about in just a little bit; and we had one flight where we put a camera on the ET. It was really a very neat view until, at separation, the BSM clouded the lens. Next chart, please. MR. HUBBARD: Before you leave that one. No. 3 there. Post-flight orbiter tile damage. Is it your understanding that the tile damage that is seen every flight mostly derives from ET debris? MR. FOSTER: Not really. Let's go to the next chart, and I'll answer it there. This is the number of hits on the lower surface of the orbiter. There's also charts for the side and the top and all that. The blue here is the total hits on the lower surface, and the red is the hits that are judged to be greater than 1 inch in diameter. There's some rather large numbers, you know, of total hits. I guess we can average somewhere in here. A lot of those are very small, that are due to other things than ET foam debris. Like on the aft end of the orbiter the heat shield, you have a lot of ice forming on the SSMEs and the aft heat shield and you get little dings, lots of those. There are areas where you get some ice, I guess, from the attach points, the orbiter ET attach areas. Usually there's a lot of dings around there. It kind of goes to a baseline number somewhere in the 13 to 25 hits greater than inch, which I'll again get to in the next chart, if we can go to that. What you'll see here is where we had ET debris events. We had some higher numbers. I'm slowly getting around to answering your question, sir. This is the same data as was on the previous charts, only this is the hits greater than 1 inch. First let me talk to this one at the very top. That's STS 27R right after we got back to flight. That was a very large number of hits. Most of that was caused by SRB debris. There was a large investigation that worked that, and so I'm really not going to talk to that particular one. We will talk about these areas where there are large numbers that we say are correlatable with the ET debris. Then the rest are very small numbers, relatively speaking. So, yes, we can tell when it's ET derived damage; and I'll show you how we have correlated some of those and what we've done about it. GEN. HESS: Before you move on, have there been any instances where you have foam striking on the RCC that have been documented? This is just tile acreage mostly, is it not? MR. FOSTER: I really can't answer that question. We use the data that's provided by KSC, the orbiter damage maps; and we're looking at the numbers here. So I'm not the right one to answer that question. MR. SPARKS: To my knowledge from the laboratory perspective, I've never been informed that the RCC was damaged due to foam debris. That's not to say that it hadn't been. I've just never had knowledge of that. MR. FOSTER: Next chart, please. This is an umbilical well photo from STS 26, where we had a very large number, 179 hits greater than an inch. Let me point out that there is an area around the flange, extending up into the intertank and then around the feed line fairing, where we have what we call two-tone foam. This was initiated when we went to the lightweight tank series, and it was an attempt to reduce the environments by filling in stringers with BX-250 foam. Then we could spray a smooth layer of the CPR on top of that and reduce the environments. That worked quite well, and these data start at the first lightweight tank. It worked quite well until STS 25. And then STS 26 -- 25 we did not have umbilical well cameras; 26, we did. These were flights that were three weeks apart. This is where we had a sub-tier vendor make a change on the isochem material that we put between the two layers of foam. And this caused a reaction and got a blister area, a void that then popped off during flight. You can see there some rather large areas where we had divots come out. So after this flight, we went to a process of drilling holes in all of these two-tone areas, on 3-inch centers, in order to relieve the pressure so the foam wouldn't divot. And it worked quite well. As you see, the numbers went down. ADM. GEHMAN: Is this an ET separation picture? MR. FOSTER: This is ET separation, umbilical well camera. ADM. GEHMAN: Oh, but it's from the umbilical well, not from the crew hand-held camera. MR. FOSTER: Yes. DR. RIDE: Can I just ask a question on your numbering system? STS 26 was return to flight? MR. FOSTER: No, that was 26R. I do have to apologize here. What I did was sorted these data by ET number; and as you're well aware, the numbering system was really messed up. So this is not in chronological order. Case in point: 27R is return to flight, and 27 was way before. So although on this chart those data would be together, you know, chronologically they're far apart. DR. RIDE: So could you just tell us what flights these referred to? MR. FOSTER: This is STS 26 -- I've put down the STS number; and a little later on, where I talk about some of the efforts we made to reduce the debris, I'll talk specifically the ET numbers here. DR. RIDE: I just needed the STS number. The flight labeled STS 26 -- MR. FOSTER: Yes. That's correct. That is STS 26. DR. RIDE: That is STS 26, the return to flight? MR. FOSTER: No, ma'am. STS 26. There's R. In our wisdom, we've flown an STS 26 and a 26R. DR. RIDE: Okay. What's STS 25? MR. FOSTER: STS 25 was flown in June of '85 and STS 26 was flow in July of '85; 27, in August of '85. So there were three of them right close together there; and then, as I said, the 27R, this one up here, wasn't until December of '88. So I apologize for not putting these in chronological order. DR. RIDE: So the one you labeled STS 25 is actually before the Challenger flight. MR. FOSTER: Yes. DR. RIDE: So it had a different designation then. MR. FOSTER: Yes, it is. DR. RIDE: And it was not the th flight. MR. FOSTER: Right. STS 25 is close to -- it's the early 20s, I think. It's hard to keep up with. I'm sorry. I'm going to redo this chart with everything done in chronological order. DR. RIDE: It would just be useful to be able to track these back to the actual flight numbers. MR. SPARKS: We can get that. MR. FOSTER: Go to the next chart. This is 32R, which is a return to flight. This one, you see we're missing a big piece of foam there that people have looked at and said, oh, that's a bipod missing. What you've actually got is -- this is, again, the two-tone foam area. We see that we have lost the foam in that two-tone area and it has taken the first part of the wedge from the bipod. So really the bipod foam loss here at the front edge is a result of another divot as opposed to being, quote, a bipod foam loss. This one here, I've got it shown 13 hits greater than an inch caused by this amount of foam coming off. MR. HUBBARD: Would you just remind us why 1 inch is an important number? MR. FOSTER: That 1 inch is -- I guess the system came up with that break point because they were getting very large numbers of total hits. So they wanted to come up with some criteria of things they should look at for trending so that they might want to take some action if they saw a large number. MR. SPARKS: I think they had numerous very small-speck hits they didn't attribute to possibly debris falling from the external tank. So they wanted another classification, and that's where they drew the line. Of course, it was easiest to say 1 inch. MR. FOSTER: Next chart, please. This is STS 47; and as you can see, there were one large divot here, a bunch of smaller ones, and even something on the outboard side. The purpose of putting this chart in here is twofold. One, the damage result was only three hits greater than an inch. What I'm attempting to show here is that it's a time-dependent thing, depending on where you lose the foam. Now, going back -- I don't have any information of exactly what time that came out, but if it's early in flight or later in the ascent flight, you're dynamic pressure is not at its maximum and so you don't put as much momentum on a piece coming off and therefore it's not going to have as much damage to the orbiter. So there's a lot of people studying the transport of debris; and it is a function of when it comes off, how much damage it can do. STS 112, we had a very large piece come off, but it never hit the orbiter at all. By the way, this second point here is that even though there were only three hits greater than an inch, an IFA was taken on this tank, to go and investigate why you're losing foam. MR. HUBBARD: Just to be sure I understood that point you just made, which I think is an important one, is that it depends on when in the flight the foam shedding occurs, how much damage a given piece might cause? MR. FOSTER: Yes, sir. Both from the trajectory -- the transport over to the orbiter. Because the flow field is constantly changing and then also the amount of entrainment you can get in the flow and therefore the more damage potential. MR. WALLACE: Sir, you said on STS 47 an IFA was taken. MR. FOSTER: Yes, sir. MR. WALLACE: Was that the decision or recommendation of the external tank project then? MR. FOSTER: Most IFAs, I believe, are a system call which the ETs along with everybody else is in the decision-making process. I don't think I can say that it was something requested by the ET here or whether it was just the system said, you know, this is a big piece of debris, we need to go look at it. I really can't answer that question. MR. WALLACE: Do you have any further recollection as to whether it was a constraint to flight or what actions were taken? MR. FOSTER: I know it was not a constraint to flight. All of the debris that we have here has been judged by the system as not a safety-of-flight issue but a maintenance issue; and we have all in the past been involved in those decisions. Rightly or wrongly, they were all declared a maintenance item and not a safety of flight. MR. WALLACE: Might affect the turn-around of the orbiter. MR. FOSTER: Yes, sir. The next chart is STS 50. This one had hits greater than an inch, but it was one where we lost the bipod but, again, in this one it was initiated in that two-tone region. Now, you've heard a lot of the two-tone. After STS 50, we changed away from the two-tone; but this one is one we looked at recently where we tried to get a solid model to show what the dimensions were. The weight calculated for this particular area, which included the front of the ramp and a little bit of the two-tone area, was about a pound. MR. HUBBARD: When in flight did this one occur? How many seconds after launch? MR. FOSTER: I don't have that information. MR. SPARKS: I don't know if we know that, Mr. Hubbard. MR. FOSTER: We asked the photo guys to go back and look at all of these; and, quite frankly, I haven't seen the results of that yet. I think, though, that they said they did not see this piece come off during flight. MR. WALLACE: So in some cases you only know that it happened when you see the separation? MR. FOSTER: Right. MR. SPARKS: I think one thing that they're additionally doing also now is if they came back with a, well, we did see it come off, I think also they're going out and saying, well, this is the window that we did not see it come off also, which would be helpful. And I think they're working that right now. MR. HUBBARD: Maybe this is a good point to ask a different version of my earlier question. If you go back to -- you don't have to go back on the slides. But on Slide 17, the data commonly available for assessment. You have ascent photos, orbiter separation photos, and post-flight tile damage. If you were to look at all the flights and say what is the preponderance of the data that you're using to assess what goes on, which one of those three would stick out as where you have the most data? MR. FOSTER: Well, basically the orbiter tile damage, you know, we have that on every flight. It's easily done. It's numbers that you bean-count. The umbilical well cameras, sometimes you're launching in darkness and so you don't get good coverage. We have one orbiter that doesn't have the umbilical well camera. So that's some information that is -- I don't even know what percentage of the time we get that. It's over 50 percent but by no means 100 percent. MR. HUBBARD: And you may or may not happen to catch it as it's coming off. MR. SPARKS: Right. Or the camera may be out of focus or a cloudy day. MR. HUBBARD: So is it a fair statement then that, by and large, we know what we know about the damage that external tank debris-shedding causes, by virtue of looking at the tiles after the fact, with some other data tied in? MR. SPARKS: Right. MR. FOSTER: We look at whatever we can to get information. MR. HUBBARD: So do you feel then, given where the data comes from and how much you have got, you feel fairly confident, then, that there is this direct connection between the tile divots, at least the larger ones, and the external tank debris? MR. FOSTER: Yes. MR. SPARKS: Let me take a cut at that because the tile count, if you will, when it gets back, is the one thing that's always consistent. You're always going to get that data, but it is confounded. That's the reason why it's so important to get ascent photography or separation photography. You know, the tile count is confounded. So any of that data that we can get upon ascent, upon separation, on crew hand-held are value added. Very much so. DR. RIDE: Could I just ask can you characterize roughly the number of flights or the percentage of flights where you've actually had ET SAP photography or ascent video that clearly shows the bipod ramp? What I'm getting at is: How do you know what percentage of flights foam has really come off the bipod? MR. FOSTER: I don't know that we can make statements with certainty. All we can say is that by looking at all these resources we have, we can see things like this that give us that information. The ones we don't know about, it would just be guesswork. However, a lot of them that we could not see, we also did not have big debris damage. So I'm not sure if there's any comfort in that. GEN. DEAL: Rephrasing her question a different way, do we know how many we have seen either through the separation or hand-held? Because we've got the ones at nighttime we definitely didn't see and we've got the ones where we didn't get the camera shots out of or where the tank had rolled around. Do we know how many we have seen? MR. SPARKS: We've got that. I don't have that, General Deal, on top of my head. I have seen it, but I just can't remember what it was. GEN. DEAL: 'Cause we throw around terms, you know, four out of 112. It may be a lot more than that 'cause we can't confirm that. DR. RIDE: Right. That's what I was getting at in a pretty badly phrased question. How many tanks shed debris where it could have come from the bipod but we just don't know because we didn't have the photography. MR. SPARKS: And I think in between 112 and 107, I believe 113 was a night launch, if I recall correctly. MR. HUBBARD: If you expand the question to the whole external tank and all of the foam that you've got there on the acreage, is it fair to say that if you look at any one of these plots that go up through more than the 100 flights there that all those little red triangles probably, or many of them, probably relate to the external tank? MR. SPARKS: I would say the majority of them do, Mr. Hubbard. That's Scotty speaking, though. MR. HUBBARD: Okay. MR. FOSTER: Let's go to the next chart. Well, before then, let me answer that question, the previous question just a little bit more. We think we have evidence of five flights, I think, where the bipod has come off. Of those, the ones I showed on the previous charts, we don't see the bipod as being the initiating mechanism. That two-tone foam was. So really it's kind of, well, we've only had a couple that we know of that were bipod alone. This chart shows STS 87, which was 109 hits. This one was the initiation of the IFA 87, it was called, because we had a lot of popcorning type foam loss on the thrust panel side of the intertank. That was worked very hard through the investigation procedure and it has been handled with the application of thousands of vent holes -- ADM. GEHMAN: This was the first flight after the shift of blowing agents, right? MR. SPARKS: It was the second flight. ADM. GEHMAN: After the shift of blowing agents? MR. SPARKS: Right. MR. FOSTER: Next chart, please. ADM. GEHMAN: Let's go back one before we get off that chart. I haven't done any kind of a scientific analysis, but we've looked at about seven or eight of these charts now with those little red diamonds down across the bottom. By rough order of magnitude, it looks to me like the number of hits greater than an inch is a straight line, a straight horizontal line. It's not obviously diminishing. MR. SPARKS: Correct. It looks like it's averaged about 16, 17, I believe, 20. I think I ran the numbers before I came in. For the CFC materials, it was 20 some odd; and for the NCFI materials since the full-up venting, it's been, I think, about 16, 17. ADM. GEHMAN: The point is the trend is not going down, not by any order of magnitude, anyway. MR. SPARKS: Correct. MR. HUBBARD: Is that taking out or leaving in the large events? MR. SPARKS: The CPR numbers are taking out that 27R event. I did take that one out. So it would run it up just a little bit. MR. FOSTER: Next chart. This is a list, a not completely comprehensive list of everything we've done but a list of efforts to reduce debris. I apologize that the font is so small on this. You could probably do better reading it on your handouts. STS 1. We had some instrumentation islands on the LOx tank. There was a concern that we were going to make ice on those. So we removed them until we could verify that instrumentation islands wouldn't form ice. So, you know, we've been concerned from Day 1 with debris formation. When we got to the lightweight tank series, which started with ET 8, this was a block change to the lightweight tank series and it enabled us to go do a few things to help reduce debris. One of the things was redesign of the bipod ramp angle from degrees to 30. Now, this was done on lightweight Tank 7. So these things I talk about as a block change are incrementally implemented; but that was to reduce the loads, the air loads on the bipod ramp. Now, STS 7, which I do not have a -- well, I guess I do have a picture somewhere in here. STS 7, at any rate, had bipod foam come off, but there was a very large repair done to the bipod ramp and it was judged that that was the key driver for losing the bipod ramp on STS 7. So we did two things. One, we incorporated the maximum repairable defect limit on the bipod ramps, said if you have to repair more than this size -- and it's a very small size -- take it off and start over again. And also we changed the ramp angle, saying that's going to reduce the air load. So those two in concert should really help the bipod ramp. Also on some of the STS 7, we saw that cable tray ice frost ramps had come off. The block change to lightweight enabled us to change to a two-step single-pour application process versus the old one-step multi-pour process, and what this did was gave better structural integrity to those ramps. We also reduced the super-light ablator areas on the tank. We had large areas of the super-light ablator running all the way down the pressurization lines, and we removed a lot of that and also deleted the anti-geyser line. So there were a lot of things done at the lightweight tank initiation, one of which was incorporating the two-tone foam configuration. That was an attempt to reduce the environments and help in foam loss prevention. ADM. GEHMAN: What does two-tone have to do with it? MR. FOSTER: Two-tone was the area that I showed around where we filled in the stringers. What that did was reduce the aerothermal environments in that region by having a smooth surface as opposed to localized stringer effects. It turns out that was probably not one of our best decisions; but, you know, we weren't planning on the vendor changing in the material. On STS 27, we saw some large intertank divots that I showed you, the umbilical well camera for STS 26. And a corrective action was to drill holes in the two-tone areas to take care of the debris use to the isochem bond line issue. STS 32R in 1990, we had the intertank and associated bipod part come off. The problem there was the vent holes that we were drilling did not go down far enough. So they pin-gauged them to make sure everything was going down the right amount, fully vent this area where we were getting de-bonds. The STS 35 in 1990, also there were ten areas on the flange where divots were observed. This started a process to investigate why we were getting flange divots, and the result was that there was an improved process to spray the foam around the flange bolts. They were getting a void underneath the bolt because of the spray pattern. They changed the technique for spraying it so that you could ensure you weren't getting a void underneath there. That helped and we're still getting flange divots, but not as many as we were before that change. So it's gone in the right direction. STS 50 in 1992. The jack-pad area, which is an area between the bipod where we have a tool helpful in holding the bipod during mating operations, when you remove that tool, you have to close out that area. The method that they were using led to void areas. They changed the process to keep from forming those void areas. Even though I don't have it on this chart, there were two or three other changes made specifically on the jack-pad to ensure we didn't get those coming out as debris, the foam in that area. And, Scotty, do you know? Have we seen jack-pad area debris recently? MR. SPARKS: It's performed very well since that configuration change. MR. FOSTER: STS 46. Again, this was the result of the observation on STS 50 that there was an intertank/bipod divot. Added some more vent holes right in front of the bipod ramp in that two-tone area to try to decouple those things and see if we could keep the intertank two-tone region from ripping off the front of the bipod. Finally in STS 54, ET 51, because of all these previous problems that we talked about on the two-tone foam on the intertank, we incorporated a two-gun spray foam application to replace the two-tone foam. So ET 51, STS 54, was where we got rid of the two-tone foam. STS 56 in '93, we saw ten large divots on the -Z intertank acreage area and there was a study that looked at that and the process was changed in order to try and reduce the rollover and crevassing that Scotty talked about a little bit earlier. MR. SPARKS: I think there were some processing changes that were made, and that process has also been approved, has improved the performance of that intertank area. MR. FOSTER: Then STS 87 was the popcorning of foam off the intertank, and there was an increase in the number of tile hits. So there was the large IFA effort that the external tank program went through, and basically we incorporated the vent holes to keep that from happening and that has worked very well. On 112 we saw the bipod foam loss. This was at 32 seconds, I think. It basically was the first bipod foam loss that we could say, you know, this was not associated with the two-tone; and it was the first thing that we had seen in quite a number of years. So there was a corrective action that was kicked off, and I won't go into what they were really going to change there. They were going to remove SLA from under the foam; and that is hypothesized as one of the factors that can lead to bipod foam loss, which I'll get into in a minute. I'm going to switch over to cryopumping and cryoingesting before we have any other questions on this. MR. HUBBARD: Before you leave that and go to cryopumping, this is a very impressive list of all the things that have been done over the last years to address the shedding of external tank debris. Nevertheless, if you go back to any of the charts that have the red triangles that indicate the divots greater than an inch, which is one of the characteristics that you look for, the line is pretty much a flat line there. I mean, whether it's 10 or 15 or 20 or whatever. So do you see any way to drive that line down to zero or near zero? MR. FOSTER: I'll let Scotty go first on that one. MR. SPARKS: Well, I think we're always trying to improve the product, but we don't want to change the product unless we're justifiably sure that that's going to improve the product. One of the things we did that's not captured on this chart is we changed from a nose cone that did contain insulation to a composite nose cone that has no insulation. That took us completely out of the realm of shedding debris, of course, in that area. So that's one of those things that you know you're going to remove a failure mode out of the way if you do that. So that's one of the things that has happened. So there have been several improvements that I think the program or project has been proactive in pursuing. Indeed, there's still a level and, you know, they're coming from those closeouts in that intertank region that seem to be problematic. We try to improve our processing to the extent possible, but thus far it's staying in that 15 to 16 range. MR. HUBBARD: To follow on that a little bit, I guess if I had a problem that, over 20 years, the average stayed essentially constant, it seems to me that that might argue something about the basic chemistry or basic properties of the thing you're dealing with, the foam itself. I mean, do you see the foam as being difficult to control in a very precise manner? MR. SPARKS: No, I don't Mr. Hubbard. Really what I'm seeing -- again, from my opinion and I think probably a generally held opinion -- is that it's an issue of trying to process that material the best you can. You know, if I had to take a guesstimate as far as the location where we're shedding the most debris, it would be in that hydrogen intertank flange area. That's just a hard area to close out. There's lot of bolts there and when you're spraying that material, a lot of potential for shadowing that foam and possibly having some voids behind that. We've always attributed that to the reason why we're losing some of that material in that area. Of course, the other closeouts. Just a little more difficult. A little bit more random as far as being able to shed that debris. Even though, say, in the hydrogen tank where it seems like the environments, as far as cryogenically are more severe, it's robotically sprayed upon, a very smooth, flat surface. It's those closeouts on complex geometries, I think, that it's tough. MR. HUBBARD: So then just to follow this to the end of my question on it is that it's the system. You know, you've got a foam and it has to be applied over a certain type of underlying structure and making that so that it is free from shedding seems to be, over the last 20 years, a tough thing to do. MR. SPARKS: Yes, sir. Generally, I mean, you've really got to go back to the beginning, as far as the design of the tank. I'm not so sure that the TPS processors were in the same room when they designed the tank, because it was designed structurally to be optimized. It's not designed for the TPS to be processed on there. If you were to redesign completely a tank, you would make the external a bit smoother, you would have those people in the same room, and you would do those trades. You know, if it's worth it, you would do it. So you've got to insulate what you've got, and I think they're doing a heck of a good job. They maintain a lot of skill in that area and, indeed, it's flat line about 15 thus far. MR. WALLACE: I think you were probably sitting in the earlier session today. MR. FOSTER: Yes, sir. MR. WALLACE: There was a discussion about whether this 112 event wouldn't be an in-flight anomaly or not. Can you speak to what the ET project's position was on that? MR. SPARKS: I think the position was that it was a random occurrence of faulty processing and that it was nothing had changed in the system to indicate that that was a systemic issue as far as processing of material. They had gone and done their homework, as far as that goes; but I think when 107 had occurred, I think that would have kicked it into another issue. If I recollect right, I think there was an issue of an IFA pending photographic analysis upon return of 107. MR. WALLACE: And with these two observations, Mr. Foster noted that this was the first time since you had changed, gotten away from the two-tone foam and it was not associated with two-tone foam and also the fact that it hadn't happened in ten years. Would that sort of argue more in favor of or make it an IFA or against that? MR. FOSTER: I guess that would have to argue in favor of making it an IFA. I can say that after the 112 the project did say, okay, we do not want to release that big a piece of debris. It hit the SRB and did no damage there, but still it was a large piece and the project said let's go look at redesign options. MR. WALLACE: When you say it did no damage there, do you mean it didn't threaten the flight? I sort of understood that it actually did some damage. MR. FOSTER: The 112 particle that came off at 32 seconds, it came down and hit the IEA box on the SRB and I believe -- and this could be secondhand information -- but I believe that it didn't do much damage at all to the foam and the TPS on top of the IEA box. I'm sure there's better information available from other people, but I don't think it was a large impact. DR. LOGSDON: Is there a program-level requirement for debris-shedding or lack of debris-shedding on the external tank? MR. FOSTER: The program-level requirement is that we shall release no debris that is harmful to the orbiter. So it's a very subjective thing; and while we have been working hand in glove with the system over the years, you know, we've worked with them on debris teams and the debris panel and all that, again, everything was judged as a maintenance item and not a safety-of-flight issue. I'm not going to say that was right or wrong in the past, but that's the way it happened. Next chart. We'll go on to cryopumping, and I'll go through these rather quickly. I'm sure you've all heard of cryopumping, but the mechanism of cryopumping is simply the transformation from a gas to a liquid at cryogenic temperatures. The little graphic shows barely a little crack from the ambient at room temperature. When you get down to low temperatures, the gases are condensed within a void or it can be a porous medium and when the air in the cavity or this porous material liquefies, which is what happens at structural temperatures below minus 297 degrees F for oxygen and minus 320 degrees F for nitrogen, it can liquefy inside the cavity and what that does, it locally reduces the pressure and basically sucks more air into the void. This is a process that continues until you can fill up the void. Now, in and of itself, that really doesn't bother you. It's what happens when that liquid tries to gasify and come out. If you have a sufficiently large vent path for the gas to come out, you know, no issue. You might see a condensation cloud. If you have not a sufficiently large vent path but one where you crack the foam and get, that way, more of an escape path, you can relieve the pressure without causing debris. But if the vent path is not sufficient, as shown in the bottom sketch, you can physically pop debris off. And we think we've seen that on a few of the flights, like in the flange region where it looks like it's a dinner plate that came out. We can recreate that in the laboratory. GEN. DEAL: Mr. Foster, say for the sake of the argument if you looked at that and you had a piece of tape or something that was blocking that from escaping, that would make it that much worse and cause a divot at that point? MR. FOSTER: Yes, sir. It's a matter of whether it's got enough vent area to get out. And cryopumping is interesting because you can slowly, you know, suck in air in hours as you're out on the pad. But when it comes time to gasify that, it usually happens quickly and you build up large pressure and it doesn't have a vent path to get out. Now, that is cryopumping. Now, we have in the bipod region created a term just so we can communicate. We call it cryoingestion. Next chart, please. This is with a postulated method for getting cryonitrogen ingested into the SLA. Let me orient you here. This is the bipod spindle. This is the super-light ablator that's over the spindle, and our heater element has a wire that comes down here and the wire runs up through this stringer into the intertank. What we're doing here, this view is a view in this direction. So you see the bipod spindle. Here's the wire that comes into the intertank, and the shaded areas are the SLA. Then you've got the foam over top of it. Next chart, please. We have a nitrogen purge in the intertank. We have an area -- during fill, you will fill up liquid hydrogen in the tank and it will go all the way up into the dome. You will get the metal surfaces cold, below the liquefication temperature of nitrogen. So we have our nitrogen purge in the intertank and you're forming liquid nitrogen down in this Y joint region. You also can get the nitrogen purge in through this single stringer associated with this bipod there. We have two bipods, so there's two stringers that have this SLA over the wire, going up into a stringer. In this area you can also get liquid nitrogen temperatures. What I've shown on the right side is that in this scenario that's postulated for cryoingestion, you get liquid nitrogen that is sitting right on top of the porous SLA material and it can absorb into the SLA. Now, this is a photograph of the flange between the hydrogen tank and the intertank. This is an area between shims so that you can have an area that goes all the way into the intertank here. I show that as also being postulated area where you can get some liquid nitrogen to come into the SLA. We don't know if that's a true hypothesis, but we're trying to look at everything to see if there's a mechanism for this thing called cryoingestion to knock off bipod foam. Next slide, please. With time, you can absorb more of the liquid nitrogen into the SLA and at some point your temperatures are going to be above the liquid nitrogen temperature and you won't fill this whole area with liquid nitrogen. The -- ADM. GEHMAN: That's all assuming the heater is on and working, but the heater doesn't work back off the top of it. MR. FOSTER: That's a true statement. This postulation here shows that we form solid nitrogen; and the timing is real critical here, you know, whether you can ingest or absorb the liquid nitrogen and how much you get in here before you get solid nitrogen forming. The key to the solid nitrogen forming is that blocks the escape path back through the stringer. So you could have an area of nitrogen here that during flight could generate pressure to try and push off this bipod. The next chart, though, shows you some temperatures. Here's the temperature on the outboard, on the top of the bipod. You see that it's basically room temperature when you launch and then it goes up with the aero heating, but what's happening outside here doesn't really transfer into this area. The blue line right here is the substrate, the aluminum substrate, and what happens is at this time the liquid level in the hydrogen tank has gone down and so your ullage temperature is warming this area up a little bit -- "warming" being a guarded term because we're still below minus 300 degrees F. The other point here is that this area which is between the SLA and the BX -- or it is that interface -- it really doesn't respond to either of these temperature changes. So there's a real critical timing in both how you get liquid nitrogen in there and how you get it out for this scenario, but it's one that we are looking at very seriously and have a bunch of tests that we're going to run to say yea or nay on this hypothesis. DR. RIDE: What's the temperature of the solid liquid transition? MR. FOSTER: Minus 346 degrees F. MR. SPARKS: Dr. Ride. Nitrogen? Minus Fahrenheit. DR. RIDE: That happens before a hundred seconds. MR. FOSTER: Let me point out that this thermal analysis here did not take into account the effect of nitrogen in the SLA, which would change the thermal conductivity a little bit and would change these numbers. We have programs to try and put liquid nitrogen in SLA and measure the conductivity, but that's a tough thing to do. But we're going about trying to get that. The next chart. Notice in big words this is preliminary graphics. We have gone through the dissection program on ET120. What I wanted to show you was that we do have some defects, rollovers, voids inside the foam and the SLA. I wanted to show you a solid model and make it real pretty, but this is an early shot at it. We'll get better in the next couple of days, but this is showing you where during our -Y bipod dissection. The yellow are little foam items that we saw. Most of these are rollovers. So don't judge anything by the shape here too much. And we had green areas, some SLA items, which are very hard to see. We had a couple right in there. The clevis itself, while it's shown as green, is not a SLA item. We'll be showing you these Thursday, I think, when you're coming down; and the graphics will be a little bit improved. Basically the intent of showing this chart was to say that we have gone through the dissections and we're proceeding on getting ready to go to dissect ET 94 to see what kind of foam we have and what kind of SLA underneath there so that we can take those into account in the testing we do to try and look at what happened on 107. Next chart. I'll let Scotty finish up with this chart here. It's the progress, I guess, we've made. MR. SPARKS: This is a chart showing -- the top picture being STS-7/ET-6. All materials were CFC-based materials. It kind of shows certainly the craftsmanship that has improved to STS 112, a separation photo. You certainly can see a lot of improvements as far as the workmanship of that material. So certainly there's been significant improvement and there's a lot to be corrected, but I think certainly the material and the processing has been improved over the years. ADM. GEHMAN: All right. Thank you very much. GEN. HESS: Y'all have a very rich history with this particular problem, and I can see visually by the chart that improvements have been made over time. My question really is: Did you ever think that it was possible to pop a big enough piece of foam off of this external tank to severely damage the shuttle itself? MR. FOSTER: I'll take a shot at it first. The answer is yes, you know. We have large areas where we have closeout materials that we know are hard to spray. So, yeah, we are always worried that there's going to be a big piece that comes out that would throw us over that maintenance item line. MR. SPARKS: Let me throw in my opinion there, too. I agree with Lee. We watch very closely ascent. That's because we know that that material could come off and cause some damage. So we understand that that's a potential and we understand that it does require a lot of focus on processing that material to make it not do that. GEN. HESS: I get a little bit lost in this characterization that it was not a safety-of-flight issue, it becomes a maintenance issue, which is what we hear on most instances, frankly. MR. SPARKS: You do hear that a lot, and maybe that was because, you know, maybe the predominance of those pieces of material coming off have been small in the recent past, but there is still a lot of concentration, a lot of focus upon not shedding debris. ADM. GEHMAN: Let me ask a question. Have you discussed or ruled out or considered a pre-formed bipod ramp piece of insulation, a molded piece that would be physically attached and that would be in some way reinforced with some structure that would not come apart. MR. FOSTER: Yeah. There's a separate group that's working the redesign options. Scotty and I have been working the investigation. I assume later on we'll transition over to looking at the redesigns. They're doing exactly what you're talking about, looking at ways to keep from having a complex geometry to have to spray or incapsulate. So all of that is working towards making sure that the spray is not too big a challenge to the techs that do it. ADM. GEHMAN: As I understand it, there are other places on the ET where there are pre-formed pieces of insulation like along the lines, for example. MR. SPARKS: Right. Right. ADM. GEHMAN: Gentlemen, again, thank you very much for being so patient with us as we worked our way through molecular structure and polymer bonds here. This is obviously a very serious issue and an issue that's going to get a lot more attention before we're finished here, and I want to thank you for answering all our questions so completely and helping us do this. I also want to wish you all the very best of luck in the two or three different hats you wear as you both do your day job and also work at finding out how we're going to fix this. So thank you very much. All right. Board, we are finished for today. See you tomorrow morning. (Hearing concluded at 5:01 p.m.) Back to Event