FDS and the Challenge of "Big Data"

Just the other day Kris Overholt stumbled across a blog post written by a post-doctoral fellow at the University of Washington, Jake VanderPlas:

http://jakevdp.github.io/blog/2013/10/26/big-data-brain-drain/

The article speaks to the difficulty of maintaining sophisticated computational tools in an academic setting, and helps explain some of the challenges we face in developing and maintaining FDS and Smokeview. In particular, the article points out how the traditional form of academic scholarship has become counter-productive:

This brings us to Academia's core problem: despite the centrality of well-documented, well-written software to the current paradigm of scientific research, academia has been singularly successful at discouraging these very practices that would contribute to its success. In the "publish-or-perish" model which dominates most research universities, any time spent building and documenting software tools is time spent not writing research papers, which are the primary currency of the academic reward structure. As a result, except in certain exceptional circumstances, those who focus on reproducible and open software are less likely to build the resume required for promotion within the academic system.

While NIST and VTT are not universities, they have much in common and share the same reward structure. Our management has, to some extent, adopted a broader set of criteria for evaluating our impact, but the fact remains that we and our collaborators are under pressure to publish papers rather than do the necessary things to maintain FDS and Smokeview. You might say that the two need not be mutually exclusive, but they often are. Take, for example, the recent request for an OpenMP (i.e., shared-memory parallel) version of FDS 6. OpenMP is a set of statements within the FDS Fortran source code that enables FDS to run on a single computer that has multiple processors or cores. This is different than the MPI version of FDS which enables multiple meshes to be processed by multiple computers. There was an OpenMP version of FDS 5 that was contributed by a volunteer, but he can no longer work on this because he has to earn a living (one does wonder where all this wonderful free stuff on the Internet comes from). OpenMP is not new, and implementing it in FDS is not going to win anyone fame, fortune, or even a published journal article. It is a thankless task (academically). OpenMP is just one example. Consider the time and effort to assemble all those verification and validation case studies, and automate the procedure for running and processing them, and prepare the manuals, and distribute the software, and so on. Much of this work is not the stuff of academic journals, but it is absolutely necessary.

So what is the solution to this problem? The first step is to develop a new definition of the word "collaboration." To most, collaboration means that we publish our work in journals, or meet at conferences and exchange ideas. In many areas of science and engineering, this has worked reasonably well for centuries. So what's wrong now? What has changed? As Jake VanderPlas points out, what has changed is the advent of complex numerical codes that cannot be maintained by a single professor and a few students; nor can they be successfully commercialized. There has been some limited success in developing useful add-ons to FDS, like the graphical user interface PyroSim by Thunderhead Engineering, but the core development work is underwritten by NIST and VTT. In our opinion, this is a good model because it allows for continued research in algorithms that probably could not be supported by a completely commercialized program. On the other hand, it means that these various "maintenance" activities fall through the cracks. They appeal to neither the core developers nor the commercial partners.

So who are to do these thankless tasks? At the moment, we the developers. This partly explains why it took us three years to release FDS 6. As you've read in the blog posts over the past few years, we have redoubled our efforts to better document our work, to develop more accurate and robust routines, and to do the necessary verification and validation work that is absolutely necessary if FDS-SMV is to be used in a regulatory setting. But there is so much more that could be done to improve this software, and what might surprise most of you is that this work IS being done, sort of, by all those students out there who are working with FDS. But the problem is, and this takes us back to the definition of the word "collaboration", that all this work is not finding its way into the FDS code, manuals, or repository. Rather, it is making its way, sort of, into the various fire journals and conference proceedings. I say "sort of" because much of it just ends with the masters or PhD thesis. Even the stuff that gets into the literature is not easily converted into genuine improvements to the code or documentation. As much as we try to open up lines of communication early in the process, we cannot break the traditional publish or perish model.

Usually, we are sent a paper describing a student's masters or PhD work with FDS only at the end of the student's tenure and asked if the work can be incorporated into the code or added to the validation suite.  This is frustrating on several levels.  First, often we have already solved the problem being addressed, but as discussed above we do not have time to write a paper every time we change a few lines of code.  Second, often the work is very good and we would like to incorporate it, but this amounts to us redoing the student's two years (plus) of work within a few days---that's about how much time we can usually spare.  Why could this work not have been done within the FDS-SMV framework to begin with?  Why won't the professors and students contact us on the front end?  The answer we usually get for this question is that they are worried about someone stealing the idea and beating them to publication.  Lest you feel we are picking on academia, in fact we find the same response from industry with their concerns about intellectual property (IP).  Here is a list of reasons why this concern is invalid:

1. Students have almost nothing to lose and a tremendous amount to gain by collaborating with us early in the process. FDS was designed as a platform for the entire research community to make small incremental improvements in fire modeling without having each and every new student re-invent the wheel.
2. The developers can usually code something up much faster than the students, but we have little time to thoroughly shake it down. Programming is not the problem – it is the work done over months, even years, to do a thorough job of verifying and validating the algorithm. This is perfect for students – they learn from watching experts, and then they develop the necessary skills to do it themselves, all while making a tangible contribution to FDS. For this to work, the algorithm must be coded into FDS. If it is not ready for prime time, that is OK, as we just hide the new algorithm with simple IF-THEN statements.
3. We have never had a case where a student’s work was hijacked by another. In fact, what students ought to worry about is being ignored, not copied. Publishing FDS-related papers in journals is NOT the best way to make a contribution in fire research. An improved algorithm in FDS is, and you cannot do that by hiding your work for 3-4 years.
4. If we adopted the same attitude toward IP, there would be no FDS. As developers we have little time to publish. And we are punished for this in various ways. Is it fair that students hide the work they are doing and publish results atop the pains-taking work of the developers to improve the core functionality of the code?  We are not asking for co-authorship, just a little give and take.  The "payment" for using our software should be to help improve it.
5. Finally, if you really think that your ideas are going to be “scooped”, understand this: all commits to the FDS repository are recorded and maintained forever. We know exactly when anyone has “touched” any routine. No one who has committed source code or experimental data to the FDS repository has ever been scooped. If it were to happen, we would be the first to write to a journal editor with undeniable proof that a researcher has violated the spirit of our open source development process.

The irony of the situation is that the majority of the students working with FDS leave with a masters degree and do not pursue academic jobs. What we have to offer these students in return for helping us develop and maintain FDS-SMV is an education in current software development practices, in addition to a far better understanding of how these models really work. Simply running a piece of software is hardly an enviable skill; but being able to get under the hood and work with its various components is. Furthermore, the skills you can acquire go way beyond fire. Many of you will move beyond fire into other areas of engineering or beyond in the next few decades. Our blogger Jake is a perfect example of a student who has acquired a tremendous amount of IT skills that go far beyond his chosen course of study. And that's a good thing because there are only so many jobs available in cosmology.

FDS 6 Official Release

Dear Users:

We are happy to announce the official release of FDS 6.

Downloads and documentation are available from the primary FDS-SMV web page:

https://code.google.com/p/fds-smv/

For installation instructions for Windows, Mac, and Linux, please visit our YouTube channel or wiki pages:

http://www.youtube.com/fdssmokeview

https://code.google.com/p/fds-smv/wiki/Installation_Notes

The release notes for FDS 6.0 are posted here:

https://code.google.com/p/fds-smv/wiki/FDS_Release_Notes

More information about the most significant changes is available in a series of blog posts on our Developer Blog:

http://blog.fds-smv.net/

Within the FDS User's Guide, you can find a table for translating FDS 5 input files to FDS 6. These changes are also mentioned in the release notes.

Please report any issues to the Issue Tracker along with a VERY simple test case, and post any general questions or comments to the Discussion Group:

https://code.google.com/p/fds-smv/issues/list

https://groups.google.com/forum/?fromgroups#!forum/fds-smv

FDS 6 has been built around a philosophy of continuous integration testing, which involves nightly automated builds of the verification and validation guides and quantitative error checking. This process results in higher quality, more reliable software for end-users, enables better collaboration with researchers, and allows our development team to focus on core model development. The nightly build status of the FDS-SMV project can be found here:

https://code.google.com/p/fds-smv/wiki/Firebot_Build_Status

Finally, thanks to all of you for your patience and support. And thanks especially to all of our beta testers. If any names are missing, please let us know and we will be sure to include you in the guide.

Best regards,

The FDS-SMV Development Team

Progress on Release of FDS 6

We are going through the final phases of our release of FDS 6. Since our first release in 2000, it has become much more difficult to issue a major new release, simply because we have put in a place a more rigorous process of testing. We now do nightly builds of both the FDS and Smokeview source codes, run all the verification cases, and compile all the manuals. This only takes a few hours to do while we sleep. Our validation cases, on the other hand, take a few weeks to run on our 256 core linux cluster, as there are approximately 900 individual cases that comprise the test suite. Some cases take only a few minutes on a single processor; some take as much as a week on multiple processors. Inevitably, Murphy's Law holds true, and there always seems to be something that hiccups. But we're nearing the end of our testing and we're almost through with the internal and external review of our manuals. Thanks to those of you who have made comments on the manuals. It's quite a task to wade through all those documents.

Those of you who have helped with the beta testing or proof reading, check to see that your name and affiliation are listed near the beginning of the new FDS User's Guide. You can find a copy that is posted nightly by following a link at the bottom of the home page, fire.nist.gov/fds. If we've forgotten to add you to the list, just post a note to the Issue Tracker thread that you used to post your test results. That's the easiest way to keep track of who did what.

If all goes well, we hope to have an official release of FDS by the end of the month.

Near-Wall Boundary Conditions in FDS 6

Happy New Year!

This blog is a continuation in our series of articles on FDS 6 Release Notes.  Here we will focus on new wall functions for turbulent momentum and heat transfer.  In early versions of FDS, the near-wall boundary conditions were treated using a "slip condition" for the tangential component of velocity and, as is still the case, empirical correlations for heat and mass transfer.  In FDS 6, the default boundary condition for the tangential velocity is based on a log law wall function for both smooth and rough walls.  In addition, a new log law wall function for heat transfer, which accounts for variation in fluid Prandtl number, is in the testing phase and is available as an option by setting HEAT_TRANSFER_MODEL='LOGLAW' on the SURF line.

With FDS 5.4, the Werner-Wengle (1991) wall model became the default model for the tangential velocity component near smooth walls.  This model assumes a power law profile for the streamwise velocity component in the wall-normal direction and then further assumes that the tangential component of velocity near the wall is equivalent to the profile filtered over the height of the first grid cell.  Because of this improvement, FDS was able reproduce the Moody Chart (friction factor versus Reynolds number) for smooth walls over a very broad range of Reynolds numbers.  For rough walls, we implemented a wall function which took the maximum stress between the fully-rough log law given in Pope (Turbulent Flows, 2000) and the smooth wall stress from WW.  This formulation was practical, but the underlying mathematical inconsistency (filtered power law versus directly sampled log law) was not ideal. For compatibility with the wall functions used by the atmospheric community (outdoor flows for wildfires being of increasing interest), in v6 we decided to abandon the Werner-Wengle model and focus on the directly sampled log law wall functions.  Further, we no longer assume a fully rough wall---the transition from smooth to rough is accounted for by the new velocity wall function.

There are two other aspects of the near-wall treatment of velocity that deserve attention: one is how we handle the near-wall eddy viscosity and the other is how we handle the vorticity at the wall (in the FDS formulation, the vorticity resides in the advective term of the momentum equation).  For dynamic models of the eddy viscosity which require "test filtering," as we require for our Deardorff model, the near-wall treatment can be tricky.  When the dynamic Smagorinsky model was developed for channel flow, the standard practice was to average the model coefficient in the two homogeneous directions---the streamwise and spanwise directions---leaving the coefficient to vary only in the wall-normal direction.  But in a typical fire scenario we do not have the luxury of homogeneous directions.  Therefore, in the bulk flow, we use a test filter of size 2*dx in all three directions to smooth the velocity field.  This is problematic near the wall.  All attempts to retain the dynamic viscosity near the wall (special test filtering and so on) led to wildly erratic pyrolysis-based fire behavior unless very fine grid resolution was used.  This could not be tolerated, as it basically canceled out all the benefits of using the dynamic model to begin with.  To address this problem, we decided to simply use the constant coefficient Smagorinsky model in the first off-wall grid cell since this model does not require any test filtering.  To overcome the issue of convergence with the constant coefficient Smagorinsky model, we employ Van Driest damping of the mixing length near the wall.

As mentioned, a second important issue related to near-wall flow behavior is the treatment of the vorticity at the wall and at sharp edges.  Given that ventilation is a zeroth-order model parameter (translation: extremely important) and because common practice in fire protection engineering is to use relatively coarse grids, the near-wall value of vorticity has a surprisingly large impact on the overall model performance.  Think of doors and windows (which, let me add, should never be placed on the exterior of the computational domain) in the fire scenario, and imagine that the value of vorticity used at the edge may implicitly change the size of the opening by a cell width (this is basically the difference between a free slip and a no slip boundary condition).  The numerical approximations used to compute the vorticity on the wall or corner edge may effectively change the flow area of the opening.  To make matters worse, this effect is---by its very nature---grid dependent.  Our best attempt to deal with this dilemma has been to explore options for the vorticity "slip condition" while comparing FDS results for a simple test case which has many problematic features.  The case is a 3D flow over a square rib in a periodic channel (ribbed_channel test series in the Verification suite).  In this test we look at the location of the reattachment zone behind the rib, as well as the mean and RMS profiles for streamwise velocity in the wall-normal and streamwise directions.  With h being the height of the rib, we examine grid resolutions from h/dx=2 to h/dx=16.  The best result is found by applying a linear average between no slip and the slip value returned by the wall function.

Last but not least, the convective heat transfer model currently employed by FDS is based on taking the max of natural and forced convection Nusselt number correlations.  An often cited criticism of this approach is that the temperature and velocities used in these correlations are, of course, the "free stream" values in the correlation, but in FDS the values in the first off-wall cell are used.  Despite the obvious shortcomings, no one has yet systematically demonstrated a better alternative, and so these correlations remain the v6 default.  It is fair to say, in fact, that the subject of heat transfer wall functions is very much a research topic in the LES community.  Over the last year, some progress has been made in this area.  For practical fire simulations, the principal developer has been Ezgi Oztekin of the Fire Research Program at William J. Hughes Technical Center.  She implemented into FDS a basic log law heat transfer model and demonstrated the viability of the approach with compartment fire calculations.  On the verification front, Jung-il Choi's group from Yonsei University visited NIST last summer and modified Oztekin's model within FDS to include Prandtl number dependence and compared FDS results with the heated channel DNS results of Kim and Moin (1987).  This work represents the first serious verification study in development of the heat transfer model in FDS.  The results are in the FDS Verification Guide, and for this reason we are reasonably optimistic that the model will survive and become the default in the near future.

FDS 6 Verification and Validation

In recent years, we have formalized the process of developing and releasing new versions of FDS and Smokeview, taking our cue from the way it is done by commercial software developers. The key to this process is a set of calculations that we divide into two categories -- relatively short, simple test cases that we use to verify that the programs are properly solving the equations we've written down, and relatively long, sometimes complex cases that we use to validate the accuracy of the model by comparing with actual experiments.

The short verification cases are the sample input files that we distribute with each new bundle. They are run each night automatically by a script we call 'firebot' to catch simple coding mistakes that we inevitably make as we develop new routines. We not only run these cases, but we also automatically regenerate all the plots that you find in the FDS and Smokeview User's Guide, plus the FDS Verification Guide. Any result that falls outside of a certain tolerance is reported to all of us via email, and we then fix the problem while it is still fresh on our minds.

The longer validation cases are run with each minor release of FDS and Smokeview because it takes between one and two weeks to complete all the cases (about 800 calculations) on our 256 core linux cluster. Because these cases are run less frequently, there almost always is something amiss about one or two cases that is more difficult to diagnose. In fact, a good part of the two years we spent preparing FDS 6 was devoted to addressing discrepancies in some of our most reliable experimental data sets. As frustrating as it was, the fact that we knew about it all was due to the new process of V&V that we initiated. In the past, it was far more difficult to detect a fundamental problem with the algorithm because we did not systematically run all the cases at once and analyze the results in a consistent way. Basically, we just eye-balled things, but that does not always reveal subtle problems.

When we released FDS version 1 in 2000, we did not have a formal process of V&V. We did develop test cases, and we did compare calculations with experiment, but we did not do it in a systematic way. We felt that papers published by ourselves and others would suffice, but we soon learned that this was not the case. This lesson was reinforced when we began working with the US Nuclear Regulatory Commission on a V&V of five different fire models that are commonly used by the nuclear industry. The most important lesson we learned is that published results using older versions of the software cannot be used to justify the use of a model to the AHJ (Authority Having Jurisdiction). We cannot republish our validation papers with the release of each new version, so we decided to develop and maintain our own versions of the V&V guides that the NRC published (NUREG-1824).

Now the FDS Verification and Validation Guides are the key volumes that quantify the robustness and accuracy of the model. When we say that FDS can or cannot do something, what we really mean is that we have documented calculations that show the range and accuracy of the model for a particular application. For those of you using FDS for design or forensic applications, the FDS V&V Guides should be the first place to look to determine if FDS is appropriate for your use. Every few days someone writes to the Discussion Group asking something like, "Does anybody know if FDS can do ...?" What that person really ought to do is check the V&V Guides. The question should not only be can it be done, but also how well can it be done. FDS has alot of potential applications, but you need to check the Guides before deciding if it is applicable.

Finally, we spend a considerable amount of work putting the V&V Guides together, running the cases, working on the statistics, the scripts, and so on. We have appealed to the user community, especially the students and profs working with FDS, to help us with V&V cases. So far, the results have been disappointing. For a variety of reasons, we are not able to capture in our Guides the work with FDS that we see submitted to journals and conferences. We suspect that the main reason for this is that a student's first objective is to graduate, then maybe publish the thesis in some way. Working with us to get the results into our Guides is either a low priority or something that no one has even considered. I sometimes meet students at meetings who do not think that their thesis work is worthy of our Guides. Think again -- not only would the work be worthy in most cases, but it is a necessity. In order for us all to continue to develop and use tools like FDS and Smokeview, we all have to contribute to their upkeep. In this case, that means extending the range of application represented by the V&V cases. If you did your thesis on smoke detector algorithms, it is in your best interest to get this work into the Guides so that we can maintain the capability in future versions. If you just hack some routine into FDS 5.3.whatever, don't expect it to be accepted down the road by the AHJ. Further, owing to the relatively small size of fire protection engineering, there are probably few other engineering disciplines where a masters degree student can have his or her work put directly into practice so readily. Ask engineers in other disciplines what kinds of software packages they use and whether or not they have any means to be involved in their development. One of the appealing things about FPE is that there are so many opportunities to have an impact. To see how you can have an impact, read through the following wiki:

http://code.google.com/p/fds-smv/wiki/Contribution_Guide