|
Home > Technologies > Software & Hardware
Getting the Most Value from Your Databy Caroline Bright, Data Management Product Manager, National Instruments
Click here for larger version.
Using tools such as DIAdem that include off-the-shelf engineering analysis
and reporting tools can save time
and money.
| Today's complex products and instruments require data acquisition throughout the research, design, development and verification process. In the past, system limitations such as processor speed and memory limited the amount of measurements engineers and scientists collected. However, multicore processors and increasingly less-expensive storage have eliminated these limitations. As a result, engineers and scientists now have the opportunity to use historical data to extract trends. But the amount of information they collect continues to grow unbounded, driving the need for well-defined and efficient data management and post-processing techniques to analyze data and extract meaningful results.
When planning a data management solution, engineers and scientists should consider three major components: the data storage method, the data organization on disk, and the data retrieval and post-processing interface. Proper planning for each of these ensures long-term efficiency and value when analyzing large amounts of data.
The first component, data storage method, refers to how measurements are stored to disk. It is important that users who choose to store data to a file evaluate different file formats to determine which one best meets their project's needs. Users should evaluate each based on the following characteristics:
• Speed of acquisition
• Available disk space
• Who or what program needs to view the data after acquisition
For example, the TDMS file format is the preferred solution for an application that needs high-speed data streaming (such as a system running on a real-time operating system), a small disk footprint, as well as the ability to load the file into a number of different common analysis packages. TDMS is a binary file format designed specifically for high-speed measurement data streaming and is structured to organize and describe information in the file, which can be extremely valuable at the post-processing stage.
The second component of planning a data management solution is determining how to organize data on disk. After storing measurements to files, many users organize them with folder hierarchies and specific file-naming conventions.
Although this solution is simple and direct, users are reducing searchability when storing to a folder rather than to a database. With databases, however, the potential is greater for time and cost overhead during the initial database design, as well as long-term maintenance requirements, due to the need to design and maintain a database schema for test results that may vary over time.
Click here for larger version.
The NI DataFinder index stores the descriptive properties saved with measurement files for quick information searching and trending. | Recently, new technology that combines the benefits of saving data to files and saving information to databases has been introduced. This technology, found in the NI DataFinder software for indexing files on disk, indexes specific information saved in files and then automatically stores it to a behind-the-scenes database. The only data stored in the database is the descriptive information saved in the files. This provides the ability to quickly access information based on these properties using a search interface, which then points back to the original file on disk.
This method gives users all the benefits of organized and searchable data within a database without having to design and maintain the database. Therefore, users can focus on the actual acquisition of the data as opposed to planning how to organize the data on disk or within a database.
After the data is stored and organized on disk, the final component of a well-designed data management system is a data-retrieval and post-processing environment where users can quickly retrieve the most relevant data, find trends, analyze data to find results and create reports. Many analysis tools, such as Microsoft Excel and NI DIAdem, software for analysis and reporting, give users the ability to connect to an external data source—whether a file or a database—and then load the data into the environment to perform analysis on the raw data.
DIAdem also connects to the NI DataFinder index and includes an interface for simple keyword searches as well as advanced parametric searches. With the advanced search, users can locate specific files or channels that meet certain criteria, allowing them to trend channels from multiple files without having to load each individually. This provides the searchability of a database but abstracts the need for any sort of database expertise, thus giving users more insight into the data that really matters within the deluge of raw data collected.
The TDMS file format contains three levels of organizational hierarchy and descriptive properties.
| Once users find the most relevant data and trends, it is important that they use post-processing tools designed specifically for engineers and scientists. Often, users will create custom file viewers and post-processing programs using programming languages such as C, .NET and the NI LabVIEW graphical programming software. With these languages, developers can create a custom analysis application that meets the specific needs of the test. However, there is a large amount of overhead in developing as well as maintaining a custom application.
A more common approach is to use an analysis package such as Microsoft Excel. Although Excel is designed for analysis and reporting, it is not designed specifically for engineers and scientists. It does not include many of the common analysis functions and graphs that engineers and scientists need daily. Therefore, users can spend large amounts of time and money writing macros to perform the needed engineering analysis.
On the other hand, DIAdem includes a wide array of preconfigured engineering analysis and graphing tools. These eliminate the need to develop custom calculations and macros for a project's data analysis. By using report templates as opposed to static reports, DIAdem updates data automatically each time new information is collected so users can focus time and efforts on what really matters—collecting and analyzing the data results. For routine processing and reporting, DIAdem can be fully automated using a scripting interface and custom dialogs. A post-processing tool developed specifically for engineers and scientists enables users to move quickly from raw data to usable results that help them to make decisions faster.
When planning a data management solution to handle the increasing amount of data collected during the design cycle, users need to consider these three major components: data storage, data organization, and the data retrieval and post processing interface. Users can save time and money in the long run by investing time in the front-end process to evaluate different data management approaches and to find the best fit for the application needs.
For more information, contact Caroline Bright, data management product manager with National Instruments, at caroline.bright@ni.com.
OnlineFor additional information on the technologies discussed in this article, visit:
• www.ni.com
Company’s Other Products
National Instruments
11500 Mopac
Austin TX 78759
Phone: 512-683-0100
Fax: 512-683-8411
http://www.ni.com
|
|
|
|
|