As more and more embedded devices evolve from single-function controllers to complex platforms supporting high-speed graphics, user interfaces, and network communications in addition to the primary application, real-time responsiveness is becoming a critical performance requirement. Although developing in-house software offers some advantages, the benefits of reduced complexity and shorter development schedules often justify the purchase of a commercial Real-Time Operating System.

The average person interacts with hundreds of embedded processors every day in phones, automobiles, home appliances, toys, cash registers, entertainment electronics, security systems, environmental controls, and personal electronics. The common link among all of these products is their ability to react in real time to the user, external events, and the communications channel.

The software for these embedded devices can be divided into application software and Operating System (OS) software. Application software makes the product unique and contains the data collection, signal processing, and hardware control routines required to make the product perform to its specification. The OS allows the programmer to break up large application programs into smaller, individually developed processes or tasks.

At the heart of an OS is the kernel, which schedules programs for execution and manages shared resources. A Real-Time OS (RTOS) processes hardware requests or interrupts from timers or external events within a guaranteed maximum time. Programmers interact with the OS through an API and set up the priorities and data dependencies. During execution, the RTOS manages the application software with a flurry of external real-time activity.

In-house code

Even with the advantages of an RTOS, homegrown OSs still occupy a non-trivial percentage of embedded real-time products. Developers have multiple incentives for bypassing a commercial RTOS entirely and writing their own real-time routines. The biggest reason developers cite for not choosing a commercial OS is lack of need. With only one task running, designers think they can easily keep track of the required hardware interaction.

Special situations sometimes justify in-house software. For example, the design objectives of a portable health care device can include low cost, low power, and a one-year battery standby life without extra memory and processing power to support a commercial RTOS. Furthermore, if a new project is an upgrade of a previous project, developers likely will want to use as much legacy code as possible.

Components that aren’t invented at the same company might also be one reason why many developers write their own OSs. Installing software from a third party into their showpiece product is like admitting they are somehow not up to the task. In addition, developers might think they’ll lose the ability to make software adjustments to compensate for hardware changes or to correct bugs. The designer can easily adjust the order of execution or drop to assembly language to solve critical timing problems with in-house developed software. However, with a commercial RTOS, the scheduler handles many of the timing issues, so developers lose the perception of being in total control. And finally, programmers list sticker shock as another reason to write their own operating software. The initial license for a full commercial RTOS and associated tools can be in the $15,000 to $20,000 range for a single development seat, plus recurring royalties for every unit shipped.

Software shortcuts

As embedded systems grow in complexity and project schedules shrink, software has displaced hardware as the highest-priced item in most embedded development projects. If design teams can buy an RTOS and eliminate the coding, debug, and documentation of the most complicated portion of the software structure, then the purchase decision should receive careful consideration. Although a commercial RTOS can be expensive, a smaller development team and shorter project time frame might create more than enough savings to justify the purchase.

An RTOS allows programmers to write independent, reusable modules to reduce software complexity and shorten the development schedule. Programmers can write each software routine independently without getting bogged down with intertask timing problems. Most RTOS vendors provide a full interactive development environment including a source code editor, code manager, linker, downloader, runtime tools, and one or more debuggers. Software vendors also supply software performance analysis tools to help profile and visualize real-time activity in application routines. Programmers can monitor which tasks are running, observe the stream of data flow, and detect when and how often a task is interrupted by a higher-priority item. RTOS vendors agree that high-quality development tools can dramatically shorten debug time.

Along with the cost savings, RTOS vendors cite multiple technical reasons to justify their products. For example, if an application involves heavy data processing, many RTOSs can be scaled easily to spread tasks across several processors for a significant performance boost. The RTOS provides communication and synchronization services to make multiprocessing transparent. In addition, an off-the-shelf RTOS working alongside multicore[] processors simplifies legacy code integration within new designs or products updates.

A commercial RTOS is modular, so users can select only those portions or features of the OS that they need. Specifying a subset of the full-blown commercial RTOS can reduce acquisition costs and the required memory footprint. With the current connectivity trend, even the simplest embedded products might need to connect to and send data over the Internet. A graphical user interface could also become standard in small embedded systems, even if just for maintenance. These features are included or optionally available in most commercial RTOSs, but can be very expensive or impossible to add to a proprietary OS. Vendors also promote product on-demand technical support as a major benefit of a commercial RTOS.

Off-the-shelf platforms

Commercial RTOSs are constantly upgraded to add new features and keep up with changing technology. For example, the popular VxWorks OS from Wind River was recently revised to deliver 64-bit computing support along with improved multicore features. VxWorks includes a shell, debugging functions, memory management, performance monitoring, and support for multiprocessing. Real-time features include a kernel for preemptive multitasking, interrupt response, interprocess communication, and a file system (see block diagram in Figure 1). Software development is enabled by the Wind River Workbench development tools suite and Intel Integrated Performance Primitives for VxWorks.

The RTOS supports various multicore configurations in Symmetrical Multi-Processing (SMP) and Asymmetrical Multi-Processing (AMP) modes or as a guest OS on top of Wind River Hypervisor. VxWorks also has a configurable and tunable small memory footprint, allowing the user to control how much of the OS to employ for each project.

In addition to offering a multitude of commercial RTOS products, the embedded systems community maintains an open-source OS based on a real-time kernel that is free for use in commercial applications. The FreeRTOS Project is under continuous active development and is distributed under the GNU General Public License with an optional exception that allows users to keep their proprietary software confidential. Free source code and the lack of recurring royalties are popular features for small, low-budget embedded projects. FreeRTOS has been ported to multiple microcontroller platforms and has minimal ROM, RAM, and processing overhead, resulting in a typical kernel binary image in the 4 KB to 9 KB range. Although FreeRTOS source code for the kernel is contained in only three C code files, the zip file download includes numerous demonstration applications to help new users get started.

The biggest complaint among potential open-source software users is the lack of a central resource to provide support similar to that offered by a commercial software vendor; however, the FreeRTOS website has an active free support forum where developers can find answers to their technical questions. In support of the open-source platform, Microchip Technology offers the FreeRTOS Microchip PIC32 Education Kit (see Figure 2). This $95 kit includes a development board that enables users to develop USB embedded host, device, and On-The-Go applications on the PIC32 microcontroller family.

Real-time future

Although programmers might get excited when considering the challenge of developing an in-house OS, the “roll your own” days might be fading away. Designers can look forward to real-time software as the norm in future embedded products.

Customer demand for faster response times, complex functionality, and instant data access continues to increase the challenge of embedded design. Advancing technology also dictates that embedded products be capable of periodic software updates as requirements change, along with the possible transfer to the next-generation hardware platform.

Developers should take the time to analyze their system requirements, development schedule, software support, expandability, communications, scalability, and future growth before embarking on an in-house software development project. An off-the-shelf commercial RTOS or even an open-source operating system could be in your future.

As designers transition to the latest generation multicore processors, software must be divided into separate partitions to gain the performance benefits of parallel execution.  With Intel® Core™ processors, developers have access to multiple techniques to enable this performance gain including symmetric or asymmetric multiprocessing and virtualization. In the symmetric multiprocessing (SMP) configuration, a single operating system allocates threads or tasks across the available cores while managing common memory and hardware resources. Asymmetric multiprocessing (AMP) allows each core to run independent software so that a single system can easily combine real-time, deterministic tasks with a graphical user interface. With virtualization, a hypervisor isolates and allocates system resources between the operating environments so that real-time, general-purpose, and legacy software can be readily integrated in a multicore system.

Each of these performance improvement techniques has some risks especially for safety-critical applications. For example, when parallel, multithreaded applications become part of the software structure they are vulnerable to “race conditions” which occur when concurrent routines have access to, and one of them modifies, a shared memory location. Software failures and bugs caused by these race conditions are not deterministic and may be extremely difficult to locate with normal testing procedures. When developing parallel software, programmers must anticipate these simultaneous activities and once a routine begins to access memory, it must lock out other activities until the transaction is completed.  Although this approach sounds feasible, in practice it may run into problems if two routines are trying to “lock out” other activities simultaneously. To tackle these somewhat subtle coding problems, a number of software vendors have integrated analysis tools into their development packages to optimize the transition from serial applications to parallelism.

For C, C++, Java, and C# software development projects, Klocwork offers the Insight static code analysis tools to automatically locate critical programming bugs and security vulnerabilities in source code (See figure 1).  Klocwork has developed several approaches to locate and eliminate coding problems associated with multicore architectures. In a whitepaper entitled “Developing Software in a Multicore & Multiprocessor World”, Klocwork CTO Gwyn Fisher discusses techniques to identify unique software problems – specifically concurrency errors and endian incompatibilities. The paper also cites a VDC Research report that multicore and multiprocessor software projects are 4.5 times more expensive and have 25% longer schedules than the single core equivalents. In addition, Klocwork offers an on-demand webinar outlining the complexity of porting software to multiprocessor architectures and demonstrating the use of their Truepath automated, whole-program analysis engine.

Unlike other source code analyzers that run as separate tools, DoubleCheck from Green Hills Software is an integrated static analyzer built directly into the MULTI Integrated Development Environment allowing compilation and defect analysis in the same pass. DoubleCheck evaluates potential execution paths through the code to determine how the values of program variables could change across these paths. To simplify the debugging of complex projects with multiple threads of execution, multiple cores, or multiple boards, the Green Hills Integrated Target List displays all system components hierarchically, making it easier to see relationships among applications, address spaces, tasks, and threads (See figure 2). Status information is displayed for all components, so you can quickly check the system state. The target list in the debugger allows you to follow application execution from one context to another with a single click. You can watch as different threads interact and sort out complex interdependencies easily.

After software has been successfully divided into secure code segments using static analysis tools, the 2nd generation Intel® Core™ architecture provides multiple features that augment the performance benefits of parallel execution.  For example, specialized Intel® functions such as Extended Page Tables (EPT) and Page Attribute Table (PAT) provide a hardware assist to the partitioning and allocation of physical memory among multiple cores. The processors also feature Intel® Virtualization Technology for flexible virtualization and Intel® QuickPath Technology to maximize multi-core performance.  If you are starting a new multicore project with and you have questions about static analysis tools, please share your concerns with fellow followers of the Intel® Embedded Community. You can also keep up with the latest technical details and product announcements at the Embedded Computing Design archives on Multicore static code analysis.

To view other community content on interoperability, see “Interoperability – Top Picks”

Warren Webb
OpenSystems Media®, by special arrangement with Intel® Embedded Alliance

Klocwork and Green Hills Software are Affiliate members of the by Intel® Embedded Alliance.

In embedded applications, virtualization software can be used to combine a real-time deterministic operating system with a high-level interactive operating system like Windows or Linux. Using virtualization platforms and tools such as those mentioned in the following discussion simplifies system upgrades and optimizes performance by independently allocating system resources to each operating environment.

As embedded technology and market expectations evolve, design engineers are constantly pressured to pack expanded functionality into smaller, reduced-power devices. In addition to the added complexity of the application software for these new projects, customers demand an interactive interface, ubiquitous connectivity, absolute security, and extreme reliability.

Embedded designers also face the challenge of combining slower legacy interface circuitry with the latest high-speed control devices and multiple displays. The resulting system often includes the original hardware with its Operating System (OS) and application software, plus a completely separate controller with software to handle the newer requirements. This approach increases component count and power requirements and does nothing to increase legacy application performance.

To deal with this increased complexity, designers are utilizing virtual processors hosting multiple OSs to ensure unimpeded, deterministic response to real-time events while simultaneously providing users and operators with a high-level, graphics-based interface. Virtualization[] is achieved by adding a Virtual Machine Monitor (VMM) software layer or hypervisor that isolates individual partitions and executes guest operating software. The hypervisor creates one or more simulated computer environments or virtual machines that can simultaneously host independent OSs and applications on a single processor.

To speed up virtual component interaction, silicon manufacturers are incorporating hardware-assisted virtualization in processor architectures tailored for extended life-cycle embedded applications. For example, the second-generation Intel Core and Intel Atom E6xx processors support Intel Virtualization Technology (Intel VT). This technology improves software-based virtualization performance and security by using hardware assist to trap and execute certain VMM instructions. Intel VT allows the VMM to allocate memory and I/O devices to specific partitions, thus decreasing the processor load and reducing virtual machine switching times.

Virtual isolation

Virtual platforms that combine real-time or safety-critical embedded functions with a large graphics-based OS must contain security provisions that allow unaffected partitions to continue operation in the event of a software failure or cyber attack. For example, LynuxWorks updated the LynxSecure separation kernel and hypervisor for various virtual machine configurations, as shown in Figure 1. This virtualization software is designed to operate in secure defense environments where data and applications with different security levels must co-reside on a single device without corruption. LynxSecure uses a hypervisor to create a virtualization layer that maps physical system resources to each guest OS, which is assigned dedicated resources such as memory, CPU time, and I/O peripherals. Another key feature is the ability to run fully virtualized 64-bit guest OSs such as Windows 7, Linux[], and Solaris across multiple cores.

TenAsys Corporation offers eVM for Windows, another embedded virtualization platform that hosts an embedded OS or Real-Time OS (RTOS) alongside Windows on the same processor platform. To ensure that critical hardware interfaces are not virtualized, eVM partitions the platform, thus guaranteeing maximum performance and deterministic response to real-time events. Installed as a standard Windows application, eVM includes all of the integration tools needed to set up, start, and stop multiple RTOS guest configurations. The Windows-based control panel also allows users to assign interrupts, allocate I/O devices, and set up disk boot images. After the system is set up, eVM provides the guest RTOS with the lowest possible interrupt latency, direct access to I/O, and non-paged RAM.

Multicore[] virtualization

Although virtualization allows designers to combine OSs and applications to reduce system power requirements and form factors, it does little to increase the performance of individual software components. One of the latest trends among designers is to incorporate multicore processors along with virtualization to boost performance through parallel processing.

With virtualization, the hypervisor isolates and allocates system resources between operating environments so that real-time, general-purpose, and legacy software can be readily integrated in a multicore system. In addition to memory and hardware device allocation, virtualization allows developers to assign multiple cores to compute-intensive applications as needed to maximize overall system performance.

Extending virtualization to multicore applications, the Wind River Hypervisor allows designers to configure and partition hardware devices, memory, and cores into virtual boards, each with its own OS, while maintaining the necessary separation (see Figure 2). These virtual boards can be run on a single processor core or distributed across multiple cores based on system needs. The Wind River Hypervisor has been applied in safety-critical applications where the system’s safety-certified and noncertified components traditionally had to be physically separated. However, embedded virtualization allows system designers to isolate the safety-certified components while still operating on a single hardware platform utilizing a certified hypervisor. Virtualization also improves the potential uptime of embedded applications by enabling individual partitions to be rebooted or even reprogrammed while other services on the same device are not affected.

Real-Time Systems also provides virtualization support for multicore processors. Leveraging Intel VT for security, the RTS Real-Time Hypervisor allows completely independent execution of more than one OS on a single multicore platform. Designers can assign individual processor cores, memory, and devices to each OS. Through a configuration file, the boot sequence can be specified, and when desired, an operating system can be rebooted independently of the others. To facilitate communication between OSs, the hypervisor also provides configurable user-shared memory, as well as a TCP/IP-based virtual network driver. The system can run multiple instances of RTOSs mixed with high-level operating software such as Windows XP/CE/7/Embedded, QNX, Linux, On Time RTOS-32, VxWorks, Microware OS-9, and Android[].

Development and debug

No matter if virtual applications run on a single processor or across multiple cores, software development and debug tools must be configured to support more than one OS and memory partition. For example, Green Hills Software updated its INTEGRITY RTOS and MULTI Integrated Development Environment (IDE) to support the latest virtualization microarchitecture. INTEGRITY RTOS is built around a partitioning architecture to provide embedded systems with enhanced reliability, security, and real-time performance. Secure partitions guarantee each task the resources it needs to protect the OS and user tasks from errant and malicious code. INTEGRITY architecture provides Asymmetrical Multi-Processing (AMP) and Symmetrical Multi-Processing (SMP) support optimized for embedded and real-time multicore processors.

MULTI IDE software tools include several C compiler options, a debugger, editor, configuration manager, code browser, and debugger in a single package. MULTI also features DoubleCheck, an integrated static analyzer that isolates bugs caused by complex interactions between code segments that might not be in the same source file. In addition, Green Hills Probe provides a multicore debug control for board bring-up, device driver development, and system-level debugging.

The next step is to incorporate multicore support by updating and streamlining the software development tool set while minimizing modifications to current code creation practices. Various software vendors provide advanced development tools and board support packages for products based on second-generation Intel Core devices. For example, the Prism software analysis tool from CriticalBlue (Figure 3) allows developers to analyze existing software applications, evaluate benefits of the new architecture, and select the appropriate processor.

Prism analyzes the behavior of existing code running on simulators or hardware development boards to assess opportunities that introduce or add further parallel code structures. For example, developers can select the appropriate member of the second-generation Intel Core processor family and analyze the impact of Intel Hyper-Threading Technology, data cache misses, and instruction throughput. Prism gives developers an estimate of the performance gain achievable by partitioning the program into multiple threads.

Design simplified, performance optimized

Virtualization is a proven way to simplify embedded designs with fewer components while integrating the framework needed to easily combine disparate operating software or future updates. Virtualization also simplifies system upgrades by isolating the hardware and software layers so that designers can easily add or modify peripherals, memory, and cores without restructuring the software architecture. A virtual machine hypervisor enables designers to optimize performance by tweaking resource mapping even after deployment.

The IDT Tsi721 is a PCI Express-to-RapidIO bridge that allows existing systems in the wireless, defense, imaging, and industrial markets to maintain their high-performance, low-latency, and highflexibility characteristics with Intel’s latest PCI Express-based architecture. Conversely, the enterprise cloud[] computing and server markets that already use PCI Express-enabled processors can now take advantage of RapidIO as a backplane interconnect. The IDT Tsi721 offers eight direct memory access and four messaging engines, each capable of transferring data and operating at line speeds of 16 Gbps. This simplifies system-level software development by enabling the allocation of multiple engines per core in a multicore[], multithreaded system. The IDT Tsi721 is available in a 13 mm x 13 mm FCBGA package and is priced at $49 each in volume. IDT offers Tsi721 software support for Linux[] and Windows.

Telecom infrastructure networks are experiencing dramatic increases in data traffic, largely driven by multimedia content from wireless and wired devices. This growth is challenging service providers to meet network performance demands while growing their average revenue per user. From the hardware perspective, the latest Intel Xeon® processors provide the efficient performance needed to deal with new demands while allowing network equipment manufacturers to consolidate application, control, and packet processing on the same platform for a more efficient solution.  Although traditional software solutions have not been optimized for network performance, telecom equipment providers must find new techniques to combine operating systems, networking software and multi-core silicon to address business and performance challenges in today’s highly competitive market.

Offering a software solution for network workload consolidation, Wind River recently announced a networking acceleration stack for Linux[] and VxWorks aimed at accelerating IP packet forwarding on carrier-grade telecommunications equipment. The Wind River Network Acceleration Platform manages processing operations over multiple cores to accelerate control and data plane activities and deliver multiple gigabit Ethernet wire-speed performance (See figure 1). By adopting the multi-core asymmetric multiprocessing (AMP) approach the Network Acceleration Platform enhances the standard Linux networking stack to support high-performance network acceleration capabilities exceeding what is possible using symmetric multiprocessing (SMP) mode alone. Offering 10 times the performance of standard Linux configurations, Wind River Network Acceleration Platform helps eliminate bottlenecks not only in moving packets through the silicon itself, but throughout the entire networking platform. The latest release expands hardware support for the Intel Xeon next-generation multicore[] processors and allows network performance to scale efficiently with the number of cores in a processor. In addition, Wind River is providing Network Acceleration developers with multi-core enabled development, testing, debugging, and simulation tools required to simulate and test systems in complex multi-core environments.

Extending packet processing performance, Intel recently announced its 32nm-fabricated Xeon® 5600 processors as the next generation upgrade to the Xeon® 5500 family. In addition to several enterprise-class versions of the Xeon® 5600, Intel introduced four models with seven-year lifecycle support for embedded applications. The 2.4GHz, 80 Watt Xeon® E5645 and the 2.0GHz, 60 Watt Xeon® L5638 each provide six cores and 12 threads. There are two quad-core embedded versions including the 4GHz, 80 Watt Xeon® E5620 and the 1.86GHz, 40 Watt Xeon® L5618. The Intel Xeon® processor 5600 series includes Intel® AES New Instructions (Intel® AES-NI), improving performance for disk and database encryption plus secure Internet transactions. The processors also feature Intel® Virtualization Technology for flexible virtualization, Intel® QuickPath Technology to maximize multi-core performance, plus Intel® Hyper-Threading Technology to deliver top performance for bandwidth-intensive applications.

6WINDGate from 6WIND is another software solution that provides packet processing optimization for networking equipment, wireless infrastructure, security appliances and data centers (See figure 2). The software provides up to ten times the packet processing performance of a standard networking stack and significantly improves the price-performance and power-performance ratios of networking equipment. 6WINDGate is compatible with standard operating system APIs to ensure that clients can migrate either from a single-core to a multi-core platform, or from one multi-core platform to another, without needing to rewrite their existing software.  On a dual-core Intel® Xeon® processor E5645 platform with a clock speed of 3.33GHz, 6WINDGate delivers over 16 million packets per second, per core of IP forwarding performance, thereby forwarding 10Gbps of network traffic in each core. This performance scales linearly with the number of cores configured to run 6WINDGate until the maximum bandwidth of the hardware platform is reached. Processor cores not used to run 6WINDGate are available to run value-added application software or Virtual Machines (VMs), resulting in an efficient and flexible system for advanced networking equipment.

Providing off-the-shelf hardware to support the transition to integrated network workloads, Emerson Network Power recently introduced a new AdvancedTCA server blade featuring a six-core processor from the Intel® Xeon® processor 5600 series to deliver optimized virtualization and power management. The dual-processor ATCA-7365 is the company’s highest-performance 10Gbps ATCA server blade to date and supports up to 96GB DDR3 memory (See figure 3).  Designed to enable network and service providers to lower their capital and operating expenses, the off-the-shelf ATCA-7365 supports a broad array of communications applications that require high network throughput such as telecom. The ATCA-7365 is available with a variety of rear transition module (RTM) variants to support different I/O configurations and can be configured with a variety of software offerings including Red Hat Enterprise Linux 5.4, Wind River Platform for Network Equipment Linux Edition 3.0 and Microsoft Windows Server 2008.

With new software solutions to complement the latest Xeon processors, telecom equipment system providers have fresh options to consolidate processing workloads and improve network infrastructure performance. You can find more information and technical articles on Intel network acceleration architecture at the Intel® Embedded Community page on Xeon processors.  If you are starting a new telecom or packet acceleration project with and you have questions, please share your concerns with fellow followers of the Intel® Embedded Community. You can also keep up with the latest technical details and product announcements at the Embedded Computing Design archives on Multi-core Packet Acceleration.

To view other community content on workload consolidation, see “Workload Consolidation – Top Picks”

Warren Webb
OpenSystems Media®, by special arrangement with Intel® Embedded Alliance

Emerson Network Power is a Premier member of the by Intel® Embedded Alliance. Wind River Systems is an Associate member and 6WIND is an Affiliate member of the Alliance.