Building phoneME Feature

Updates: -- Main.terrencebarr - 02 Jul 2007

-- Main.terrencebarr - 04 Jun 2008

Table of Contents

• Introduction
• Tested versions of phoneME Feature
• Tools Setup on Windows
• Applying fixes to the configuration files
• A working setup
• Build scripts
• Running the resulting phoneME Feature binary on Windows

Introduction

This wiki topic is a place to collect up-to-date information on how to build phoneME Feature. It assumes that you are already familiar with following phoneME documentation and pages:

• Description of phoneME Feature MR2

• phoneME Feature FAQ

In particular, a thorough explanation of the steps required to build phoneME feature are located in the Getting Started Guide:

• Getting Started Guide for phoneME Feature MR2

This wiki topic here is meant as additional information to complement the Getting Started Guide and to add late-breaking information, tips, and workarounds when building phoneME Feature.

Tested phoneME Feature Versions

Currently, these instructions here have been verified using the MR2 b04 build available here:

• Dowbload link for phoneme_feature-mr2-dev-src-b04-21_dec_2006.zip

as well as

• The MR2 b13 as well as b20 build available via the subversion repository

Tools Setup on Windows

In addition to building on Linux phoneME Feature can also be built on Windows. Both Windows 2000 Professional SP4 as well as Windows XP have been successfully used.

An important aspect is the tool chain used. The Windows build currently assumes the use of Microsoft tools. The build system has been used predominantely with Microsoft Visual C/C++ 6.0 and works fine with it. Unfortunately, Visual C/C++ 6.0 is no longer available. It has been replaced by Visual Studio 2005 and Visual Studio 2005 Express, both of which are based on version 8.0 of the C/C++ tool chain.

Unfortunately, Microsoft has made backwards-incompatible changes to the C/C++ compiler and linker so that the attempt to build the phoneME Feature sources "out-of-the-box" will fail. The phoneME sources and build system have not yet been updated fully to support Visual Studio 2005 so you will need to make a few small modifications to some build configuration files to build successfully.

Also, the phoneME build system requires various UNIX utilities such as make and sh. Therefore, a package that provides these UNIX utilities on Windows must be used. cygwin is an open source project that provides such UNIX utilities. However, cygwin has it's share of bugs and is somewhat tricky to deal with so we'll describe possible problems and workarounds here as well.

Microsoft Visual Studio 2005 Issues

You have two choices of Microsoft Visual Studio:

• Visual Studio C++ Express (no cost) and install the missing ml.exe MASM from here MASM tool

• Or use Visual Studio 2005 Professional or above

NOTE: Visual Studio Standard does not include the ml.exe MASM program and I have not found any way to add it (the MASM for Visual C++ Express refuses installation onto Visual Studio Standard and manually copying ml.exe doesn't work)

The compiler in Visual Studio 2005 has backward-incompatible changes from the one the phoneME Feature engineering team has been using. As explained above these difference require several small modifications to the build configuration files:

• Standard library functions like strcpy() are now deprecated in favor of versions that accept array lengths and check for array overruns.
• Executables are now generated in a new way so that an executable has an associated manifest that describes its DLL dependencies. This may cause trouble with romgen.exe
• A number of compiler options are now deprecated or have changed, such as: /GB and /DEBUGTYPE
• Also, it seems the linker now includes standard libraries so when CLDC or MIDP provide alternate definitions of symbols the linker fails with conflicting symbols. Standard libraries now must be explicitly excluded with the /nodefaultlib linker option.

UNIX tools (cygwin and cyg4me):

cygwin is a Linux-like environment for Windows. It consists of two parts: A DLL (cygwin1.dll) which acts as a Linux API emulation layer providing substantial Linux API functionality as well as a collection of tools which provide Linux look and feel.

For use with phoneME Feature the Sun engineering team has created a purpose-built version called cyg4me. It is highly recommended to use cyg4me instead of cygwin (cygwin will seems to work initially but cause you trouble later). Download cyg4me here.

However, even cyg4me has its share of problems which it inherits from cygwin. In particular, it is unreliable and inconsistent in it's handling of paths:

• Be very careful with paths within cyg4me and shells you run within cyg4me. There seems to be various flaky behavior around paths, e.g. _path_ vs. c:/cygwin/_path_ vs. \cygwin\_path_ vs. /cygdrive/c/_path_ should all be the same but depending on the stage of the build and the tools used they seem to be evaluated differently by different tools resulting in puzzling build failures. A source of much fiddling and irritation. Make sure you enter all paths as described below.
• Also, don't use shell environment variables except for a variable for the top of your source tree. Pass all variables to make on the command line directly.
• Your executable path in cyg4me bash should look like this: j2sdk1.4.2_13/bin, followed by cyg4me/bin, followed by the stuff defined by Visual C++ vsvargs32.bat
• Make sure j2sdk1.4.2_13/bin is in your excutable path and check the correct version by running java -version from the comand line. Also, pass in JDK_DIR seperately on the make command line. Dropping either will result in failures.
• Finally, sometimes the phoneME Windows build will fail with a STATUS_ACCESS_VIOLATION something like this:

         4 [main] sh 1560 handle_exceptions: Exception: 
STATUS_ACCESS_VIOLATION
       653 [main] sh 1560 open_stackdumpfile: Dumping stack trace to 
sh.exe.stackdump
   make: *** 

This seems to be cause by some bug in cyg4me with handling paths. In particular, the sh.exe program crashes and the build fails. The workaround is to shorten or lengthen the path of your phoneME Feature source tree root. Try the following: delete the build output directory, move the source tree to some other directory, and re-run the build. This bug can be very frustrating and it might take several tries to avoid it.

Update on STATUS_ACESS_VIOLATION:

I have been able to mostly avoid this problem now by doing the following. Instead of:

   $TOP=<_path_>
   $OUTPUT=$TOP/output

   $TOP=<_path_>
   $OUTPUT=<_short_path_>

Applying Fixes to the Configuration Files

Because of the bckwards-incompatible changes in the Microsoft tools described above a couple of fixes to the build configuration files need to be made in order for the build to execute successfully.

Fixing javacall

Edit javacall/build/vc/rules.gmk

In lines 53 and 54, change the compiler flags definition. Add the flags _CRT_SECURE_NO_DEPRECATE and _CRT_NONSTDC_NO_DEPRECATE and change the treat-warnings-as-errors level from /W3 to /W2. It should now look something like this:

LOCAL_CFLAGS_debug = -MDd -Gm -GX -Zi -Od -D_DEBUG -D_MBCS -D_LIB -YX -FD -GZ -W2 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_NONSTDC_NO_DEPRECATE
LOCAL_CFLAGS_nodebug = -MD -GX -O2 -D_MBCS -D_LIB /YX -FD -W2 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_NONSTDC_NO_DEPRECATE

Fixing PCSL

Edit pcsl/makefiles/share/vc.gmk

In lines 46-50, change the compiler flags definition. Add the flags _CRT_SECURE_NO_DEPRECATE and _CRT_NONSTDC_NO_DEPRECATE and change the treat-warnings-as-errors level from /W3 to /W2. It should now look something like this:

ifeq ($(USE_DEBUG), true)
CFLAGS= /WX /c /W2 -Zi -Od -MDd /D_CRT_SECURE_NO_DEPRECATE /D_CRT_NONSTDC_NO_DEPRECATE
else
CFLAGS= /WX /c /W2 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_NONSTDC_NO_DEPRECATE
endif

The PCSL build should run fine now.

Fixing CLDC

For CLDC, be sure to set the environment variable USE_VS2005 to true (as done in the build-phoneME.sh above) before starting the build. It takes care of a lot of compiler warnings.

After the CLDC build successfully completes, you may need to copy romgen.exe.manifest from $JVMBuildSpace/cldc/win32_i386/romgen/app to $JVMBuildSpace/win32_i386/cldc/dist/bin. Otherwise romgen.exe may fail during the MIDP build with a message about a missing DLL.

Also possible:

cp $OUTPUT/cldc/win32_i386/romgen/app/romgen.exe.manifest $OUTPUT/cldc/win32_i386/dist/bin

Update: Copying romgen.exe.manifest appears necessary on Windows XP systems but not necessary on Windows 2000.

Update: The CLDC build failed when I attempted the SJWC JavaCall build. I get the same link error as for MIDP below. I fixed it by adding a line to cldc/build/javacall_i386_vc/javacall_i386_vc.cfg:

LINK_FLAGS+=-nodefaultlib:libcmt.lib

Note: Towards the end of the CLDC build there are still a lot of warning messages around compiler and linker options, such as /GB, /DEBUGTYPE, and multiple defined symbols regarding LIBCMT. The build should complete successfully despite these warnings.

Fixing MIDP

The MIDP link will not work unless you add a flag. Do this by adding the following line to midp/build/win32/Platform.gmk:

LD_FLAGS+=-nodefaultlib:libcmt.lib

After this, the MIDP build should proceed successfully.

You will get lots of warnings about method deprecation and an unsupported compiler option /GB, but it should stagger through to the end without errors.

A Working Setup

• cygwin.bat

@echo off
C:
call "/Program Files/Microsoft Visual Studio 8/Common7/Tools/vsvars32.bat"
set PATH=c:\cyg4me\bin;%PATH%
bash --login -i

• .bash_profile

export JDK_DIR=/cygdrive/c/j2sdk1.4.2_13
PATH=$JDK_DIR/bin:$PATH:.

• Resulting bash shell $PATH

/cygdrive/c/j2sdk1.4.2_13/bin:
/cygdrive/c/cyg4me/bin:
/cygdrive/c/Program Files/Microsoft Visual Studio 8/Common7/IDE:
/cygdrive/c/Program Files/Microsoft Visual Studio 8/VC/BIN:
/cygdrive/c/Program Files/Microsoft Visual Studio 8/Common7/Tools:
/cygdrive/c/Program Files/Microsoft Visual Studio 8/Common7/Tools/bin:
/cygdrive/c/Program Files/Microsoft Visual Studio 8/VC/PlatformSDK/bin:
/cygdrive/c/Program Files/Microsoft Visual Studio 8/SDK/v2.0/bin:
/cygdrive/c/WINNT/Microsoft.NET/Framework/v2.0.50727:
/cygdrive/c/Program Files/Microsoft Visual Studio 8/VC/VCPackages:
/cygdrive/c/cyg4me/bin:/cygdrive/c/WINNT/system32:/cygdrive/c/WINNT:
/cygdrive/c/WINNT/System32/Wbem:
.

Build Scripts

phoneME Feature offers a multitude of configuration, build options, and flags to allow you to create binaries to your exact requirements. For detailed information please see the Getting Started Guide for phoneME Feature MR2, and the phoneME Feature MR2 build guide.

Here are example build-phoneME.sh scripts for selected configurations:

Example 1

phoneME Feature MR2 with CLDC + MIDP, not using javacall, and without debug support. This is the most basic build configuration.

#build-phoneME.sh
export TOP=c:/cygwin/home/work/phoneME_feature_MR2
export OUTPUT=c:/cygwin/out
export LOG=$TOP/log.txt
# make PCSL
cd $TOP/pcsl
make ENABLE_PCSL=true PCSL_PLATFORM=win32_i386_vc PCSL_OUTPUT_DIR=$OUTPUT/pcsl 
# make CLDC
cd $TOP/cldc/build/win32_i386
make JDK_DIR=c:/j2sdk1.4.2_13 ENABLE_PCSL=true \
JVMWorkSpace=$TOP/cldc JVMBuildSpace=$OUTPUT/cldc \
USE_VS2005=true PCSL_OUTPUT_DIR=$OUTPUT/pcsl 
# make MIDP
cd $TOP/midp/build/win32
make JDK_DIR=c:/j2sdk1.4.2_13 TOOLS_DIR=$TOP/tools \
CLDC_DIST_DIR=$OUTPUT/cldc/win32_i386/dist \
USE_VS2005=true PCSL_OUTPUT_DIR=$OUTPUT/pcsl \
MIDP_OUTPUT_DIR=$OUTPUT/midp

Example 2

phoneME Feature MR2 with CLDC + MIDP, using javacall, and with debug support enabled. This is a configuration that can be integrated with NetBeans so that you can run and debug MIDlets on phoneME Feature as an additional emulator platform from within NetNeans:

#build-phoneME.sh
export TOP=c:/cygwin/home/work/phoneME_feature_MR2
export OUTPUT=c:/cygwin/out
export LOG=$TOP/log.txt
# javacall
cd $TOP/javacall/configuration/phoneMEFeature/win32_x86; make USE_DEBUG=true \
JAVACALL_DIR=$TOP/javacall JAVACALL_OUTPUT_DIR=$OUTPUT/javacall
# PCSL
cd $TOP/pcsl; make USE_DEBUG=false ENABLE_PCSL=true \
PCSL_PLATFORM=javacall_i386_vc JAVACALL_OUTPUT_DIR=$OUTPUT/javacall \
PCSL_OUTPUT_DIR=$OUTPUT/pcsl 
# CLDC
cd $TOP/cldc/build/javacall_i386_vc; make JDK_DIR=c:/j2sdk1.4.2_13 \
ENABLE_PCSL=true PCSL_PLATFORM=javacall_i386_vc PCSL_OUTPUT_DIR=$OUTPUT/pcsl \
JAVACALL_OUTPUT_DIR=$OUTPUT/javacall JVMWorkSpace=$TOP/cldc \
JVMBuildSpace=$OUTPUT/cldc USE_VS2005=true ENABLE_ISOLATES=true \
ENABLE_JAVA_DEBUGGER=true ENABLE_SYSTEM_CLASSES_DEBUG=true \
ENABLE_ROM_JAVA_DEBUGGER=true USE_DEBUG=true 
# MIDP
cd $TOP/midp/build/javacall; make JDK_DIR=c:/j2sdk1.4.2_13 TOOLS_DIR=$TOP/tools \
CLDC_DIST_DIR=$OUTPUT/cldc/javacall_i386_vc/dist JAVACALL_PLATFORM=win32_i386_vc \
JAVACALL_OUTPUT_DIR=$OUTPUT/javacall PCSL_OUTPUT_DIR=$OUTPUT/pcsl \
MIDP_OUTPUT_DIR=$OUTPUT/midp USE_VS2005=true USE_MULTIPLE_ISOLATES=true \
USE_DEBUG=true USE_JAVA_DEBUGGER=true

You can start a complete MIDP build with:

   sh ./build-phoneME.sh

If you have fixed up the configuration files according to the previous steps the build should run all the way through and result in a completely built phoneME stack located in the $OUTPUT/midp directory.

Running the resulting phoneME Feature binary on Windows

Running the phoneME executable on a system that has Microsoft Visual Studio installed

Just go to the $OUTPUT/midp/bin/i386 directory and run runMidlet.exe as described in the Getting Started Guide. It should work fine.

Running the phoneME executable on a system that does not have Microsoft Visual Studio installed

Binaries built with Microsoft Visual Studio require dynamic linking of additional libraries which come with Microsoft Visual Studio. On standard Windows installations these libraries may not be present. Here is a blog on the issue:

• Running C++ application built by VC++ Express on another computer

One solution is to download and install on the target Windows machine the Microsoft Visual Studio redestributable library package:

• Visual C++ 2005 Redistributable Package

Update:

There appears to be an incompatibility in the Windows DLL mechanism for certain system DLLs between Windows 2000 and Windows XP.

• If you build phoneME Feature as above on a Windows 2000 system the resulting phoneME binary will work fine on another WIndows 2000 system. However, if will likely not work on a Windows XP system because of the way the Windows XP DLL loading mechanism works (at least, that is my current understanding).
• If you build phoneME Feature on a Windows XP machine the resulting phoneME binary appears to run fine on other WIndows XP systems as well as Windows 2000 systems.

-- Main.terrencebarr - 26 Apr 2007

Windows Platform SDK is required when building the source code. It can be downloaded and installed from Microsoft's website: Windows Platform SDK

-- Main.emil_kvarnhammar - 26 May 2007

You can merge the manifest-file and the exe-file with mt.exe. After that, the manifest-file is no longer needed. Works great on Windows Vista, but I have not tried it on Windows XP. This can by applied to any exe-file created for phoneME. The syntax is:

mt.exe -manifest romgen.exe.manifest -outputresource:romgen.exe

-- Main.michael_heinrichs - 26 Nov 2007

If using MR3 then set the environment variable USE_VS2005 to true when building MIDP and the manifest-file will automatically incorporated into the exe-file during build time.

-- Main.randomaniac777 - 14 Nov 2008