This is probably a sort of “me too” story. There are several posts on the Open CASCADE forum explaining how different developers struggled with making it work on Mac. These posts were great sources for me to make my way. As Isaac Newton said “If I have seen further it is only by standing on the shoulders of giants”. So this post is primarily to give credit back to those experimentors and to help those who will follow up.

This entire effort has started to meet a CAD Exchanger SDK prospect who requested its version on MacOS. CAD Exchanger SDK was Windows-only and to fill this gap I first started Linux port (as Linux is officially supported by OCC, and I had Linux experience unlike Mac). “Eat an elephant – one bite at a time”. Linux port helped to port the entire build system, to solve some compilation problems (gcc is often more paranoic than Microsoft compiler), to solve some non-Windows specific problems (like crash of an application with MMGT_REENTRANT set to 1. If you are on Linux or Mac, can you try to see it happens ;-) ?)

My major starting points for Mac were the following posts:
http://www.opencascade.org/org/forum/thread_15606/
http://www.opencascade.org/org/forum/thread_14128/. So many thanks go to Emmanuel, Torsten Sadowski, pythonOCC team (jelle and Thomas Paviot), and others.

I have eventually used MacOS 10.5 (Leopard) with gcc4.0.1. My exact sequence of actions was the following:

Made sure that there is a /usr/X11 directory (I first used a machine with only /Developer/SDKs/MacOSX10.4u.sdk/usr/X11R6 and faced with issue of isysroot and other headache. Though build was almost fine, the executable won’t start and eventually I gave up)
Downloaded and unpacked OpenCASCADE_src.tgz (e.g. into /users/rlygin/dev/Dev/OCC/fix-mac32)
export CASROOT=/users/rlygin/dev/Dev/OCC/fix-mac32
# remove shipped ltmain.sh – see previous post
cp ${CASROOT}/make/ltmain.sh ${CASROOT}/make/ltmain.sh.sav
ln –s /usr/share/libtool/ltmain.sh ${CASROOT}/make/ltmain.sh
cd ${CASROOT}
aclocal
automake
autoconf
export INSTALL_DIR=${CASROOT}/install
export CPPFLAGS="-I/usr/X11/include -arch i386"
export LDFLAGS="-arch i386 -Wl,-dylib_file,/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib:/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib"
The latter option is to work-around 10.5 OpenGL bug described here and here.
For release mode:
./configure --with-xmu-include=/usr/X11R6/include --with-xmu-library=/usr/X11R6/lib --with-gl-include=/usr/X11R6/include --with-gl-library=/usr/X11R6/lib --with-dps-include=/usr/X11R6/include --with-dps-library=/usr/X11R6/lib --with-x-include=/usr/X11R6/include --with-x-libraries=/usr/X11R6/lib --with-x --enable-static=no --enable-shared=yes --enable-production=yes --without-tcl --without-tk --disable-draw --disable-wok --disable-wrappers --prefix=${INSTALL_DIR}
Though I have explicitly specified all those --with-xmu- etc options they seemed to have no effect and what matters is CPPFLAGS and LDFLAGS. Notice also that I disabled Draw, WOK and the Wrappers modules.

Watch for the output and ensure that all your intended modules are correctly configured. Then type:
make
make install

On a properly configured machine the above worked just fine and after a few hours there was a working build. However lauching the unit test framework I quickly faced with an exception on a test case related to use of Standard_Transient in multi-threaded mode. The root-cause came up quickly – race condition in Standard_Transient reference count. See Standard_Atomic.hxx:
//===================================================
// Default stub implementation, not atomic actually
//===================================================
#else

inline void Standard_Atomic_Increment (int volatile* var)
{
++(*var);
}

...

Thus, it was simply not thread-safe at all. Fortunately the fix was easy to make – just use Linux implementation (as gcc is also used on MacOS):
#elif defined(LIN) || defined(__APPLE__)

inline void Standard_Atomic_Increment (int volatile* var)
{
// C equivalent:
// ++(*var);

__asm__ __volatile__
(
"lock incl %0"
: "=m"(*var) // out
: "m" (*var) // in
);
}
...

So, hopefully the Open CASCADE team will integerate this fix as it already kindly did for all other fixes previous pioneers did for MacOS.
After the fix everything worked as expected.

I have currently only ported CAD Exchanger SDK. GUI and Command Line (CLI) remain Windows-only. So no real attempts to make OpenGL-related code work on Mac yet. Perhaps it will uncover some new problems. If anyone knows this one upfront it would be great to learn their experience …