Amir Masoud Abdol’s Personal Website

Apple Notes vs. Hyperlinks

2022-07-11T00:00:00+00:00

Over the past years, Apple Notes has transformed from a simple notepad to a full featured note-taking app. It provides a long list of useful features like Formatted Text, Support for Attachments/Photos/Videos, Folders, Smart Folders, Handwritten Notes, Quick Notes, Scribble, Secure Notes, Note Sharing, and Live Collaboration. In iOS 16, Apple Notes will have an improved Secure Notes integration, and the Quick Notes is making its way into the iPhone. Overall, Apple Notes is really starting to be a great app, and after using so many note-taking apps (Drafts, Agenda, GoodNotes, Notability, DEVONthink, Workflowy), I just prefer something simple, and well-integrated just to make simple notes. Apple Notes’ cross-platform support is sublime, the Apple Pencil support and integration is unmatched.

Unfortunately though, these days, like most Apple software you start to notice inconsistencies and annoying behaviors here and there, e.g., there is no easy way to export your notes from Apple Notes. Unlike Apple Photos where you can simply export or drag/drop a photo out of the app, Apple Notes does not let you do that. Your only option is to export your entire library as a PDF, or use a third-party app which you’ll never know when it stops working.

Apple Notes does not know how to deal with hyperlinks.

Besides the lack of export functionality, one aspect of the app that I am particularly annoyed with is how Apple Notes manages hyperlinks. Apple Notes does not know how to deal with hyperlinks, and its behavior varies across systems and between methods. Here, I have listed the most consistent interactions and behaviors that I have found, but even these may not result in same outcomes based on the page that you are creating link from, or the time of the day I suppose!

macOS

Method 1

Copying (Cmd + C) a link from Safari’s address bar results in a formatted link with the hyperlink itself being the text, e.g., https://support.apple.com/en-us/HT205773.

You cannot neither change the text, nor the link afterward, at least not trivially. You can edit the text, but if you are not careful you might delete the link itself.

Method 2

By using the “Cmd + K” command on a selected text, you can create your own text and link it to a destination as you can do within most apps.

After creating a link, you cannot change the text anymore, at least not trivially. You can still change the link itself by using the “Cmd + K” command.

Method 3

You can create a rich link object by using the special “Add a link” toolbar button in the Apple Notes app. In this case, Apple Notes tries to guess which page you want to create a link from, and upon the selection, it appends a rich link object to your note.

You cannot interact much with this object. You can only tell it to use the small image instead of the large image, or vice verse.
You cannot make a list of these objects.
The “Add a link” option is only available via the toolbar, and it should not be confused by “Add link” shortcut which you can instantiate using “Cmd + K”.
Notice how Apple Notes picks up some random information from the Yahoo! Finance app, and does not give me the opportunity to see all the open tabs on Safari for instance.

iOS and iPadOS

Method 1

Copying a link from Safari’s address bar results in a formatted link with the hyperlink itself being the text, e.g., https://support.apple.com/en-us/HT205773.

You cannot neither change the text, nor the link; at least not trivially. You can very carefully edit it though.
You get the same behavior if you use “Cmd + C” shortcut when using an external keyboard.

Method 2

Copying a link using the “Copy” option of the Shared Menu, most of the times, results in a link with the page’s header as the text, e.g., Use Notes on your iPhone, iPad, and iPod touch.

If you are copying a link to a list, this action will break the list and it creates a new line!

Method 3

You can create a rich link object by using the Share Menu, and creating a new note, or appending the rich link object to an already existing notes.

Again, you cannot interact with this object much, and all the macOS rules are applied here.

From iOS to macOS, and vice versa

With Universal Control, it is possible to copy a link from iOS, and bring it to the macOS; however,

If you copy a link from the address bar of Safari on macOS and paste it into a note on iOS, you will get the first behavior of the above lists, i.e., a formatted link with the hyperlink itself being the text, e.g., https://support.apple.com/en-us/HT205773.
If you copy the link using the “Copy” option from the Share Menu on iOS/iPadOS, you again you get the first behavior and not the link with the title.

Possible Solution

I can imagine that they wanted to keep the Notes app as simple as possible, and they didn’t want to add extra unnecessary features to it; however, here, it feels like that the hyperlink interactions’ have been mostly overlooked. I say it is not about simplicity because Apple Notes has a proper table creation tools which arguably is quite complicated, certainly more than providing a box to rename a text/link.

On both macOS and iOS, Apple Notes comes with a text formatting option, and text formatting menu; however, there is no option for making or interacting with a link there. Long pressing a rich link object offers you an option to use a small or large image, but not an option to convert the item to a normal link. Similarly, long pressing a “normal” link does not offer you anything, e.g., converting to a rich link object, or pulling the title of the page, or editing the text!

Apple Notes’ UI is well-suited to deal with these problems and it can offer a consistent behavior but at the moment it does not, and that is frustrating! Maybe we can just have what it has been there for years in Pages, or Mails app, and stop this madness, please!

From qmake to CMake: A Nontrivial Case Study

2022-05-15T00:00:00+00:00

In this article, I will walk you through the process of rewriting JASP’s build system from scratch using CMake. It is a long journey of moving the entire build and deployment process from qmake to CMake. In addition, I will discuss how we have used CMake to work with R framework, and how we have used Conan to manage our dependencies in order to achieve a robust cross-platform, multi-architecture build, and deployment.

While this writing is very much focused on JASP project, and it might not directly apply to your project, I hope you can find bits and pieces of if helpful when you decide to go through the same process in your project. I should mention that I will not be covering every aspects of the software or the build process equally.

About JASP, and Its Architecture

JASP is an open-source alternative to SPSS, SAS, and other lookalikes. It is built on top of the Qt Framework, and R. This means that the main body of the app is written in C++, i.e., GUI, Data Management, Engine. In contrast, all the statistical analyses are done in R, and the results are being communicated back to the GUI using JSON objects. JASP modules are specialized R packages consist of R, and some QML codes. The R part is responsible for calculation, and it is being executed in an R instance using the JASPEngine, and the QML part is being parsed by JASP to construct an ocean of check boxes¹ and controls for the user to interact with!

JASPEngine runs as a separate process alongside JASP, and uses libR-Interface ≅ R + RInside + Rcpp to run the R analyses and commands, and ultimately retrieve their results. The libR-Interface is a library developed by JASP Team. It encapsulates an interface to R, as well as some logic on how the two apps needs to communicate to each other.

I am not intending to describe the entire architecture of JASP. My goal is to provide a high level overview of its architecture and components in order for you to be able to observe its complexity and hopefully find similar situations or solutions to/for your project.

R Framework, R Packages and JASP Modules

JASP uses R as its computational back-end. Every command or analysis, one way or another, is being passed to an instance of R (via JASPEngine), and after a successful computation, the results are being communicated back to JASP’s GUI using a JSON object (via JASPEngine). Upon the retrieval of results, JASP processes the results and generates a HTML page, and finally it displays it inside a Chromium WebEngine, i.e., QtWebEngine.

R Framework

On both Windows, and macOS, a copy of R Framework is being embedded into JASP’s binary so that JASP can communicate with R without needing to rely on system’s R, or asking the user to install an instance of R. On Linux, JASP uses Flatpak to manage a local copy of R.

On Windows, the entire R internal is being shipped. The libR-Interface.dll links itself against three main libraries (R.dll, RInside.dll and Rcpp.dll) to be able to talk to R, and performs its calculation. One complication that raises here is the fact that libR-Interface.dll needs to be linked against R libraries, and that means that it needs to be compiled with GCC, or MinGW ² and linked against the JASP GUI that, in our case, is compiled with MSVC. At the same time, Qt 6 does not like to be compiled using GCC; therefore, we ought to deal with two compilers. Later on, I will explain this problem and discuss how I used CMake toolchains to overcome this issue.
On macOS, the R.framework is being embedded into the Mac App Bundle, and libR-Interface.dylib dynamically links against R.dylib (a.k.a R.framework), RInside.dylib and Rcpp.dylib.³

In both systems, I use CMake to glue everything together, more on this later.

I will be using “R Framework” to refer to the R/ folder on Windows, and R.framework on macOS.

R Packages, and JASP Modules

R packages that are necessary to the function of JASP needs to be installed inside the R Framework and shipped within the final binary.⁴ Some of the R packages like RInside and Rcpp are needed during the build, as they need to be linked against the libR-Interface and consequently JASPEngine; as a results they need to be treated differently. More on this later.

JASP modules are basically extended R packages with some QML files that are being used by JASP to construct the graphic user interface for each module, i.e., all those controls and checkboxes. In addition, they know how to prepare, and pass their results back to JASP. These modules need to be installed and shipped within the final binary, and they do not necessary need to be available during the build.

The process of installing both the R packages and JASP modules are being handled by CMake either during the configuration or at the build step, more on this later.

Qt 5, qmake, and Dependencies

Prior to JASP 0.16.2, JASP was based on Qt 5, and it was using qmake as its build system generator. In order to manage the libraries, a pool of pre-built libraries, headers, and frameworks for every platform and architecture was maintained by the developers, and in every build they were being manually linked to the final binary.

As we decided to upgrade to Qt 6 and CMake, I set to get rid of the manually maintained pool of libraries and headers. Fortunately, CMake is much better at dealing with third-party libraries than qmake. In addition, I have adapted Conan as our package manager to be able to build the dependencies for every target individually.

Qt6, CMake, and Conan

As of Qt 6, CMake is the default build system generator of the Qt Framework. Qt team has started adapting the CMake in their internal processes, and projects and they are continuously improving their CMake toolsets in past years. As Qt 6 is becoming more mature, more and more Qt-specific CMake commands are being added, and the gap between qmake and CMake is getting smaller.

From here onward, I assume that you are familiar with CMake, Qt, and at least have read this article and are familiar with basics of CMake.

CMake Setup

Now that we are somewhat familiar with the architecture of JASP, and its bits and pieces, let’s start looking into how the project is setup, and what are the tasks that needs to be done.

jasp-desktop/
  ├── CMakeLists.txt
  ├── Common
  │    └── CMakeLists.txt
  ├── R-Interface
  │    └── CMakeLists.txt
  ├── Engine
  │    └── CMakeLists.txt
  ├── Desktop
  │    └── CMakeLists.txt
  ├── Modules/
  ├── Resources/
  ├── Tools
  │    └── CMake
  │        ├── R.cmake
  │        ├── Conan.cmake
  │        ├── Modules.cmake
  │        ├── Programs.cmake
  │        ├── Libraries.cmake
  │        ├── Dependencies.cmake
  │        └── ...
  ├── R (R/ or R.framework)
  └── conanfile.txt

As you can see, JASP consists of several sub-projects. In addition to the following four main sub-projects, we have the Modules/ folder that contains the JASP Modules, and finally a generic Resources/ folder.

Common, leading to the generation of the libCommon
R-Interface, which compiles into the libR-Interface.
Engine, which compiles into JASPEngine executable
and Desktop, which is the main GUI of the app, and it mainly consists of Qt codes

Below you can see some of the main interactions and dependencies between all the entities of the project. I did not go into full length to describe whether the libraries are linked statically or dynamically; however, in general,

on macOS most libraries are linked statically except a few dependencies and Qt itself.
on Windows most libraries are linked dynamically except libCommon and few other exceptions (if I recall correctly)

JASP's Dependency Graph

Dependencies

Before we start plugging things together using CMake, we need to make sure that all our dependencies are ready and available. I categorize the dependencies into 3 main categories,

R Framework
Dependencies that can be managed using Conan
and dependencies that needs special care, e.g., JAGS, ReadStat
- These are dependencies that do not offer a versatile build script, or cannot be found in Conan’s cellar

R Framework

R Framework is a package consists of libraries, binaries, header files, help files, and anything else that needed for R to function. Its main library is R.dll (or libR.dylib on macOS) and its main executable is R.exe (or bin/R on macOS, see here for more on the anatomy of R.framework). As mentioned, libR-Interface and some other libraries in the JASP project will be linking to R Framework; therefore it needs to be available during the build process.

In order to get the most recent version of the R Framework, and create a reproducible build, we use CMake to identify the architecture of the system, then download the R package, and place it inside the build/.

R.cmake is a CMake module written to manage this task. Based on the host/target system, it downloads the R package, unpack it, and place it in the build/ folder. Additionally, if necessary, it downloads extra dependencies of R, e.g., Tk/Tcl, FORTRAN; and add them to the final package. Furthermore, it sets some of the R related parameters of the project.

On macOS

On macOS, a few key points needs to be dealt with:

Setting path parameters
Downloading the right R.framework based on the host system.
Downloading the corresponding FORTRAN compiler on the host system.
Unpacking everything, and moving them into the build/ folder.
Patching the libraries, and resigning them using the Apple Developer Certificate
Downloading and installation of RInside and Rcpp inside the R.framework for later usage, and linking. Read more here.

Setting parameters:

set(R_FRAMEWORK_PATH "${CMAKE_BINARY_DIR}/Frameworks")
set(R_HOME_PATH "${R_FRAMEWORK_PATH}/R.framework/Versions/${R_DIR_NAME}/Resources")
set(R_LIBRARY_PATH "${R_HOME_PATH}/library")
set(R_OPT_PATH "${R_HOME_PATH}/opt")
set(R_EXECUTABLE "${R_HOME_PATH}/R")
set(R_INCLUDE_PATH "${R_HOME_PATH}/include")
set(RCPP_PATH "${R_LIBRARY_PATH}/Rcpp")
set(RINSIDE_PATH "${R_LIBRARY_PATH}/RInside")

Declaring the FetchContent object for downloading the R Framework:

fetchcontent_declare(
  r_pkg
  URL ${R_DOWNLOAD_URL}
  URL_HASH SHA1=${R_PACKAGE_HASH}
  DOWNLOAD_NO_EXTRACT ON
  DOWNLOAD_NAME ${R_PACKAGE_NAME})

Unpacking and moving the R.framework to the build/ folder:

execute_process(
  WORKING_DIRECTORY ${r_pkg_SOURCE_DIR}
  COMMAND tar -xf tcltk.pkg/Payload -C ${r_pkg_r_home}/)

execute_process(
  WORKING_DIRECTORY ${r_pkg_SOURCE_DIR}
  COMMAND tar -xf texinfo.pkg/Payload -C ${r_pkg_r_home}/)

make_directory(${CMAKE_BINARY_DIR}/Frameworks)
execute_process(
  WORKING_DIRECTORY ${r_pkg_SOURCE_DIR}
  COMMAND cp -Rpf R.framework ${CMAKE_BINARY_DIR}/Frameworks)

Installing the RInside and Rcpp inside the R.framework:

file(
  WRITE ${MODULES_RENV_ROOT_PATH}/install-RInside.R
  "install.packages(c('RInside', 'Rcpp'), type='binary', repos='${R_REPOSITORY}', INSTALL_opts='--no-multiarch --no-docs --no-test-load')"
)

execute_process(
  ERROR_QUIET OUTPUT_QUIET
  WORKING_DIRECTORY ${R_HOME_PATH}
  COMMAND ${R_EXECUTABLE} --slave --no-restore --no-save
          --file=${MODULES_RENV_ROOT_PATH}/install-RInside.R)

On Windows

Similarly, on Windows, we need to download the R package, unpack it and place it inside the build/ folder, and finally we need to install the RInside and Rcpp inside the build/R folder.

Downloading, unpacking and moving the R package to the build/ folder:

fetchcontent_declare(
  r_win_exe
  URL ${R_DOWNLOAD_URL}
  URL_HASH SHA1=${R_PACKAGE_HASH}
  DOWNLOAD_NO_EXTRACT ON
  DOWNLOAD_NAME ${R_PACKAGE_NAME})

execute_process(
  WORKING_DIRECTORY ${r_win_exe_SOURCE_DIR}
  COMMAND ${R_PACKAGE_NAME} /CURRENTUSER /verysilent /sp
          /DIR=${r_win_exe_BINARY_DIR}/R)

file(COPY ${r_win_exe_BINARY_DIR}/R DESTINATION ${CMAKE_BINARY_DIR})

Installing the RInside and Rcpp inside the R/:

file(
  WRITE ${MODULES_RENV_ROOT_PATH}/install-RInside.R
  "install.packages(c('RInside', 'Rcpp'), type='binary', repos='${R_REPOSITORY}' ${USE_LOCAL_R_LIBS_PATH}, INSTALL_opts='--no-multiarch --no-docs --no-test-load')"
)

execute_process(
  ERROR_QUIET OUTPUT_QUIET
  WORKING_DIRECTORY ${R_BIN_PATH}
  COMMAND ${R_EXECUTABLE} --slave --no-restore --no-save
      --file=${MODULES_RENV_ROOT_PATH}/install-RInside.R)

I am omitting a lot of details here; especially when it comes to how R packages installation works. As far as CMake concerns, you need to execute a process to call the ${R_EXECUTABLE} and make sure that it can install your requested package.

On Linux

On Linux, since JASP uses Flatpak, we can rely on system’s R; therefore we only need to located the R libraries, and set some paths for later usage, so that we can link everything and use the headers when needed.

Conan

Now that we have the R Framework ready to be used, and linked against. We need to deal with the rest of dependencies. Except a few libraries, all JASP dependencies can be found in ConanCenter, and therefore can simply be defined inside a conanfile.txt.

[requires]
boost/1.78.0
libiconv/1.16
libarchive/3.5.2
zlib/1.2.11
zstd/1.5.2
jsoncpp/1.9.5
openssl/1.1.1m
bison/3.7.6
brotli/1.0.9

[generators]
cmake_paths
cmake_find_package

[options]
brotli:shared=True

While most of the time, running conan install .. inside the build directly is enough for setting up the libraries, we need to make sure that we can handle different architectures, especially on macOS where you can (need to) build both for x86_64 and arm64.

A savvy Conan user can often run the appropriate commands and set up the right environment, however, that’s not always straightforward; therefore, I decided to delegate this task to a CMake module, Conan.cmake. Here, based on the architecture, build type, and target properties’, Conan.cmake decides what Conan command should be used:

if(WIN32)

  message(STATUS "  ${CONAN_COMPILER_RUNTIME}")

  execute_process(
    COMMAND_ECHO STDOUT
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
    COMMAND
      conan install ${CONAN_FILE_PATH} -s build_type=${CMAKE_BUILD_TYPE} -s
      compiler.runtime=${CONAN_COMPILER_RUNTIME} --build=missing)

elseif(APPLE)

  if(CROSS_COMPILING)

    execute_process(
      COMMAND_ECHO STDOUT
      WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
      COMMAND
        conan install ${CONAN_FILE_PATH} -s build_type=${CMAKE_BUILD_TYPE} -s
        os.version=${CMAKE_OSX_DEPLOYMENT_TARGET} -s os.sdk=macosx -s
        arch=${CONAN_ARCH} -s arch_build=${CONAN_ARCH} --build=missing)

  else()

    execute_process(
      COMMAND_ECHO STDOUT
      WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
      COMMAND
        conan install ${CONAN_FILE_PATH} -s build_type=${CMAKE_BUILD_TYPE} -s
        os.version=${CMAKE_OSX_DEPLOYMENT_TARGET} -s os.sdk=macosx
        --build=missing)

  endif()

endif()

Integrating Conan is often as simple as this. If you make sure that you are selecting the right libraries, and setting the right flags for your conan install .., you can simply use find_package command to link your Conan libraries to your project.

Notice that I have used two generators in the Conan file, cmake_paths and cmake_find_package. This is because some of those libraries are not properly set up to work with CMake projects, and therefore I need to use Conan environment variables to tap into their header or libraries folders. In addition, as we see later on during the Windows build, I needed a more fine-grained control over which libraries to select, so this was necessary.

Misc./Problematic Libraries

One of the problematic libraries that we had to deal with was ReadStat. ReadStat neither has support for CMake, nor (a proper) PkgConfig setup. This makes it challenging to integrate into a project, especially because we had to make sure that libreadstat is build correctly for our selected host and target. Therefore, we had to dive deeper into its native build system, and make sure that appropriate parameters are set.

As you can see below, based on the system and CMake configurations, we instruct the autoconf to configure the ReadStat’s Makefile accordingly. Dependencies.cmake:

set(READSTAT_CFLAGS
    "-g -O2 -arch ${CMAKE_OSX_ARCHITECTURES} -mmacosx-version-min=${CMAKE_OSX_DEPLOYMENT_TARGET}"
)
set(READSTAT_EXTRA_FLAGS_1 "--with-sysroot=${CMAKE_OSX_SYSROOT}")
set(READSTAT_EXTRA_FLAGS_2 "--target=${CONFIGURE_HOST_FLAG}")
set(READSTAT_CXXFLAGS "${READSTAT_CFLAGS}")

add_custom_command(
  WORKING_DIRECTORY ${readstat_SOURCE_DIR}
  OUTPUT ${readstat_BINARY_DIR}/include/readstat.h
         ${readstat_BINARY_DIR}/lib/libreadstat.a
  COMMAND
    export CFLAGS=${READSTAT_CFLAGS} && export CXXFLAGS=${READSTAT_CXXFLAGS}
    && ./configure --enable-static --prefix=${readstat_BINARY_DIR}
    ${Iconv_FLAGS_FOR_READSTAT} ${READSTAT_EXTRA_FLAGS_1}
    ${READSTAT_EXTRA_FLAGS_2}
  COMMAND ${MAKE}
  COMMAND ${MAKE} install
  COMMENT "----- Preparing 'readstat'")

add_custom_target(readstat
                  DEPENDS ${readstat_BINARY_DIR}/include/readstat.h)

set(LIBREADSTAT_INCLUDE_DIRS ${readstat_BINARY_DIR}/include)
set(LIBREADSTAT_LIBRARY_DIRS ${readstat_BINARY_DIR}/lib)
set(LIBREADSTAT_LIBRARIES ${LIBREADSTAT_LIBRARY_DIRS}/libreadstat.a)

Similar situation raised when we started dealing with JAGS. With JAGS, on macOS, we had to provide an appropriate FORTRAN compiler as well, and make sure that the FORTRAN compiler as well is aware of the environment variables. We went to this length to make sure that our build is consistent and we can cross-compile our software for x86_64 and arm64. Modules.cmake:

fetchcontent_declare(
  jags
  URL "https://sourceforge.net/projects/mcmc-jags/files/JAGS/4.x/Source/JAGS-4.3.0.tar.gz"
  URL_HASH
    SHA256=8ac5dd57982bfd7d5f0ee384499d62f3e0bb35b5f1660feb368545f1186371fc
)

message(CHECK_START "Downloading 'jags'")

fetchcontent_makeavailable(jags)

if(jags_POPULATED)

  message(CHECK_PASS "successful.")

  set(JAGS_F77_FLAG "F77=${FORTRAN_EXECUTABLE}")
  set(JAGS_CFLAGS
      "-g -O2 -arch ${CMAKE_OSX_ARCHITECTURES} -mmacosx-version-min=${CMAKE_OSX_DEPLOYMENT_TARGET}"
  )
  set(JAGS_EXTRA_FLAGS_1 "--with-sysroot=${CMAKE_OSX_SYSROOT}")
  set(JAGS_EXTRA_FLAGS_2 "--target=${CONFIGURE_HOST_FLAG}")
  set(JAGS_CXXFLAGS "${JAGS_CFLAGS}")

  add_custom_command(
    JOB_POOL sequential
    WORKING_DIRECTORY ${jags_SOURCE_DIR}
    OUTPUT ${jags_VERSION_H_PATH}
    COMMAND
      export CFLAGS=${READSTAT_CFLAGS} && export
      CXXFLAGS=${READSTAT_CXXFLAGS} && ${JAGS_F77_FLAG} ./configure
      --disable-dependency-tracking --prefix=${jags_HOME}
      ${JAGS_EXTRA_FLAGS_1} ${JAGS_EXTRA_FLAGS_2}
    COMMAND ${MAKE}
    COMMAND ${MAKE} install
    COMMAND
      ${CMAKE_COMMAND} -D
      NAME_TOOL_PREFIX_PATCHER=${PROJECT_SOURCE_DIR}/Tools/macOS/install_name_prefix_tool.sh
      -D PATH=${jags_HOME} -D R_HOME_PATH=${R_HOME_PATH} -D
      R_DIR_NAME=${R_DIR_NAME} -D
      SIGNING_IDENTITY=${APPLE_CODESIGN_IDENTITY} -D
      SIGNING=${IS_SIGNING} -D
      CODESIGN_TIMESTAMP_FLAG=${CODESIGN_TIMESTAMP_FLAG} -P
      ${PROJECT_SOURCE_DIR}/Tools/CMake/Patch.cmake
    COMMENT "----- Preparing 'jags'")

R is very particular about the FORTRAN compiler, and it uses gfortran 8 on x86_64 architecture, and gfortran 12 on arm64 architecture. To overcome this complication, R.cmake downloads the right FORTRAN, places it inside the R.framework, and makes sure that R toolchain can find and use it. On Windows, things are simpler as Rtools42 bundles everything R needs into a MSYS environment, and we can just use that instead. Also, the ARM Windows is not yet a thing!

Managing the CMake Project

Now that we have all our dependencies ready, let’s talk CMake project management. As shown below, we start by building each library separately using its own dedicated CMakeLists.txt configuration file.

jasp-desktop/
  ├── CMakeLists.txt
  ├── Common
  │    └── CMakeLists.txt
  ├── R-Interface
  │    └── CMakeLists.txt
  ├── Engine
  │    └── CMakeLists.txt
  ├── Desktop
  │    └── CMakeLists.txt
  ├── Modules/
  ├── Resources/
  ├── Tools
  │    └── CMake
  │        ├── R.cmake
  │        ├── Conan.cmake
  │        ├── Modules.cmake
  │        ├── Programs.cmake
  │        ├── Libraries.cmake
  │        ├── Dependencies.cmake
  │        └── ...
  ├── R (R/ or R.framework)
  └── conanfile.txt

The top level CMakeLists.txt file starts with the project definition, and system variable detection routines, e.g., arch, build type. Besides that it mainly consists of several include and add_subdirectory commands loading various aspects of the project as you can see in the simplified version of the file below.

# CMake Modules
###############

include(Config)
include(Conan)
include(Programs)
include(Libraries)
include(Dependencies)
include(JASP)
include(R)

# Sub Directories / Projects
############################

add_subdirectory(Common)

if(NOT WIN32)
  add_subdirectory(R-Interface)
else()
  add_custom_target(
    R-Interface
    WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/R-Interface
    BYPRODUCTS ${CMAKE_BINARY_DIR}/R-Interface/libR-Interface.dll
               ${CMAKE_BINARY_DIR}/R-Interface/libR-Interface.dll.a
    COMMAND
      ${CMAKE_COMMAND} -G "MinGW Makefiles" -S . -B
      ${CMAKE_BINARY_DIR}/R-Interface "-DMINGW_PATH:PATH=${MINGW_PATH}"
      "-DCMAKE_C_COMPILER:PATH=${MINGW_C_COMPILER}"
      "-DCMAKE_CXX_COMPILER:PATH=${MINGW_CXX_COMPILER}"
      "-DCMAKE_MAKE_PROGRAM:PATH=${MINGW_MAKE_PROGRAM}"
      "-DJASP_BINARY_DIR:PATH=${CMAKE_BINARY_DIR}"
      "-DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}"
    COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR}/R-Interface
    COMMENT "------ Configuring and Building the libR-Interface")
endif()

add_subdirectory(Engine)
add_subdirectory(Desktop)

include(Modules)

# Installation and Packing
##########################
 
include(Install)

if(NOT LINUX)
  include(Pack)
endif()

The first part, CMake Modules, makes sure that dependencies are downloaded, prepared and if needed built on the host system for the target system.
The second part, Sub Directories / Projects, loads every sub project one by one and run their dedicated CMakeLists.txt and upon successful execution prepares the corresponding library or executable for the next subproject, e.g., libCommon is needed for successful creation of libR-Interface. In addition, the Modules CMake module installs all the JASP Modules.
The third part, Installation and Packing, mocks the install process, deploys the project and prepares the it for the creation of the WIX Installer or an App Bundle on macOS.

If you know the basic of CMake, you can see that nothing mysterious is happening in those CMakeLists.txt files; except maybe how some of those dependencies have been handled. There, I have utilized the cmake-generator-expressions to link the appropriate libraries based on what is available where on different systems and/or different configurations. E.g., you can see below that $<$<BOOL:${USE_CONAN}>:jsoncpp::jsoncpp> instructs CMake to use jsoncpp::jsoncpp target if Conan has been used, i.e., on Windows and macOS; however, on Linux where we rely on system libraries, we use PkgConfig’s variables to access the header files, and link to libraries, $<$<PLATFORM_ID:Linux>:${_PKGCONFIG_LIB_JSONCPP_INCLUDEDIR}>.

target_include_directories(
  Common
  PUBLIC # JASP
         jaspColumnEncoder
         # R
         ${R_INCLUDE_PATH}
         ${R_HOME_PATH}/include
         ${RCPP_PATH}/include
         #
         $<$<PLATFORM_ID:Linux>:${_PKGCONFIG_LIB_JSONCPP_INCLUDEDIR}>)

target_link_libraries(
  Common
  PUBLIC # jsoncpp
         $<$<BOOL:${USE_CONAN}>:jsoncpp::jsoncpp>
         $<$<PLATFORM_ID:Linux>:${_PKGCONFIG_LIB_JSONCPP_LIBRARIES}>
         $<$<PLATFORM_ID:Linux>:${_PKGCONFIG_LIB_JSONCPP_LINK_LIBRARIES}>
         #
         LibArchive::LibArchive
         # Boost
         Boost::filesystem
         Boost::system
         Boost::date_time
         Boost::timer
         Boost::chrono
         #
         $<$<BOOL:${JASP_USES_QT_HERE}>:Qt::Core>)

Building R-Interface on Windows, Rtools42, and Other Considerations for Windows

As briefly discussed, building libR-Interface on Windows is not a trivial task. This is due to the fact that the libR-Interface needs to be linked against R.dll, RInside.dll and Rcpp.dll. In addition, by adding their headers to the project, we trigger their preprocessors which are going to stop us from compiling if we are not compiling against a GCC-compatible compiler.

As a result, we need to build the libR-Interface differently (using GCC) and link it dynamically against our other libraries and executable. Fortunately, or unfortunately, R team maintains their own MSYS system. This allows us to use GCC friendly compilers on Windows and build the R-Interface.dll there and finally links it against the JASP executable (+ Qt6) which is going to be built using MSVC.

In order to do this dance, we use the following code in the top-level CMakeLists.txt:

if(NOT WIN32)
  add_subdirectory(R-Interface)
else()
  add_custom_target(
    R-Interface
    WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/R-Interface
    BYPRODUCTS ${CMAKE_BINARY_DIR}/R-Interface/libR-Interface.dll
               ${CMAKE_BINARY_DIR}/R-Interface/libR-Interface.dll.a
    COMMAND
      ${CMAKE_COMMAND} -G "MinGW Makefiles" -S . -B
      ${CMAKE_BINARY_DIR}/R-Interface "-DMINGW_PATH:PATH=${MINGW_PATH}"
      "-DCMAKE_C_COMPILER:PATH=${MINGW_C_COMPILER}"
      "-DCMAKE_CXX_COMPILER:PATH=${MINGW_CXX_COMPILER}"
      "-DCMAKE_MAKE_PROGRAM:PATH=${MINGW_MAKE_PROGRAM}"
      "-DJASP_BINARY_DIR:PATH=${CMAKE_BINARY_DIR}"
      "-DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}"
    COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR}/R-Interface
    COMMENT "------ Configuring and Building the libR-Interface")
endif()

Here, we tap out of our main CMake build process, and start a new process based on the R-Interface/CMakeLists.txt which itself is constructed such that it behaves as a standalone CMake projects on Windows, see below. By asking the CMake to construct a “MinGW Makefiles”, and setting all the necessary paths to MINGW’s C and C++ compiler, we initiate the configuration and build process of the R-Interface project from our main CMake project. If the build is successful, the build/R-Interface will contains the libR-Interface.dll and it is ready to be used by the rest of the project.

Note that in the snippet above we are passing several of our local variables to the ${CMAKE_COMMAND} when we are initiating the config/build of the libR-Interface. This is because when you are taping out of the main CMake process, the new process does not have access to your local variable anymore and you need to provide them to the other project manually, or using a template file.

if(WIN32)

  set(QT_CREATOR_SKIP_CONAN_SETUP ON)

  cmake_minimum_required(VERSION 3.21)

  project(
    R-Interface
    VERSION 11.5.0.0
    LANGUAGES C CXX)

  list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/../Tools/CMake")

  # More CMake commands for building the R-Interface

else()

  # Building for macOS, and Linux without defining a new projects

endif()

It is worth noting that there is a lot of detailed involved in here, and if you are interested you should study the R-Interface/CMakeLists.txt, Programs.Cmake and the main CMakeLists.txt file. One of the complication that we faced was the fact that libR-Interface.dll needs to be linked against Boost, and it could not be linked against the Boost that was already prepared by Conan for the main project using MSVC. Instead, it needed to be linked against a Boost that is build inside the MSYS (i.e., Rtools42) environment. Therefore, we had to make sure that Boost (and jsoncpp) is available inside the MSYS system. Since Rtools42 is a full-featured MSYS environment, we can achieve this by installing those packages using pacman. After that, we only had to make sure that a copy of all those libraries sit next to the JASP executable because Windows. So, some manual (via CMake’s file command) copy/paste of libraries were needed.

I have mentioned Rtools42, but at the time that we implemented this method, R 4.0 was not released yet. Therefore, we had to use the combination of Rtool35 and MSYS to achieve a complete build. Most links to CMake files in this article are pointing to tags/0.16.2. If you are interested on how Rtools42 has simplified this process, you can study this pull request, Support for R-4.2.0. In this RP, I have replaced the MSYS environment with Rtools42. (The PR is closed, however, it is content is merged into this pull request.)

Build

Now that all our dependencies, libraries, and packages ready, we should be able to simply run the CMake command, and compile everything. I am not going to go into details on how to initiate the build, and I assume that you are familiar with the following sequence, or, you are going to open the project in Qt Creator, and rely on its integration! 🤞🏼

mkdir build && cd build
cmake .. -GNinja -DCMAKE_PREFIX_PATH=<path-to-qt>
ninja

Installation and Deployment

The one last thing to do is to deploy the project. I will not go into details of the deployment, rather I mention a few common issues or obstechales that we had to deal with, and leave you with a link to Install and Pack modules.

Deployment of Qt projects are done using the macdeployqt and windeployqt. Unfortunately, neither of these tools are robust enough to do everything and you will end up having to do several things manually. CPack is designed to help you with the deployment of your project; however, to take advantages of the CPack, you must have a very standard project tree, libraries, and dependencies. In our case, the R.framework and exceptions around the libR-Interface did not allow us to use CPack successfully.

On macOS, CPack can construct a .app bundle form your project; however, you have a slim chance of getting this to work if you are using an external Framework, in our case R.framework.
On Windows, we have already had a WIX template and we still ran into problem due to the complication caused by R/ and some of the modules.

In both system, our approach was to stage the installation and fully deploy the project, and all its libraries inside the Install/ folder, and pass everything to WIX on Windows, and on macOS put everything precisely where they need to be according to Apple’s Bundle Structure.

The install command is your friend. Set up your project structure, and handle as much as your installation logic using the install command, e.g.,

set(MACOS_BUNDLE_NAME JASP)
set(JASP_INSTALL_PREFIX "${CMAKE_INSTALL_PREFIX}/${MACOS_BUNDLE_NAME}.app")
set(JASP_INSTALL_BINDIR "${JASP_INSTALL_PREFIX}/Contents/MacOS")
set(JASP_INSTALL_RESOURCEDIR "${JASP_INSTALL_PREFIX}/Contents/Resources")
set(JASP_INSTALL_FRAMEWORKDIR "${JASP_INSTALL_PREFIX}/Contents/Frameworks")
set(JASP_INSTALL_MODULEDIR "${JASP_INSTALL_PREFIX}/Contents/Modules")
set(JASP_INSTALL_DOCDIR "${JASP_INSTALL_RESOURCEDIR}")

install(
  TARGETS JASP JASPEngine
  RUNTIME DESTINATION ${JASP_INSTALL_BINDIR}
  BUNDLE DESTINATION .)

install(
  DIRECTORY ${_R_Framework}
  DESTINATION ${JASP_INSTALL_FRAMEWORKDIR}
  REGEX ${FILES_EXCLUDE_PATTERN} EXCLUDE
  REGEX ${FOLDERS_EXCLUDE_PATTERN} EXCLUDE
  REGEX ".*/bin/R(Script)?$" EXCLUDE
  )

Caveats

As mentioned before, due to the requirements of JASP project, we had to deal with several type of dependencies. Some managed by Conan, others fetched using the FetchContent, and the rest were downloaded directly from their repository. I have to emphasize that this is not an ideal scenario, and you should avoid it if you can. You should try to manage all your dependencies with one dependency manager. In our case, we could have improved our setup by writing a Conan recipe for ReadStat, and let Conan handle its build as well. However, we could have not moved everything to Conan, e.g., R Framework and JAGS (due to its FORTRAN dependency).

Additionally, I should reiterate the fact that, ideally you should not interact with Conan within your CMake files, and educate your team to use Conan command line instead. However, in our case, due to time pressure and because the team was not familiar with Conan, I decided to manage some of the Conan parameters with CMake. This is not ideal because you risk replying on deprecated Conan features’ as Conan evolves.⁵ In our case, I have made a note/task of this fact and encouraged the team to deprecate the Conan.cmake module, study Conan, and setup their environment manually for each build.

Summary

CMake offers a lot more freedom compared to qmake. I am aware that CMake is not everyone’s cup of tea and many people do not like it. However, I feel if you have tried CMake before and had similar experience, you need to give it another try. A lot has changed from early days, and large and complicated projects like Qt or KDE heavily rely on it. In Modern CMake⁶, you have much more control over your targets, and libraries. CPack, although not perfect, can facilitate your deployment greatly if you have a relatively standard project. Finally, dependency managers like Conan are improving the CMake integration every day, and can generate CMake config files for packages that do not come with CMake support of the box.⁷

I should also reiterate that most of what I have discussed might not directly apply to your project; however, I hope this could come handy if you are trying to resolve some of your build system issues’, or rewriting it from scratch because you want to move away from qmake to CMake. Or at very least, I hope it gives you an idea on how far you can get with CMake when it comes to a rather complicated and irregular project, which is about every C/C++ project out there! 😅

P.S. I was expecting this article to become long, but not this long! Feel free to reach out on Twitter, or Cpplang Slack where I occasionally spend some time, and try to learn from very knowledgeable members of #conan, #cmake and #qt channels! 🙏🏼

If OpenSSL were a GUI ↩
R is a GNU Project and therefore it can only be compiled using the GNU GCC compiler. This means, anything that needs to be linked directly to R libraries needs to use a GCC-compatible compiler. ↩
I have already covered the complicated and cumbersome process of successfully embedding the R.framework into the app bundle, as well as the process of signing and notarizing the app bundle for distribution. ↩
Most of the R packages are being installed via Renv. ↩
Conan 2.0 ↩
I highly recommend Professional CMake: A Practical Guide book by Craig Scott. I personally have learned a lot from it. You can often find him answering some question on CMake Discourse as well. ↩
The quality of the config file you get out of Conan depends on the quality of the Conan script but in general, chances are high that you get a good, and working config file, so that you can simply use the target, e.g., jsoncpp::jsoncpp. ↩

Embedding R.framework in a Qt Mac App, and Cross Compiling for Two Architectures

2022-04-18T00:00:00+00:00

Developing for Apple platform can be fun if your app lives solely inside Xcode, and you only utilize standard libraries, or don’t interact with any alien libraries or frameworks. In that case, there is a high chance that you have never been worried about compatibility issues, linking problems, signing struggles and the notarization!

However, if your software is built on top of a different GUI framework rather than UIKit, e.g., Qt Framework, and you need all sort of other libraries; then you are familiar with the pain of getting them to know each other, link together properly, and build into one coherent piece of software. Finally, you are going to prepare your App Bundle, sign it, and send it to Apple for notarization and start praying! Now, add the R.framework or any other non-standard framework to the mix, and you are very close to the definition of nightmare.¹

JASP is one of these software, where in addition to embedding the R.framework, we include series of R packages and Modules. Some can be downloaded as binaries from CRAN, some need compilation, and some needs third-party libraries, e.g. JAGS. Now, add to all that the fact that we need to do all that for two architectures, i.e., arm64 and x86_64, and three platforms.

In this article, I go through some of the challenges that we had to overcome to get all this to working in a stable manner on macOS. Most of the heavy-lifting of the build system is being taken care of by CMake which is not discussed here; instead, in this post I focus on preparing and embedding the R.framework.

R.framework

R.framework is inherently not portable. This means the entire Framework is built with hardcoded paths. It expects to be installed in /Library/Frameworks/R.framework, and every other libraries and packages expect it to be there so that they can link to the R libraries and therefore function correctly. Placing the R.framework to any other location on your system is merely asking for trouble. One of the biggest consequences of installing R.framework in a non-standard place is that none of the CRAN binary packages can navigate their ways to R’s home directory, consequently they cannot find the libR.dylib and therefore they won’t work.

For us, this is a big issue as we need to embed the R.framework to our app such that it can sit and function inside the JASP.app and work in our user’s system. In addition, we need to download several pre-build R packages from CRAN and inform them about the whereabouts of libR.dylib. Finally, we need to make sure that JASP modules can be built on our build machines and work inside our app bundle.

Anatomy of R.framework

R.framework, i.e., R, is a complicated and busy framework. It is designed to comply with Apple’s Framework Bundles Structure, but underneath its Apple’s flavored anatomy, everything is organized according to a standard GNU Install Structure where binaries lives in bin/ and headers are in include/, etc. This means when you are inside the GNU environment things get Linux-y as much as they could be on a Darwin subsystem.

Apple’s Framework Bundle is wrapper around a dynamic library, and all the miscellaneous files needed by the library. In the case of the R.framework, everything from include files, fonts, libraries and modules are included in the Framework. One crucial aspect of designing a Framework is to make sure that components of the framework can reliably find each other. In R.framework, this is insured by having everything route through R’s home directory, defined by the R_HOME environment variable. If you install the R.framework in its default place, R_HOME is set to /Library/Frameworks/R.framework/Resources which is a symbolic link to /Library/Frameworks/R.framework/Versions/A/Resources where A is the version of the R that you are using, e.g., 4.1 or 4.1-arm64.

Here, we can see that everything is neatly placed in their respectable folder:

Frameworks/R.framework/Versions/4.1-arm64/Resources
├── bin
  ├── exec
    └── R
  └── R.sh
├── doc
├── etc
  ├── ldpaths
  └── Makeconf
├── fontconfig
├── include
├── lib
├── library
├── man1
├── modules
├── opt
├── share
└── tests

Inside the bin/ folder, you find exec/R which is the actual R executable. However, this file is not simply being called by its own, instead, it is called from a bash script, bin/R.sh. This script does a lot of work, and one of them is to set the R_HOME variable and to inform the bin/exec/R about it.

Another crucial step that bin/R.sh takes is to load the etc/ldpaths. This file then makes sure that bin/exec/R and all the other libraries needed during the execution of R can be found by the system during the loading and execution process. If you are familiar with the concept of library pathing, then ldpaths sets the DYLD_FALLBACK_LIBRARY_PATH so that when bin/exec/R requests a library, the system knows where to look for them.

Another important file is placed in etc/ folder, etc/Makeconf. This file sets several environment and R variables, from compiler flags, library paths, header paths, to path of the Fortran compiler. These are useful and necessary information for when you are requesting R to build a package from source. More on this later.

Embedding R.framework

In order to make the framework portable, we need to make a few changes to the bin/R.sh. First, R_HOME has to correctly point to our desired path, second, we need to make sure that bin/exec/R and all the other libraries inside the R.framework can find each other during the loading process.

On the very top of the bin/R.sh, we need to set the R_HOME_DIR to our custom R’s home directory path. Then, I set the following variables as follow

R_HOME="${R_HOME_DIR}"
R_SHARE_DIR=${R_HOME_DIR}/share
R_INCLUDE_DIR=${R_HOME_DIR}/include
R_DOC_DIR=${R_HOME_DIR}/doc 

If you are planning to simply run R from a different place in your system, and do not want to install any packages, you are done! However, if you are planning to install any other packages, and later place the R.framework inside an App Bundle, your journey is just starting!

Your first step is to disable the default LD_PATH that is being set in etc/ldpaths. You can do this by commenting the following line so that bin/R.sh doesn’t try to overwrite/set the library path for you. While this is not strictly needed, it gives you more granular controls over your library pathing and simplifies your debugging later on.

# . "${R_HOME}/etc${R_ARCH}/ldpaths"

After this, if you try to run the bin/R.sh, you notice that the executable expectedly won’t work because system cannot find the libraries requested by the R executable.

Now, in the rest of this post, we are going to take care that libraries inside the R.framework can find each other even though we have disabled the ldpaths; in addition, we are going to modify the CRAN binaries upon installation and replace their hard-coded paths. To be able to follow the upcoming section, you need to be familiar with the concept of RPATH and LD_LIBRARY_PATH.²

Patching Library Path and `install_name_tool`

At this point, we know that bin/R.sh doesn’t know where libR.dylib and other libraries are. You can check the internal of a binary file using the otool command on macOS. Running otool -L bin/exec/R lists all the libraries that are linked to the executable:

❯ otool -L bin/exec/R
exec/R:
	/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libR.dylib (compatibility version 4.1.0, current version 4.1.3)
	/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRblas.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

We see that bin/exec/R is looking for libR.dylib in the default path /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/ which is in fact ${R_HOME}/lib but since we have changed it, it will not be able to find any of them, and will be terminated before it can get far into its execution process.³

After reading the above footnote, and refreshing your LD_LIBRARY_PATH knowledge, you now know how everything works, and that we are going to use the install_name_tool to change the library path! 😬

Since we are going to embed the R.framework into an App Bundle, we are going to use the @executable_path variable as our anchor point. Our executable is located in JASP.app/Contents/MacOS/JASP and our embedded R.framework will be placed in JASP.app/Contents/Frameworks/R.framework, so, by changing all /Library/Frameworks prefix with @executable_path/../Frameworks we make sure that if JASP executable calls the R executable, libraries can be found in the relative path to the main binary, ie. JASP, therefore

❯ otool -L bin/exec/R
exec/R:
	@executable_path/../Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libR.dylib (compatibility version 4.1.0, current version 4.1.3)
	@executable_path/../Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRblas.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

However, this is not yet enough. If we are in bin/exec/ and we try to run the already modified R executable, the @executable_path/../Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libR.dylib does not lead to anything. To overcome this issue, we need to create a symlink to the Frameworks folder inside the ${R_HOME}/bin/ that points to JASP.app/Contents/Frameworks. By doing so, when R executable is being called, and system starts looking for libR.dylib in @executable_path/../Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libR.dylib there will be a symlink that leads to the location of the libR.dylib. Basically, we are making sure that Frameworks folder is placed in the same relative location to R executable, and JASP executable, and @executable_path/../Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libR.dylib is always well-defined regardless of which executable is being called.

├── bin
  ├── Frameworks -> ../../../../../../Frameworks
  ├── exec
    └── R
  ├── .
  ├── .
  ├── .
  └── R
├── lib
    ├── libR.dylib
    ├── libRblas.dylib

Now, you need to do the same things for all the libraries that are inside the lib/ folder.

❯ otool -L libR.dylib
libR.dylib:
	@executable_path/../Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libR.dylib (compatibility version 4.1.0, current version 4.1.3)
	@executable_path/../Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRblas.dylib (compatibility version 0.0.0, current version 0.0.0)
	@executable_path/../Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1853.0.0)
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.8)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libicucore.A.dylib (compatibility version 1.0.0, current version 68.2.0)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)

Dealing with Other Libraries in `library/`, `modules/` and `opt/`

If you look around the R.framework, you will notice that there are a lot more libraries that all need to be patched with the same procedure. Most libraries are linked against libR.dylib and other libraries located in R_HOME/lib, so replacing the /Library/Frameworks/ with @executable_path/../Frameworks will resolves a lot of issues. However, it does not cover all the edge cases. At JASP, we use a CMake script to make sure that all libraries are patched.

CRAN Packages

When you are installing a CRAN binary package, you are requesting a pre-built version of the library. As discussed, this means, for instance, MASS.so will except to find the libR.dylib in /Library/Frameworks/Resources/lib. If you have the R.framework at its preferred place, every CRAN binary package will find the R library by following the R_HOME.

However, if R.framework is sitting in a non-default path, every CRAN binary package comes with a wrong library path; therefore, you have to patch every single of them manually as you did for all the first-party libraries and modules, as described before.

macOS Security and Signing

Modern macOS is quite strict about the identity and origin of the binaries and libraries that are allowed to run in a system. Every Framework or App needs to be signed, verified and notarized by Apple and it’s then and only then that macOS will fully trust and run them without complaining.[^This was a big issue for us where at some point due to the complexity of JASP.app, we couldn’t continue notarizing our app with Apple, so our users ended up getting scary warnings when they tried to run the app.]

R.framework comes pre-signed and notarized; however, any tempering with its guts will avoid its notarizations and signature. In fact, everything that we did so far falls into the definition of tempering. On a Mac with all default security options enabled, the modified bin/exec/R will not run, and macOS refuses to verify any of the libraries, e.g., libR.dylib, and therefore terminates the R executable at runtime.

Fortunately, we can resign a binary/library, and get around this issue. This means that you need to remember to sign every single binary/library that you are modifying during this process. For us, at JASP, we need to sign every libraries on the fly as we build our software. This is because at the same time where we are building our Desktop app, we are also calling the R executable to install our Modules inside the R.framework. You can see part of this process here where we are using a CMake module to automate this process during the configuration and build stages.

CMake, Qt, R.framework, and Cross Compilation

Cross compilation is the process of compiling a program for different architectures and platforms. Most C/C++ compilers are capable of cross compiling, and most build system and build system generators, e.g., automake, CMake, simplify the cross compilation by providing higher level flags, or variables. For instance, when using CMake on macOS, setting the MACOSX_DEPLOYMENT_TARGET variable ensures that the compiler builds your program for the specific architecture.

JASP core is written in C++, and we are using Qt Framework and other C++ libraries; and on top of that, CMake is used to generate the build system command; therefore, we can target different architecture and version by configuring CMake correctly. However, R.framework as an external pre-built framework doesn’t have any ideas about CMake, and it is only being linked to JASP App Bundle. The other problem is the Fortran compiler which is not fully stable on Apple Silicon; therefore its cross compilation is not to be trusted.

In the rest of this post, I briefly walk through the steps necessary for treating the R.framework such that it can correctly cross compile R source packages.

Cross Compilation of R Source Packages

With JASP 0.16.2, we rewrote our build system from scratch, see here. We committed to this effort to resolve some of the underlying issues in our backend. In addition, we wanted to have a native version of JASP for Apple Silicon, and that meant we had to deal with two different architectures, two different R.frameworks and a whole lot of other details. I am planning to write another post about this transition; here though, I would like to discuss what this cross compilation mean for R.framework and the process of embedding it to JASP.

Everything that I have mentioned above applies to both versions of R.framework, ie., x86_64 and arm64. Libraries and binaries should be patched similarly, the resigning procedure is as important and it should be done on the fly; and bin/R.sh and R_HOME need attentions. R.framework in different architectures do a good job of downloading the right binary from CRAN; so, if you have already taken care of the library paths, you embed the right R.framework to the right JASP.app, either x86_64 or arm64, and you can install R packages and you are guaranteed to get the right binary.

However, the problem arises when you have to compile something! Since R.framework does not come with its own compiler toolchain, you need to make sure that during the cross compilation, R’s internal are aware of your compiler flags.

Notes on `Makeconf` and `fortran`

As mentioned, CRAN is doing a great job of providing binary packages, and the only thing you need to do is to patch and resign your downloaded libraries, and make sure that R can find them. However, sometimes, you need to build a package, in our case our JASP Modules. This means that R needs to have access to compilers, and development toolchains.

On macOS, you can get your C/C++ compiler by downloading and installing Xcode, and after that, R can use system variables and find the compiler with no problem. However, Xcode doesn’t ship with gfortran and you often cannot use the Fortran that comes with Homebrew as you may ran into ABI incompatibilities, even if your build passes. In addition, R.framework is very particular about the gfortran. So, what you want to do is to download the right Fortran, install it somewhere, and make sure that R can find it. This is a problematic step even without making R portable, or cross-compilation attempt. Even worst, on Apple Silicon, the Fortran is not yet stable, so you need to go with an experimental version.

Since we are cross compiling, we need to have a Fortran compiler that is capable of cross compilation; however, considering the way R uses the Fortran compiler, at this moment, I would not recommend using the Fortran compiler for cross compilation. Instead, we need to download two versions of Fortran, ie., x86_64 and arm64, and link the right compiler to the right R.framework.

Now, the problem is how to inform the R.framework about the location of Fortran.

Enters `etc/Makeconf`

As mentioned, most of the compiler configurations, including the host system and R.framework’s build flags are set in etc/Makeconf. In order to guide the R.framework to the right gfortran, or any other tools, we need to modify the etc/Makeconf file, and that’s what we are doing at JASP. During the configuration step, we set the target architecture, ie., x86_64 and arm64, download the right Fortran compiler, unpack and patch it; and finally, modify the etc/Makeconf such that the FC variable points to the right gfortran executable.

To be precise, we download the gfortran installer, unpack it inside the R.framework, e.g., ${R_HOME}/opt/local/gfortran or ${R_HOME}/opt/R/arm64/gfortran and make sure that any references to the Fortran compiler (in etc/Makeconf) points to the right path, e.g.,

sed -i.bak -e "s/\\/opt\\/R\\/arm64/$(R_HOME)\\/opt\\/R\\/arm64/g" Makeconf

This guarantees that any R packages that needs building is being build for the target architecture, even if it needs to compile Fortran code.

Summary

In this post, I have tried to briefly summarize some of the most important steps needed to successfully patch and embed the R.framework into a macOS App Bundle. I mainly focused on the process of preparing the R.framework, making sure that it can work from a non-default location, patching its executable and libraries, and modifying the CRAN binary packages such that they can live inside the altered framework. In addition, I touched upon the topic of cross compilation, and what needs to be done when it comes to R source packages.

I hope that in the next post, I can dig deeper into the role of CMake, and explain how we utilize CMake tools and capabilities to glue different aspects of the build together. In the meantime, if you are struggling with R.framework please feel free to reach out on Twitter, or write to me. I’d be happy if I could be of any help!

Embedding R.framework is such a cumbersome task that as far as I can tell, even the RStudio team gave up on it, and simply ask you to download the R.framework if you don’t have it. ↩
You can refresh your knowledge by reading Shared Libraries: Understanding Dynamic Loading. ↩
If you are not familiar with these concepts, you first need to read and understand a few things about how libraries are linked together, and how they can find each other if they are not placed in the same directory, see here and here. ↩

San Francisco Themes

2020-12-26T00:00:00+00:00

There are many, many, many different color palettes available for {ggplot2}, and the list is growing daily. In short, {sfthemes} is technically an another set of colors and palettes that is inspired by iOS/macOS/watchOS system colors. These palettes are vibrant and are designed to be easily distinguishable in differnt screen sizes and brightness settings. Moreover, they offer optimized variants for light and dark mode.

From the project website:

sfthemes is a pair of ggplot2 themes, theme_sf_light(), theme_sf_dark(), and a collection of color scales based on Apple’s Human Interface Guidelines. Themes are inspired by the occasional appearances of charts used by Apple within different contexts, e.g., Apple Health app, Screen Time on iOS and macOS. And colors are adapted from iOS, macOS, and watchOS system colors. Each color scale has a light and dark variant as well that compliments the light/dark themes, scale_colour_ios_light() and scale_colour_ios_dark(), respectively. In addition, sfthemes uses dynamic type sizing in order to provide a consistent and elegant overall resizing and re-scaling of plot elements. Read more here.

Project website: {sfthemes} - An accessible ‘pkgdown’ template
GitHub Repo: amirmasoudabdol/sfthemes

Preferably

2020-12-20T00:00:00+00:00

During this lockdown period with long hours of screentime every day, my eyes started to become very sensitive to white background. Luckily, these days most websites and operation systems offer Dark Mode by default. However, I realized that the documentation websites of most R packages do not support dark theme, and since I’m working with R packages every day, I had to do something about it. So, I decided to develop a template for {pkgdown} that can take advantage of some of the modern capabalities of modern browsers.

From the project website:

preferably is an accessible template for pkgdown documentation websites. It uses a light and a dark theme and utilizes the prefers-color-scheme CSS variable to automatically serve either of the two based on users’ operating system setting, or allowing them to manually toggle between them.

Besides offering light and dark mode, I spent some time making the overall reading experience of ‘pkgdown’ documentations just a bit nicer, by using richer fonts, adopting a better color scheme for codes, etc.

Project website: {preferably} - An accessible ‘pkgdown’ template
GitHub Repo: amirmasoudabdol/preferably

Spatialtemporal Clustering of Gene Expression Level

2018-05-10T00:00:00+00:00

I like data, colors, composition, design, and story-telling. Alternatively, I can say that I like Data Visualization. To me, data visualization is the process of designing — or extracting — a story from the data. As a good design is selective, focused — and minimal —, a good visualization is clever, simple, and discerning. It often condenses a colossal dataset into bite-size visual facts; or it tells a gentle and informative story of a common statistic. Of course there are many other types and categories but I feel this definition and examples are quite descriptive.

During my Ph.D. project, I have tried to look at my dataset differently, clarify my results and produce nice charts and visualizations. I am quite proud of a few of them, namely, the spatiotemporal clusters visualization. It clearly shows the distinction between the spatial and temporal clusters, and it makes it relatively easy to track the spatial or temporal correlation of genes. In the figure below, there are 4 spatial clusters, i.e., nodes color, and 9 temporal clusters, i.e., edges color.

You probably need some knowledge of biology and gene regulatory network ¹, but the main idea here is that if genes are connected to each others and are having the same colors, then are correlated both in time, and space, i.e., appearing at the same time and place in the organism.

Amir M Abdol et al. “A novel technique to combine and analyse spatial and temporal expression datasets: A case study with the sea anemone Nematostella vectensis to identify potential gene interactions”. English. In: (June 2017). DOI: 10.1016/j.ydbio. 2017.06.004. ↩

Ph.D. Thesis

2018-05-08T00:00:00+00:00

Abstract

Over the last decade, the sea anemone Nematostella vectensis has become a popular model to study bilaterian evolution, development and more recently also regeneration. Understanding genetic interactions during the early development of N. vectensis is the first step toward unveiling the details of its early developmental processes, e.g., polarization, the formation of the blastula and initiation of the gastrulation process. Furthermore, the collective knowledge of gene interactions allows researchers to speculate about possible Gene Regulatory Networks (GRNs) governing each process. The knowledge of gene interactions also provides an opportunity to reverse engineer gene interactions in a GRN model which potentially leads to detailed understanding of the early developmental processes, e.g., pattern formation and the mechanics of gastrulation. There is still a limited amount of knowledge available regarding N. vectensis gene interactions and possible GRNs involved in each developmental process. This thesis introduces a method for extracting spatial gene expression profiles from in situ hybridization images of N. vectensis embryo. My collaborators and I have introduced a systematic procedure to combine and process the available data from different sources (e.g., in situ and qPCR) in order to understand gene interactions and reconstruct testable hypotheses for GRNs controlling development.

A. M. Abdol. Toward Reverse Engineering Spatiotemporal Gene Regulatory Networks of Nematostella Vectensis. Supervised by P. M. A. Sloot and J. A. Kaandorp. ISBN: 9789402810110. UvA-Dare

NimbleBox

2012-06-14T00:00:00+00:00

In early ages of the Evernote, I was trying to store and organize everything on the Evernote. I remember being inspired by the Getting Things Done technique and I set up a very sophisticated GTD system. I’ve defined several notebooks and tags for everything, Inbox, Later, Today, etc. The Evernote and my GTD system was working reasonably in my favor, I have collected a few thousand notes and managed to organize and finish some projects while keeping my sanity!

One of the problem with Evernote client at that time was adding stuff to it was not very convenient, especially when you have large number of notes then the client would have been sluggish and rather buggy to work with. The Mac version was faster and more well-behaved. It was also equipped with this small version of Evernote called Evernote Mini. The Evernote Mini was a small app living in the menubar and it can be summoned by a HotKey to quickly capture a note, screenshot, or an audio recording. This was perfect for capturing quite notes and just dump them into the Inbox for later processing.

I love the idea of Evernote Mini, and I’ve tried to convince Evernote team to implement something similar for the Windows client. Unfortunatly that was not the priority at the time and it is still not in the Windows version. So, I’ve decided to implement something similar for Windows.

I’ve started with the main idea, a simple and minimal app that can be used to quickly capture notes and send them to the Evernote. Back then I was doing some C# programming and has never touched a real web API, but I knew about UI Design and I had the core experience in mind. Also, I was inspired by the iA Writer and I wanted to reproduce the same simplicity and experience. The video below shows my attempts to recreate Evernote Mini for the Windows. I called it NimbleBox.

In term of features, I’ve added a few neat tricks:

Markdown
Support for Right-to-Left languages like Farsi, Arabic, and Hebrew.
Inline shortcut for adding #Tags, sending a note to a @Notebook or adding a ^DueDate to a note.
Twitter support
Etc.

I was and I am still very happy with the outcome. Unfortunately, at around one year after I released the project, Aug 2013, one of the API that I was using stopped working and I didn’t have time to reimplement it myself. So the project effectively dead after that.

Amir Masoud Abdol’s Personal Website

Apple Notes vs. Hyperlinks

macOS

Method 1

Method 2

Method 3

iOS and iPadOS

Method 1

Method 2

Method 3

From iOS to macOS, and vice versa

Possible Solution

From qmake to CMake: A Nontrivial Case Study

About JASP, and Its Architecture

R Framework, R Packages and JASP Modules

R Framework

R Packages, and JASP Modules

Qt 5, qmake, and Dependencies

Qt6, CMake, and Conan

CMake Setup

Dependencies

R Framework

On macOS

On Windows

On Linux

Conan

Misc./Problematic Libraries

Managing the CMake Project

Building R-Interface on Windows, Rtools42, and Other Considerations for Windows

Build

Installation and Deployment

Caveats

Summary

Embedding R.framework in a Qt Mac App, and Cross Compiling for Two Architectures

R.framework

Anatomy of R.framework

Embedding R.framework

Patching Library Path and install_name_tool

Dealing with Other Libraries in library/, modules/ and opt/

CRAN Packages

macOS Security and Signing

CMake, Qt, R.framework, and Cross Compilation

Cross Compilation of R Source Packages

Notes on Makeconf and fortran

Enters etc/Makeconf

Summary

San Francisco Themes

Preferably

Spatialtemporal Clustering of Gene Expression Level

Ph.D. Thesis

Abstract

NimbleBox

Patching Library Path and `install_name_tool`

Dealing with Other Libraries in `library/`, `modules/` and `opt/`

Notes on `Makeconf` and `fortran`

Enters `etc/Makeconf`