Building AOSP on ARM64: From an 8 GB Jetson to DGX Spark

At Actinis, we build AOSP constantly. That makes build-machine choices more than an infrastructure detail: a slow or unreliable build host becomes a daily tax on engineering work.

When NVIDIA announced the RTX Spark class of machines, the hardware looked interesting for our workload. The public RTX Spark material describes slim laptops and small desktops with up to a 20-core Grace CPU, a Blackwell RTX GPU, up to 128 GB of unified memory, and up to 1 PFLOP of FP4 AI performance. NVIDIA’s announcement also frames RTX Spark as a Windows PC platform, which matters for the caveats below. ^{1 2}

The question we cared about was narrower and more practical:

Can a modern ARM64 workstation-class host build AOSP natively, or is the AOSP host-side build still too x86-centric?

AOSP’s own setup requirements still describe the development workstation as a 64-bit x86 system, with at least 400 GB of free disk and a minimum of 64 GB of RAM. The same page requires a 64-bit Linux distribution with glibc 2.17 or later and states that modern Android OS development on macOS is not supported. ³

That is the official baseline. Our experiment deliberately stepped outside it.

We first brought up a native ARM64 AOSP build on a Jetson Orin Nano Super Developer Kit 8GB. That was the worst-case environment: 6 ARM cores, 8 GB of RAM, and a large NVMe-backed swap file. Once the hard problems were solved there, we applied the resulting patch set to a clean tree on DGX Spark, a much more realistic ARM64 workstation-class proxy. DGX Spark is documented as a Grace Blackwell system with a 20-core Arm processor and 128 GB of unified LPDDR5x memory. ⁴

The result: AOSP 17 built natively on ARM64. The Android Emulator also built natively on ARM64. On the Jetson, the natively built emulator booted the natively built AOSP image under KVM to the Android launcher, with adb reporting sys.boot_completed=1.

This does not prove that future RTX Spark laptops will be turnkey AOSP build machines. It does reduce the main architectural risk: AOSP 17 can be pushed through on a Linux/aarch64 host. The remaining risks are the actual Linux support story on shipping RTX Spark laptops, KVM availability, thermals, storage, and whether the local workaround patch set can be cleaned up or upstreamed.

Executive summary

We built the target:

sdk_phone64_arm64-trunk_staging-userdebug

on an ARM64 host, using the Android 17 / SDK 37 trunk-staging tree we had at the time of the experiment.

The experiment had three stages:

The Jetson bring-up took about three days of wall-clock time, including debugging and repeated failed attempts. We did not rerun a fresh clean benchmark on the 8 GB board just to measure it; based on the observed pace, a clean run would likely be on the order of 24–36 hours. Treat that as an estimate, not a benchmark.

The DGX Spark result is the more meaningful performance signal: after the Jetson had exposed the host-architecture problems, the same target built from a clean tree on DGX Spark in 2h13m. That is close enough to the i9-14900K desktop’s 1h45m to make ARM64 workstation-class hardware interesting for our AOSP workload.

The key caveats:

ALLOW_MISSING_DEPENDENCIES=true
SKIP_ABI_CHECKS=true
THINLTO_USE_MLGO=false

We also used a clang-real shim to strip x86-only flags from generated Cronet build rules, and we made LFI verification non-fatal for one development image. This is not an upstream-supported production configuration. It is a technical bring-up proving that the host-side AOSP build graph can be made to run natively on ARM64.

Why start on an 8 GB Jetson?

DGX Spark was the machine we wanted as a proxy for future RTX Spark-class hardware. But while waiting for access, we started on a much harsher target: Jetson Orin Nano Super Developer Kit 8GB.

That choice made the bring-up slower, but it was useful. If the build failed there, it failed loudly. Memory leaks, missing host prebuilts, x86-only host tools, JVM heap defaults, path assumptions, and emulator issues all surfaced quickly. The Jetson was not a fair performance target; it was a fault amplifier.

The host environment was:

The AOSP checkout was a blobless partial clone, about 132 GB before the build, with 1084 repo projects. The target product was the ARM64 emulator product:

lunch sdk_phone64_arm64-trunk_staging-userdebug
m sdk_phone64_arm64-trunk_staging-userdebug

The important decision was to build natively on ARM64, not to emulate an x86 host.

We considered two paths:

Path A won. AOSP already contains enough ARM64-host concepts to make the path plausible: Soong has ARM64 Linux host configs, linux_musl_arm64 exists as a host variant, and microfactory.bash already selects prebuilts/go/linux-arm64 on aarch64. The problem is that the open manifest does not provide all the host prebuilts and not all build logic is wired for this path.

The first wall: AOSP knows about ARM64 hosts, but the manifest does not ship the complete host toolchain

The starting inventory looked like this:

One subtle point: the x86 Clang prebuilt project still mattered even though we were not going to execute its x86 binaries. Its Android.bp files define host LLVM modules that the rest of the tree references during Soong analysis. In our partial clone, that worktree had not materialized correctly after an interrupted sync, so Soong failed before compilation. Restoring the x86 Clang project fixed the module-definition side of the problem.

The actual ARM64 compiler came from the upstream platform/prebuilts/clang/host/linux-arm64 project, which exists on android.googlesource.com but was not in our manifest. The tree wanted clang-r584948; the ARM64 prebuilt branch provided clang-r584948b, so the local tree used a symlink:

prebuilts/clang/host/linux-arm64/clang-r584948 -> clang-r584948b

Go had a similar trap. A stock upstream Go linux-arm64 tarball was not enough. AOSP’s microfactory bootstrap invokes the Go compiler in a way that needs precompiled standard-library archives under $GOROOT/pkg/linux_arm64/*.a. The upstream Go tarball did not ship those. The version-matched AOSP Go prebuilt did.

At this point the strategy was clear: provide the missing ARM64 host prebuilts, then patch the places where Kati, Soong, and host-tool module definitions still assumed x86.

Patching the build system into recognizing the ARM64 host

The first source patches taught the build system that aarch64 is a valid host and that host artifacts should live under out/host/linux-arm64, not a mixture of linux-x86, linux_musl-arm64, and legacy x86 naming.

The first two patches are on the Make/Kati side. envsetup.sh had to return linux-arm64 for the host prebuilt tag, and envsetup.mk had to set HOST_ARCH=arm64 and HOST_PREBUILT_ARCH=arm64.

The next patches enabled ARM64 variants of prebuilt build tools and JDK tools, made Soong install host outputs under the same tag as Make, and ensured sandboxed host tools received libc_musl.so when the build host was forced to the LinuxMusl OS type.

Subject: [arm64-host 0001] envsetup.sh: get_host_prebuilt_prefix -> linux-arm64 on aarch64

Fixes `one-true-awk: cannot execute binary file: Exec format error` when sourcing
build/envsetup.sh on an aarch64 host (it was using the x86 build-tools prefix). The root
`build/envsetup.sh` is a symlink, so the patch targets the real file in `build/make`.

--- a/build/make/envsetup.sh
+++ b/build/make/envsetup.sh
@@ -137,8 +137,13 @@
 function get_host_prebuilt_prefix
 {
   local un=$(uname)
+  local um=$(uname -m)
   if [[ $un == "Linux" ]] ; then
-    echo linux-x86
+    if [[ $um == "aarch64" ]] ; then
+      echo linux-arm64
+    else
+      echo linux-x86
+    fi
   elif [[ $un == "Darwin" ]] ; then
     echo darwin-x86
   else

Subject: [arm64-host 0002] build/make: aarch64 host arch, prebuilt tag, and clang runtimes

Fixes `build/make/core/envsetup.mk:195: error: unknown variable: HOST_ARCH` on an aarch64 host
(no arm64 branch existed, so HOST_ARCH stayed unset and .KATI_READONLY failed). Also makes
HOST_PREBUILT_ARCH=arm64 on arm64 so Kati's HOST_OUT / HOST_PREBUILT_TAG resolve to linux-arm64,
matching soong's PrebuiltOS().

Also adds the arm64 host clang runtime makefile that Kati later includes from
`build/make/core/clang/config.mk`.

--- a/build/make/core/envsetup.mk
+++ b/build/make/core/envsetup.mk
@@ -177,10 +177,16 @@ ifneq (,$(findstring x86_64,$(UNAME)))
   HOST_2ND_ARCH := x86
   HOST_IS_64_BIT := true
 else
+ifneq (,$(findstring aarch64,$(UNAME)))
+  HOST_ARCH := arm64
+  HOST_2ND_ARCH :=
+  HOST_IS_64_BIT := true
+else
 ifneq (,$(findstring i686,$(UNAME))$(findstring x86,$(UNAME)))
 $(error Building on a 32-bit x86 host is not supported: $(UNAME)!)
 endif
 endif
+endif
 
 ifeq ($(HOST_OS),darwin)
   # Mac no longer supports 32-bit executables
@@ -245,7 +251,11 @@ endif
 endif
 
 # We don't want to move all the prebuilt host tools to a $(HOST_OS)-x86_64 dir.
+ifeq ($(HOST_ARCH),arm64)
+HOST_PREBUILT_ARCH := arm64
+else
 HOST_PREBUILT_ARCH := x86
+endif
 # This is the standard way to name a directory containing prebuilt host
 # objects. E.g., prebuilt/$(HOST_PREBUILT_TAG)/cc
 # This must match the logic in get_host_prebuilt_prefix in envsetup.sh

--- /dev/null
+++ b/build/make/core/clang/HOST_arm64.mk
@@ -0,0 +1,2 @@
+HOST_LIBPROFILE_RT := $(LLVM_RTLIB_PATH)/libclang_rt.profile-aarch64.a
+HOST_LIBCRT_BUILTINS := $(LLVM_RTLIB_PATH)/libclang_rt.builtins-aarch64.a

Subject: [arm64-host 0003] soong/rust: wire arm64 host into prebuilt rust-sysroot-std

Makes `prebuilt_libstd` (rust_stdlib_prebuilt_host) enable for the linux_musl_arm64 host.
Upstream toolchain_library.go only enumerates x86/darwin host platforms, so on an arm64 build
prebuilt_libstd was always disabled (everything else hit the else->disable branch). Adds a
Linux_musl_arm64 target field + an arm64 branch keyed on BuildOS==LinuxMusl && BuildArch==Arm64,
and guards the existing musl-x86 branch with BuildArch==X86_64 so it stops matching the arm64 build.

Requires a sibling `prebuilts/rust-toolchain/linux-arm64/Android.bp` so `platform == "linux-arm64"`,
plus the arm64 musl Rust toolchain under `prebuilts/rust-toolchain/linux-arm64/<ver>/`.

--- a/build/soong/rust/toolchain_library.go
+++ b/build/soong/rust/toolchain_library.go
@@ -153,6 +153,7 @@ type hostPrebuiltProps struct {
 		Linux_glibc_x86    hostPrebuiltTargetProps
 		Linux_musl_x86_64  hostPrebuiltTargetProps
 		Linux_musl_x86     hostPrebuiltTargetProps
+		Linux_musl_arm64   hostPrebuiltTargetProps
 		Darwin_x86_64      hostPrebuiltTargetProps
 	}
 }
@@ -176,9 +177,11 @@ func constructLibProps(rlib, solib bool) func(ctx android.LoadHookContext) {
 		if platform == "linux-x86" && ctx.Config().BuildOS == android.Linux {
 			p.Target.Linux_glibc_x86_64.addPrebuiltToTarget(ctx, name, rustDir, "linux-x86", "x86_64-unknown-linux-gnu", rlib, solib)
 			p.Target.Linux_glibc_x86.addPrebuiltToTarget(ctx, name, rustDir, "linux-x86", "i686-unknown-linux-gnu", rlib, solib)
-		} else if platform == "linux-musl-x86" && ctx.Config().BuildOS == android.LinuxMusl {
+		} else if platform == "linux-musl-x86" && ctx.Config().BuildOS == android.LinuxMusl && ctx.Config().BuildArch == android.X86_64 {
 			p.Target.Linux_musl_x86_64.addPrebuiltToTarget(ctx, name, rustDir, "linux-musl-x86", "x86_64-unknown-linux-musl", rlib, solib)
 			p.Target.Linux_musl_x86.addPrebuiltToTarget(ctx, name, rustDir, "linux-musl-x86", "i686-unknown-linux-musl", rlib, solib)
+		} else if platform == "linux-arm64" && ctx.Config().BuildOS == android.LinuxMusl && ctx.Config().BuildArch == android.Arm64 {
+			p.Target.Linux_musl_arm64.addPrebuiltToTarget(ctx, name, rustDir, "linux-arm64", "aarch64-unknown-linux-musl", rlib, solib)
 		} else if platform == "darwin" && ctx.Config().BuildOS == android.Darwin {
 			p.Target.Darwin_x86_64.addPrebuiltToTarget(ctx, name, rustDir, "darwin-x86", "x86_64-apple-darwin", rlib, solib)
 		} else {

--- /dev/null
+++ b/prebuilts/rust-toolchain/linux-arm64/Android.bp
@@ -0,0 +1,5 @@
+rust_stdlib_prebuilt_host {
+    name: "libstd",
+    crate_name: "std",
+    sysroot: true,
+}

Subject: [arm64-host 0004] prebuilts/build-tools: enable bison/flex/m4/make for arm64 host

These prebuilt_build_tool modules were gated `arch: { x86_64: { enabled: true } }` with
`target.linux.src` hardcoded to linux-x86. This adds arm64 enablement and a linux_musl_arm64
target for bison/flex/m4/make.

On Orin the `linux-arm64/bin/{bison,flex,m4}` files were also replaced with glibc host binaries,
with the original musl binaries kept as `.musl`. That binary replacement is not represented in
this source patch.

--- a/prebuilts/build-tools/Android.bp
+++ b/prebuilts/build-tools/Android.bp
@@ -84,12 +84,19 @@ prebuilt_build_tool {
         x86_64: {
             enabled: true,
         },
+        arm64: {
+            enabled: true,
+        },
     },
     target: {
         darwin: {
             src: "darwin-x86/bin/bison",
             deps: ["darwin-x86/lib64/libc++.dylib"],
         },
+        linux_musl_arm64: {
+            src: "linux-arm64/bin/bison",
+            deps: ["linux-arm64/lib64/libc_musl.so"],
+        },
         linux: {
             src: "linux-x86/bin/bison",
             deps: ["linux-x86/lib64/libc++.so"],
@@ -105,12 +112,19 @@ prebuilt_build_tool {
         x86_64: {
             enabled: true,
         },
+        arm64: {
+            enabled: true,
+        },
     },
     licenses: ["prebuilts_build-tools_flex_license"],
     target: {
         darwin: {
             src: "darwin-x86/bin/flex",
         },
+        linux_musl_arm64: {
+            src: "linux-arm64/bin/flex",
+            deps: ["linux-arm64/lib64/libc_musl.so"],
+        },
         linux: {
             src: "linux-x86/bin/flex",
         },
@@ -125,12 +139,19 @@ prebuilt_build_tool {
         x86_64: {
             enabled: true,
         },
+        arm64: {
+            enabled: true,
+        },
     },
     licenses: ["prebuilts_build-tools_gnu_license"],
     target: {
         darwin: {
             src: "darwin-x86/bin/m4",
         },
+        linux_musl_arm64: {
+            src: "linux-arm64/bin/m4",
+            deps: ["linux-arm64/lib64/libc_musl.so"],
+        },
         linux: {
             src: "linux-x86/bin/m4",
         },
@@ -144,11 +165,18 @@ prebuilt_build_tool {
         x86_64: {
             enabled: true,
         },
+        arm64: {
+            enabled: true,
+        },
     },
     target: {
         darwin: {
             src: "darwin-x86/bin/make",
         },
+        linux_musl_arm64: {
+            src: "linux-arm64/bin/make",
+            deps: ["linux-arm64/lib64/libc_musl.so"],
+        },
         linux: {
             src: "linux-x86/bin/make",
         },

Subject: [arm64-host 0005] prebuilts/jdk/jdk21: enable javap for arm64 host

The `javap` prebuilt_build_tool was x86-only. This adds arm64 enablement and a linux_musl_arm64
target pointing at the real arm64 JDK 21 drop under `prebuilts/jdk/jdk21/linux-arm64`.

The JDK directory itself is a prebuilt drop on Orin, not a source patch. Final Orin state has
`prebuilts/jdk/jdk21/linux-arm64` as a real directory, not the earlier abandoned jdk25 symlink.

--- a/prebuilts/jdk/jdk21/Android.bp
+++ b/prebuilts/jdk/jdk21/Android.bp
@@ -19,8 +19,35 @@ prebuilt_build_tool {
         x86_64: {
             enabled: true,
         },
+        arm64: {
+            enabled: true,
+        },
     },
     target: {
+        linux_musl_arm64: {
+            src: "linux-arm64/bin/javap",
+            deps: [
+                "linux-arm64/lib/libjli.so",
+                "linux-arm64/lib/jrt-fs.jar",
+                "linux-arm64/lib/jvm.cfg",
+                "linux-arm64/lib/server/libjvm.so",
+                "linux-arm64/lib/libverify.so",
+                "linux-arm64/lib/libjava.so",
+                "linux-arm64/lib/libzip.so",
+                "linux-arm64/lib/libjimage.so",
+                "linux-arm64/lib/modules",
+                "linux-arm64/lib/libnio.so",
+                "linux-arm64/lib/libnet.so",
+                "linux-arm64/lib/tzdb.dat",
+                "linux-arm64/lib/libawt.so",
+                "linux-arm64/lib/libawt_headless.so",
+                "linux-arm64/lib/libjavajpeg.so",
+                "linux-arm64/lib/liblcms.so",
+                "linux-arm64/lib/libmanagement.so",
+                "linux-arm64/lib/libmanagement_ext.so",
+                "linux-arm64/conf/security/java.security",
+            ],
+        },
         linux: {
             src: "linux-x86/bin/javap",
             deps: [

Subject: [arm64-host 0006] soong/paths: host-out dir naming for the arm64 build host

Two related fixes in pathForInstall() so Make and Soong agree on out/host/<tag> on an arm64 host
(everything should be "linux-arm64", matching the envsetup.mk #8 patch / HOST_PREBUILT_ARCH:=arm64):

(1) Collapse the LinuxMusl OsType name to "linux" for the build's own host (not just under the
    UseHostMusl opt-in, which is false on arm64 where BuildOS is forced to LinuxMusl). Fixes the kati
    SOONG_ZIP make-vars check (Make linux-arm64 vs Soong linux_musl-arm64).
(2) For Common/X86_64 host modules (arch-neutral java tools etc.) the archName was hardcoded "x86"
    (legacy, tied to the old HOST_PREBUILT_ARCH:=x86). On an arm64 build host use "arm64" so e.g.
    signapk.jar installs to out/host/linux-arm64/framework/ where consumers (apex etc.) look for it.

--- a/build/soong/android/paths.go
+++ b/build/soong/android/paths.go
@@ -2031,9 +2031,10 @@
 			// instead of linux_glibc
 			osName = "linux"
 		}
-		if os == LinuxMusl && ctx.Config().UseHostMusl() {
+		if os == LinuxMusl && (ctx.Config().UseHostMusl() || os == ctx.Config().BuildOS) {
 			// When using musl instead of glibc, use "linux" instead of "linux_musl".  When cross
-			// compiling we will still use "linux_musl".
+			// compiling we will still use "linux_musl". (arm64-host: BuildOS is forced to LinuxMusl
+			// without the UseHostMusl opt-in, so also collapse for the build host itself.)
 			osName = "linux"
 		}
 
@@ -2044,6 +2045,11 @@
 		archName := arch.String()
 		if os.Class == Host && (arch == X86_64 || arch == Common) {
 			archName = "x86"
+			if ctx.Config().BuildArch == Arm64 {
+				// arm64 build host: HOST_PREBUILT_ARCH is arm64 (envsetup.mk patch), so
+				// arch-neutral / x86_64 host modules install under linux-arm64, not linux-x86.
+				archName = "arm64"
+			}
 		}
 		partitionPaths = []string{"host", osName + "-" + archName, partition}
 	}

--- a/build/soong/android/rule_builder.go
+++ b/build/soong/android/rule_builder.go
@@ -1412,7 +1412,7 @@
 //
 //	cmd.Tool(ctx.Config().HostToolPath(ctx, tool))
 func (c *RuleBuilderCommand) BuiltTool(tool string) *RuleBuilderCommand {
-	if c.rule.ctx.Config().UseHostMusl() {
+	if c.rule.ctx.Config().UseHostMusl() || c.rule.ctx.Config().BuildOS == LinuxMusl {
 		// If the host is using musl, assume that the tool was built against musl libc and include
 		// libc_musl.so in the sandbox.
 		// TODO(ccross): if we supported adding new dependencies during GenerateAndroidBuildActions

After these patches and the prebuilt drops, the build reached real compilation. That is where the next class of bugs appeared: not “ARM64 host unsupported” in the abstract, but individual generated rules or host tools carrying old x86 assumptions.

Cronet as a canary for x86 assumptions

The first large compile wall was Cronet. Its generated Blueprint files carried x86 SIMD flags such as:

-msse3
-mfpmath=sse
-msse2
-msse4.*
-mavx*
-maes
-mpclmul
-mf16c

Those flags leaked into host variants compiled for aarch64-linux-musl, and Clang correctly rejected them.

We did not want to regenerate all of Cronet’s build metadata just for this bring-up. For the sdk_phone64_arm64 target, nothing in the build graph legitimately needs x86 SIMD host flags: host and target compilation are ARM/ARM64. So we replaced the ARM64 Clang prebuilt’s clang-real symlink with a bash shim that strips x86-only -m* flags and then execs the real clang-22.

That shim needed two follow-up fixes:

1. It had to preserve the full-path argv[0] with exec -a "$0". A bare clang++-real made Clang derive the wrong resource directory, which broke libc++ includes.
2. It had to be pure bash. Some Rust link actions run under a hermetic environment with no PATH; using external tools such as dirname or basename caused the shim itself to fail.

Later, Cronet also invoked hardcoded x86 llvm-objdump and llvm-nm paths. A direct binary copy of the ARM64 tools was not enough because those binaries had musl loader dependencies relative to their original tree. The final fix was wrapper scripts in the x86 Clang bin directory that execute the real ARM64 tools in place.

These were not upstream-quality fixes. They were targeted bring-up shims that avoided a long re-analysis cycle and carried the build forward.

Rust: the real wall

The hardest problem was Rust, specifically the host-side Rust graph used by ART/dex2oat.

The failing edge was a generated Rust static library linked into dex2oat. The errors looked like this class of Rust sysroot inconsistency:

E0464: multiple candidates for rlib dependency core
E0460: found possibly newer version of crate core/std
E0152: duplicate lang item panic_impl in std

The underlying problem was “Rust sysroot double vision”.

AOSP’s intended host Rust model mixes a prebuilt libstd with from-source sysroot crates such as core, alloc, and compiler_builtins. The upstream ARM64 Rust toolchain we dropped in, however, shipped a complete prebuilt sysroot under:

prebuilts/rust-toolchain/linux-arm64/1.93.1/lib/rustlib/aarch64-unknown-linux-musl/lib

So the build could see both:

from-source core/std/sysroot rlibs
raw prebuilt core/std/proc_macro from the upstream Rust sysroot

The obvious first idea was to remove the raw prebuilt sysroot -L path. That failed. Normal host rlib-std crates relied on that path to find prebuilt core. Removing it caused E0463 failures.

The second idea was to force all host Rust onto from-source libstd and corelibs. That also failed. proc_macro is prebuilt-only and tied to the raw prebuilt std/core Strict Version Hash, so pushing the graph toward from-source std/core broke proc-macro crates.

The final conclusion was the opposite of the first instinct:

For this host toolchain, keep host Rust all-prebuilt. Do not mix from-source sysroot rlibs into generated host staticlibs.

Patch 0009 implements that pivot. In TransformRlibstoStaticlib, it drops from-source .rust_sysroot and libstd/linux_musl_arm64 link dirs and rlibs, then pins the raw prebuilt sysroot dir. Patches 0008 and 0010 remain in our source pack only as historical reverted records.

Subject: [arm64-host 0008] REVERTED: rust/toolchain_library: omit raw prebuilt sysroot -L

Final Orin state: REVERTED. Do not apply this as part of the final arm64-host patch set.

This was an intermediate #22 attempt to omit the raw prebuilt rust sysroot directory from
`Link_dirs` for `aarch64-unknown-linux-musl`. It was wrong for two reasons:

- host rlib-std crates such as `anstyle` rely on the raw prebuilt `-L` to find prebuilt core;
- the raw prebuilt directory still leaked into generated staticlibs through per-rlib paths anyway.

The final #22 fix is the all-prebuilt staticlib pinning in patch 0009. Orin's final
`build/soong/rust/toolchain_library.go` keeps the normal `Link_dirs` behavior, with only the
arm64-host prebuilt target wiring from patch 0003.

Subject: [arm64-host 0009] rust/builder: pin generated staticlibs to the prebuilt Rust sysroot

Final #22 fix. Keep host Rust all-prebuilt because proc_macro is prebuilt-only and SVH-tied to
the raw prebuilt std/core. For generated staticlibs, drop from-source sysroot link dirs and rlibs,
then pin the raw prebuilt sysroot directory so std/core resolve from one coherent toolchain.

--- a/build/soong/rust/builder.go
+++ b/build/soong/rust/builder.go
@@ -17,6 +17,7 @@ package rust
 import (
 	"path/filepath"
 	"runtime"
+	"slices"
 	"strings"
 
 	"github.com/google/blueprint"
@@ -206,6 +207,27 @@ func TransformRlibstoStaticlib(ctx android.ModuleContext, mainSrc android.Path,
 
 	mod := ctx.Module().(cc.LinkableInterface)
 	toolchain := config.FindToolchain(ctx.Os(), ctx.Arch())
+
+	if ctx.Host() && ctx.Arch().ArchType.String() == "arm64" {
+		// all-prebuilt (#22): proc_macro is prebuilt-only and rigidly tied to the raw prebuilt
+		// std/core SVH, so the entire host rust graph (rlibs + proc-macros) uses the raw prebuilt
+		// sysroot. This synthetic staticlib otherwise pulls a FROM-SOURCE libstd + *.rust_sysroot
+		// (different SVH) that conflicts with its prebuilt rlib inputs (E0464 dup / E0460 SVH).
+		// Drop the from-source sysroot link-dirs + the from-source std/sysroot rlibs, then pin the
+		// raw prebuilt sysroot dir so std/core resolve from the one consistent prebuilt toolchain.
+		rustPathDeps.linkDirs = slices.DeleteFunc(rustPathDeps.linkDirs, func(d string) bool {
+			return strings.Contains(d, ".rust_sysroot/") || strings.Contains(d, "/libstd/linux_musl_arm64")
+		})
+		rustPathDeps.RLibs = slices.DeleteFunc(rustPathDeps.RLibs, func(l RustLibrary) bool {
+			p := l.Path.String()
+			return strings.Contains(p, ".rust_sysroot") || strings.Contains(p, "/libstd/linux_musl_arm64")
+		})
+		rawDir := filepath.Join("prebuilts/rust-toolchain", ctx.Config().PrebuiltOS(),
+			config.RustDefaultVersion, "lib/rustlib", toolchain.RustTriple(), "lib")
+		if !slices.Contains(rustPathDeps.linkDirs, rawDir) {
+			rustPathDeps.linkDirs = append(rustPathDeps.linkDirs, rawDir)
+		}
+	}
 	t := transformProperties{
 		// Crate name can be a predefined value as this is a staticlib and
 		// it does not need to be unique. The crate name is used for name

Subject: [arm64-host 0010] REVERTED: rust/compiler: force host rust to from-source libstd/corelibs

Final Orin state: REVERTED. Do not apply this as part of the final arm64-host patch set.

This was an intermediate #22 attempt to force all arm64 host Rust modules onto from-source
`libstd` plus `config.Corelibs`. It solved one generated-staticlib SVH mismatch but regressed
proc-macro crates: `proc_macro` is prebuilt-only and tied to the raw prebuilt std/core SVH.

The final #22 direction is the opposite: keep the host Rust graph all-prebuilt and make
`TransformRlibstoStaticlib` drop from-source sysroot inputs before pinning the raw prebuilt
sysroot directory. See patch 0009.

After this change, the build passed the Rust wall: proc macros, hundreds of crates, and the dex2oat generated_rust_staticlib/librustlibs.a all built successfully.

This was the most important technical result of the platform build. Most other failures were path assumptions or host tool portability bugs. Rust required choosing one coherent sysroot strategy and making the generated staticlib path obey it.

The 8 GB machine made JVM defaults visible

Some failures were not architecture, but memory-class problems.

On a normal 64 GB or 128 GB build host, default JVM heap sizing often gets away with being vague. On an 8 GB Jetson with heavy swap, it did not.

The first JVM wall was SystemUI KSP:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

The Soong KSP rule invoked kotlin-ksp-client without forwarding the same heap flags used by Kotlin compilation. Patch 0011 added ${config.KotlincHeapFlags} to that invocation.

Later, metalava failed in the same way while generating test API stubs. The shared java_binary_host wrapper did not set a default -Xmx, so the JVM used its default heap sizing. Patch 0014 added a default -Xmx8g while still allowing tool-specific -J-Xmx... flags to override it.

The KSP patch was necessary on the 8 GB Jetson. It was not needed on DGX Spark, which has enough memory for this class of tool.

--- a/build/soong/java/kotlin.go
+++ b/build/soong/java/kotlin.go
@@ -476,6 +476,7 @@ var kspProcessingRule = pctx.AndroidRemoteStaticRule("ksp", android.RemoteRuleSu
 			inputDeltaCmd + ` && ` +
 			kotlinZipSyncCmd + ` && ` +
 			`${config.KotlinKspClientBinary} ` +
+			` ${config.KotlincHeapFlags} ` + // arm64-host (#25): avoid KSP JVM OOM on big modules
 			` -jvm-target=$kotlinJvmTarget ` +
 			` -project-base-dir=. ` +
 			` -module-name=$name ` +

Subject: [arm64-host 0012] sepolicy: use int for getopt results on aarch64

On aarch64, plain `char` is unsigned. Storing `getopt()` / `getopt_long()`'s `-1`
end-of-options sentinel in `char` turns it into `0xff`, so option parsing runs one
extra iteration and falls into the usage/error path. Use `int`, as getopt requires.

--- a/system/sepolicy/tools/checkfc.c
+++ b/system/sepolicy/tools/checkfc.c
@@ -444,7 +444,7 @@ int main(int argc, char **argv)
   bool allow_empty = false;
   bool compare = false;
   bool test_data = false;
-  char c;
+  int c;
 
   filemode mode = filemode_file_contexts;
 
--- a/system/sepolicy/tools/sepolicy-analyze/attribute.c
+++ b/system/sepolicy/tools/sepolicy-analyze/attribute.c
@@ -75,7 +75,7 @@ int attribute_func (int argc, char **argv, policydb_t *policydb) {
     int rc = -1;
     int list = 0;
     int reverse = 0;
-    char ch;
+    int ch;
 
     struct option attribute_options[] = {
         {"list", no_argument, NULL, 'l'},
--- a/system/sepolicy/tools/sepolicy-analyze/neverallow.c
+++ b/system/sepolicy/tools/sepolicy-analyze/neverallow.c
@@ -493,7 +493,7 @@ static int check_neverallows_string(policydb_t *policydb, char *string, size_t l
 
 int neverallow_func (int argc, char **argv, policydb_t *policydb) {
     char *rules = 0, *file = 0;
-    char ch;
+    int ch;
 
     struct option neverallow_options[] = {
         {"debug", no_argument, NULL, 'd'},
--- a/system/sepolicy/tools/sepolicy-check.c
+++ b/system/sepolicy/tools/sepolicy-check.c
@@ -236,7 +236,7 @@ int main(int argc, char **argv)
 	policydb_t policydb;
 	struct policy_file pf;
 	sidtab_t sidtab;
-	char ch;
+	int ch;
 	int match = 1;
 
 	struct option long_options[] = {

Subject: [arm64-host 0013] lfi-verify: make verification failures non-fatal for arm64-host build

The arm64-host build hit an LFI verification failure in the final tail. For this dev image, keep
the verifier diagnostics but do not fail the build on the experimental LFI hardening check.

--- a/external/lfi/lfi-verifier/tools/lfi-verify/main.c
+++ b/external/lfi/lfi-verifier/tools/lfi-verify/main.c
@@ -143,8 +143,8 @@ verify(struct LFIVerifier *v, const char *filename)
                 }
 
                 if (!lfiv_verify(v, segment, phdr.p_filesz, phdr.p_vaddr)) {
-                    fprintf(stderr, "verification failed\n");
-                    return false;
+                    // arm64-host (#27): LFI verify non-fatal (bionic asm not LFI-rewritten)
+                    fprintf(stderr, "warning: LFI verification failed (non-fatal on arm64-host build)\n");
                 }
                 segments++;

Subject: [arm64-host 0014] jar-wrapper: default host Java tools to -Xmx8g

Metalava OOMed on the 8 GB arm64 host because java_binary_host wrappers ran with the JVM default
heap. Add a default cap before forwarded `-J` options so tool-specific `-J-Xmx...` values still win.

--- a/build/soong/scripts/jar-wrapper.sh
+++ b/build/soong/scripts/jar-wrapper.sh
@@ -55,4 +55,6 @@ while expr "x$1" : 'x-J' >/dev/null; do
     shift
 done
 
-exec java "${javaOpts[@]}" -jar ${jardir}/${jarfile} "$@"
+# arm64-host (#28): default max-heap so heap-hungry tools (metalava) do not OOM on the 8 GB box;
+# a tool-supplied -J-Xmx is appended after and overrides this (JVM uses the last -Xmx).
+exec java -Xmx8g "${javaOpts[@]}" -jar ${jardir}/${jarfile} "$@"

The sepolicy patch in the same group is a different class of bug and a classic portability issue. Several host tools stored the result of getopt() in char. On aarch64, plain char is unsigned in the relevant toolchain configuration, so the -1 end-of-options sentinel became 0xff, the parser took one extra iteration, and the tool fell into usage/error handling. The correct type for getopt() results is int, which is what patch 0012 fixes.

The LFI patch is the least production-like change in the series. An LFI verification step rejected a binary containing hand-written AArch64 assembly using dc zva in a form the verifier did not accept. For this emulator development image, we kept the diagnostics but made the verifier failure non-fatal. That is a caveat, not a general recommendation.

Platform build success

The platform build eventually completed on the Jetson:

ninja: Build Succeeded: 9154 steps
M_BUILD_EXIT=0
0 FAILED

The image artifacts were produced under:

out/target/product/emu64a/

The produced image set was about 13 GB and included:

super.img
system.img
system-qemu.img
vendor.img
product.img
system_ext.img
userdata.img
vbmeta.img
ramdisk.img
vendor_boot.img
dtb.img

This proved that the platform-side graph could run natively on ARM64: Soong and Kati analysis, host tool builds, C/C++, Rust, ART/dex2oat, metalava, d8/r8, KSP, APEX/APK packaging, dexpreopt, signing, AVB, and image assembly.

But a built image is not the same as a booted image. To close the loop, we also needed an ARM64-host Android Emulator.

Building the Android Emulator natively on ARM64

AOSP does not ship an ARM64-host Android Emulator prebuilt in the open tree we used. Building the platform image was only half of the experiment; running it required building the emulator from the emulator source tree.

The emulator tree was emu-master-dev. The build required a separate set of fixes, which we grouped as E1–E8. They were mostly practical host-build issues: missing or mismatched CMake behavior, host library path assumptions, Qt/graphics dependencies, and packaging details. The important point is that the final emulator binary was native ARM64, not an x86 binary running under user-space emulation.

After the emulator built, KVM was the next gate. The Jetson had /dev/kvm, but the local user initially did not have access. Without KVM, the emulator fell back to TCG and exited because gic-version=host requires KVM. Granting the user KVM access fixed that.

Then we hit the most interesting emulator bug: QEMU crashed during VM reset.

The first stack pointed at:

object_get_class
virtio_current_cpu_endian
virtio_mmio_reset
qdev_reset_one
qbus_walk_children
qemu_devices_reset
qemu_system_reset

The direct bug was that virtio_current_cpu_endian() dereferenced current_cpu during system reset. On this native ARM64/KVM path, reset runs on the main loop thread and current_cpu can be null. Patch E9 added a guard and returned the target default endian value.

That was not enough. The crash recurred with current_cpu non-null but stale. The deeper issue was QEMU’s system-reset invariant: during system reset on the main loop thread, current_cpu should be NULL. On this native ARM64/KVM build, the thread-local current_cpu could carry stale non-null state into reset, making virtio take the wrong branch and dereference garbage.

Patch E10 saved current_cpu, forced it to NULL across qemu_system_reset(), then restored the saved value afterwards.

Subject: [emulator arm64-host E9] virtio: guard current_cpu during system reset

Recovered from the final Orin emulator tree. Apply from the Android Emulator
`external/qemu` git root.

Fixes an arm64/KVM reset crash where `virtio_current_cpu_endian()` ran on the
main loop thread during system reset and dereferenced an invalid `current_cpu`.

diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index 31e9b99b..388be22b 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -1185,6 +1185,11 @@ static enum virtio_device_endian virtio_default_endian(void)
 
 static enum virtio_device_endian virtio_current_cpu_endian(void)
 {
+    if (!current_cpu) {
+        /* arm64-host (B2): reset runs on the main thread with current_cpu==NULL for a
+           KVM aarch64 guest; CPU_GET_CLASS(NULL) would segfault. Use target default. */
+        return virtio_default_endian();
+    }
     CPUClass *cc = CPU_GET_CLASS(current_cpu);
 
     if (cc->virtio_is_big_endian(current_cpu)) {

Subject: [emulator arm64-host E10] vl: preserve current_cpu NULL invariant across system reset

Recovered from the final Orin emulator tree. Apply from the Android Emulator
`external/qemu` git root.

Fixes an arm64/KVM reset crash by forcing the main-loop reset path to run with
`current_cpu == NULL`, matching QEMU's system-reset invariant, then restoring
the saved TLS value after reset.

diff --git a/vl.c b/vl.c
index 18e6033e..36eed5e1 100755
--- a/vl.c
+++ b/vl.c
@@ -1937,6 +1937,13 @@ static int qemu_debug_requested(void)
 void qemu_system_reset(ShutdownCause reason)
 {
     MachineClass *mc;
+    /* arm64-host (B2): a system reset runs on the main loop thread, where QEMU's invariant is
+       current_cpu==NULL ("System reset", not a guest-initiated reset). On this native-arm64/KVM
+       build the main-thread TLS current_cpu can hold a stale non-NULL pointer, so virtio_reset()
+       takes the guest-reset branch and CPU_GET_CLASS(current_cpu) dereferences garbage -> SIGSEGV.
+       Force the correct invariant for the duration of the reset, then restore. */
+    CPUState *saved_current_cpu = current_cpu;
+    current_cpu = NULL;
 
     mc = current_machine ? MACHINE_GET_CLASS(current_machine) : NULL;
 
@@ -1952,6 +1959,8 @@ void qemu_system_reset(ShutdownCause reason)
                               &error_abort);
     }
     cpu_synchronize_all_post_reset();
+
+    current_cpu = saved_current_cpu;
 }
 
 void qemu_system_guest_panicked(GuestPanicInformation *info)

With these patches, the emulator booted the natively built AOSP image under ARM64 KVM.

The verification points were:

adb -s emulator-5554 shell getprop sys.boot_completed
# 1

adb -s emulator-5554 shell getprop ro.product.cpu.abi
# arm64-v8a

adb -s emulator-5554 shell getprop ro.build.version.sdk
# 37

Userspace was up: surfaceflinger, zygote, and system_server were running, the boot animation had stopped, and a screenshot confirmed that the launcher rendered.

For remote viewing, the emulator’s ADB ports were loopback-only on the Jetson, so we used SSH local forwarding from a workstation and then connected with the local adb / scrcpy stack:

ssh -f -N \
  -o ServerAliveInterval=15 \
  -o ServerAliveCountMax=3 \
  -o ExitOnForwardFailure=yes \
  -L 127.0.0.1:5555:127.0.0.1:5555 \
  -L 127.0.0.1:5554:127.0.0.1:5554 \
  <jetson-host>

adb connect localhost:5555
adb -s localhost:5555 shell getprop sys.boot_completed

That closed the loop: platform image built natively on ARM64, emulator built natively on ARM64, image booted on ARM64 host KVM.

DGX Spark validation

After the Jetson bring-up, we repeated the platform build on DGX Spark using a clean AOSP tree and the final patch set discovered on the Jetson.

The DGX Spark validation did not require the KSP heap workaround because memory was no longer the constraint. The clean platform build completed in:

2h13m

For comparison, the same target on an i9-14900K desktop with 128 GB of RAM completed in:

1h45m

We did not validate emulator boot on DGX Spark because KVM was unavailable on that particular machine. Emulator boot was validated on the Jetson.

The point of the DGX Spark run was not to rediscover the bugs. The point was to answer a practical question: once the ARM64-host patch set exists, does a workstation-class ARM64 system build AOSP in a time range that makes sense for daily work?

For us, the answer was yes.

What this says — and does not say — about RTX Spark laptops

The result is encouraging for RTX Spark-class machines, but it should be interpreted conservatively.

RTX Spark laptops are publicly positioned as Windows PCs. Our experiment validated a Linux/aarch64 host path. DGX Spark is a useful hardware proxy because it shares the ARM64 Grace Blackwell class and 128 GB unified-memory profile, but it is not proof of future laptop OS support, KVM availability, sustained thermals, or vendor-supported Linux configuration.

What the experiment does show is narrower and still useful:

1. AOSP 17 can be built natively on an ARM64 Linux host.
2. The missing pieces are mostly host prebuilt availability, build-system wiring, and long-tail x86 assumptions, not a fundamental impossibility.
3. A 128 GB ARM64 workstation-class system can build the target in a practical time range.
4. The Android Emulator can also be built and made to boot an ARM64 AOSP image on an ARM64 host, at least on the Jetson with KVM available.

That reduces the risk that RTX Spark-class hardware will be unusable for AOSP work. It does not eliminate integration risk.

Lessons from the bring-up

1. AOSP has partial ARM64-host support already

This was not a from-zero port. Soong knows about ARM64 Linux hosts. linux_musl_arm64 exists. Microfactory already tries to use prebuilts/go/linux-arm64 on aarch64.

The gap is that the open manifest does not ship the full set of ARM64 host prebuilts, and several surrounding build rules still assume x86.

2. Host prebuilts are not interchangeable

The Go failure is the cleanest example. A normal upstream Go tarball was the right architecture and version family, but it did not contain the precompiled standard-library archives AOSP’s bootstrap path expected. The AOSP prebuilt did.

The same pattern repeated with Rust and Clang. Matching the architecture is not enough; the prebuilt has to match the build system’s assumptions.

3. Rust sysroot consistency mattered more than “source vs prebuilt” ideology

The final Rust solution was not “build more from source.” It was “stop mixing sysroots.” With this toolchain, proc_macro forced the host graph toward the raw prebuilt sysroot. Once we accepted that and kept generated host staticlibs all-prebuilt, the Rust wall disappeared.

4. x86 assumptions hide in generated metadata

Cronet did not fail because AOSP globally lacks ARM64-host support. It failed because generated .bp metadata carried x86 SIMD flags and x86 tool paths into host variants that were no longer x86.

This is the kind of bug that only appears when the host architecture actually changes.

5. Low memory exposed missing JVM heap plumbing

The KSP and metalava failures were not ARM64 bugs. They were default-heap bugs. A big x86 build machine hides them. An 8 GB ARM64 board makes them visible.

6. “Built” is not “booted”

A platform image build success is important, but emulator boot exposed another layer of host-architecture assumptions in QEMU. Boot verification changed the result from “the build graph can finish” to “the produced image can actually start userspace.”

Final patch set

For the platform tree, the final applied source patches were:

0001
0002
0003
0004
0005
0006
0007
0009
0011
0012
0013
0014

The reverted historical records were:

0008
0010

The safe apply list is therefore explicit:

git apply -p1 \
  patches/0001-envsetup-host-prebuilt-prefix-arm64.patch \
  patches/0002-envsetup-mk-host-arch-arm64.patch \
  patches/0003-soong-rust-toolchain-library-arm64-host.patch \
  patches/0004-build-tools-arm64-host-tools.patch \
  patches/0005-jdk21-javap-arm64-host.patch \
  patches/0006-soong-paths-host-out-musl-arm64.patch \
  patches/0007-rule_builder-stage-libc_musl-in-sbox-on-forced-musl.patch \
  patches/0009-builder-pin-prebuilt-rust-sysroot-staticlib.patch \
  patches/0011-kotlin-ksp-client-heap-flags.patch \
  patches/0012-sepolicy-getopt-int-arm64.patch \
  patches/0013-lfi-verify-nonfatal-arm64-host.patch \
  patches/0014-jar-wrapper-default-xmx8g.patch

The emulator QEMU patches apply from the Android Emulator external/qemu root:

git apply -p1 \
  emulator-patches/E9-qemu-virtio-current-cpu-endian-reset-fix.patch \
  emulator-patches/E10-qemu-system-reset-current-cpu-invariant.patch

The final state also depended on non-patch changes:

• restored prebuilts/clang/host/linux-x86 for Soong module definitions;
• dropped in prebuilts/go/linux-arm64 from a version-matched AOSP Go prebuilt;
• dropped in prebuilts/clang/host/linux-arm64 and linked clang-r584948 -> clang-r584948b;
• added a real ARM64 JDK 21 under prebuilts/jdk/jdk21/linux-arm64;
• added an upstream ARM64 musl Rust toolchain under prebuilts/rust-toolchain/linux-arm64/1.93.1;
• used wrapper scripts for Cronet’s hardcoded x86 llvm-objdump / llvm-nm paths;
• used a clang-real shim to strip x86-only flags from ARM64 host compile actions.

Those are not presented here as a productized recipe. They are part of the bring-up record.

Conclusion

The experiment started with a simple procurement question: could upcoming ARM64 NVIDIA workstation/laptop hardware plausibly replace x86 build machines for our AOSP work?

The answer is not “yes, buy anything with RTX Spark on the box.” That would be too strong.

The answer is:

AOSP 17 can be made to build natively on Linux/aarch64. The Android Emulator can be built natively on ARM64 and can boot the resulting ARM64 emulator image under KVM. On DGX Spark, the patched clean build completed in 2h13m, close enough to our i9-14900K desktop’s 1h45m to make ARM64 workstation-class hardware a credible option for daily AOSP work.

For future RTX Spark laptops, the remaining questions are not primarily “can AOSP ever build on ARM64?” We have answered that. The remaining questions are platform integration questions: native Linux support, KVM, thermal behavior, storage, and how much of this patch set can be replaced by proper upstream support.

That is a much better risk profile than we had before the experiment.

References