github.com/bytedance/sonic Go module: is it safe?

Extensive use of unsafe.Pointer and pointer arithmetic (e.g., rt.UnpackEface, (*rt.GoString)(unsafe.Pointer(&self.s)), native.Value calls with raw pointers) bypasses Go's memory safety guarantees. While typical for high-performance JSON libraries, this increases the risk of memory corruption or exploitable bugs if the native code has flaws.

Calls to internal/native functions (native.Value, native.SkipOne, native.SkipOneFast, native.GetByPath) execute platform-specific assembly or C code that is not visible in this file. Such native code could contain hidden malicious behavior or vulnerabilities that cannot be audited from the provided source alone.

Extensive use of unsafe package with //go:nocheckptr directives and direct pointer arithmetic to bypass bounds checking. While typical for performance-critical JSON parsing libraries, this significantly increases risk of memory corruption if inputs are not perfectly validated. The skipBlank, decodeInt64, decodeFloat64, skipString, and skipPair functions manipulate pointers directly without bounds verification.

In skipString, when encountering a backslash escape character, the code does 'sp += 2' without checking if sp+1 is still within bounds. This could read past the end of the string buffer if a string ends with an unescaped backslash near the buffer boundary.

In skipPair, when encountering a backslash, the code does 'sp += 2' without verifying sp+1 remains within the buffer bounds, potentially reading beyond the allocated memory.

The code uses unsafe.Pointer to store a pointer to a local string variable in newError and newSyntaxError. The string variable 'msg' is allocated on the stack and its address is taken. After the function returns, the pointer stored in the Node struct points to a stack location that may be reused or invalidated, leading to a dangling pointer. When Error() dereferences this pointer later, it can cause memory corruption, crashes, or potentially leak sensitive information from other stack frames. This is a memory safety issue that could be exploited.

Similar unsafe.Pointer issue in newSyntaxError: the local variable 'msg' is stored via unsafe.Pointer in the Node returned. The pointer becomes invalid after function return, causing undefined behavior when Error() is later called.

In unwrapError, when err is not *Node or Node, a local variable 'msg' is created and its address stored via unsafe.Pointer in a new Node. This again creates a dangling pointer after function return, leading to potential memory safety issues.

The mem2ptr function uses the unsafe package to convert a byte slice to an unsafe.Pointer by accessing internal runtime slice representation. This bypasses Go's type safety and could lead to memory corruption, out-of-bounds access, or undefined behavior if used improperly. The //go:nosplit directive prevents stack growth checks, further increasing risk. While this appears to be a legitimate utility for a high-performance JSON library (Sonic), unsafe pointer manipulation is a common vector for exploitation if the library is compromised or if callers misuse the function.

This file is part of a JIT (Just-In-Time) code assembler that generates and executes native x86-64 machine code at runtime. It uses the golang-asm library to emit raw assembly instructions, construct executable machine code, and jump to dynamically generated code addresses. While this is a legitimate performance optimization for the sonic JSON decoder, JIT compilation is a dual-use capability that could be abused to execute arbitrary code if the instruction stream is attacker-controlled. The _Program instruction stream originates from internal decoder logic, not external input, but the mechanism itself is a high-risk pattern.

The assembler emits raw machine code bytes directly (e.g., self.Byte(0xcc) for INT3 debug breakpoints, self.Byte(0x4c, 0x8d, 0x0d) for LEAQ instructions) and uses unsafe.Pointer extensively. The _asm_OP_debug function emits an INT3 (0xcc) breakpoint instruction. While these are standard for a JIT assembler, direct byte emission and unsafe pointer arithmetic bypass Go's memory safety guarantees and could lead to memory corruption if the generated code is incorrect.

The code uses unsafe.Pointer extensively with type-unsafe memory reinterpretation, including crafting slices from raw pointers in vs() and reconstructing Go structures from raw memory. While this is typical for JIT/compiler-style JSON decoders like sonic, it is inherently dangerous and could enable memory corruption if fed untrusted input that manipulates the compiled instruction stream.

The embeddedFieldPtrDecoder.FromDom function performs raw pointer arithmetic using unsafe.Pointer and uintptr to walk struct fields. This is inherent to the library's design (it is part of sonic, a high-performance JSON library by bytedance that uses unsafe for speed), but it carries memory-safety risk if field metadata is inconsistent. No malicious intent detected.

The decoder heavily uses unsafe.Pointer, PtrOffset, rt.Mapassign, and related low-level runtime/reflection primitives to build Go maps from JSON AST nodes. While this is a known performance pattern in libraries like sonic (ByteDance), it bypasses Go's type safety guarantees. Incorrect handling of key memory layouts (e.g., for pointer-key map types) could lead to memory corruption or crashes if inputs are adversarial. The code also depends on runtime internals (rt.GoMapType, rt.Mapassign) that can vary across Go versions.

Several decoder methods use unsafe casts like *(*unsafe.Pointer)(vp) = nil and *(*uint32)(unsafe.Pointer(&key)) with assumptions about the underlying type layout (e.g., mapI32KeyDecoder). If the mapType metadata does not match the actual destination type (e.g., through reflection misuse by callers), this could result in memory corruption rather than a safe type error.

The code makes extensive use of the unsafe package, performing pointer arithmetic, manual memory allocation via rt.Mallocgc, rt.Memmove, and type punning (rt.UnpackType, rt.Str2Mem, rt.Mem2Str). While this is typical for high-performance JSON parsers and not inherently malicious, it creates memory-safety risks including potential out-of-bounds reads/writes that could be exploited if the native parser (native.ParseWithPadding) is fed crafted input.

In Parser.parse(), a new node buffer is allocated with size derived from the remaining JSON length (calMaxNodeCap). If jsonSize is extremely large, this can lead to excessive memory allocation, a potential denial-of-service vector when parsing untrusted JSON input.

Extensive use of unsafe.Pointer, uintptr, ptrCast, and //go:nocheckptr for direct memory manipulation without bounds or safety checks. While typical for high-performance JSON parsers, this pattern can lead to memory corruption, out-of-bounds reads/writes, and potential exploitation if input parsing logic has flaws.

Custom boundedStack implementation with direct index arithmetic and unsafe.Pointer storage. Missing bounds checks on index (e.g., Pop decrements index without verifying > 0, Push increments without verifying < len). Could cause stack underflow/overflow leading to memory corruption.

efacePool allocates and reuses memory via unsafe operations. GetMap/GetSlice use unsafe.Pointer conversions; ConvTSlice/ConvF64 etc. write directly to interface slots. If pool sizing (from untrusted JSON stats) is wrong, buffer overflows may occur.

AsEfaceFast writes to map/slice interface slots using rt.Mapassign_faststr and direct *vt = ...; *vp = ... assignments. Combined with manual pointer arithmetic (rt.PtrAdd), this bypasses Go's type safety and could corrupt memory if node structure assumptions are violated.

The file heavily relies on the unsafe package and raw pointer arithmetic (e.g., unsafe.Pointer(uintptr(elems) + uintptr(i)*d.elemType.Size)) to manually compute element addresses in slices and arrays. While this is typical for high-performance decoders, it bypasses Go's memory safety guarantees. If the size/type calculations are ever incorrect or inputs are malformed, this could lead to out-of-bounds memory access, memory corruption, or potential exploitation. The //go:nocheckptr directive in arrayDecoder.FromDom further disables runtime pointer checks, increasing the risk surface. No direct malicious intent is evident, but the pattern warrants caution and careful auditing.

The Stack type implements manual stack pointer arithmetic using unsafe.Pointer and uintptr (Top/Cur/Push/Pop). If sp is corrupted or the slab size assumptions are violated, reads/writes could occur out of bounds of the sb array, leading to memory corruption.

ResetStack uses rt.MemclrNoHeapPointers over an assumed StackSize. If StackSize is incorrect, this could clear beyond the object and corrupt adjacent heap memory.

Pop sets *st = State{} but Push stores raw unsafe.Pointer values (p, q) into the stack. FreeStack only resets sp and returns the Stack to the pool; it does not clear the sb array, so stale unsafe.Pointer values may be reused, creating potential use-after-free semantics if those pointers are read later.

The file uses //go:linkname to access unexported runtime functions (runtime.checkptrBase, runtime.findObject). This is a fragile and dangerous pattern that bypasses Go's type safety and can break between Go versions. While used here for debugging pointer checks, it violates encapsulation and could be misused for memory manipulation.

The code uses unsafe.Pointer and uintptr conversions to inspect pointers (checkptr, findobj). This is inherently unsafe and can lead to memory corruption or information disclosure if pointer values are printed or manipulated incorrectly.

The file is part of a JIT (Just-In-Time) assembler that dynamically generates and loads executable machine code into memory. While this is a legitimate part of the Sonic JSON library for performance, runtime code generation can be abused if the assembler input is not strictly controlled. The presence of a custom loader (loader.LoadOne) that maps generated machine code into executable memory is a high-risk pattern if inputs can be influenced externally.

Functions like Byte, From, Emit, and resolve directly manipulate memory and patch machine code at runtime. This low-level memory access can be used to bypass security mechanisms or execute arbitrary instructions if the assembler is fed untrusted input.

This file is part of a JIT backend that assembles x86 machine code at runtime using golang-asm. It constructs executable code directly into memory (sym.P) and returns it for execution. While this is the intended purpose of the library (sonic JSON JIT), dynamic code generation inherently carries risk if input is attacker-controlled. The code itself does not appear malicious, but any vulnerability in the assembler or caller could lead to arbitrary code execution.

File is part of a JIT runtime (github.com/bytedance/sonic/internal/jit) that emits machine code at runtime using golang-asm. While not malicious on its own, dynamic code generation can be abused to execute obfuscated payloads and is a common vector for advanced threats.

The code uses unsafe.Pointer and direct uintptr conversions to obtain raw addresses of runtime types (GoType), itabs, and function values. This bypasses Go's type safety and could be leveraged to corrupt memory or construct arbitrary jump targets if the surrounding JIT engine is misused or fed untrusted input.

The function F_f64toa is declared as a package-level function variable and S_f64toa as a package-level uintptr. These are presumably assigned at init time or via linking to native assembly code. The use of unsafe.Pointer with rt.NoEscape to call an indirect function pointer is a common pattern in high-performance libraries but also could be used to hide malicious native code execution. This file alone does not contain the assignment, making the ultimate behavior opaque from this file.

The variable F_i64toa is declared and expected to be assigned at runtime by native assembly code. If an attacker can control the assignment of this function pointer (e.g., via a malicious binary or improper initialization), it could lead to arbitrary code execution. However, in the context of the ByteDance sonic library, this is a standard pattern for SIMD-optimized routines.

The code uses unsafe.Pointer for low-level memory access. While this is common in performance-optimized libraries and appears legitimate for SIMD/AVX2 operations, it bypasses Go's memory safety guarantees and could lead to memory corruption if misused.

Use of unsafe.Pointer and rt.NoEscape to pass a *string to an external function pointer without memory layout guarantees. This bypasses Go's type safety and could lead to memory corruption if the underlying assembly/native implementation mishandles the pointer.

The file heavily uses //go:linkname to bind to assembly functions and unsafe.Pointer for memory access. While this is a legitimate optimization pattern in the Sonic JSON library, it bypasses Go's type safety and could lead to memory corruption if the linked functions are not correctly implemented. This is not inherently malicious but represents a high-risk coding pattern.

The code uses the unsafe package and passes raw pointers to a function pointer loaded from a native assembly implementation, which could potentially allow memory corruption if the native code is not properly implemented or validated.

This file binds Go function stubs to C functions contained in source files like sse/f64toa.c, sse/parse_with_padding.c, etc. The loader package dynamically links and executes native code at runtime. While this is a standard pattern for high-performance SIMD libraries like sonic, it introduces a trust boundary: the bundled C code could perform arbitrary operations (file system access, network calls, process spawning) that would not be visible in this Go file. The security of the package depends entirely on the integrity of the referenced .c files and the loader implementation.

The Use() function registers numerous native C functions with the Go runtime via loader.WrapGoC. If Use() is invoked during package initialization (or from an init() in the same package), native code paths become active before any application-level validation. This matches the red-flag category 'code that runs at install, build or import time'.

The use of the unsafe package to pass string and int pointers to a native function (F_skip_one_fast) bypasses Go's type safety. Although this is a common pattern in high-performance libraries (e.g., bytedance/sonic), it could theoretically be exploited if the native function is malicious or if pointers are misused to cause memory corruption. The function rt.NoEscape explicitly prevents escape analysis, which is typical for avoiding allocations but can obscure pointer lifetimes.

The call to F_skip_one_fast, a function pointer likely set during initialization, executes native code. While not inherently malicious, this pattern can be used to execute arbitrary native code if the function pointer is overwritten by an attacker or if the library is compromised. No evidence of such tampering in this file.

The file is marked 'Code generated by scripts, DO NOT EDIT' and only declares a symbol pointer to an externally supplied implementation (S_validate_utf8_fast). The actual executable behavior comes from an unresolved native symbol, meaning the auditable surface here is incomplete and the true implementation cannot be reviewed in this file.

This file uses unsafe.Pointer and runtime linkage (rt.NoEscape, function pointer F_validate_utf8_fast populated from a native/assembly implementation) to call platform-specific UTF-8 validation code. While this is typical for performance libraries like Sonic, the package replaces a safe Go implementation with a native call visible only through symbol pointers, which complicates static auditing and can be used to hide behavior outside the Go source tree.

The variable F_vstring is declared as a function pointer but not assigned in this file. It is likely populated at runtime via dynamic linking or assembly linkage, which introduces a risk of loading untrusted native code if the linkage mechanism can be influenced by external inputs or environment. This pattern is common in performance-critical libraries but should be audited to ensure the symbol resolution is secure.

The code uses the unsafe package to pass Go pointers directly into a native function (F_vstring). Although the //go:nosplit directive and rt.NoEscape wrapper are intended to prevent heap escape analysis, this pattern bypasses Go's memory safety guarantees. If the native function retains any of these pointers or misuses the JsonState structure, it could lead to memory corruption, information disclosure, or arbitrary code execution.

The code imports github.com/cloudwego/base64x, a third-party package, and directly uses its encoding functions and internal symbols. If this dependency is compromised, it could lead to arbitrary code execution or data exfiltration during base64 encoding/decoding operations. The use of //go:linkname increases the attack surface by allowing access to unexported functions that may not be intended for external use.

The file uses //go:linkname to reference unexported symbols (_subr__b64decode and _subr__b64encode) from the github.com/cloudwego/base64x package. This bypasses Go's type safety and encapsulation, creating a fragile and potentially exploitable dependency on internal implementation details. If the base64x package changes its internal symbols, this could cause undefined behavior or crashes. While not inherently malicious, this technique is often used to access hidden functionality.

The code uses unsafe.Pointer and direct memory manipulation (GoEface, GoSlice, Mallocgc) to convert values into interface{} representations. This bypasses Go's type safety and could lead to memory corruption, crashes, or arbitrary code execution if an attacker can influence the types or values passed to these functions. Functions like Conv, ConvNum, and ConvT64 write directly to memory addresses without bounds checking.

The code references internal types like GoEface, GoSlice, GoType, Mallocgc, BytesType, StringType, JsonNumberType, Uint64Type, and BoolType that are not part of the public Go standard library. These are copied from runtime internals and are version-specific. Using such internals can cause undefined behavior, crashes, or security issues when the Go runtime changes, and may be exploited if the expected memory layout differs.

The file uses unsafe.Pointer extensively to reinterpret Go reflect.Type and interface internals, including direct memory layout assumptions (GoType, GoIface, GoEface, GoItab). It also uses go:linkname to access the runtime internal function runtime.getitab. This bypasses Go's type safety and relies on unstable runtime internals, which is fragile and can lead to memory corruption or arbitrary memory access if types are mismatched. While this is a known pattern in high-performance JSON libraries like Sonic (ByteDance), it presents a significant security risk if untrusted input influences the types being manipulated.

The code uses //go:linkname to directly access unexported runtime functions and variables (runtime.gcWriteBarrier2 and runtime.writeBarrier). This bypasses Go's type safety and encapsulation, and relies on internal runtime implementation details that may change across Go versions. It also uses the unsafe package for pointer arithmetic and memory access. While this may be legitimate for low-level runtime manipulation, it is a red flag because such techniques are commonly used in malicious code to hide behavior or manipulate runtime state.

The GcwbAddr function disassembles machine code at runtime using x86asm to locate the writeBarrier variable by scanning for a specific instruction pattern (CMP with memory operand and zero immediate). This involves reading and interpreting raw memory within the Go runtime, which is highly unusual for normal application code and could be used to locate and modify critical runtime structures.

The file uses the //go:linkname compiler directive to access unexported runtime internals (runtime.gcWriteBarrier, runtime.writeBarrier) from the standard library. This is a fragile and undocumented mechanism that bypasses Go's type and package safety guarantees. While this specific usage appears to be a legitimate low-level GC write barrier implementation (consistent with ByteDance/Sonic high-performance JSON library patterns), linkname access to runtime symbols can be abused by malicious packages to modify or read internal runtime state, hook memory management, or escape sandboxed execution contexts.

The code uses the unsafe package and //go:linkname to access the unexported runtime.growslice function. This bypasses Go's type safety and internal runtime encapsulation. While this is a legitimate performance optimization technique used by packages like Sonic (ByteDance), it directly manipulates memory layout and relies on runtime internals that could change between Go versions, potentially leading to memory corruption or crashes.

The //go:linkname growslice runtime.growslice directive depends on the exact signature and behavior of an internal runtime function. If the Go runtime implementation changes (e.g., growslice signature or GoSlice/GoType layout changes), this could cause undefined behavior, memory corruption, or arbitrary memory writes. This is a legitimate but risky pattern.

The file uses //go:linkname to bind GrowSlice directly to runtime.growslice, an undocumented internal runtime symbol. This bypasses Go's type safety and version compatibility guarantees. While this pattern is common in performance-oriented libraries (this appears to be from ByteDance's sonic JSON library), linkname to runtime internals can break across Go versions, can be used to circumvent memory safety, and is difficult to audit for malicious behavior. In this file, no exfiltration, credential harvesting, network calls, subprocess execution, init() side effects, or obfuscated payloads are present, so the immediate risk is low; however, the same mechanism could be abused in a malicious variant.

The code uses unsafe.Pointer and manual pointer arithmetic without bounds checking. This can lead to memory corruption, arbitrary memory read/write, or crashes if inputs are not validated. The GetSlice method does not verify that the requested size is positive or that index+size stays within pool bounds, which could result in out-of-bounds access. While not inherently malicious, such patterns are dangerous and often associated with exploitation.

The calculation uintptr(self.index)*AsGoType(self.typ).Size could overflow on 32-bit platforms or with large values, leading to pointer arithmetic underflow and out-of-bounds memory access.

The file uses //go:linkname directives extensively to access unexported runtime and reflect functions (runtime.memmove, runtime.mapiternext, runtime.mallocgc, reflect.makemap, runtime.throw, etc.). This bypasses Go's type safety and encapsulation guarantees, tightly coupling the package to specific Go runtime internals. This is characteristic of high-performance libraries (this appears to be ByteDance's sonic JSON library), but it is a fragile and potentially dangerous pattern: it can break with any Go version change, and similar techniques have been abused by malicious packages to manipulate memory, bypass security checks, or invoke privileged runtime behavior. It also complicates security auditing because the actual behavior depends on undocumented runtime internals.

The code performs direct memory operations via unsafe.Pointer (add, Memmove, MemclrHasPointers, MemclrNoHeapPointers, MakeSlice, GrowSlice) and type-puns slices via unsafe.Pointer(&emptyBytes). Such low-level manipulations can lead to memory corruption, use-after-free, or out-of-bounds access if misused. While consistent with the intended purpose of a runtime-reflection helper package, these primitives are the same ones that malicious code would leverage to tamper with process memory, bypass sandboxing, or execute arbitrary payloads. No actual exfiltration, credential harvesting, network activity, process spawning, or dynamic code execution is present, so the risk is limited to fragility and potential memory-safety hazards rather than direct malice.

The IntoBytes function checks capacity but not the length of the destination slice. It uses cap(*m) < len(s) as a guard, but then writes into the slice via unsafe pointer, bypassing slice length checks. This could lead to writing beyond the slice's logical length if the caller expects the length to be respected, potentially corrupting adjacent memory.

The code uses Go's unsafe package with pointer arithmetic to reinterpret slices and strings (rt.GoSlice, rt.GoString). While this is a known performance optimization technique in the sonic library, it bypasses Go's memory safety guarantees. If the native.Unquote function has a bug or is exploited, it could lead to memory corruption or arbitrary code execution.