Skip to content
niso

icu_segmenter: the `new_for_non_complex_scripts` segmenters print a data error for every complex run

Generated from repros/icu-segmenter-non-complex-warns/README.md when the site was built at 7aa23e2.

What niso does

Section titled “What niso does”

The workspace turns on blitz-dom’s complex-scripts feature, which turns on parley’s. Parley then uses new_dictionary word and line segmenters. CJK gets dictionary word breaks, and Thai, Khmer, Lao and Myanmar get dictionary line breaks, the same as in Chrome.

Without the feature, every debug build of niso printed ICU4X data error: No segmentation model for complex script: Chinese/Japanese to stderr for every text run in those scripts, and the same for Thai and the others. blitz-dom’s text-transform: capitalize still uses its own new_for_non_complex_scripts word segmenter, so capitalizing CJK text still prints the message.

Expected vs actual

Section titled “Expected vs actual”

WordSegmenter::new_for_non_complex_scripts and LineSegmenter::new_for_non_complex_scripts are documented as having no support for complex scripts, and they treat each complex run as one segment. Choosing them is how a caller says it doesn’t want the models. Even so, ComplexPayloadsBorrowed::select in src/complex/mod.rs logs a DataError for every run it has no model for. Without the logging feature, icu_provider turns log::warn! into eprintln! in debug builds, so the error lands on stderr.

Segmenting 日本語のテキスト by word and ภาษาไทย by line in a debug build:

expected stderr: (empty)
actual stderr:
ICU4X data error: No segmentation model for complex script: Chinese/Japanese
ICU4X data error: No segmentation model for complex script: Thai

Release builds print nothing.

Run

Section titled “Run”
Terminal window
CARGO_TARGET_DIR=~/.capy/work/repro-target cargo run

Exit status 1 while the bug reproduces.

Upstream main

Section titled “Upstream main”

unicode-org/icu4x main has the same code:

  • select still calls with_display_context when a model is missing.
  • provider/core/src/lib.rs still maps the log macros to eprintln! in debug builds without logging.

The repro wasn’t built against main, because cloning it ran the test machine out of disk.

Existing issue or PR

Section titled “Existing issue or PR”

No ICU4X issue was found about the non-complex constructors logging.

Issue text

Section titled “Issue text”

Title: new_for_non_complex_scripts segmenters print “No segmentation model” for every complex run in debug builds

Body:

WordSegmenter::new_for_non_complex_scripts and LineSegmenter::new_for_non_complex_scripts have no complex-script models by design. Even so, ComplexPayloadsBorrowed::select logs a DataError every time it meets a run it has no model for. Without the logging feature, icu_provider prints that with eprintln! in debug builds. So every CJK word segmentation, and every Thai, Khmer, Lao or Myanmar line segmentation, prints a line to stderr. linebender/parley#623 and slint-ui/slint#11638 are downstream reports of this.

// Cargo.toml: icu_segmenter = "2.3"
use icu_segmenter::options::{LineBreakOptions, WordBreakInvariantOptions};
use icu_segmenter::{LineSegmenter, WordSegmenter};


fn main() {
    let words = WordSegmenter::new_for_non_complex_scripts(WordBreakInvariantOptions::default());
    let lines = LineSegmenter::new_for_non_complex_scripts(LineBreakOptions::default());
    let _: Vec<usize> = words.segment_str("日本語のテキスト").collect();
    let _: Vec<usize> = lines.segment_str("ภาษาไทย").collect();
    // stderr (debug build):
    // ICU4X data error: No segmentation model for complex script: Chinese/Japanese
    // ICU4X data error: No segmentation model for complex script: Thai
}

Tested with icu_segmenter 2.3.0 and icu_provider 2.3.1. select and the debug eprintln! mapping are unchanged on main. Suggested fix: when the segmenter was built without models, don’t log in select (for example, make the error silent there), because the caller chose that fallback.