From 5278fcab685723099e9950d56cb605807db5aa2a Mon Sep 17 00:00:00 2001 From: fufesou Date: Fri, 18 Sep 2026 12:43:07 +0800 Subject: [PATCH] fix(cursor): correct native cursor sizing and validate received images (#16213) * fix(flutter): shrink the unzoomed remote cursor by DPR on macOS and Linux With "Zoom cursor" off in Adaptive or Custom view, the remote cursor bitmap was registered at scale 1.0. NSCursor and GdkCursor treat the bitmap size as logical pixels, so on a HiDPI controller the cursor was drawn DPR times larger than in Original view (which already passes 1/DPR) and than on Windows (whose cursor path is in physical pixels). A HiDPI remote such as KDE Wayland sends a 48-64 px bitmap, which then showed up 3-4x too big on a Retina Mac. Scale the bitmap by 1/DPR in that case, and scale the Flutter-painted cursor used while the peer moves the mouse the same way so its size does not jump. The new branch is an identity at DPR 1 and the Windows paths are untouched. Fixes https://github.com/rustdesk/rustdesk/discussions/15363 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01K8HSHTEHo27mDcJXVyVfSP * fix(flutter): check the cursor height against the min cursor size `_checkUpdateScale` computed the scaled height from `width`, so the min-size clamp never looked at the height. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01K8HSHTEHo27mDcJXVyVfSP * fix(flutter): keep the painted cursor hotspot in place when zoom cursor is off `CursorPaint` subtracted the hotspot in remote pixels and then scaled it by the canvas scale, but drew the image at scale 1.0, so the hotspot landed hotx * (1 - scale) logical pixels away from the remote cursor position. Cursors with a centered hotspot (I-beam, crosshair) were off by up to half their size in Adaptive view. Subtract the hotspot after scaling the position instead. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01K8HSHTEHo27mDcJXVyVfSP * fix(flutter): read the live DPR and clamp the painted cursor like the native one CanvasModel caches devicePixelRatio and only refreshes it when the view style changes, so after the window moves to a monitor with a different DPR the unzoomed cursor kept the previous monitor's scale. Read it from MediaQuery instead, which also rebuilds the cursor when it changes. The native path clamps the scaled bitmap to kMinCursorSize; apply the same clamp to the painted cursor so a small cursor does not change size when the peer moves the mouse. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01K8HSHTEHo27mDcJXVyVfSP * build(cursor): declare existing Zstd dependency for bounded decoding * fix(cursor): validate and bound received cursor images * fix(cursor): preserve thin cursor sizes when scaling * fix(cursor): limit view changes to native cursor sizing * fix(cursor): apply long-edge minimum to Web cursor sizing * fix: preserve Linux cursor alpha and match Windows Custom scale * fix(cursor): keep scaled buffers and raster dimensions in sync * fix(cursor): preserve Windows peer alpha when resizing * fix(cursor): preserve mixed alpha during downsampling * comments Signed-off-by: fufesou * fix(cursor): match desktop cursor size to peer display density * refactor(cursor): clarify desktop and web scale branches * fix(cursor): correct adaptive display scaling and fixed cursor size * fix(cursor): apply all-display adaptive density on every desktop * fix(cursor): normalize unzoomed all-display cursor density * fix(cursor): synchronize original view on DPR changes * fix(cursor): match original zoom to the active renderer * fix: align Wayland software scrolling with input coordinates * fix: preserve scroll offsets when switching scrolling modes * fix: correct cursor size and pointer mapping with custom scale Apply the hovered Linux display density to Custom cursor zoom in All Displays. Refresh scroll fractions after layout so changing the Custom percentage uses current scrollbar extents and detached controllers. Validated with macOS and Windows component tests, formatting, and static analysis. * fix(linux): pad tall native cursors to prevent clipping * fix(cursor): preserve macOS point size in unzoomed views * fix(linux): pad rectangular cursors to square canvases * fix(cursor): divide dpr on Linux -> macOS Signed-off-by: fufesou * fix: cursor size test Signed-off-by: fufesou * fix(cursor): cursor size of controlled side macOS Signed-off-by: fufesou * fix(cursor): limit received cursor allocations * test(cursor): retain reverse raster transition coverage * fix(cursor): bound compressed cursor input * fix(cursor): restrict the scaled size of the cursor Signed-off-by: fufesou * fix(cursor): align hotspots with resized raster dimensions Calculate each hotspot axis from the actual raster-to-source ratio. Update existing boundary tests and run cursor tests in Flutter CI. Signed-off-by: fufesou * fix(cursor): align Sciter hotspots with raster dimensions Calculate native and overlay hotspots from the final raster size. Preserve input coordinate scaling and cursor refresh order. Signed-off-by: fufesou * fix(cursor): use `1.0` instead of `1.0/dpr` on Linux -> macOS, Zoom off, Scale adaptive The mouse cursor currently appears somewhat large. However, this is difficult to adjust because the cursor size is fixed while the window size varies; its relative size depends on the specific desktop environment. We can modify it if users actually provide feedback. Ideally, we should check the "Zoom cursor" option. Further adjustments may also be needed later based on cursor density. Signed-off-by: fufesou --------- Signed-off-by: fufesou Co-authored-by: rustdesk Co-authored-by: Claude Fable 5.1 --- .github/workflows/flutter-build.yml | 5 + Cargo.lock | 1 + Cargo.toml | 1 + flutter/lib/desktop/pages/remote_page.dart | 114 +++++-- flutter/lib/models/model.dart | 118 +++++-- flutter/lib/native/custom_cursor.dart | 29 +- flutter/test/cursor_size_test.dart | 368 +++++++++++++++++++++ src/client/io_loop.rs | 47 ++- src/flutter.rs | 2 +- src/ui/remote.rs | 2 +- src/ui/remote.tis | 36 +- 11 files changed, 676 insertions(+), 47 deletions(-) create mode 100644 flutter/test/cursor_size_test.dart diff --git a/.github/workflows/flutter-build.yml b/.github/workflows/flutter-build.yml index 83cacd1cb..779971f2d 100644 --- a/.github/workflows/flutter-build.yml +++ b/.github/workflows/flutter-build.yml @@ -871,6 +871,11 @@ jobs: name: bridge-artifact path: ./ + - name: Test cursor sizing + if: ${{ !inputs.upload-artifact && matrix.job.arch == 'aarch64' }} + working-directory: flutter + run: flutter test --timeout 60s test/cursor_size_test.dart + - name: Setup vcpkg with Github Actions binary cache uses: lukka/run-vcpkg@b1a0dd252f06b9e25b3c022a9a03bd7a427fb6a2 # v11 with: diff --git a/Cargo.lock b/Cargo.lock index 018e86e1d..cb6b0e270 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -7196,6 +7196,7 @@ dependencies = [ "wol-rs", "x11-clipboard 0.8.1", "x11rb 0.12.0", + "zstd", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 3bbbf782d..afd619322 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -62,6 +62,7 @@ cfg-if = "1.0" lazy_static = "1.4" sha2 = "0.10" repng = "0.2" +zstd = "0.13" parity-tokio-ipc = { git = "https://github.com/rustdesk-org/parity-tokio-ipc" } magnum-opus = { git = "https://github.com/rustdesk-org/magnum-opus" } dasp = { version = "0.11", features = ["signal", "interpolate-linear", "interpolate"], optional = true } diff --git a/flutter/lib/desktop/pages/remote_page.dart b/flutter/lib/desktop/pages/remote_page.dart index 3e98418b1..55277caa0 100644 --- a/flutter/lib/desktop/pages/remote_page.dart +++ b/flutter/lib/desktop/pages/remote_page.dart @@ -1005,8 +1005,8 @@ class _RemotePageState extends State bool get wantKeepAlive => true; } -/// A widget that tracks the view size and updates CanvasModel.updateViewStyle() -/// and InputModel.updateImageWidgetSize() only when size actually changes. +/// Tracks view size and DPR to update CanvasModel.updateViewStyle() +/// and InputModel.updateImageWidgetSize() only when either changes. /// This avoids scheduling post-frame callbacks on every LayoutBuilder rebuild. class _ViewStyleUpdater extends StatefulWidget { final CanvasModel canvasModel; @@ -1026,10 +1026,12 @@ class _ViewStyleUpdater extends StatefulWidget { class _ViewStyleUpdaterState extends State<_ViewStyleUpdater> { Size? _lastSize; + double? _lastDevicePixelRatio; bool _callbackScheduled = false; @override Widget build(BuildContext context) { + final devicePixelRatio = MediaQuery.devicePixelRatioOf(context); return LayoutBuilder( builder: (context, constraints) { final maxWidth = constraints.maxWidth; @@ -1039,8 +1041,9 @@ class _ViewStyleUpdaterState extends State<_ViewStyleUpdater> { return widget.child; } final newSize = Size(maxWidth, maxHeight); - if (_lastSize != newSize) { + if (_lastSize != newSize || _lastDevicePixelRatio != devicePixelRatio) { _lastSize = newSize; + _lastDevicePixelRatio = devicePixelRatio; // Schedule the update for after the current frame to avoid setState during build. // Use _callbackScheduled flag to prevent accumulating multiple callbacks // when size changes rapidly before any callback executes. @@ -1088,6 +1091,7 @@ class ImagePaint extends StatefulWidget { class _ImagePaintState extends State { bool _lastRemoteCursorMoved = false; + final _cursorDisplayScale = 1.0.obs; String get id => widget.id; RxBool get zoomCursor => widget.zoomCursor; @@ -1101,25 +1105,19 @@ class _ImagePaintState extends State { final m = Provider.of(context); var c = Provider.of(context); final s = c.scale; + // Read the live DPR while the canvas's post-frame update is pending. + final dpr = MediaQuery.devicePixelRatioOf(context); - bool isViewAdaptive() => c.viewStyle.style == kRemoteViewStyleAdaptive; bool isViewOriginal() => c.viewStyle.style == kRemoteViewStyleOriginal; mouseRegion({child}) => Obx(() { double getCursorScale() { var c = Provider.of(context); - var cursorScale = 1.0; - if (isWindows) { - // debug win10 - if (zoomCursor.value && isViewAdaptive()) { - cursorScale = s * c.devicePixelRatio; - } + if (isDesktop) { + return _getDesktopCursorScale(c, dpr); } else { - if (zoomCursor.value || isViewOriginal()) { - cursorScale = s; - } + return zoomCursor.value || isViewOriginal() ? s : 1.0; } - return cursorScale; } return MouseRegion( @@ -1193,11 +1191,70 @@ class _ImagePaintState extends State { } } + /// Matches desktop cursors to the rendered image and native pixel units. + /// + /// Windows cursor pixels are physical; NSCursor/GdkCursor use logical pixels. + /// Unzoomed Adaptive/Custom views preserve macOS cursors' point dimensions. + double _getDesktopCursorScale(CanvasModel c, double dpr) { + final peer = widget.ffi.ffiModel; + if (!zoomCursor.value && + peer.pi.platform == kPeerPlatformMacOS && + (c.viewStyle.style == kRemoteViewStyleAdaptive || + c.viewStyle.style == kRemoteViewStyleCustom)) { + return isWindows ? dpr : 1.0; + } + if (peer.isPeerLinux && peer.pi.currentDisplay == kAllDisplayValue) { + if (!zoomCursor.value || c.viewStyle.style == kRemoteViewStyleOriginal) { + // Remove the host output's density without applying canvas zoom. + final scale = 1.0 / _cursorDisplayScale.value; + return isWindows ? scale : scale / dpr; + } + final scale = c.scale / _cursorDisplayScale.value; + return isWindows ? scale * dpr : scale; + } + if (!zoomCursor.value || c.viewStyle.style == kRemoteViewStyleOriginal) { + // Original and unzoomed views do not apply canvas zoom. + final scale = _getCursorScaleForDisplay(1.0); + return isWindows ? scale : scale / dpr; + } + final scale = _getCursorScaleForDisplay(c.scale); + return isWindows ? scale * dpr : scale; + } + + double _getCursorScaleForDisplay(double scale) { + final peer = widget.ffi.ffiModel; + // All Displays can mix densities; no single display scale applies. + if (peer.pi.currentDisplay == kAllDisplayValue) return scale; + final displays = peer.pi.getCurDisplays(); + if (displays.isEmpty) return scale; + if (peer.pi.platform == kPeerPlatformMacOS) { + // macOS sends NSImage.size in points (src/platform/macos.rs:640,682), + // while HiDPI screen frames use backing pixels (libs/scrap/src/quartz/display.rs:37). + return scale * displays.first.scale; + } + if (!peer.isPeerLinux) return scale; + // Match rendering when a Wayland host + // with multiple outputs reports a display scale > 1. + // A single-output host keeps scale at 1.0 to preserve physical uinput + // coordinates, even with OS scaling enabled. This counts host outputs, + // not the selected displays.length. + // See src/server/display_service.rs:650 (update_sync_displays) and + // src/server/drm_capturer.rs:1523 (DRM's matching convention). + return scale / displays.first.scale; + } + Widget _buildScrollbarNonTextureRender( ImageModel m, Size imageSize, double s) { + double sizeScale = s; + if (widget.ffi.ffiModel.isPeerLinux) { + final displays = widget.ffi.ffiModel.pi.getCurDisplays(); + if (displays.isNotEmpty) { + sizeScale = s / displays[0].scale; + } + } return CustomPaint( size: imageSize, - painter: ImagePainter(image: m.image, x: 0, y: 0, scale: s), + painter: ImagePainter(image: m.image, x: 0, y: 0, scale: sizeScale), ); } @@ -1242,11 +1299,14 @@ class _ImagePaintState extends State { top: (displays[i].y - rect.top) * s + offset.dy, width: displays[i].width * sizeScale, height: displays[i].height * sizeScale, - child: Obx(() => Texture( - textureId: textureId.value, - filterQuality: - isViewOriginal ? FilterQuality.none : FilterQuality.low, - )), + child: _trackCursorDisplay( + Obx(() => Texture( + textureId: textureId.value, + filterQuality: + isViewOriginal ? FilterQuality.none : FilterQuality.low, + )), + displays[i], + ), )); } } @@ -1257,6 +1317,20 @@ class _ImagePaintState extends State { ); } + Widget _trackCursorDisplay(Widget child, Display display) { + final peer = widget.ffi.ffiModel; + if (!isDesktop || + !peer.isPeerLinux || + peer.pi.currentDisplay != kAllDisplayValue) { + return child; + } + return MouseRegion( + onEnter: (_) => _cursorDisplayScale.value = display.scale, + onHover: (_) => _cursorDisplayScale.value = display.scale, + child: child, + ); + } + MouseCursor _buildCustomCursor(BuildContext context, double scale) { final cursor = Provider.of(context); final cache = cursor.cache ?? preDefaultCursor.cache; diff --git a/flutter/lib/models/model.dart b/flutter/lib/models/model.dart index 3572b9728..30e3849c1 100644 --- a/flutter/lib/models/model.dart +++ b/flutter/lib/models/model.dart @@ -166,6 +166,7 @@ class FfiModel with ChangeNotifier { bool get isPeerMobile => isPeerAndroid; bool get isPeerLinux => _pi.platform == kPeerPlatformLinux; + bool get isPeerWindows => _pi.platform == kPeerPlatformWindows; bool get viewOnly => _viewOnly; bool get showMyCursor => _showMyCursor; @@ -2401,7 +2402,9 @@ class CanvasModel with ChangeNotifier { // ViewStyle fields and is not captured by the equality check. Therefore, we must // allow updates to proceed when style == kRemoteViewStyleCustom, even if the // rest of the ViewStyle fields are unchanged. - if (_lastViewStyle == viewStyle && style != kRemoteViewStyleCustom) { + if (_lastViewStyle == viewStyle && + _devicePixelRatio == ui.window.devicePixelRatio && + style != kRemoteViewStyleCustom) { return; } if (_lastViewStyle.style != viewStyle.style) { @@ -2461,6 +2464,8 @@ class CanvasModel with ChangeNotifier { _resetScroll(); Future.delayed(duration, () async { + // Layout updates scroll extents and detaches scrollbars no longer needed. + await SchedulerBinding.instance.endOfFrame; updateScrollPercent(); }); } @@ -2472,7 +2477,8 @@ class CanvasModel with ChangeNotifier { style != null ? ScrollStyle.fromString(style) : ScrollStyle.scrollauto; if (_scrollStyle != ScrollStyle.scrollauto) { - _resetScroll(); + // Scrollbar and Scroll Edge share controllers and retain their positions. + updateScrollPercent(); } notifyListeners(); @@ -2852,6 +2858,9 @@ class CanvasModel with ChangeNotifier { // data for cursor class CursorData { + // At most 4 MiB of RGBA, including Linux's square cursor padding. + static const _maxRasterSize = 1024; + final String peerId; final String id; final img2.Image image; @@ -2863,6 +2872,12 @@ class CursorData { double hoty; final int width; final int height; + int _rasterWidth; + int _rasterHeight; + bool _scaleLimitReported = false; + + int get rasterWidth => _rasterWidth; + int get rasterHeight => _rasterHeight; CursorData({ required this.peerId, @@ -2874,57 +2889,109 @@ class CursorData { required this.hotyOrigin, required this.width, required this.height, - }) : hotx = hotxOrigin * scale, + }) : _rasterWidth = (width * scale).ceil(), + _rasterHeight = (height * scale).ceil(), + hotx = hotxOrigin * scale, hoty = hotyOrigin * scale; int _doubleToInt(double v) => (v * 10e6).round().toInt(); + double _limitScale(double requestedScale) { + final valid = requestedScale.isFinite && requestedScale > 0; + // Invalid requests retain the last valid raster and hotspot. + final limitedScale = valid + ? min(requestedScale, _maxRasterSize / max(width, height)) + : scale; + final limited = !valid || limitedScale != requestedScale; + if (limited && !_scaleLimitReported) { + debugPrint( + 'Cursor $id: rejected scale $requestedScale for ${width}x$height; ' + 'using $limitedScale (maximum raster side $_maxRasterSize).'); + } + _scaleLimitReported = limited; + return limitedScale; + } + double _checkUpdateScale(double scale) { - double oldScale = this.scale; + scale = _limitScale(scale); if (scale != 1.0) { - // Update data if scale changed. - final tgtWidth = (width * scale).toInt(); - final tgtHeight = (width * scale).toInt(); - if (tgtWidth < kMinCursorSize || tgtHeight < kMinCursorSize) { - double sw = kMinCursorSize.toDouble() / width; - double sh = kMinCursorSize.toDouble() / height; - scale = sw < sh ? sh : sw; - } + // A thin cursor must not grow just to make its short edge reach the minimum. + scale = max(scale, kMinCursorSize / max(width, height)); } - if (_doubleToInt(oldScale) != _doubleToInt(scale)) { + final targetWidth = (width * scale).ceil(); + final targetHeight = (height * scale).ceil(); + if (_rasterWidth != targetWidth || _rasterHeight != targetHeight) { if (isWindows) { data = img2 .copyResize( image, - width: (width * scale).toInt(), - height: (height * scale).toInt(), + width: targetWidth, + height: targetHeight, interpolation: img2.Interpolation.average, ) .getBytes(order: img2.ChannelOrder.bgra); + } else if (isDesktop && scale < 1.0 && !image.hasPalette) { + data = Uint8List.fromList( + img2.encodePng(_resizeWithAlpha(targetWidth, targetHeight))); } else { data = Uint8List.fromList( img2.encodePng( img2.copyResize( image, - width: (width * scale).toInt(), - height: (height * scale).toInt(), + width: targetWidth, + height: targetHeight, interpolation: img2.Interpolation.average, ), ), ); } + _rasterWidth = targetWidth; + _rasterHeight = targetHeight; } this.scale = scale; - hotx = hotxOrigin * scale; - hoty = hotyOrigin * scale; + hotx = hotxOrigin * _rasterWidth / width; + hoty = hotyOrigin * _rasterHeight / height; return scale; } + img2.Image _resizeWithAlpha(int targetWidth, int targetHeight) { + final resized = + img2.Image.fromResized(image, width: targetWidth, height: targetHeight); + final dx = image.width / targetWidth; + final dy = image.height / targetHeight; + // Use the average filter's sample area, but weight RGB by alpha so + // transparent pixels do not darken visible edges in the straight-alpha PNG. + for (final pixel in resized) { + final x = (pixel.x * dx).toInt(); + final y = (pixel.y * dy).toInt(); + final sampleWidth = ((pixel.x + 1) * dx).toInt() - x; + final sampleHeight = ((pixel.y + 1) * dy).toInt() - y; + final samples = image.getRange(x, y, sampleWidth, sampleHeight); + num r = 0; + num g = 0; + num b = 0; + num a = 0; + while (samples.moveNext()) { + final sample = samples.current; + r += sample.r * sample.a; + g += sample.g * sample.a; + b += sample.b * sample.a; + a += sample.a; + } + if (a == 0) { + pixel.setRgba(0, 0, 0, 0); + continue; + } + pixel.setRgba(r / a, g / a, b / a, a / (sampleWidth * sampleHeight)); + } + return resized; + } + String updateGetKey(double scale) { scale = _checkUpdateScale(scale); - return '${peerId}_${id}_${_doubleToInt(width * scale)}_${_doubleToInt(height * scale)}'; + return '${peerId}_${id}_${_doubleToInt(width * scale)}_${_doubleToInt(height * scale)}_${rasterWidth}_$rasterHeight'; } } @@ -3466,6 +3533,17 @@ class CursorModel with ChangeNotifier { return false; } data = imgBytes.buffer.asUint8List(); + if (isDesktop && + (parent.target?.ffiModel.isPeerLinux == true || + parent.target?.ffiModel.isPeerWindows == true)) { + // PNG decoding supplies straight alpha for Linux/Windows cursor resizing. + final decoded = img2.decodePng(data); + if (decoded == null) { + debugPrint('Unable to decode cursor $id PNG for resizing'); + return false; + } + imgOrigin = decoded; + } } final cache = CursorData( peerId: peerId, diff --git a/flutter/lib/native/custom_cursor.dart b/flutter/lib/native/custom_cursor.dart index e85d42a55..5ea6a3951 100644 --- a/flutter/lib/native/custom_cursor.dart +++ b/flutter/lib/native/custom_cursor.dart @@ -5,7 +5,9 @@ import 'package:flutter_custom_cursor/flutter_custom_cursor.dart'; import 'package:flutter/foundation.dart'; import 'package:flutter/services.dart'; +import 'package:flutter_hbb/common.dart' show isLinux; import 'package:flutter_hbb/models/model.dart'; +import 'package:image/image.dart' as img; deleteCustomCursor(String key) => custom_cursor_manager.CursorManager.instance.deleteCursor(key); @@ -23,6 +25,11 @@ MouseCursor buildCursorOfCache( if (data == null) { return MouseCursor.defer; } + // Square canvases avoid clipping or stray edge pixels on Linux. + final width = isLinux && cache.rasterWidth < cache.rasterHeight + ? cache.rasterHeight + : cache.rasterWidth; + final height = isLinux ? width : cache.rasterHeight; debugPrint( "Register custom cursor with key $key (${cache.hotx},${cache.hoty})"); // [Safety] @@ -32,9 +39,12 @@ MouseCursor buildCursorOfCache( custom_cursor_manager.CursorManager.instance .registerCursor(custom_cursor_manager.CursorData() ..name = key - ..buffer = data - ..width = (cache.width * cache.scale).toInt() - ..height = (cache.height * cache.scale).toInt() + ..buffer = + width == cache.rasterWidth && height == cache.rasterHeight + ? data + : _padCursor(data, width) + ..width = width + ..height = height ..hotX = cache.hotx ..hotY = cache.hoty); cursor.addKey(key); @@ -42,3 +52,16 @@ MouseCursor buildCursorOfCache( return FlutterCustomMemoryImageCursor(key: key); } } + +Uint8List _padCursor(Uint8List data, int size) { + final bitmap = img.decodePng(data); + if (bitmap == null) { + throw const FormatException('Invalid native cursor PNG'); + } + final padded = img.copyExpandCanvas(bitmap, + newWidth: size, + newHeight: size, + position: img.ExpandCanvasPosition.topLeft, + toImage: img.Image(width: size, height: size, numChannels: 4)); + return Uint8List.fromList(img.encodePng(padded)); +} diff --git a/flutter/test/cursor_size_test.dart b/flutter/test/cursor_size_test.dart new file mode 100644 index 000000000..7df1cf74e --- /dev/null +++ b/flutter/test/cursor_size_test.dart @@ -0,0 +1,368 @@ +import 'dart:convert'; +import 'dart:io'; +import 'dart:ui' as ui; + +import 'package:flutter/services.dart'; +import 'package:flutter/widgets.dart'; +import 'package:flutter_hbb/consts.dart'; +import 'package:flutter_hbb/desktop/pages/remote_page.dart'; +import 'package:flutter_hbb/models/input_model.dart'; +import 'package:flutter_hbb/models/model.dart'; +import 'package:flutter_hbb/native/custom_cursor.dart'; +import 'package:flutter_hbb/utils/image.dart'; +import 'package:flutter_test/flutter_test.dart'; +import 'package:get/get.dart'; +import 'package:image/image.dart' as img; +import 'package:provider/provider.dart'; + +class _Canvas extends ChangeNotifier implements CanvasModel { + _Canvas(String style) + : viewStyle = ViewStyle( + style: style, + width: 200, + height: 160, + displayWidth: 400, + displayHeight: 320); + @override + final ViewStyle viewStyle; + @override + final devicePixelRatio = 1.0; + @override + double scale = 0.25; + @override + final imageOverflow = false.obs; + @override + bool get cursorEmbedded => false; + @override + Size get size => const Size(200, 160); + @override + double get x => 0; + @override + double get y => 0; + @override + dynamic noSuchMethod(Invocation invocation) => super.noSuchMethod(invocation); +} + +class _Image extends ChangeNotifier implements ImageModel { + @override + bool get useTextureRender => false; + @override + ui.Image? get image => null; + @override + dynamic noSuchMethod(Invocation invocation) => super.noSuchMethod(invocation); +} + +class _Input extends Fake implements InputModel { + @override + final relativeMouseMode = false.obs; +} + +class _Display extends Display { + @override + double scale = 1.0; +} + +class _Peer extends Fake implements FfiModel { + @override + final pi = PeerInfo(); + @override + bool get isPeerLinux => pi.platform == kPeerPlatformLinux; + @override + bool get isPeerWindows => pi.platform == kPeerPlatformWindows; +} + +class _FFI extends Fake implements FFI { + _FFI(this.canvasModel); + @override + final CanvasModel canvasModel; + @override + final ffiModel = _Peer(); + @override + final inputModel = _Input(); +} + +class _Cursor extends CursorModel { + _Cursor(this.cache, FFI ffi) : super(WeakReference(ffi)); + @override + final CursorData cache; +} + +void main() { + final binding = TestWidgetsFlutterBinding.ensureInitialized(); + final channel = Platform.isWindows + ? SystemChannels.mouseCursor + : const MethodChannel('flutter_custom_cursor'); + final registrations = >[]; + setUp(() { + registrations.clear(); + binding.defaultBinaryMessenger.setMockMethodCallHandler(channel, + (call) async { + if (!call.method.startsWith('createCustomCursor')) return null; + final args = call.arguments as Map; + registrations.add(args); + return args['name']; + }); + }); + tearDown(() => + binding.defaultBinaryMessenger.setMockMethodCallHandler(channel, null)); + for (final scenario in [ + ((4, 64), 0.5, (2, 32)), + ((64, 4), 0.5, (32, 2)), + ((1, 64), 0.25, (1, 16)), + ((8, 8), 0.5, (12, 12)), + ((4, 64), 1.0, (4, 64)), + ((1, 512), 10.0, (2, 1024)), + ]) { + test('native cursor size $scenario', + () => _checkSize(scenario, registrations)); + } + test('native raster boundaries rebuild buffers and distinguish cache keys', + () => _checkRasterTransitions(registrations)); + test('cursor resize limits preserve the last valid raster', () async { + for (final size in [(30, 20), (20, 30)]) { + await _checkResizeLimits(size, registrations); + registrations.clear(); + } + }); + for (final pattern in [ + ([128, 0, 0, 128], 128), + ([255, 0, 0, 255, 0, 0, 0, 0], 127), + ]) { + test('Windows peer cursor alpha survives resizing $pattern', + () => _checkWindowsPeerAlpha(pattern, registrations)); + } + for (final style in [ + kRemoteViewStyleOriginal, + kRemoteViewStyleAdaptive, + kRemoteViewStyleCustom + ]) { + for (final zoom in [false, true]) { + testWidgets('$style zoom=$zoom follows the live DPR and peer scale', + (tester) => _checkView(tester, (style, zoom), registrations)); + } + } +} + +CursorData _data((int, int) size, {Offset hotspot = Offset.zero}) { + final image = img.Image(width: size.$1, height: size.$2, numChannels: 4); + for (final pixel in image) { + pixel.setRgba(64, 32, 16, 255); + } + image.getPixel(0, 0).setRgba(255, 0, 0, 128); + return CursorData( + peerId: 'size', + id: '$size', + image: image, + scale: 1, + data: Platform.isWindows + ? image.getBytes(order: img.ChannelOrder.bgra) + : Uint8List.fromList(img.encodePng(image)), + hotxOrigin: hotspot.dx, + hotyOrigin: hotspot.dy, + width: size.$1, + height: size.$2); +} + +Future _dispose(CursorModel cursor) async { + for (final key in cursor.cachedKeys) { + await deleteCustomCursor(key); + } + cursor.dispose(); +} + +Future _checkWindowsPeerAlpha( + (List, int) pattern, List> registrations) async { + const sourceSize = 64; + const dpr = 2.0; + const channels = 4; + final ffi = _FFI(_Canvas(kRemoteViewStyleAdaptive)); + ffi.ffiModel.pi.platform = kPeerPlatformWindows; + final cursor = CursorModel(WeakReference(ffi))..id = 'alpha'; + addTearDown(() => _dispose(cursor)); + addTearDown(cursor.disposeImages); + addTearDown(ffi.canvasModel.dispose); + await cursor.updateCursorData({ + 'id': 'alpha', + 'hotx': '0', + 'hoty': '0', + 'width': '$sourceSize', + 'height': '$sourceSize', + 'colors': jsonEncode(List.generate(sourceSize * sourceSize * channels, + (i) => pattern.$1[i % pattern.$1.length])), + }); + buildCursorOfCache(cursor, 1.0 / dpr, cursor.cache); + await Future.delayed(Duration.zero); + final args = registrations.single; + final targetSize = (sourceSize / dpr).ceil(); + _expectSize(args, (targetSize, targetSize)); + final bytes = args['buffer'] as Uint8List; + if (Platform.isWindows) { + expect(bytes.sublist(0, channels), [0, 0, pattern.$2, pattern.$2]); + } else { + final pixel = img.decodePng(bytes)!.getPixel(0, 0); + expect([pixel.r, pixel.g, pixel.b, pixel.a], [255, 0, 0, pattern.$2]); + } +} + +Future _checkSize(((int, int), double, (int, int)) scenario, + List> registrations) async { + final ffi = _FFI(_Canvas(kRemoteViewStyleAdaptive)); + final cursor = _Cursor(_data(scenario.$1), ffi); + addTearDown(() => _dispose(cursor)); + addTearDown(ffi.canvasModel.dispose); + buildCursorOfCache(cursor, scenario.$2, cursor.cache); + await Future.delayed(Duration.zero); + final (width, height) = scenario.$3; + expect( + (cursor.cache.rasterWidth, cursor.cache.rasterHeight), (width, height)); + final args = registrations.single; + final padded = Platform.isLinux && width != height; + final side = width > height ? width : height; + _expectSize(args, (padded ? side : width, padded ? side : height)); + if (padded) { + final bitmap = img.decodePng(args['buffer'] as Uint8List)!; + final artwork = + img.copyCrop(bitmap, x: 0, y: 0, width: width, height: height); + expect(artwork.getBytes(), img.decodePng(cursor.cache.data!)!.getBytes()); + expect(bitmap.where((p) => p.x >= width || p.y >= height).map((p) => p.a), + everyElement(0)); + } +} + +void _expectSize(Map args, (int, int) expected) { + expect((args['width'], args['height']), expected); + final bytes = args['buffer'] as Uint8List; + if (Platform.isWindows) { + const channels = 4; + expect(bytes.length, expected.$1 * expected.$2 * channels); + } else { + final bitmap = img.decodePng(bytes)!; + expect((bitmap.width, bitmap.height), expected); + } +} + +Future _checkRasterTransitions( + List> registrations) async { + const delta = 3e-8; + const scaleAboveOne = (2.25 / 1.75) / 2.25 * 1.75; + final ffi = _FFI(_Canvas(kRemoteViewStyleAdaptive)); + final cursor = _Cursor(_data((64, 64)), ffi); + addTearDown(() => _dispose(cursor)); + addTearDown(ffi.canvasModel.dispose); + for (final (scale, expected) in [ + (scaleAboveOne, (65, 65)), + (0.5 - delta, (32, 32)), + (0.5 + delta, (33, 33)), + (1.0, (64, 64)), + (scaleAboveOne, (65, 65)), + (1.0, (64, 64)), + ]) { + buildCursorOfCache(cursor, scale, cursor.cache); + await Future.delayed(Duration.zero); + final key = cursor.cache.updateGetKey(scale); + _expectSize( + registrations.singleWhere((args) => args['name'] == key), expected); + } + expect(registrations.length, 4); +} + +Future _checkResizeLimits( + (int, int) size, List> registrations) async { + const maxSide = 1024; + const sourceLongEdge = 30; + const validScale = maxSide / sourceLongEdge; + final ffi = _FFI(_Canvas(kRemoteViewStyleAdaptive)); + const hotspot = Offset(4, 7); + final cursor = _Cursor(_data(size, hotspot: hotspot), ffi); + addTearDown(() => _dispose(cursor)); + addTearDown(ffi.canvasModel.dispose); + buildCursorOfCache(cursor, validScale, cursor.cache); + await Future.delayed(Duration.zero); + final raster = ((size.$1 * validScale).ceil(), (size.$2 * validScale).ceil()); + final expectedHotspot = + (hotspot.dx * raster.$1 / size.$1, hotspot.dy * raster.$2 / size.$2); + _expectSize( + registrations.single, Platform.isLinux ? (maxSide, maxSide) : raster); + expect((registrations.single['hotX'], registrations.single['hotY']), + expectedHotspot); + // Fail on a small allocation before reaching unsafe sizes without the guard. + for (final scale in [ + (maxSide + 1) / sourceLongEdge, + double.maxFinite, + double.infinity, + double.nan, + 0.0, + -1.0, + ]) { + buildCursorOfCache(cursor, scale, cursor.cache); + await Future.delayed(Duration.zero); + expect(cursor.cache.scale, validScale); + expect((cursor.cache.rasterWidth, cursor.cache.rasterHeight), raster); + expect((cursor.cache.hotx, cursor.cache.hoty), expectedHotspot); + } + buildCursorOfCache(cursor, 1.0, cursor.cache); + await Future.delayed(Duration.zero); + expect(cursor.cache.scale, 1.0); + _expectSize(registrations.last, + Platform.isLinux ? (sourceLongEdge, sourceLongEdge) : size); +} + +const _viewCases = [ + (1.0, kPeerPlatformMacOS, 2.0), + (1.25, kPeerPlatformLinux, 2.0), + (2.0, kPeerPlatformMacOS, 1.0), +]; + +Future _checkView(WidgetTester tester, (String, bool) mode, + List> registrations) async { + const sourceSize = 64, customScale = 4.0; + final canvas = _Canvas(mode.$1); + final ffi = _FFI(canvas); + final display = _Display(); + ffi.ffiModel.pi.displays.addAll([Display(), display]); + ffi.ffiModel.pi.currentDisplay = 1; + final cursor = _Cursor(_data((sourceSize, sourceSize)), ffi); + addTearDown(() => _dispose(cursor)); + addTearDown(canvas.dispose); + addTearDown(tester.view.resetDevicePixelRatio); + for (final (dpr, peer, peerScale) in _viewCases) { + ffi.ffiModel.pi.platform = peer; + display.scale = peerScale; + canvas.scale = mode.$1 == kRemoteViewStyleCustom + ? customScale / dpr + : mode.$1 == kRemoteViewStyleOriginal + ? 1.0 / dpr + : canvas.viewStyle.scale; + tester.view.devicePixelRatio = dpr; + await tester.pumpWidget(MediaQuery( + data: MediaQueryData(devicePixelRatio: dpr), + child: MultiProvider( + providers: [ + ChangeNotifierProvider(create: (_) => _Image()), + ChangeNotifierProvider.value(value: canvas), + ChangeNotifierProvider.value(value: cursor), + ], + child: ImagePaint( + ffi: ffi, + id: 'size', + zoomCursor: mode.$2.obs, + cursorOverImage: true.obs, + keyboardEnabled: true.obs, + remoteCursorMoved: false.obs, + ), + ), + )); + final video = tester.widget(find.byType(CustomPaint)).painter + as ImagePainter; + final scale = video.scale * (mode.$2 ? 1.0 : 1.0 / (canvas.scale * dpr)); + final size = sourceSize * (peer == kPeerPlatformMacOS ? peerScale : 1.0); + final w = peer == kPeerPlatformMacOS && + !mode.$2 && + mode.$1 != kRemoteViewStyleOriginal + ? (sourceSize * (Platform.isWindows ? dpr : 1.0)).ceil() + : (size * scale * (Platform.isWindows ? dpr : 1.0)).ceil(); + final key = cursor.cache.updateGetKey(cursor.cache.scale); + _expectSize(registrations.singleWhere((v) => v['name'] == key), (w, w)); + } + await tester.pumpWidget(const SizedBox.shrink()); +} diff --git a/src/client/io_loop.rs b/src/client/io_loop.rs index e893a4780..880a0260f 100644 --- a/src/client/io_loop.rs +++ b/src/client/io_loop.rs @@ -1511,7 +1511,11 @@ impl Remote { _ => {} }, Some(message::Union::CursorData(cd)) => { - self.handler.set_cursor_data(cd); + let id = cd.id; + match decode_cursor_data(cd) { + Ok(cd) => self.handler.set_cursor_data(cd), + Err(err) => log::warn!("Rejected cursor {id}: {err}"), + } } Some(message::Union::CursorId(id)) => { self.handler.set_cursor_id(id.to_string()); @@ -2545,6 +2549,47 @@ impl Remote { } } +// Both UI handlers receive validated, uncompressed RGBA from the receive loop. +fn decode_cursor_data(data: CursorData) -> hbb_common::ResultType { + use hbb_common::{anyhow::anyhow, bail}; + + // Limit decoded cursor data to 1 MiB before JSON serialization. + const MAX_CURSOR_SIZE: i32 = 512; + const RGBA_CHANNELS: usize = 4; + + let mut cd = data; + if !(1..=MAX_CURSOR_SIZE).contains(&cd.width) || !(1..=MAX_CURSOR_SIZE).contains(&cd.height) { + bail!("invalid source size {}x{}", cd.width, cd.height); + } + if !(0..cd.width).contains(&cd.hotx) || !(0..cd.height).contains(&cd.hoty) { + bail!( + "hotspot ({},{}) is outside the cursor image", + cd.hotx, + cd.hoty + ); + } + let expected = (cd.width as usize) + .checked_mul(cd.height as usize) + .and_then(|pixels| pixels.checked_mul(RGBA_CHANNELS)) + .ok_or_else(|| anyhow!("cursor RGBA size overflow"))?; + let max_compressed_size = zstd::zstd_safe::compress_bound(expected); + if cd.colors.len() > max_compressed_size { + bail!( + "compressed cursor data too large: {} bytes (limit {max_compressed_size})", + cd.colors.len() + ); + } + let colors = zstd::bulk::decompress(&cd.colors, expected)?; + if colors.len() != expected { + bail!( + "invalid RGBA length: expected {expected}, got {}", + colors.len() + ); + } + cd.colors = colors.into(); + Ok(cd) +} + struct RemoveJob { files: Vec, path: String, diff --git a/src/flutter.rs b/src/flutter.rs index a6d3496dd..5c453d468 100644 --- a/src/flutter.rs +++ b/src/flutter.rs @@ -642,7 +642,7 @@ impl FlutterHandler { impl InvokeUiSession for FlutterHandler { fn set_cursor_data(&self, cd: CursorData) { - let colors = hbb_common::compress::decompress(&cd.colors); + let colors = &cd.colors; self.push_event( "cursor_data", &[ diff --git a/src/ui/remote.rs b/src/ui/remote.rs index 2f36f5d81..1bdf90a55 100644 --- a/src/ui/remote.rs +++ b/src/ui/remote.rs @@ -122,7 +122,7 @@ impl SciterHandler { impl InvokeUiSession for SciterHandler { fn set_cursor_data(&self, cd: CursorData) { - let mut colors = hbb_common::compress::decompress(&cd.colors); + let mut colors: Vec = cd.colors.into(); if colors.iter().filter(|x| **x != 0).next().is_none() { log::info!("Fix transparent"); // somehow all 0 images shows black rect, here is a workaround diff --git a/src/ui/remote.tis b/src/ui/remote.tis index 87c543eb0..f1306633b 100644 --- a/src/ui/remote.tis +++ b/src/ui/remote.tis @@ -386,14 +386,35 @@ var cur_local_x = 0; var cur_local_y = 0; var cursors = {}; var image_binded; +const MAX_CURSOR_RASTER_SIZE = 1024; +var cursor_scale_limited = false; + +function limitCursorImageScale(img, factor) { + // Subtraction rejects NaN and Infinity without converting them to integers. + var valid = factor > 0 && factor - factor == 0; + var side = img.width > img.height ? img.width : img.height; + var max_factor = MAX_CURSOR_RASTER_SIZE.toFloat() / side; + // Invalid factors use the validated source image at its original size. + var limited_factor = valid ? (factor > max_factor ? max_factor : factor) : 1.; + var limited = !valid || limited_factor != factor; + if (limited && !cursor_scale_limited) { + stdout.println("Cursor image scale " + factor + " limited to " + limited_factor + + " (maximum raster side " + MAX_CURSOR_RASTER_SIZE + ")"); + } + cursor_scale_limited = limited; + return limited_factor; +} function scaleCursorImage(img) { var factor = cursor_scale; if (cursor_img.style#display != 'none') { factor /= scaleFactor; } + factor = limitCursorImageScale(img, factor); var w = (img.width * factor).toInteger(); var h = (img.height * factor).toInteger(); + if (w < 1) w = 1; + if (h < 1) h = 1; cursor_img.style.set { width: w + "px", height: h + "px", @@ -413,7 +434,10 @@ function updateCursor(system=false) { if (system) { handler.style#cursor = undefined; } else if (cur_img) { - handler.style.cursor(cur_img, (cur_hotx * cursor_scale).toInteger(), (cur_hoty * cursor_scale).toInteger()); + var img = cursors[cur_id][0]; + handler.style.cursor(cur_img, + (cur_hotx * cur_img.width.toFloat() / img.width).toInteger(), + (cur_hoty * cur_img.height.toFloat() / img.height).toInteger()); } } @@ -468,6 +492,16 @@ handler.setCursorPosition = function(x, y) { var y = cur_y - cur_hoty; x *= cursor_scale / scaleFactor; y *= cursor_scale / scaleFactor; + // The overlay may still be hidden, so compute its raster before refreshCursor(). + // Keep screen coordinates on the video scale and only adjust the hotspot. + var img = cursors[cur_id][0]; + var hotspot_scale = limitCursorImageScale(img, cursor_scale / scaleFactor); + var w = (img.width * hotspot_scale).toInteger(); + var h = (img.height * hotspot_scale).toInteger(); + if (w < 1) w = 1; + if (h < 1) h = 1; + x += cur_hotx * (cursor_scale / scaleFactor - w.toFloat() / img.width); + y += cur_hoty * (cursor_scale / scaleFactor - h.toFloat() / img.height); cursor_img.style.set { left: x + "px", top: y + "px",