Skip to content

Commit 4809ba1

Browse files
craiglabenzPiinksdcharkes
authored
Language Detector task (#22)
* Adds mediapipe_core package (#11) * adds mediapipe_core package * adds makefile for all packages * fixes typo in Makefile * Apply suggestions from code review * Update Makefile * Apply suggestions from code review * updated generated code's location and license (for 3P status) * code review responses including: * comments * licensing * resolving testing nits * updated README * setup sharing of base analysis_options * Update packages/mediapipe-core/lib/src/containers.dart Co-authored-by: Kate Lovett <[email protected]> * Update packages/mediapipe-core/lib/src/containers.dart Co-authored-by: Kate Lovett <[email protected]> * add free methods to core structs * code review updates to options * adds publish blocker * Add GitHub actions for CI/CD (#14) * adds CI/CD for mediapipe_core * add newlines * moves file into workflows dir * uncomment flutter doctor * testing PR config to run this now * added master and beta CI scripts * add executable permissions to CI scripts * adds ffiwrapper to ci/cd --------- Co-authored-by: Kate Lovett <[email protected]> * Add utility to collect headers from google/mediapipe (#10) * adds cmd to pull header files from google/mediapipe * polish and missing parts from git surgery * More comments and touch ups * Apply suggestions from code review * moves build command into `tool/` directory and renames folder `build_cmd` -> `builder` * complete build_cmd -> builder rename * Update readme * Added licenses * Adds DownloadModelCommand --------- Co-authored-by: Kate Lovett <[email protected]> * [FFI] MediaPipe SDKs finder automation (#16) * adds sdks_finder command to builder utility * propagates changes to existing commands * adds manifest files generated by new sdks_finder command * updates in response to code review * Adds mediapipe_text package (#12) * adds mediapipe_text package * Update .vscode/settings.json added newline * resync headers * regenerated core bindings * Apply suggestions from code review * Adds example to `mediapipe-task-text` (#15) * initial commit of example * build file changes from `flutter pub get` * update main.dart * removes commented code * updates example for isolates design * Use `native-assets` to vendor MediaPipe SDK (#9) * adding bare structure for native assets * add MVP / first draft of build.dart * build.dart updates * update build.dart TODO: stream file * model memory troubleshooting * vendoring script tweak * remove development logging * removes pointless build method * Add utility to collect headers from google/mediapipe (#10) * adds cmd to pull header files from google/mediapipe * polish and missing parts from git surgery * More comments and touch ups * Apply suggestions from code review * moves build command into `tool/` directory and renames folder `build_cmd` -> `builder` * complete build_cmd -> builder rename * Update readme * Added licenses * Adds DownloadModelCommand --------- Co-authored-by: Kate Lovett <[email protected]> * adds mediapipe_text package * Update .vscode/settings.json added newline * resync headers * regenerated core bindings * native assets troubleshooting this commit is broken * Removes redundant count field * update build.dart for correct bindings path * download text classification model for CI * better memory freeing in executor * added SafeArea to example * added CI to PRs into text package * ci tooling change * remove accidentally commited model * more CI shenanigans * lowers minimum Dart version for builder * added smoke test for text example * Added CI/CD for examples * More CI tweaks * entering "please work" territory * d'oh * trying more random stuff * one more time * it'd be funny if this helped * more print statements * enable reaching new print statements * more logging * see what's in build dir * another test * adding flutter config list * turn off fail-fast for beta and master * moar logs * way moar prints * moare things * commit rest of rename * moar whatevers * adds manifest files generated by new sdks_finder command * adds sdks_finder command to builder utility * propagates changes to existing commands * updates in response to code review * updates to build.dart and tests * add Android runtime * sdks_finder logging improvement for when build folders change names * refreshed symbols from google/mediapipe * cleanup * loosens closeness thresholds in integration tests * separate build commands for macos architectures * restores fail-fast setting to CI * removed stale logging statements from CI * removes accidentally committed lines * add formatting of sdk_downloads.dart for CI * fixes broken example test * code touch ups from @Piinks code review --------- Co-authored-by: Kate Lovett <[email protected]> * added base Dart class ClassificationResult to consolidate results logic in `mediapipe_core` in doing so, removed meaningless TextClassifierResult.timestamp field * Update Makefile * Update Makefile * Update packages/mediapipe-task-text/build.dart Co-authored-by: Kate Lovett <[email protected]> * Update packages/mediapipe-task-text/lib/src/tasks/text_classification/text_classification_executor.dart Co-authored-by: Kate Lovett <[email protected]> * code review responses * formatting * removes stale comment * adds Dart to Native converters, with tests * changes from code review * updates mediapipe-core to prepare for IO/web split * Improves memory management in core tests * updated ffigen / bindings * refactors text package for better memory management and eventual web/io split * removed stale test * cleanup on aisle COMMENTS * Comments and documentation improvements * Moved log statement * Removed native memory management helpers in favor of `free` extensions on pointers * Renamed abstract classes to have Base prefix * sorted out class constructors * moved `fake` constructor to default unnamed constructor * leaning on the fact that the native constructors will be hidden by conditional exports, reducing confusion * Improved docstrings explaining memory ownership * Convert lists to lazy iterable / generators * added missing licenses * Update packages/mediapipe-task-text/example/test/widgets_test.dart Co-authored-by: Kate Lovett <[email protected]> * Update packages/mediapipe-task-text/example/lib/main.dart Co-authored-by: Kate Lovett <[email protected]> * completed return style change * CI troubleshooting * moved around debugging code * logging tweak * cat native-assets.yaml * moar logs * removed bad echo * fixed native-assets typo should be underscore! * Removes CI debugging statements --------- Co-authored-by: Kate Lovett <[email protected]> * Native Assets CI fix (#20) * adds native assets debugging statements * Try only downloading target arch * Revert "adds native assets debugging statements" This reverts commit b2bc215. --------- Co-authored-by: Daco Harkes <[email protected]> * Text Embedding task (#21) * updated and re-ran generators * added embedding concepts to mediapipe-core * fixed embedding header file and bindings * adds text embedding classes to text pkg * updates example with text embedding * removed dead file * added more embedding tests * added embedding model download to CI script * touch ups * Update packages/mediapipe-core/lib/src/io/containers.dart Co-authored-by: Kate Lovett <[email protected]> * Update packages/mediapipe-task-text/example/.gitignore Co-authored-by: Kate Lovett <[email protected]> * Update packages/mediapipe-task-text/example/lib/text_embedding_demo.dart Co-authored-by: Kate Lovett <[email protected]> * moved worker dispose method to base class * docstring & comment improvements * throw exceptions in impossible code paths instead of returning null * class hierarchy improvements * fixed outdates tests * cleaned up dispose methods * various tidying * fixed deprecation warning * moves repeated widgets into helper method --------- Co-authored-by: Kate Lovett <[email protected]> * updated and re-ran generators * fixed embedding header file and bindings * adds text embedding classes to text pkg * moved worker dispose method to base class * class hierarchy improvements * cleaned up dispose methods * initial commit of language detection task * finishes language detection impl * adds language detection demo * backfilling improvements to classification and embedding * adds language detection tests * add new model download to CI script * fixes stale classification widget test, adds language detection widget test * copied latest headers from mediapipe * Update packages/mediapipe-task-text/lib/src/io/tasks/language_detection/language_detector_executor.dart Co-authored-by: Kate Lovett <[email protected]> * comments / documentation improvements --------- Co-authored-by: Kate Lovett <[email protected]> Co-authored-by: Daco Harkes <[email protected]>
1 parent 6d9c37a commit 4809ba1

31 files changed

+1324
-190
lines changed

Makefile

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ headers:
66
models:
77
cd tool/builder && dart bin/main.dart model -m textclassification
88
cd tool/builder && dart bin/main.dart model -m textembedding
9-
9+
cd tool/builder && dart bin/main.dart model -m languagedetection
1010

1111
# Runs `ffigen` for all packages
1212
generate: generate_core generate_text

packages/mediapipe-core/lib/universal_mediapipe_core.dart

+1-1
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ class Classifications extends BaseClassifications {
6262
});
6363

6464
@override
65-
Iterable<BaseCategory> get categories => throw UnimplementedError();
65+
Iterable<Category> get categories => throw UnimplementedError();
6666

6767
@override
6868
int get headIndex => throw UnimplementedError();

packages/mediapipe-core/third_party/mediapipe/tasks/c/components/containers/classification_result.h

-2
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@ struct Classifications {
2828
// The array of predicted categories, usually sorted by descending scores,
2929
// e.g. from high to low probability.
3030
struct Category* categories;
31-
3231
// The number of elements in the categories array.
3332
uint32_t categories_count;
3433

@@ -58,7 +57,6 @@ struct ClassificationResult {
5857
// exceed the maximum size that the model can process: to solve this, the
5958
// input data is split into multiple chunks starting at different timestamps.
6059
int64_t timestamp_ms;
61-
6260
// Specifies whether the timestamp contains a valid value.
6361
bool has_timestamp_ms;
6462
};

packages/mediapipe-core/third_party/mediapipe/tasks/c/components/processors/classifier_options.h

-2
Original file line numberDiff line numberDiff line change
@@ -41,15 +41,13 @@ struct ClassifierOptions {
4141
// category name is not in this set will be filtered out. Duplicate or unknown
4242
// category names are ignored. Mutually exclusive with category_denylist.
4343
const char** category_allowlist;
44-
4544
// The number of elements in the category allowlist.
4645
uint32_t category_allowlist_count;
4746

4847
// The denylist of category names. If non-empty, detection results whose
4948
// category name is in this set will be filtered out. Duplicate or unknown
5049
// category names are ignored. Mutually exclusive with category_allowlist.
5150
const char** category_denylist;
52-
5351
// The number of elements in the category denylist.
5452
uint32_t category_denylist_count;
5553
};

packages/mediapipe-task-text/example/lib/enumerate.dart

+24-2
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,34 @@
22
// Use of this source code is governed by a BSD-style license that can be
33
// found in the LICENSE file.
44

5-
extension Enumeratable<T> on List<T> {
6-
Iterable<S> enumerate<S>(S Function(T, int) fn) sync* {
5+
extension EnumeratableList<T> on List<T> {
6+
/// Invokes the callback on each element of the list, optionally stopping
7+
/// after [max] (inclusive) invocations.
8+
Iterable<S> enumerate<S>(S Function(T, int) fn, {int? max}) sync* {
79
int count = 0;
810
while (count < length) {
911
yield fn(this[count], count);
1012
count++;
13+
14+
if (max != null && count >= max) {
15+
return;
16+
}
17+
}
18+
}
19+
}
20+
21+
extension EnumeratableIterable<T> on Iterable<T> {
22+
/// Invokes the callback on each element of the iterable, optionally stopping
23+
/// after [max] (inclusive) invocations.
24+
Iterable<S> enumerate<S>(S Function(T, int) fn, {int? max}) sync* {
25+
int count = 0;
26+
for (final T obj in this) {
27+
yield fn(obj, count);
28+
count++;
29+
30+
if (max != null && count >= max) {
31+
return;
32+
}
1133
}
1234
}
1335
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
// Copyright 2014 The Flutter Authors. All rights reserved.
2+
// Use of this source code is governed by a BSD-style license that can be
3+
// found in the LICENSE file.
4+
5+
import 'dart:async';
6+
import 'dart:typed_data';
7+
import 'package:flutter/material.dart';
8+
import 'package:getwidget/getwidget.dart';
9+
import 'package:mediapipe_text/mediapipe_text.dart';
10+
import 'enumerate.dart';
11+
12+
class LanguageDetectionDemo extends StatefulWidget {
13+
const LanguageDetectionDemo({super.key, this.detector});
14+
15+
final LanguageDetector? detector;
16+
17+
@override
18+
State<LanguageDetectionDemo> createState() => _LanguageDetectionDemoState();
19+
}
20+
21+
class _LanguageDetectionDemoState extends State<LanguageDetectionDemo>
22+
with AutomaticKeepAliveClientMixin<LanguageDetectionDemo> {
23+
final TextEditingController _controller = TextEditingController();
24+
final Completer<LanguageDetector> _completer = Completer<LanguageDetector>();
25+
final results = <Widget>[];
26+
String? _isProcessing;
27+
28+
@override
29+
void initState() {
30+
super.initState();
31+
_controller.text = 'Quiero agua, por favor';
32+
_initDetector();
33+
}
34+
35+
Future<void> _initDetector() async {
36+
if (widget.detector != null) {
37+
return _completer.complete(widget.detector!);
38+
}
39+
40+
ByteData? bytes = await DefaultAssetBundle.of(context)
41+
.load('assets/language_detector.tflite');
42+
43+
final detector = LanguageDetector(
44+
LanguageDetectorOptions.fromAssetBuffer(
45+
bytes.buffer.asUint8List(),
46+
),
47+
);
48+
_completer.complete(detector);
49+
bytes = null;
50+
}
51+
52+
void _prepareForDetection() {
53+
setState(() {
54+
_isProcessing = _controller.text;
55+
results.add(const CircularProgressIndicator.adaptive());
56+
});
57+
}
58+
59+
Future<void> _detect() async {
60+
_prepareForDetection();
61+
_completer.future.then((detector) async {
62+
final result = await detector.detect(_controller.text);
63+
_showDetectionResults(result);
64+
result.dispose();
65+
});
66+
}
67+
68+
void _showDetectionResults(LanguageDetectorResult result) {
69+
setState(
70+
() {
71+
results.last = Card(
72+
key: Key('prediction-"$_isProcessing" ${results.length}'),
73+
margin: const EdgeInsets.all(10),
74+
child: Column(
75+
children: [
76+
Padding(
77+
padding: const EdgeInsets.all(10),
78+
child: Text(_isProcessing!),
79+
),
80+
Padding(
81+
padding: const EdgeInsets.all(10.0),
82+
child: Wrap(
83+
children: <Widget>[
84+
...result.predictions
85+
.enumerate<Widget>(
86+
(prediction, index) => _languagePrediction(
87+
prediction,
88+
predictionColors[index],
89+
),
90+
// Take first 4 because the model spits out dozens of
91+
// astronomically low probability language predictions
92+
max: predictionColors.length,
93+
)
94+
.toList(),
95+
],
96+
),
97+
),
98+
],
99+
),
100+
);
101+
_isProcessing = null;
102+
},
103+
);
104+
}
105+
106+
static final predictionColors = <Color>[
107+
Colors.blue[300]!,
108+
Colors.orange[300]!,
109+
Colors.green[300]!,
110+
Colors.red[300]!,
111+
];
112+
113+
Widget _languagePrediction(LanguagePrediction prediction, Color color) {
114+
return Padding(
115+
padding: const EdgeInsets.only(right: 8),
116+
child: GFButton(
117+
onPressed: null,
118+
text: '${prediction.languageCode} :: '
119+
'${prediction.probability.roundTo(8)}',
120+
shape: GFButtonShape.pills,
121+
color: color,
122+
),
123+
);
124+
}
125+
126+
@override
127+
Widget build(BuildContext context) {
128+
super.build(context);
129+
return Scaffold(
130+
body: SafeArea(
131+
child: Padding(
132+
padding: const EdgeInsets.all(16.0),
133+
child: SingleChildScrollView(
134+
child: Column(
135+
children: <Widget>[
136+
TextField(controller: _controller),
137+
...results.reversed,
138+
],
139+
),
140+
),
141+
),
142+
),
143+
floatingActionButton: FloatingActionButton(
144+
onPressed:
145+
_isProcessing != null && _controller.text != '' ? null : _detect,
146+
child: const Icon(Icons.search),
147+
),
148+
);
149+
}
150+
151+
@override
152+
bool get wantKeepAlive => true;
153+
}
154+
155+
extension on double {
156+
double roundTo(int decimalPlaces) =>
157+
double.parse(toStringAsFixed(decimalPlaces));
158+
}

packages/mediapipe-task-text/example/lib/main.dart

+31-19
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import 'package:flutter/material.dart';
2+
import 'language_detection_demo.dart';
23
import 'logging.dart';
34
import 'text_classification_demo.dart';
45
import 'text_embedding_demo.dart';
@@ -31,7 +32,7 @@ class TextTaskPages extends StatefulWidget {
3132
class TextTaskPagesState extends State<TextTaskPages> {
3233
final PageController controller = PageController();
3334

34-
final titles = <String>['Classify', 'Embed'];
35+
final titles = <String>['Classify', 'Embed', 'Detect Languages'];
3536
int titleIndex = 0;
3637

3738
void switchToPage(int index) {
@@ -61,28 +62,39 @@ class TextTaskPagesState extends State<TextTaskPages> {
6162
children: const <Widget>[
6263
TextClassificationDemo(),
6364
TextEmbeddingDemo(),
65+
LanguageDetectionDemo(),
6466
],
6567
),
66-
bottomNavigationBar: ColoredBox(
67-
color: Colors.blueGrey,
68-
child: Row(
69-
mainAxisAlignment: MainAxisAlignment.spaceEvenly,
70-
children: <Widget>[
71-
TextButton(
72-
onPressed: () => switchToPage(0),
73-
child: Text(
74-
'Classify',
75-
style: titleIndex == 0 ? activeTextStyle : inactiveTextStyle,
68+
bottomNavigationBar: SizedBox(
69+
height: 50,
70+
child: ColoredBox(
71+
color: Colors.blueGrey,
72+
child: Row(
73+
mainAxisAlignment: MainAxisAlignment.spaceEvenly,
74+
children: <Widget>[
75+
TextButton(
76+
onPressed: () => switchToPage(0),
77+
child: Text(
78+
'Classify',
79+
style: titleIndex == 0 ? activeTextStyle : inactiveTextStyle,
80+
),
7681
),
77-
),
78-
TextButton(
79-
onPressed: () => switchToPage(1),
80-
child: Text(
81-
'Embed',
82-
style: titleIndex == 1 ? activeTextStyle : inactiveTextStyle,
82+
TextButton(
83+
onPressed: () => switchToPage(1),
84+
child: Text(
85+
'Embed',
86+
style: titleIndex == 1 ? activeTextStyle : inactiveTextStyle,
87+
),
8388
),
84-
),
85-
],
89+
TextButton(
90+
onPressed: () => switchToPage(2),
91+
child: Text(
92+
'Detect Languages',
93+
style: titleIndex == 2 ? activeTextStyle : inactiveTextStyle,
94+
),
95+
),
96+
],
97+
),
8698
),
8799
),
88100
);

0 commit comments

Comments
 (0)