source: trunk/src/org/apollo/agents/MelodySearch.java@ 1434

Last change on this file since 1434 was 1434, checked in by bln4, 5 years ago

Implementation of ProfileManager. Refactor + additional content for how new profiles are created. The refactoring split out the creation of the default profile from user profiles. Refactoring revealed a long term bug that was causing user profiles to generate with incorrect information. The additional content fixed this bug by introducing the ${USER.NAME} variable, so that the default profile frameset can specify resource locations located in the users resource directory.

org.expeditee.auth.AuthenticatorBrowser
org.expeditee.auth.account.Create
org.expeditee.gui.Browser
org.expeditee.gui.management.ProfileManager
org.expeditee.setting.DirectoryListSetting
org.expeditee.setting.ListSetting
org.expeditee.settings.UserSettings

Implementation of ResourceManager as a core location to get resources from the file system. Also the additional variable ${CURRENT_FRAMESET} to represent the current frameset, so that images can be stored in the directory of the current frameset. This increases portability of framesets.

org.expeditee.gui.FrameIO
org.expeditee.gui.management.ResourceManager
org.expeditee.gui.management.ResourceUtil
Audio:

#NB: Audio used to only operate on a single directory. This has been updated to work in a same way as images. That is: when you ask for a specific resouce, it looks to the user settings to find a sequence of directories to look at in order until it manages to find the desired resource.


There is still need however for a single(ish) source of truth for the .banks and .mastermix file. Therefore these files are now always located in resource-<username>\audio.
org.apollo.agents.MelodySearch
org.apollo.audio.structure.AudioStructureModel
org.apollo.audio.util.MultiTrackPlaybackController
org.apollo.audio.util.SoundDesk
org.apollo.gui.FrameLayoutDaemon
org.apollo.io.AudioPathManager
org.apollo.util.AudioPurger
org.apollo.widgets.FramePlayer
org.apollo.widgets.SampledTrack

Images:

org.expeditee.items.ItemUtils

Frames:

org.expeditee.gui.FrameIO

Fixed a error in the FramePlayer class caused by an incorrect use of toArray().

org.apollo.widgets.FramePlayer


Added several short cut keys to allow for the Play/Pause (Ctrl + P), mute (Ctrl + M) and volume up/down (Ctrl + +/-) when hovering over SampledTrack widgets.

org.apollo.widgets.SampledTrack


Changed the way that Authenticate.login parses the new users profile to be more consistance with other similar places in code.

org.expeditee.auth.account.Authenticate


Encapsulated _body, _surrogateItemsBody and _primaryItemsBody in Frame class. Also changed getBody function to take a boolean flag as to if it should respect the current surrogate mode. If it should then it makes sure that labels have not changed since last time getBody was called.

org.expeditee.gui.Frame
File size: 13.2 KB
Line 
1package org.apollo.agents;
2
3import java.io.File;
4import java.io.FileNotFoundException;
5import java.io.IOException;
6import java.util.Collections;
7import java.util.LinkedList;
8import java.util.List;
9
10import javax.sound.sampled.AudioFormat;
11
12import org.apollo.ApolloSystem;
13import org.apollo.meldex.DynamicProgrammingAlgorithm;
14import org.apollo.meldex.McNabMongeauSankoffAlgorithm;
15import org.apollo.meldex.MeldexConversion;
16import org.apollo.meldex.Melody;
17import org.apollo.meldex.StandardisedMelody;
18import org.apollo.meldex.WavSample;
19import org.apollo.util.ExpediteeFileTextSearch;
20import org.apollo.util.TextItemSearchResult;
21import org.apollo.widgets.LinkedTrack;
22import org.apollo.widgets.SampledTrack;
23import org.apollo.widgets.TrackWidgetCommons;
24import org.expeditee.agents.SearchAgent;
25import org.expeditee.core.Colour;
26import org.expeditee.gui.DisplayController;
27import org.expeditee.gui.Frame;
28import org.expeditee.gui.FrameIO;
29import org.expeditee.gui.management.ResourceManager;
30import org.expeditee.items.Item;
31import org.expeditee.items.ItemUtils;
32import org.expeditee.items.widgets.Widget;
33import org.expeditee.items.widgets.WidgetCorner;
34import org.expeditee.items.widgets.WidgetEdge;
35import org.expeditee.settings.folders.FolderSettings;
36
37/**
38 * Performs a melody search against track widgets within a whole frameset.
39 *
40 * Uses meldex. Thanks David Bainbridge.
41 *
42 * The agent runs a query on the given track widget that launched it.
43 * If the track launches it it does a full search for all tracks on the current frameset.
44 *
45 * @author Brook Novak
46 *
47 */
48public class MelodySearch extends SearchAgent {
49
50 private long firstFrame = 1;
51
52 private long maxFrame = Integer.MAX_VALUE;
53
54 /** Either querry from raw audio or track widget: */
55 private SampledTrack querryTrack = null;
56
57 /** Either querry from raw audio or track widget: */
58 private byte[] querryRawAudio = null;
59 private AudioFormat querryRawAudioFormat = null;
60
61 public static final String MELODY_METAFILE_PREFIX = ".";
62 public static final String MELODY_METAFILE_SUFFIX = ".mel";
63
64 /** The default score cut-off for omitting bad results above this value*/
65 private static final float DEFAULT_SCORE_THRESHOLD = 400.0f;
66
67 public MelodySearch() {
68 super("MelodySearch");
69 }
70
71 /**
72 *
73 * @param firstFrame
74 * The first frame number to start searching from (inclusive)
75 *
76 * @param maxFrame
77 * The max frame number to start searching from (inclusive)
78 */
79 public MelodySearch(long firstFrame, long maxFrame) {
80 super("MelodySearch");
81 this.firstFrame = firstFrame;
82 this.maxFrame = maxFrame;
83 }
84
85 public void useRawAudio(byte[] querryRawAudio, AudioFormat querryRawAudioFormat) {
86
87 if (querryRawAudio == null || querryRawAudioFormat == null) return;
88 this.querryRawAudio = querryRawAudio;
89 this.querryRawAudioFormat = querryRawAudioFormat;
90 }
91
92 /**
93 * {@inheritDoc}
94 */
95 @Override
96 public boolean initialise(Frame frame, Item item)
97 {
98 if (!super.initialise(frame, item)) return false;
99
100 // Get the track to querry .. if given one
101 querryTrack = null;
102
103 if (item != null) {
104
105 Widget iw = null;
106
107 if (item instanceof WidgetCorner) {
108
109 iw = ((WidgetCorner)item).getWidgetSource();
110 } if (item instanceof WidgetEdge) {
111
112 iw = ((WidgetEdge)item).getWidgetSource();
113 }
114
115 if (iw != null && iw instanceof SampledTrack) {
116 querryTrack = (SampledTrack)iw;
117 }
118
119 }
120
121 return true;
122 }
123
124 /**
125 * {@inheritDoc}
126 */
127 @Override
128 protected Frame process(Frame frame) {
129
130 try {
131 if(frame == null) {
132 frame = FrameIO.LoadFrame(_startName + '0');
133 }
134
135 String path = frame.getPath();
136
137 int count = FrameIO.getLastNumber(_startName);
138
139 String trackPrefix = ItemUtils.GetTag(ItemUtils.TAG_IWIDGET) + ": ";
140 String linkedTrackPrefix = trackPrefix;
141
142 trackPrefix += SampledTrack.class.getName();
143 linkedTrackPrefix += LinkedTrack.class.getName();
144
145 Melody querryMelody = null;
146 // Maps FrameName -> MelodySearchResult
147 List<MelodySearchResult> melodyScores = new LinkedList<MelodySearchResult>();
148
149 // If querrying a track widget then get its melody
150 if (querryTrack != null) {
151 assert(querryRawAudio == null);
152 assert(querryRawAudioFormat == null);
153
154 try {
155 querryMelody = MeldexConversion.toMelody(querryTrack);
156 } catch (IOException e) {
157 e.printStackTrace();
158 } catch (OutOfMemoryError ex) {
159 ex.printStackTrace();
160 }
161
162 if (querryMelody == null) { // abort - failed to get audio
163 overwriteMessage("Melody search aborted: Failed to load tracks audio");
164 _results.addText("Melody search aborted: querry data not good enough to search with",
165 Colour.RED, null, null, false);
166 _results.addText("Click here for help on melody searches",
167 Colour.FromRGB255(0, 180, 0), ApolloSystem.HELP_MELODYSEARCH_FRAMENAME, null, false);
168
169 _results.save();
170 return null;
171 }
172
173 // If querrying raw audio then get its melody
174 } else if (querryRawAudio != null && querryRawAudioFormat != null) {
175 assert(querryTrack == null);
176
177 try {
178 querryMelody = MeldexConversion.toMelody(querryRawAudio, querryRawAudioFormat);
179 } catch (IOException e) {
180 e.printStackTrace();
181 } catch (OutOfMemoryError ex) {
182 ex.printStackTrace();
183 }
184
185 if (querryMelody == null) { // abort - failed to get audio
186 overwriteMessage("Melody search aborted: Failed to proccess querry data");
187 _results.addText("Melody search aborted: querry data not good enough to search with", Colour.RED, null, null, false);
188 _results.addText("Click here for help on melody searches",
189 Colour.FromRGB255(0, 180, 0), ApolloSystem.HELP_MELODYSEARCH_FRAMENAME, null, false);
190 _results.save();
191 return null;
192 }
193
194
195 }
196
197 // Support range searching... i.e. frame100 - frame500
198 for (long i = firstFrame;i <= maxFrame && i <= count; i++) {
199
200 // Has requested stop?
201 if (_stop) {
202 break;
203 }
204
205 String frameName = _startName + i;
206
207
208 overwriteMessage("Searching " + frameName); // RISKY
209 // Note: cannot invoke later otherwise can congest the swing queue!
210
211 // Perform prefix search
212 List<TextItemSearchResult> results = null;
213 try {
214
215 String fullpath = getFullPath(frameName, path);
216
217 if (fullpath != null) {
218 results = ExpediteeFileTextSearch.prefixSearch(
219 fullpath,
220 new String[] {trackPrefix, linkedTrackPrefix});
221 }
222
223 } catch (FileNotFoundException e) {
224 e.printStackTrace();
225 } catch (IOException e) {
226 e.printStackTrace();
227 }
228
229 if (results == null) continue; // frame does not exist or an error occured
230
231 // If the frame exists / succeeded to perform a prefix search then increment the
232 // searched frame count...
233 _frameCount++;
234
235 // Is doing a full search for all tracks?
236 if (querryMelody == null) {
237 if(!results.isEmpty()) {
238 _results.addText(frameName + "(" + results.size() + ")", null, frameName, null, false);
239 DisplayController.requestRefresh(true);
240 }
241
242 } else { // meldex querry
243
244 MelodySearchResult bestScore = null;
245
246 for (TextItemSearchResult res : results) {
247 if (res.data == null) continue;
248
249 // should this widget be indexed?
250 if (res.containsData(SampledTrack.META_DONT_INDEX_AUDIO_TAG)) continue;
251
252 // get the local filename
253 String localFilename = null;
254 String trackName = null;
255
256 // Parse meta
257 for (String data : res.data) {
258 if (data.startsWith(SampledTrack.META_LOCALNAME_TAG) &&
259 data.length() > SampledTrack.META_LOCALNAME_TAG.length()) {
260 localFilename = data.substring(SampledTrack.META_LOCALNAME_TAG.length());
261 if (trackName != null) break;
262 } else if (data.startsWith(TrackWidgetCommons.META_NAME_TAG) &&
263 data.length() > TrackWidgetCommons.META_NAME_TAG.length()) {
264 trackName = data.substring(TrackWidgetCommons.META_NAME_TAG.length());
265 if (localFilename != null) break;
266 }
267
268 }
269
270 if (localFilename == null)
271 continue;
272
273 // Safety: omit this if it is infact the very widget we are searching
274 if (querryTrack != null && querryTrack.getLocalFileName().equals(localFilename))
275 continue;
276
277 Melody testMelody = null;
278
279 // Get cached melody from file if it is up to date
280 String metaFileRelativePath = MELODY_METAFILE_PREFIX + localFilename + MELODY_METAFILE_SUFFIX;
281 String metaFilePath = ResourceManager.getAudioResource(metaFileRelativePath, DisplayController.getCurrentFrame()).getAbsolutePath();
282
283 //File localFile = new File(AudioPathManager.AUDIO_HOME_DIRECTORY + localFilename);
284 File localFile = ResourceManager.getAudioResource(localFilename, DisplayController.getCurrentFrame());
285 if (!localFile.exists()) continue;
286
287 File metaFile = new File(metaFilePath);
288
289 // If there is a metafile that contains the serialized melody and is up to date...
290 if (metaFile.exists() && metaFile.lastModified() >= localFile.lastModified()) {
291
292 try {
293 StandardisedMelody sm = StandardisedMelody.readMelodyFromFile(metaFilePath);
294 if (sm != null && sm instanceof Melody) {
295 testMelody = (Melody)sm;
296 }
297 } catch (FileNotFoundException e) {
298 e.printStackTrace();
299 } catch (IOException e) {
300 e.printStackTrace();
301 } catch (ClassNotFoundException e) {
302 e.printStackTrace();
303 }
304
305 }
306
307 if (_stop) break;
308
309 // If did not manage to load any melody meta ... calculate the meta
310 if (testMelody == null) {
311
312 // Load wave file
313 WavSample sampleTrack = new WavSample();
314 if (!sampleTrack.loadFromFile(localFile)) {
315 continue;
316 }
317 if (_stop) break;
318
319 // Transcribe melody
320 try {
321 testMelody = MeldexConversion.toMelody(sampleTrack);
322 } catch (IOException ex) {
323 ex.printStackTrace();
324 } catch (OutOfMemoryError ex) {
325 ex.printStackTrace();
326 }
327
328 if (testMelody == null) continue;
329 if (_stop) break;
330
331 // Save meta
332 if (metaFile.exists()) metaFile.delete();
333
334 try {
335 StandardisedMelody.writeMelodyToFile(metaFilePath, testMelody);
336 } catch (FileNotFoundException e) {
337 e.printStackTrace();
338 } catch (IOException e) {
339 e.printStackTrace();
340 }
341
342 }
343
344 assert(testMelody != null);
345 assert(querryMelody != null);
346
347 // Omit bogus melody transcriptions for which audio must not of have enough rests
348 // to give a strong enough representation. This avoids bogus meta getting better results
349 // than something actually more meaningful.
350 if (testMelody.getLength() <= 1) continue;
351
352 DynamicProgrammingAlgorithm dpa = new McNabMongeauSankoffAlgorithm();
353
354 float score = dpa.matchToPattern(querryMelody, testMelody, DynamicProgrammingAlgorithm.MATCH_ANYWHERE);
355
356 if (bestScore == null || bestScore.getScore() < score) {
357
358 bestScore = new MelodySearchResult(
359 frameName,
360 score,
361 trackName,
362 localFilename
363 );
364 }
365
366
367 } // next result in frame
368
369 if (_stop) {
370 break;
371 }
372
373 if (bestScore != null) {
374 melodyScores.add(bestScore);
375 }
376
377 }
378
379
380 } // Next frame
381
382
383 // Did do melody matching? add ordered results
384 if (querryMelody != null) {
385
386 float threshold = DEFAULT_SCORE_THRESHOLD; // TODO: Have tolerance option
387
388 // Order results descending
389 Collections.sort(melodyScores);
390
391 if (melodyScores.isEmpty() || melodyScores.get(0).getScore() > threshold) {
392 _results.addText("No matches", Colour.RED, null, null, false);
393 _results.addText("Click here to find out how to improve your melody searches",
394 Colour.FromRGB255(0, 180, 0), ApolloSystem.HELP_MELODYSEARCH_FRAMENAME, null, false);
395 } else {
396
397 int rank = 1;
398 for (MelodySearchResult melRes : melodyScores) {
399
400 if (melRes.getScore() <= threshold) {
401
402 String name = melRes.getTrackName();
403 if (name == null || name.length() == 0)
404 name = "Unnamed";
405
406 _results.addText(rank + ": " + melRes.getParentFrame() + " ("
407 + name + ")",
408 null,
409 melRes.getParentFrame(), null, false);
410
411
412 }
413
414 rank ++;
415
416 }
417
418 }
419
420 }
421
422 if (_stop) {
423 _results.addText("Search cancelled", Colour.RED, null, null, false);
424 }
425
426 // Spit out result(s)
427 _results.save();
428
429 String resultFrameName = _results.getName();
430 if (_clicked != null)
431 _clicked.setLink(resultFrameName);
432
433 return _results.getFirstFrame();
434
435 }
436 finally {
437 querryRawAudio = null; // Free memory
438 }
439
440 }
441
442 /**
443 * Gets the full path from a given framename and path
444 * @param frameName
445 * @param path
446 * @return
447 */
448 private String getFullPath(String frameName, String path) {
449
450 String fullPath = null;
451 if (path == null) {
452 for (String possiblePath : FolderSettings.FrameDirs.getAbsoluteDirs()) {
453 fullPath = FrameIO.getFrameFullPathName(possiblePath, frameName);
454 if (fullPath != null)
455 break;
456 }
457 } else {
458 fullPath = FrameIO.getFrameFullPathName(path, frameName);
459 }
460
461 return fullPath;
462
463 }
464}
Note: See TracBrowser for help on using the repository browser.