FlightGear wiki:Instant-Refs

Instant-Cquotes user script in Firefox

Screenshot showing the web-extension port of the InstantRefs user-script in FireFox.

Instant-Cquotes screenshot prototyping runtime format selection

The Instant-Refs (renamed in 11/2016, originally, Instant-cquotes) script is a browser addon (as of late 2017, a so called web-extension) implemented in JavaScript that started out as a user script in order to convert excerpts (created via copy&paste) from FlightGear forum or mailing list postings into MediaWiki markup/quotes to be used on the FlightGear wiki.

It is being developed and maintained by a group of volunteers involved in maintaining the wiki and in trying to provide more up-to-date information to end-users who may not be as involved in the various FlightGear-related communication channels.

Background and motivation

FlightGear's development is, at best, "self-coordinated", meaning that contributors discuss ideas and make proposals to contribute in a certain fashion and then team up to implement certain features and building blocks, often just temporarily.

Unfortunately, due to a lack of development manpower, many ideas are not implemented immediately; it is, thus, important to know their pros and cons, as well as who originally proposed them and/or might help with their implementation, even after a long time, preferably with links to the original discussions so that new contributors can decide whether to get involved in some effort or not. The project documentation, however, is in great need of improvement and usually significantly lacking behind: many core developers update it seldomly, if not anymore, as it takes time to write it, as well as an understanding of the inner workings of a complex system such as FlightGear. Furthermore, this task might not be attractive due to its short term impact on the project, and it is overwhelming to think about creating documentation that would address the needs of many different kinds of contributors with different backgrounds, experience levels and goals.

Forum and mailing lists discussions have therefore become the only up-to-date (albeit difficult to filter) source of information about recent development progress; this makes it tricky to know what is going on, what needs fixing, what were the decisions taken by the developers.

The aim of the Instant-Refs script is to help wiki editors to copy relevant excerpts from such sources, formatting them as proper quotations, and help bootstrap new articles collecting them until a dedicated rewrite is made. It can also be used to reuse announcements to update the changelogs, newsletter or the Release plan/Lessons learned page.

After being away from the wiki system for some time it becomes more of an effort to re-learn how to start a new file and figure out how to format it , with what headings, etc. Sometimes it is almost too much to do the original write up of the original post and all the work of that layout and effort to have to also duplicate it in a wiki article. If I was a little more current and affluent with the wiki editor, I might not be so hesitant to create the work there instead of the forum.

In a few cases, such collections of quotes helped not only create bootstrap new articles, but even actual features, too.

In other cases, quotes have been used to update documentation of features whose maintainers may not be actively involved in FlightGear (e.g. Rembrandt), to help document discussions that are taking place in the meantime, and provide some background information for people interested in the corresponding feature.

Configuration

GreaseMonkey menu shown in FireFox with instanct cquotes menu items

The configuration dialog for the Instant-Cquotes script

As of version 0.30, a dedicated configuration dialog is in the process of being added, so that certain script features can be dynamically configured, without having to edit the script. For now, this is merely a placeholder that provides a checkbox to easily enable/disable the debug mode. In the future, we are hoping to also expose other features this way.

Usage

Instant-Cquotes script, with updates contributed by Red Leader

Screenshot showing Instant-Cquotes 0.30 at work

1. Go to some mailing list archive URL, for example [1] or any forum message, such as Re: Get objects to show up on Map/Radar topic on the forum .
2. Select the relevant portion of text.
3. When you release the mouse button, a box will appear containing the converted text (for now, mainly properly-referenced quotes for the wiki).
4. The text will be automatically selected and copied to the clipboard.
5. Paste the text into the desired wiki page.

For example, by selecting part of the forum post in the link above you can get the following quotation:

The upcoming FlightGear version (3.2) will contain a canvas-based map dialog, including a modular "plugin" system for creating custom map layers and charts with roughly ~50 lines of code, most of it boilerplate. This is entirely XML/Nasal based (scripted) - symbols can be pretty much anything, raster or vector images (png or svg), but even animated. Styling can be customied, too. For more info, I suggest to check out: MapStructure#Porting the map dialog — Hooray (Jun 14th, 2014). Re: Get objects to show up on Map/Radar. (powered by Instant-Cquotes)

On quoting

See FlightGear wiki:Quoting Guidelines for the main article about this subject.

Using the Instant-Cquotes script is a good way to bootstrap and write some preliminary notes; however, while quotes might be useful to understand how undocumented subsystems and features work and are definitely better than nothing, they are not meant to replace proper, structured and well-written wiki articles.

One way to convert pages bootstrapped using quotes is to extract relevant information from them and keep citations only as references; in case important details are missing, they can be asked for on the mailing lists (on the forum, the chance to get a complete answer might be lower). Another option might be moving the quotes to the Talk page for each entry, which would preserve the sources without clogging up the articles.

As a matter of fact, the whole paragraph above was assembled using this approach; to see for yourself, look up the references at the end of this page. For another example, see TerraSync#News.

Development

Note  A Chrome/Chromium-specific extension that will not need Tampermonkey installed is under development.

Resources

• https://www.mediawiki.org/wiki/API:Changing_wiki_content
• https://www.mediawiki.org/wiki/API:Edit

Adding sources

Adding a new source is pretty straightforward if you understand how xpath and regexes work - basically, you only need an archive (e.g. gmane), and then determine the xpath of each relevant field, as well as the regular expression to process the extracted fields (optional).

The basic steps are these:

1. open the user script in an editor
2. navigate to the meta header of the user script
3. add a new URL to the top of the script, e.g. by copying/adapting an existing line like this:

// @match       https://sourceforge.net/p/flightgear/mailman/*

Once copied, add the new URL:

// @match       http://thread.gmane.org/gmane.games.flightgear.devel/*

Next, you need to navigate to the configuration hash to add a new website to it.

Again, it makes sense to simply take an existing configuration hash and adapt it as needed (ignore/omit the tests vector for now by keeping it empty):

Caution  the following example may meanwhile be outdated, so be sure to look at the actual code instead

  'Sourceforge Mailing list': {
    enabled: true,
    type: 'archive',
    event: 'document.onmouseup', // when to invoke the event handler
    event_handler: instantCquote, // the event handler to be invoked
    url_reg: '^(http|https)://sourceforge.net/p/flightgear/mailman/.*/',
    content: {
      xpath: 'tbody/tr[2]/td/pre/text()',
      selection: getSelectedText,
      idStyle: /msg[0-9]{8}/,
      parentTag: [
        'tagName',
        'PRE'
      ],
    transform: [] // vector with transformation callbacks
    }, // content recipe
    // vector with tests to be executed for sanity checks (unit testing)
    tests: [
    ], // end of vector with self-tests
    // regex/xpath and transformations for extracting various required fields
    author: {
      xpath: 'tbody/tr[1]/td/div/small/text()',
      transform: [extract(/From: (.*) <.*@.*>/)]
    },
    title: {
      xpath: 'tbody/tr[1]/td/div/div[1]/b/a/text()'
      transform: []
    },
    date: {
      xpath: 'tbody/tr[1]/td/div/small/text()',
      transform: [extract(/- (.*-.*-.*) /)]
    },
    url: {
      xpath: 'tbody/tr[1]/td/div/div[1]/b/a/@href',
      transform: [prepend('https://sourceforge.net')]
    }
  }, // end of mailing list profile

Now, we need to review/adapt the profile according to the new archive we'd like to see supported.

for starters, that means:

• changing the name of the profile, e.g. to read gmane (instead of sourceforge)
• change the url_reg field to the gmane URL (this can be a regular expression)

Next, it makes sense to use an XPath checker, so that we can look up the xpath expression for various HTML elements, and add those to the configuration hash above.

For testing purposes, you will probably go to the setup dialog and enable the DEBUG mode, and use your browser's console to see what is going on.

Getting involved

While having some experience with JavaScript/HTML and jQuery will definitely be useful, JavaScript is close enough to FlightGear scripting (Nasal), so that people can get involved pretty easily.

Most maintenance work will typically involve reviewing/maintaining a few configuration hashes, that contain meta information for each supported archive (mailing list/forum).

Usually, each hash contains a combination of xpath/regex expressions to look up the relevant information, as well as vector of optional transformations that are applied (in order) to convert contents to a different format (e.g. dates).

In addition, there is growing library of utility functions, a handful wrappers for useful stuff, for example:

• Host.dbLog(message_string) - log a message to the console if the DEBUG flag is set
• Host.download(url, callback, method='GET') - will download the URL and pass the downloaded content to the callback specified
• Host.downloadPosting(url, callback) - will download the posting URL and pass the extracted and transformed author/content and date fields in a hash to the callback specified, the URL must be one supported in the CONFIG hash (i.e. forum/sourceforge for now)
• Host.make_doc(string, type="text/html") - will turn a string/blob into a DOM that can be queried
• Host.eval_xpath(document, xpath_expression, type=XPathResult.STRING_TYPE) - will apply the xpath expression to the document specified, returning the requested type (defaulted to string)
• Host.set_persistent(key,value) - stores a key/value pair
• Host.get_persistent(key,default_value) - retrieves a value using the key specified, falling back to the default value specified

Self checks (unit testing)

For regression testing purposes, there's a dedicated "self check" dialog that will be extended over time. For now it will download a few profile/website specific postings using a vector called "tests" and then log the posting's title, author and date to the console.

The next step will be actually showing that information in the dialog itself - there's a separate helper function to accomplish that, so that the corresponding div layer can be updated with the results.

automatically executed sanity checks

Debug mode

Instant-Cquotes debug mode

AJAX (live page editing)

Work in progress This article or section will be worked on in the upcoming hours or days. See history for the latest developments.

• http://wiki.flightgear.org/api.php
• http://wiki.flightgear.org/api.php?action=parse&page=Frequently%20asked%20questions&prop=sections

For now, the setup dialog will try to obtain a login token for the wiki and show a message if successful.

In addition, the profile/website hash also contains a new wiki entry for the FlightGear wiki, whose event_handler callback will be invoked once the FG wiki is visited - the console/log will show a greeting, so that is where other code can be added - e.g. to help clean up/rewrite FGCquote-based articles automatically etc.

There is a vector of "modes", whose members are a hash containing trigger/handler fields, linked to two callbacks - the trigger callback can be used to check some condition, while the handler will be invoked if the trigger returns true.

This can be used to support an arbitrary number of modes, whose triggers are evaluated during page load - all triggers that return true, will have their handlers invoked, which is how the following snippet works to rewrite/augment wiki edit handles:

 'FlightGear.wiki': {
    type: 'wiki',
    enabled: false,
    event: 'document.onmouseup', // when to invoke the event handler
    event_handler: function () {
      console.log('FlightGear wiki handler active (waiting to be populated)');
      // this is where the logic for a wiki mode can be added over time (for now, it's a NOP)
    
    //for each supported mode, invoke the trigger and call the corresponding handler
    [].forEach.call(CONFIG['FlightGear.wiki'].modes, function(mode) {
      //dbLog("Checking trigger:"+mode.name);
      if(mode.trigger) {
        mode.handler();
      }
    });
      
    }, // the event handler to be invoked
    url_reg: '^(http|https)://wiki.flightgear.org', // ignore for now: not currently used by the wiki mode
    
    modes: [
      { name:'process-editSections',
        trigger: function() {return true;}, // match URL regex - return true for always match
       
        // the code implementing the mode
        handler: function() {
                
    var editSections = document.getElementsByClassName('mw-editsection');
    console.log('FlightGear wiki article, number of edit sections: '+editSections.length);
   
    // for now, just rewrite edit sections and add a note to them
   
     [].forEach.call(editSections, function (sec) {
       sec.appendChild(
         document.createTextNode(' (instant-cquotes is lurking) ')
       );
     }); //forEach section
        } // handler
       
       
      } // process-editSections
      // TODO: add other wiki modes below 
      
    ] // modes
    
  }, // end of wiki profile

See User:Red Leader/Sandbox/AJAX test

• https://www.mediawiki.org/wiki/API:Changing_wiki_content
• https://www.mediawiki.org/wiki/Extension:VisualEditor
• https://en.wikipedia.org/wiki/Wikipedia:Creating_a_bot

Issues/limitations

Bugs

• It's eating characters, apparently related to regex/xpath handling - e.g. words like "analyzing" are turned into "analying" [2] [3]

Non working URLs

• forum2wiki img conversion does not work for all embedded images, e.g. see https://forum.flightgear.org/viewtopic.php?f=4&t=25747&p=285956#p285956
• image matching/extraction: http://forum.flightgear.org/viewtopic.php?p=276221#p276221
• http://sourceforge.net/p/flightgear/mailman/message/34754961/
• http://forum.flightgear.org/viewtopic.php?f=18&t=27054&start=90#p273972 → selecting from “As promised, two sample installation sessions on Linux” to “That's it.” towards the end of the message causes Iceweasel (Firefox) 44.0.2 to display a dialog box reading “A script on this page may be busy, or it may have stopped responding. You can stop the script now, open the script in the debugger, or let the script continue.” The line below reads “Script: chrome://greasemonkey-modules/...quotes/instant_cquotes.user.js:544”. Choosing Continue doesn't help: the same message reappears a few seconds afterwards.

Feature requests & ideas

• resolve footers [4]: "I believe that because of [x] ... [x]: " -> "I believe that because of (x)"
• make transformations customizable/editable via the UI (think git links to repo url template)
• split into separate files [5][]
• replace temporary references:
  • yesterday/today/tomorrow
  • last/this/next week, month/year/release
• Article wizard, optionally append References section with Appendix
• Template integration to turn templates into interactive wizards to create/add to pages, for example:
  • Template:Canvas Snippet
  • Template:Nasal doc
• try to recognize list items [6] (heuristics: look for colon/asterisk, dashes and CR/LF)
• should add Template:News to the article dropdown for announcements [7]
• add a mode that will download screenshots from forum postings and automatically upload/categorize them, see Birds
• split the article dropdown into sections, and also populate it with the user's watchlist [8]
• mailing list templates, listed at [9]
• expose the cquote/ref markup via the UI so that it can be edited/customized and treated like a template Done (0.36+)
• identify common/repeated links and automatically create link/infrastructure templates and use those (should be straightforward using the AJAX mode) [10]
  • use simple naming heuristic: Author_title_date
• add a devel/maintainer mode where it will return the xpath for a selection [11] [12]
• move openlink,dblog helpers to Environment hash Done
• identify CLI arguments like --aircraft=c172p and wrap them in between code tags --aircraft=c172p [13] (note that we can simply download options.xml via openlink() and use that, which is kinda of neat...) (see downloadOptionsXML() in the code)
• introduce "layouts" (templates) for different purposes: newsletter, changelog, wiki article, The Manual (LaTex)  ?
• use wikipedia template if possible [14]
• the new tests vector could also contain vectors for tests to test the extract/transform* utilities, see Environment.APITests
• move environment specific APIs (browser, script host etc) into some kind of Environment hash to encapsulate things (Red Leader was working on a pure Chrome-only version at some point IIRC)
• encode script settings in created markup, for future processing/updating of quotes
• look up [x] references and replace with the corresponding link (titled) [15] [16]
  • convert footnotes into Abbr templates [17]
• support named refs for combining identical refs [18]
• adopt {{forum link}}
• implement a less obnoxious quoting mode, without quotes, where only the ref part would be added, e.g. see the example at Graphics Card Profiles (it's still 99% quotes, but much less annoying) Done
• attachment support: identify attachments and link to them: [19] [20] [21] [22]
• bulletin points: if there is a colon (:) followed by at least two dashes (-), split up everything after the colon to turn each dash into an asterisk (wiki markup for bulletin points), followed by a newline [23]
• generic URL/template matching, e.g. for for sourceforge commit IDs
• make filters/conversions configurable via checkboxes (nowiki, wrap in alert/note boxes)
• make syntax highlighting configurable (language, mode) ?
• consider using something like the Roles template to turn contributor names into tooltips where contributor roles are shown (core dev, fgdata committer etc)
• introduce support for tag clouds to help categorize/classify related quotes
• consider making transformations optional/configurable using check boxes in the jQuery dialog
• add new input method, for quotes that need to be updated/converted (added script version specifically for this purpose), should also add extraction/processing date
• investigate why not all mailing list archives/postings are supported correctly: http://sourceforge.net/p/flightgear/mailman/message/8090479/ (problem traced to getPostID())
• explore having a ref-only mode without using cquotes, i.e. just copy/paste quotes with proper ref tags and a references section, to rewrite the whole thing (possibly with templates for different purposes, e.g. newsletter/changelog) Done
• should use {{forum link}} (Category:link templates)
• should be updated to use the new repo/flightgear file templates created by Johan & RedLeader Not done
• resolve links to forum threads to look up the title for the topic/posting, so that the posting's title can be used for those links, instead of just the URL - we can probably do that by making an AJAX call to open URLs asynchronously and extract the title for the thread/posting to come up with something like A call to developers-Lockheed -L188 Electra ([24])
• convert quoted bug tracker URLs to use the issue template on the wiki Not done
• do regex/xpath validation, and display any errors (e.g. template/theme changes on the forum would currently break the script)
• increased focus on supporting different output formats, maybe using a simple jQuery based wizard (wiki, forum, newsletter, changelog) Not done
• token matching for keywords/acronyms to link to the corresponding wiki articles (e.g. Nasal, Canvas, FG_ROOT, FG_HOME etc) Not done
• Add support for tickets, merge requests comments and FGUK forum. (also see FlightGear wiki talk:Instant-Cquotes#more sources)
• GET-encoded SID arguments should be stripped from forum URLs. Not done
• Links to repositories should be converted to use wiki templates. Not done
• The regexes used may fail if the HTML DOM of the source changes (e.g., phpBB/theme update)
  • Show a warning when that's the case. (see the self-check dialog)
  • Try multiple regexes in order. (see the self-check dialog)
• Use the script to update previously created Cquotes automatically
  • Instead of using the getSelection() helper, we could register a match for wiki.flightgear.org with action=edit set, so that we can directly process all text of an edited page, using AJAX calls to open the URL in the background. (see the self-check dialog, available via the greasemonkey menu)
  • See MW:API:Edit § Editing via Ajax

The script

• 

  Instant cquotes: Revamped user interface exposes script internals to make the script better configurable at runtime

• 

  Instant cquotes now features a simple built-in template editor with support for variable substitution, so that wiki templates can be more easily customized

• 

  Screenshot showing instant-cquotes with wikimedia API integration to fetch articles/sections and populate dropdown menus accordingly

• 

  Screenshot showing the JQuery-mode of the instant cquotes script with built-in support for fetching a user's wiki/watchlist to edit/update articles in a semi-automated fashion

Note  The userscript times are over meanwhile, the script has been ported to use the web extension framework - please don't update the user script. thanks !

References