Quick Script to Migrate from SimpleNotes

Moving hundreds or thousands of notes from SimpleNotes (to Apple Notes in my case)? This script aught to help clean up a few things for a better structure after importing.

  • Converts tags in *.txt files to #tag format for easier searching. Preserves original modified timestamp.
  • Updates the modified timestamps to a UTC offset (if specified)
  • Moves each note file into a subfolder based on its first tag (if specified)

This will only modify txt files, and should be run against an extracted export of notes from SimpleNote, so it should be fairly safe. But keep the archive and/or backups just in case. Use as follows:

chmod +x convert-tags.php
./convert-tags.php --path ./notes --timezone "-0500" --usefolders

Then import the notes folder through Notes.app.


#!/usr/bin/env php
<?
/**
 * This script helps migrate from SimpleNotes to Apple Notes 
 * (and potentially others) in a few ways:
 * 
 * 1) Converts tags in exported SimpleNotes *.txt files
 * to #tag format for easier searching. Preserves original modified timestamp.
 * 2) Updates the modified timestamps to a UTC offset (if specified)
 * 3) Moves each note file into a subfolder based on its first tag (if specified)
 * 
 * Arguments: (in this order)
 * --path - Location of the exported txt files
 * --timezone - an offset value (eg: -0500) from UTC to correct timestamps
 * --usefolders - moves notes into a subfolder based on their first tag
 */

$tags_pattern = '/Tags:\n\s\s(.+)$/';
$updated_files = 0;
$updated_timestamps = 0;
$moved_files = 0;
$errors = 0;
$skipped_files = 0;

$options = getopt(null, ['path:', 'timezone:', 'usefolders:']);

if (!isset($options['path']) || is_dir($options['path']) !== true) {
    throw new Exception('--path argument is missing or invalid.');
}

$base = trim($options['path'], '/') . '/';

if ( isset($options['timezone']) ) {
    if ( preg_match('/[-+]\d\d:?\d\d/', $options['timezone']) === 1 ) {
        // Calculate interval in seconds for the supplied timezone offset 
        $dtz_utc = new DateTimeZone('UTC');
        $dtz_offset = new DateTimeZone($options['timezone']);
        $dt_utc = new DateTime('now', $dtz_utc);
        $utc_offset_seconds = $dtz_offset->getOffset($dt_utc);

        $di_offset = DateInterval::createFromDateString( abs($utc_offset_seconds) . ' seconds');

        if ($utc_offset_seconds < 0) {
            $di_offset->invert = 1;
        }

        echo "Modified timestamps will be adjusted by {$utc_offset_seconds} seconds.\n";
    } else {
        throw new Exception('Timezone format should be -0500.');
    }
} else {
    echo "Not adjusting timezone of modified timestamps.\n";
    $di_offset = null;
}

$use_folders = in_array('--usefolders', $argv);

if ($use_folders) {
    echo "Moving notes into subfolders\n";
} else {
    echo "Leaving notes in $base\n";
}


if ($handle = opendir($base)) {
    echo "Updating *.txt in $base\n";
    echo "---\n";

    while (false !== ($entry = readdir($handle))):

        // Skip non-txt files
        if (preg_match('/^.*\.txt$/', $entry) !== 1) {
            echo "Skipping '$entry'\n";
            $skipped_files++;
            continue;
        }

        if ($contents = file_get_contents($base . $entry) ):
            $tags = [];

            // Find the tags string, replace with new format, and stash tags for later use
            $new_contents = preg_replace_callback($tags_pattern, function ($matches) use (&$tags) {
                $tags = explode(', ', $matches[1]);
                return '#' . implode(' #', $tags);
            }, $contents);

            $original_mod_time = filemtime($base.$entry);

            if ($contents !== $new_contents) {
                if (file_put_contents($base . $entry, $new_contents) !== false) {
                    echo "Updated '$entry'\n";
                    $updated_files++;
                }
            }
            
            // Resetting modified timestamp back to original 
            $date = new DateTime( '@'.$original_mod_time );
            
            // Correct modified timestamp's timezone
            if ($di_offset) {
                $date->add($di_offset);
            }
            
            // Set the timestamp
            if ( touch($base.$entry, $date->getTimestamp()) ) {
                $updated_timestamps++;
            }

            // If specified, move the note into a folder based on the first tag
            if ($use_folders && count($tags)) {
                $folder = $base . $tags[0] . '/';

                if (!is_dir($folder)) {
                    mkdir ($folder);
                }

                if ( rename($base.$entry, $folder.$entry) ) {
                    echo "Moved '$entry' to '" . $folder . $entry . "'\n";
                    $moved_files++;
                } else {
                    echo "*** Could not move '$entry' to '" . $folder . $entry . "'\n";
                    $errors++;
                }
            }
        endif;
    endwhile;

    closedir($handle);
}

echo "---\n";
echo "Skipped $skipped_files files/folders.\n";
echo "Updated $updated_files files.\n";
if ($use_folders) {
    echo "Moved $moved_files files.\n";
}
if ($di_offset) {
    echo "Modified and reset timestamps on $updated_timestamps files.\n";
} else {
    echo "Reset timestamps on $updated_timestamps files.\n";
}
if ($errors) {
    echo "** Encountered $errors errors.\n";
}

iOS Shortcuts for Firefly III

I built a couple of iOS shortcuts for Firefly III (a self-hosted personal finance web app) to let me add transactions on the go. I’ve shared some more details and iCloud links to the shortcuts below.

The first Shortcut is more or less a function that returns a list of accounts. That list is used in the second shortcut to submit the new transaction to Firefly. I wanted to keep the process as quick as possible, so that shortcut requests the least information possible to submit the transaction.


I’ve only just recently switched to using Firefly III; prior to that I had been using Mint for about 8 years. I’ve been increasingly uncomfortable with giving up that much data – and my bank credentials – for a number of years, so I’d been on the lookout for a replacement for quite a while. I’ve probably tried them all, but for one reason or another I couldn’t find an app that fit my requirements.

Firefly is a bit of a change in workflow over Mint, but I’ve found that it’s encouraging me to take a more active role in managing my finances. In part because it doesn’t recommend auto-importing transactions. All-in-all though, it’s working quite well so far.

There are a few places where I’d do things differently in Firefly, but part of the appeal (for me at least) is that it’s built on PHP (Laravel, to be specific) so I could feasibly contribute to it, or at least modify my own fork. It also has a nice REST API and great documentation to go with it, which of course is what these shortcuts are using.

I knew I wanted the ability to add transactions on the go, but logging into the web app is a bit too much friction while waiting in line for a $2 coffee. And let’s face it, if the transaction isn’t added immediately I’ll probably forget about it. So that’s what I’ve solved with these shortcuts.

Shortcut Requirements

  • Your Firefly III instance must be accessible over the internet (I would not do this without using HTTPS)
  • The API supports using a Personal Access Token rather than OAuth, which must be created in Firefly > Options > Profile
  • The Firefly URL and token must be added to both shortcuts

Shortcut can’t find an account?

If no accounts are returned, it’s most likely that the Personal Access Token was denied; either the token was incorrect or, as is common after a Firefly update, the Personal Access Token was essentially deactivated. I often have to delete and recreate it after updates.

Shortcut Downloads

I’ve shared these via iCloud Drive. Load up this post and download the links on an iOS device. Plug your Personal Access Token and the URL to your Firefly III instance into both shortcuts, then test out the Accounts shortcut. It should return a list the asset accounts in Firefly III, along with the current balances.

If you’re not on an iOS device, here’s the what the two shortcuts look like—warning, they’re long:

Screenshot of an iOS shortcut to display a listing of accounts and balances from Firefly III
Accounts shortcut

Screenshot of a Firefly III shortcut that adds a new transactions to a selected account.
New transaction shortcut

Self-Hosting Fathom analytics behind Apache

I’m hosting Fathom on my domain at /fathom. It runs its own web server, so I’ve done this using a reverse proxy that makes the Fathom server accessible at that virtual directory. Their self-hosting instructions do have an example configuration for using Fathom with a reverse proxy through NGINX, but not Apache.

Fortunately the idea’s the same, so adapting it for Apache doesn’t involve too much. It does mean modifying the httpd.conf (or appropriate site-specific config file), and enabling a few additional Apache modules.

The Apache modules needed are mod_proxy, mod_proxy_http and mod_substitute. These can be enabled using a2enmod and restarting Apache:

sudo a2enmod proxy proxy_http substitute && sudo service apache2 restart

Now to enable the reverse proxy, I’ve added a new Location directive the VirtualHost in my site’s config file:

 <Location "/fathom">
        # Host and port of the Fathom instance
        ProxyPass http://127.0.0.1:8999

        # Updates redirect headers in responses from Fathom
        ProxyPassReverse http://127.0.0.1:8999

        # Send original Host header to Fathom
        ProxyPreserveHost On

        # Enables replacing of URLs in HTML elements
        ProxyHTMLEnable On

        # Load default mapping of attribute -> HTML elements that need replacing
        Include /etc/apache2/mods-available/proxy_html.conf

        # Define the regex in which to use to replace URLs in HTML
        ProxyHTMLURLMap ^(.*)$ /fathom/$1 [R]

        # Update the location of the /api/ calls in JavaScript responses from Fathom
        AddOutputFilterByType SUBSTITUTE application/javascript
        Substitute "s|/api/|/fathom/api/|niq"
</Location>

A little more on what each of those directives are doing:

  • ProxyPass http://127.0.0.1:8999
    Creates an entry point at the URL in the Location directive (the first line, in this case it’s /fathom), which in turn forwards those requests on to the Fathom server defined in this ProxyPass directive.
  • ProxyPassReverse http://127.0.0.1:8999
    Replaces redirects sent from Fathom (in the Location header) with the location that the client browser can access. In other words, when Fathom tries to redirect to an internal Login page (127.0.0.1:8999/login), this directive tells Apache to replace that with an externally-accessible URL (/fathom/login).
  • ProxyPreserveHost On
    Fathom’s example NGINX configuration explicitly passes on the Host header from the proxy to the upstream server (Fathom), so it’s safe to assume we should too.
  • ProxyHTMLEnable On
    Include /etc/apache2/mods-available/proxy_html.conf
    ProxyHTMLURLMap ^(.*)$ /fathom/$1 [R]

    These next few directives replace URLs in various HTML elements before sending them back to the browser. For example, <img> and <script> tags will need to be updated to point to the virtual directory (subfolder). Including the proxy_html.conf file imports a default mapping of attributes to elements so that Apache knows which attributes on which elements to update.
  • AddOutputFilterByType SUBSTITUTE application/javascript
    Substitute "s|/api/|/fathom/api/|niq"

    These final directives tell Apache to update URLs in JavaScript files. This is specifically needed for Fathom’s assets/js/scripts.js file, which makes multiple requests to files in the API folder. Just a note that Substitute uses a sed-like syntax for replacing text.