4.0 blog

Author: Billy Charlton

docs/CHANGELOG-header.md

@@ -10,3 +10,4 @@ The list of changes is generated automatically based on the commit messages them
 - See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
 - The SimWrapper code repository containing the full commit history is on GitHub at <https://github.com/simwrapper/simwrapper>.
+

docs/CHANGELOG.md

@@ -10,11 +10,12 @@ The list of changes is generated automatically based on the commit messages them
 - See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
 - The SimWrapper code repository containing the full commit history is on GitHub at <https://github.com/simwrapper/simwrapper>.
+
 # Changelog
 
 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
-## 4.0.0 (2025-06-10)
+## 4.0.0 (2025-06-11)
 
 Bump to version 4.0. New front page, file view, color scheme, event viewer (beta), map builder
 (beta), and updated Python (pip) package for running standalone or in server environments.

docs/assets/default-dashboard.png

@@ -0,0 +1,60 @@
+---
+id: guide-cloud-storage
+title: Network/cloud storage
+---
+
+There are many ways to connect SimWrapper to cloud storage, and we have worked with several agencies that had different needs and requirements. This page summarizes our findings from those implementations.
+
+_This is work in progress, more to come_
+
+## Internal file server
+
+At VSP / TU Berlin, we have a huge on-premise file server for storing MATSim runs. The server runs Apache as a front-end, with Subversion managing updates. Subversion is old, yes, but it works very well with the enormous file sizes that MATSim produces (especially compared to "modern" solutions like Git).
+
+With this setup, Apache/subversion provide a web front-end to the directory and file system. SimWrapper then can be configured to point to this as a base URL for its Data Source.
+
+This approach has worked well since the inception of SimWrapper and continues as the primary file storage for VSP.
+
+## SSHFS -- mount an SSH system
+
+Also at VSP there is a supercompute "cluster" to which we only have SSH access.
+
+Every desktop OS has a way to mount SSH files using something like "SSHFS", the SSH filesystem. Search on `SSHFS [your OS]` to find the right solution for your situation.
+
+
+## AWS S3 / Microsoft Azure / Other Clouds
+
+The many giant cloud providers all have some file storage solution under various marketing names. We spent a lot of time trying to connect to AWS S3 and Azure in particular; and while those solutions worked, they felt like a lot of extra effort for the big players, and then we hadn't addressed NextCloud or any of the other players.
+
+And then we found [RClone.org](https://rclone.org/) which provides an open-source solution to mounting more than 70 different cloud providers' data storage as a local folder mount on your server/computer.
+
+RClone has an "rclone mount" program which mounts file or "blob" storage _as if it is local_ but does not actually have to sync any files until you request them. It even allows partial file syncs, which is absolutely ideal for giant OMX/HDF5 files of multiple gigabytes in size (but which users generally only look at a small slice). Details such as cache size, number of threads, and timeouts are all configurable.
+
+Not only does using RClone mean we can support ALL cloud service providers, but in practice it turned out to be faster an more reliable that the cloud vendors own file-mount products.
+
+### Using Rclone
+
+The short version is:
+
+- Run `rclone config` to specify which cloud provider you want to connect to, and to provide all of the authentication details needed. This file can be encrypted, supports two-factor auth, and more. The authentication information is stored locally on your server. Each config can be named, such as `AZURE-MODEL-STORAGE`
+- Then run `rclone mount` to mount a specific config to a folder of your choosing in your filesystem.
+
+The mount command has many, many configuration options to optimize and experiment with here. The example below shows one set of parameters that worked well for us with Azure.
+
+The mount can be set up as an autostart program using `systemd` on linux so that the storage even survives server reboots.
+
+**Example rclone command**
+```bash
+rclone mount AZURE-MODEL-STORAGE: /mnt/azure \
+  --read-only \
+  --fast-list \
+  --no-modtime \
+  --vfs-cache-mode full \
+  --vfs-fast-fingerprint \
+  --vfs-cache-max-age 7d \
+  --vfs-cache-max-size 5G \
+  --vfs-read-chunk-size 1M \
+  --vfs-read-chunk-streams 16
+```
+
+We will add more details to this page as we get more experience with the various cloud vendors.

docs/assets/example-folder.jpg

@@ -1,6 +1,6 @@
 ---
 id: guide-dashboards
-title: 2. Dashboards in depth
+title: Dashboards in depth
 ---
 
 A dashboard is a page laid out with multiple charts, plots, and visualizations all together. You define the layout with a YAML configuration file which contains the types of plots and their configurations all in one place.

docs/guide-cloud-storage.md

@@ -1,44 +1,45 @@
 ---
 id: guide-getting-started
-title: 1. Getting started tutorial
+title: Getting started tutorial
 ---
 
-Welcome to SimWrapper! Let's get you up and running with the basics.
+Welcome to SimWrapper! Let's get you up and running with the basics. This guide uses sample data that's hopefully a lot like the data you have for your projects
 
-- This guide uses sample data that's hopefully a lot like the data you would have for your projects
-- Use Google Chrome or MS Edge for the guide. Other browsers (Firefox, Safari) require a [separate local HTTP server](file-management) to access local files; for now that's just a stumbling block to getting started.
+> ➡ Use a Chromium-based browser: **Chrome/Edge/Brave** and others, for this guide.<br/><br/>Other browsers (Firefox, Safari) can work, but require a [separate local HTTP server](file-management) to access local files on your computer. For now, that's just a stumbling block to getting started. Safari is a particularly bad browser for data visualization, since Apple doesn't support the latest APIs nor local file access, and Safari has the slowest Javascript engine! 🐳
 
 ## How it works: SimWrapper and file-based configuration
 
-Most MATSim/ActivitySim outputs such as the `*.xml.gz` files are too large to open in a web browser, so SimWrapper provides a set of _visualization plugins_ which can display something useful for you. Plugins exist for lots of things and the list is growing: link volumes, agent animations, aggregate area summaries, and more.
+Your transport simulation produces a lot of files for every run, some of them quite large! MATSim/ActivitySim outputs such as the `*.xml.gz` files are usually too big to open in a web browser. SimWrapper provides a set of _visualization plugins_ which load the raw data, or more often, load post-processed files which contain only the data that you wish to display. Plugins exist for lots of things and the list is growing: link volumes, agent animations, aggregate area summaries, and more.
 
-Here's how it works: For every visualization you want to create, you write a small _configuration file_ and store it in the same folder as the inputs for that visualization. We use the YAML text format, which is a common configuration file format. For each properly named YAML file, one visualization thumbnail will appear in that folder when you navigate to the folder in SimWrapper. Clicking on the thumbnail will open that visualization full-screen.
+On top of this, you can collect various visualizations into _dashboards_ that you can lay out logically to help your users find the data they need.
 
-Generally, a viz will require a specific set of inputs, and those inputs are usually the result of some _post-processing_ of the raw simulation outputs. It's up to you to do that post-processing and store the files in the same folder as your config file.
+Here's how it works: For every visualization or dashboard you want to create, you write a small _configuration file_ and store it in the same folder as the inputs for that visualization. These configs are in the simple [YAML text format](https://yaml.org/) and SimWrapper looks for YAML files with specific names to learn what it should read and display (more on this below).
+
+Generally, a viz will require a specific set of inputs, and those inputs are either raw simulation outputs or the result of some _post-processing_ of the outputs. It's up to you to do that post-processing and store the resulting files in the same folder as your YAML config files.
 
 Let's get started with some sample data.
 
 ## 1. Get the sample data and open it in SimWrapper
 
 - Download [simwrapper-example-project.zip](https://github.com/simwrapper/simwrapper-example-project/archive/refs/heads/main.zip) from GitHub
 - Unzip the file somewhere you can find it easily - Desktop, home folder, etc.
-- Go to [simwrapper.github.io](https://simwrapper.github.io/site) and click `Add folder...` and browse to the folder you just created. Grant access to the folder so the SimWrapper site can see the files!
-  - (If you are using Firefox, `cd` to the data folder and run `simwrapper here` to start the local HTTP server)
+- Go to https://simwrapper.app and click `View local files...` and browse to the folder you just created. Grant access to the folder so the SimWrapper website can see the files!
+- Click on the `data` folder. You will find dashboard examples and some raw data from a few MATSim runs.
+
+If you are using Firefox, there is no "View local files" button. Instead `cd` to the data folder and run `simwrapper here` to start the local HTTP server. This requires first running `pip install simwrapper` to get the simwrapper  command-line helper app. I suggest you use Chrome/Edge/Brave for this tutorial!
 
 You should now see something similar to this:
 
 <img src="assets/example-folder.jpg" style="border: 1px solid #ccc"><i>Example data folder</i></img>
 
-## 2. Explore the samples
-
 Each of the subfolders in the example project shows different map views and capabilities of SimWrapper -- network link plots, statistical charts, area maps (shapefiles), dashboards, and so on.
 
-- Experiment with the various knobs and configuration settings to see how the visualizations can be manipulated
-- From your PC file browser, open up the `viz-*.yaml` files in each subfolder to see how each of the visualizations is defined in a readable text format.
+- Experiment with the various knobs and configuration settings in the visualizations to see how they can be manipulated
+- From your PC file explorer / Finder, find the YAML text files named`viz-*.yaml` in each subfolder to see how each of the visualizations is defined in a clear, readable text format.
 - Every visualization type has a different filename "prefix" to help you find them: e.g, `viz-map-*.yaml` are for shapefiles, `viz-link-*.yaml` are for MATSim network plots, and so on.
-- You can edit these YAML files, save, and click Reload on your browser to see how your changes affect the visualizations.
+- Try making a copy and editing these YAML files. Save and click `Reload` on your browser to see how your changes affect the visualizations.
 
-## 3. Create a dashboard with some charts
+## 2. Create a dashboard with some charts
 
 The dashboards subfolder shows how you can combine multiple visualizations into cohesive dashboards.
 
@@ -49,17 +50,29 @@ See the [Dashboards in Depth](guide-dashboards) article to learn more about buil
 
 ## 4. Configuring dashboard templates for multiple run folders
 
-In SimWrapper, everything is folder-based. So `viz-*.yaml` and `dashboard-*.yaml` files in a folder will automatically be detected and loaded based on their filenames.
+In SimWrapper, everything is file- and folder-based. So `viz-*.yaml` and `dashboard-*.yaml` files in a folder will automatically be detected and loaded, based on their filenames.
 
 If you want to define dashboards that will be used for **multiple folders**, such as several runs for a particular project:
 
-- **Create a folder** named `simwrapper` in the parent project directory.
+- **Create a folder** named `simwrapper` or `.simwrapper` in the parent project directory.
 - Move all dashboard, viz, and template YAMLs into that folder
 - Tweak any file paths as necessary, so that relative file names resolve properly.
 - The base folder for a dashboard is the _folder you are viewing_, not the dashboard template folder.
-- You can have multiple `simwrapper` folders all the way up your folder hierarchy; dashboard panels will be generated based on filename, and each found dashboard will be displayed as a tab on the folder view.
+- You can have multiple `simwrapper` folders all the way up your folder hierarchy; dashboard panels will be generated based on filename, and each found dashboard will be displayed as a tab on the dashboard.
+
+## 5. More on YAML filenames
 
-## 5. More details on visualizations and their YAML files
+SimWrapper looks in your folder for YAML files that follow specific naming conventions. That way, it won't try to read unrelated YAML files that you might have. But this means you must name your configuration files carefully! SimWrapper will **silently ignore all YAML files** that do not match these conventions:
+
+- **`dashboard-*.yaml`** - these contain the configuration settings for dashboard panes. Each found file results in a left-side tab on the dashboard page, and that tab contains the dashboard contents specified in this file. For this reason, many users tend to number their dashboard configs so that the tabs always appear in a predictable order: `dashboard-1-summary.yaml`, `dashboard-2-mode-choice.yaml`, `dashboard-3-traffic.yaml` and so on.
+- **`viz-*.yaml`** - these contain individual map-based visualizations. Each of these will appear on the Files tab of a folder.
+  - Each visualization plugin has a specific subname that must follow `viz-subname-...` in the filename: for example, the shapefile viewer would be `viz-map-*.yaml` while the xy-hexagon viewer is `viz-xy-*.yaml`. See the API reference for each plugin for examples.
+  - Use these `viz-*` configs if you don't have the need to collate multiple vizes into dashboards, or if you want single visualizations to have their own URL that you can link to directly.
+- **`simwrapper-config.yaml`** - is a special top-level configuration for project sites that is described in detail on [Building a project site](guide-project-sites).
+
+Note that in most cases, both `.yaml` and `.yml` file extensions are valid and honored.
+
+## 6. Example roadway links visualization
 
 Here is an example YAML config file for a link-volume summary:
 
@@ -80,8 +93,6 @@ If you wanted to look at several different link volume plots from the same model
 
 This is a very different paradigm than most "point and click" GIS tools, but we have found that the ability to script and cut/paste the config files has been a huge time saver and also reduces manual errors.
 
-> Make sure that your files are allowed to be "world-readable" before you publish anything to public-svn! Once files are pushed to public-svn, they are not secured in any way; anyone on the internet can access them!
-
 ### Visualization types
 
 Each visualization is described in the API Reference section of this documentation, including how to post-process your outputs if necessary, and how to define any configuration settings for your visualizations.