Field surveys web app set-up instructions

What is this app about?

We developed an interactive web application to facilitate multi-taxon field sampling and data collection using the Shiny for Python platform. This app serves the following purposes:

It allows researchers, citizen scientists, students, and others to easily design and conduct standardized surveys for various organisms, such as arthropods, birds, and plants.
The app is accessible on any device with an internet connection, including desktops, laptops, tablets, and smartphones.

This user-friendly application streamlines the process of field sampling and data collection, making it more efficient and accessible to a wide range of users, regardless of their technical expertise or location.

Demo on a mobile device — *Demo on Mobile*

App Features

Seamless Integration

Google Drive Integration: Store images of specimens effortlessly during the survey, ensuring all your visual data is securely backed up.
Google Sheets Integration: Record observations directly into a Google Sheet during the survey, eliminating the need for manual data entry later.

Precise Location Tracking

High-Accuracy GPS: Record your location with precision, thanks to the app’s ability to pinpoint your GPS coordinates within 15 meters.

Comprehensive Specimen Information

Identification Images and Notes: Access identification images and detailed notes about the selected specimen, providing you with valuable insights during your survey.
Attribute Recording: Easily record additional attributes such as canopy cover and vegetation height (in cm) for comprehensive vegetation surveys.
Specimen Lookup: Quickly look up specimens by their Genus-Species or Common Name while recording observations, ensuring accurate data collection.

Data Security and Restoration

Observation Restoration: In case of unexpected data loss while in the field, you can restore your recorded observations, ensuring no valuable data is lost.

Environmental Data Integration

Weather Information: Look up and store weather information for your detected GPS location, providing valuable environmental context to your survey data.

With its seamless integration, precise location tracking, comprehensive specimen information, data security measures, and environmental data integration, this app empowers you to conduct efficient and accurate surveys, streamlining your field data collection process.

How does the app work?

Can I see a demo of the app?

If you want to play around with the app before setting up one for your case, feel free to go over to this link and explore the app and the features it offers.

Prerequisites

Python 3.8 , 3.9, 3.10 or 3.11 (3.12 is not tested and might not work as expected)
Check python version on your machine
Run this command on your Terminal on a macOS/Linuxmachine or Windows PowerShell/Command Prompt on a Windows machine.

Checking python version on windows machine
python --version
OR
python3 --version
If Python is not installed

Visit the official Python downloads page and download the desired version of Python 3. Run the downloaded package and follow the installation wizard. Check the previous command to ensure python is installed successfully on the machine. Make sure you select the option to `Add Python to PATH` during installation.

Add python to path

Note: You will need to add Python to PATH if you’ve installed Python but did not select that option, follow instructions here to amend that.
Wheel python package

Install this package by typing this command on your Terminal on a macOS/Linuxmachine or Windows PowerShell/Command Prompt on a Windows machine.
```
pip install wheel
```
Data CSV files (Data CSV files)
survey.yaml that is customized to your use case (Make sure Data CSV files are ready before this step)

Create your unique survey.yaml file

Please complete all the required information fields and then you can download your file from this location.

Do not rename survey.yaml file

The file you download should be named survey.yaml. Please do not rename it, else the app won’t work as expected.
keyfile.json (keyfile.json)

Data CSV files

If you do not know what a CSV file is or how to create one, read this article to understand what CSV files are. The app requires users to upload data CSV files. These CSV files should contain the data that the user wants to collect during the survey. The CSV files must have the following headers (first line) in the specified order:

Common Name
Genus
Species

The app also supports an additional column called Alpha Code, but it is not mandatory.

An example is shown below:

Alpha Code,Common Name,Genus,Species
PLOC,American sycamore,Platanus,occidentalis
SALA,Arroyo willow,Salix,lasiolepis
PEAM,Avocado,Persea,americana
ARGL,Big berry manzanita,Arctostaphylos,glauca
ACMA,Bigleaf maple,Acer,macrophyllum

Ensure your CSV file follows the required format before uploading. You can use multiple CSV files, one for each survey type (e.g., plants, trees, grasses).

Creating your CSV Data file

Create your desired CSV file using this website or Google sheets

*Using tableconvert website for CSV generation*

Once the CSV file is downloaded, rename the file to something more readable like birds.csv, trees.csvfrom the existing name of tableConvert.com_ypnog1.csv

Do it for other files if you have more than one data CSV file.

keyfile.json

To download the keyfile.json for your Google Cloud account as a first-time user, follow these steps:

Enable API Access for a Project

Head to Google Developers Console and create a new project (or select the one you already have). The project name is for your reference only and can be anything you’d like.
In the box labeled Search for APIs and Services, search for Google Drive API and enable it.

Enable Google Drive API
In the box labeled Search for APIs and Services, search for Google Sheets API and enable it.

Enable Google sheets API

Using Service Account

A service account is a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in Google APIs.

Since it’s a separate account, by default it does not have access to any spreadsheet until you share it with this account. Just like any other Google account.

Here’s how to get one:

Enable API Access for a Project if you haven’t done it yet.
Go to APIs & Services > Credentials and choose Create credentials > Service account key.
Fill out the form
Click Create and Done.
Press “Manage service accounts” above Service Accounts.
Press on ⋮ near recently created service account and select “Manage keys” and then click on “ADD KEY > Create new key”.
Select JSON key type and press “Create”.

You will automatically download a JSON file with credentials. It may look like this:

{
    "type": "service_account",
    "project_id": "api-project-XXX",
    "private_key_id": "2cd … ba4",
    "private_key": "-----BEGIN PRIVATE KEY-----\nNrDyLw … jINQh/9\n-----END PRIVATE KEY-----\n",
    "client_email": "473000000000-yoursisdifferent@developer.gserviceaccount.com",
    "client_id": "473 … hd.apps.googleusercontent.com",
    ...
}

Navigate to the path to the downloaded credentials filerename and rename this file to keyfile.json
Open the keyfile.json file and get the value of client_email from this file.
Proceed to the next section to configure your Google sheets and Google drive folder.

Google sheets and Google Drive folder configured

Create a new Google spreadsheet that you want your survey data to live in. You can create a new workbook by going to Google Sheets and clicking on the + button for blank spreadsheet. You can also use an existing workbook if you have one. Give it an appropriate name and make sure it has the same name as the one you provide in the survey.yaml file.
Using that spreadsheet name and share it with a client_email from the step above. Just like you do with any other Google account. Make sure the service account has editor access to the Google sheet.
Create a new Google Drive folder where you want to store the specimen images during the survey. You can create a new folder by going to Google Drive and clicking on the “+ New” button for new folder. You can also use an existing folder if you have one. Give it an appropriate name.
To share that Google Drive folder, right-click on the folder and select Share. Then, share it with the client_email from the step above. Make sure the service account has editor access to the Google Drive folder.

survey.yaml file

*Demo of the Survey App config generator*

This is the config file that allows the user to configure the app to their specific needs. The survey.yaml file should be created using the Survey App config generator over here. The Survey App config generator allows the user to create a survey.yaml file by providing the following information in a step-by-step process:

Surveyor names
Survey locations
Survey plots
Survey points
Survey sides
Survey data sources
Company logo url
Database link
Google Workbook Name
Google Drive Folder Id

Company logo url

To get the image URL directly from the browser, follow these steps:

Right-click on the image you want to get the URL for.
From the context menu that appears, select Copy Image Address or something similar.
That’s it! The URL of the image is now copied to your clipboard.

Installation

fieldsurveys is a python package that can be installed on a user’s machine by satisfying all prerequisites and running this command in the Terminal on a macOS/Linuxmachine or Windows PowerShell/Command Prompt on a Windows machine.

pip install fieldsurveys

Run the app on your machine

To instantiate the app, run the following command in your terminal:

fieldsurveys

Follow the instructions on the screen to select your survey.yaml, data CSV files, and keyfile.json. It will then ask you for the directory where you want to store the app. Once you have provided all the necessary information, the app will be created in the specified directory.

Cannot run ‘fieldsurveys’ command on Windows

If running fieldsurveys on the Windows Powershell/Command Prompt throws an error stating

fieldsurveys : The term 'fieldsurveys' is not recognized as the name of a cmdlet, function, script file, or operable
program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

OR

'fieldsurveys' is not a recognized as an internal or external command, operable program or batch file.

You might need to add Python scripts location to the PATH.
Locate the Scripts Directory: First, you need to find the directory where your Python scripts are located. You can find out the location of the scripts directory by running this command on your Terminal/ Command Prompt

python -c "import sys; print(sys.prefix + '/bin' if sys.platform != 'win32' else sys.prefix + '/Scripts')"

this command will return something like C:\Users\username\AppData\Local\Programs\Python3XX\Scripts

Open System Properties: Right-click on the “This PC” or “My Computer” icon on your desktop or File Explorer and select “Properties”. Alternatively, you can search for “System” in the Start menu and select “System” from the search results.
Open Advanced System Settings: In the System window, click on “Advanced system settings” on the left side. This will open the System Properties window.
Open Environment Variables: In the System Properties window, click on the “Environment Variables…” button.
Edit System Variables: In the Environment Variables window, under the “System variables” section, find the Path variable and select it. Then click on the “Edit…” button.
Add Python Scripts Directory: In the Edit Environment Variable window, click on the “New” button and then paste the path to the directory where your Python scripts are located. For example, if your scripts are in C:\Users\username\AppData\Local\Programs\Python3XX\Scripts, you would add C:\Users\username\AppData\Local\Programs\Python3XX\Scripts to the list of paths.
Apply Changes: Click “OK” to close all the windows and apply the changes.
Verify: Open a new command prompt window and type the name of one of your Python scripts (i.e. `fieldsurveys`). If the script executes without errors, it means the path has been successfully added to the system PATH.

Deploying the app on the web (so it can be accessed from anywhere)

To deploy the app on the web, you can use the Shinyapps.io platform. The platform allows you to host your field survey application for 25 hours every month. Further usage is restricted on free tier. To deploy the app, follow the instructions below:

Create an Account: Go to Shinyapps.io website and click on the “Sign Up” button. Fill out the required information to create a new account.
Verify Your Email: Fill all the required information and click on the Sign Up button. After creating your account, you’ll receive a verification email. Open the email and click on the verification link to confirm your email address.
Log in to Your Account: Once your email is verified, go back to Shinyapps.io website and click on the Log In button in the top right corner. Enter your credentials to log in to your account.
Select Account name: The first time you sign in, shinyapps.io prompts you to set up your account. shinyapps.io uses the account name as the domain name for all of your apps. Account names must be between four and 63 characters and can only contain letters, numbers, and dashes (-).
Create token: Click on the the user name on the top right of the page and select Tokens from the drop down menu.
Add token: Click on the + Add Token button. It will add the new token below. Click on the Show button to reveal the token and the secret.
Copy command to clipboard: Click on the With Python tab and click on the Copy to clipboard button.
Open your Terminal on a macOS/Linuxmachine or Windows PowerShell/Command Prompt on a Windows machine and paste the copied command and hit enter to store the credentials on your machine that will be used for deployment.
In the Terminal on a macOS/Linuxmachine or Windows PowerShell/Command Prompt, navigate to the path where your app is located
```
cd path/to/your_app
```
Then, run the following command to deploy your app
```
rsconnect deploy shiny . --name <NAME> --title my-app
```

<NAME> and my-app values

replace <NAME> with the account name you created in step 4. For the title of your app, select any desired name in place of my-app. This will also be used as the url of your app.

Once the deployment goes through, it will open the url of your deployed app. Save that url and use that for future surveys.

If you want to redeploy changes to the app

In the scenario if you already deployed this app and want to make changes, archive the app and delete it using the shinyapps.io dashboard before following the deployment steps again.

Issues/Feature requests

If you encounter any issues or have any feature requests, please feel free to open an issue on the GitHub repository here